Quick way to transfer huge data from one file to another.

[Sakharam Phapale]

Hi All,

I want to read data from one file and write that into another file.

Which is a way to do it quickly in terms of processing time?


Data to be transfer is huge one. E.g. 2147483458 bytes.
So I can't use array as temporary data type.
Since looping takes too much time, Is there any way to directly use
StreamReader and StreamWriter in conjunction.

Any help will be appreciated.

Thanks and Regards
Sakharam Phapale

 

[Larry Serflaten]

I want to read data from one file and write that into another file.

Is the Copy method in the File class out of the question?

Since looping takes too much time, Is there any way to directly use
StreamReader and StreamWriter in conjunction.

To do what? Do you need to process the data? Then yeah, use
the streams. If not, then copy it outright....

LFS

 

[Cor Ligthert]

Sakharam,

I can not understand direct from, your message if you mean a complete file
to a new or a part.

In the first situation, than why not just filecopy

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vblr7/html/vafctfilecopy.asp

I hope this helps?

Cor

 

[Sakharam Phapale]

Hi Cor,

I want to transfer given part of file.
From byte no to byte no.

Thanks and Regards
Sakharam Phapale

 

[Cor Ligthert]

Sakharam,

When you do not use a kind of database (even a very classic one which
adresses bytes direct on disk) you and it is something you can append. Than
the only thing (as Larry already wrote I saw now) is using the streamreader
and the streamwriter (There are more methods which do the same, however see
that yourself on this page, all this pages have direct links to other
methods)

http://msdn.microsoft.com/library/d.../html/frlrfsystemiostreamwriterclasstopic.asp

That can by reading first everything in memory or doing it line by line,
both methods can be used with the classes above.

I hope this helps?

Cor

 

[Sakharam Phapale]

Hi Cor,

First of all, thanks for reply.

I have used following two ways to transfer part of data from one
file to another.
If I used first option, performance is very slow.

Since, I am using big files, If I use second option i.e. using
array, I get error of out of memory because size of data is very big.

So, How can I improve the spped of transfering data?
Is there any other way around, or any suggetions then please let
me know.

Thanks and Regards
Sakharam Phapale

1) Using for Loop

Dim fsReader As FileStream
Dim br As BinaryReader
Dim fsWriter As FileStream
Dim bw As BinaryWriter

fsReader = New FileStream(TxaFileName, FileMode.Open,
FileAccess.Read)
br = New BinaryReader(fsReader)

fsWriter = New FileStream(m_strTxtFile, FileMode.CreateNew,
FileAccess.Write)
bw = New BinaryWriter(fsWriter)

Dim myByte As Byte

br.BaseStream.Seek(lngFromByte, SeekOrigin.Begin)
bw.BaseStream.Seek(0, SeekOrigin.Begin)

Do While br.BaseStream.Position <> lngFileLen
Try
myByte = br.ReadByte
Catch
Exit Do
End Try
bw.Write(value:=myByte)
Loop

bw.Close()
fsWrite.Close()
br.Close()
fsRead.Close()


2) Using buffer (array)


Dim arrCopiedWaveData(lngDataSize - 1)

Dim fsRead As New FileStream(TxaFileName, FileMode.Open,,
FileAccess.Read)
Dim br As New BinaryReader(fsRead)

br.BaseStream.Seek(lngFromByte, SeekOrigin.Begin)
br.Read(arrCopiedWaveData, 0, arrCopiedWaveData.Length)

bw.BaseStream.Seek(0, SeekOrigin.Begin)
bw.Write(arrCopiedWaveData, 0, arrCopiedWaveData.Length)

bw.Close()
fsWrite.Close()
br.Close()
fsRead.Close()

 

[Larry Serflaten]

So, How can I improve the spped of transfering data?
Is there any other way around, or any suggetions then please let me know.

Dim myByte As Byte

Do While br.BaseStream.Position <> lngFileLen
Try
myByte = br.ReadByte
Catch
Exit Do
End Try
bw.Write(value:=myByte)
Loop

Instead of reading 1 byte at a time, use ReadBytes to read 4096 or 8192 bytes
at a time. (Using multiples of disk cluster size increases performance over other
odd amounts (like 1000, or 5000 etc...))

Pseudo code:

' Open streams ...
' Seek to position ...

size = 8192
Try
Do While lngFileLen > 0
If lngFileLen < size then size = lngFileLen
bw.Write(br.ReadBytes(size))
lngFileLen -= size
Loop
Catch
' Report error ...
Finally
' Close streams...
End Try

 

[Sakharam Phapale]

Thanks Larry, Cor for your kind help.

Regards
Sakharam Phapale

 

[Jay B. Harlow [MVP - Outlook]]

Sakharam,
In addition to the other comments about using a buffer to copy chunks of the
file at a time.

You may want to review the asynchronous IO sample from MSDN.

http://msdn.microsoft.com/library/d...n-us/cpguide/html/cpconasynchronousfileio.asp

As asynchronous IO can, in some cases, be quicker then normal synchronous
IO.

Hope this helps
Jay

 

[Tom Shelton]

Hi Cor,

I want to transfer given part of file.
From byte no to byte no.

Thanks and Regards
Sakharam Phapale

Out of curiosity... Are you saying that you want to transfer a chunk of
the file with a given byte postion to a given byte position? If so,
are these positions fixed, or known before hand?

If so, you could simply seek to the start position, read the chunk, and
write it to the new file. Of course, I may be totally misunderstanding
your requirements