MD5 hash on very large files 500mb to 4gb+

[Paul Spielvogel]

I need to compute the MD5 hash on VERY large files 500mb to 4gb+

I have found two ways but neither one of them does what i need.

Private Function ComputeDataMD5(ByVal path As String) As String
Dim fi As New FileInfo(path)
Dim fs As FileStream = fi.OpenRead()
fs = fi.OpenRead

Dim Md5 As New MD5CryptoServiceProvider
Dim hash As String =
BitConverter.ToString(Md5.ComputeHash(fs)).Replace("-", "")

'fs.Close()
ComputeDataMD5 = hash.ToLower
End Function

This function uses the filestream object to create the hash from,
problem is that it locks up the application and does not allowe me to
show/update a progress bar.


Function GetHash(ByVal path As String) As String

Dim cs As CryptoStream
Dim ms As MemoryStream = New MemoryStream
Dim md5Hash As MD5CryptoServiceProvider = New
MD5CryptoServiceProvider

Dim fi As New FileInfo(path)
Dim fs As FileStream = fi.OpenRead()

Try

fs = fi.OpenRead

Dim buffer(1024) As Byte
Dim size As Integer

Do While fs.Position <> fs.Length
size = fs.Read(buffer, 0, 1024)

cs = New CryptoStream(ms, md5Hash,
CryptoStreamMode.Write)
cs.Write(buffer, 0, size)
Loop

cs.FlushFinalBlock()
Return BitConverter.ToString(md5Hash.Hash()).Replace("-",
"").ToLower


Catch ex As Exception

MsgBox("Error during hash operation: " + ex.ToString())

Finally
If Not (fs Is Nothing) Then fs.Close()
If Not (cs Is Nothing) Then cs.Close()
If Not (md5Hash Is Nothing) Then md5Hash.Clear()
End Try
End Function

This function reads a block of data and places it into the
CryptoStream object, after we are done reading the file we compute the
MD5. Problem with this function is that it reads the whole file into
memory, 500mb file = 500mb in ram.
Since i need to compute hash on files that are in the range of 4gb
this method is useless.

 

[Imran Koradia]

This function uses the filestream object to create the hash from,
problem is that it locks up the application and does not allowe me to
show/update a progress bar.

generate the hash on another thread so that the main thread is free to
update the form and its controls and wont lock up.


Imran.

 

[Samuel Neff]

As Imran said, put it in a separate thread to leave the main thread
open to update the UI.

As for generating a progress bar, you'll need to get the progress of
the Hash. To do this you can create a custom stream that wraps
FileStream and broadcasts progress events. Or you can use an
indeterminate progress bar that just shows something is hapenning
without showing the percent completed.

HTH,

Sam