Marshal large byte array into array of structs?

[twawsico]

I have a piece of code that needs to read the contents of a binary file
(that I've created with another app) into an array of structures. The
binary data in the file represents just a series of singles that
correspond to those in my structure detailed below. So when I load the
file, all that I know for certain is that there will be some multiple
of these eight singles represented in the binary data.
My code below will read the data correctly, but only marshals one
structure (8 singles) at a time, forcing me to loop through the binary
array in chunks. With the large amount of data I need to read, this is
very slow.

In C++, I can load data from the same file into an array of these
structures with a single call to fread() and it's very fast. Can anyone
give me a tip on how to accomplish this in .NET?

Thanks in advance.


Code that loads one struct at a time:

<StructLayout(LayoutKind.Sequential)> Public Structure MyStruct
<MarshalAs(UnmanagedType.R4)> Dim X As Single
<MarshalAs(UnmanagedType.R4)> Dim Y As Single
<MarshalAs(UnmanagedType.R4)> Dim Z As Single
<MarshalAs(UnmanagedType.R4)> Dim tu As Single
<MarshalAs(UnmanagedType.R4)> Dim tv As Single
<MarshalAs(UnmanagedType.R4)> Dim NX As Single
<MarshalAs(UnmanagedType.R4)> Dim NY As Single
<MarshalAs(UnmanagedType.R4)> Dim NZ As Single
End Structure

Public Sub ReadBinaryIntoStructArray(ByVal FileName As String,
ByRef ArrayOfStructs() As MyStruct)
Dim bFileOpen As Boolean = False
Dim bHeapAlloc As Boolean = False
Dim fstm As FileStream
Dim binaryData() As Byte
Dim bytesRead As Long
Dim iStructSize As Int32
Dim iCurVertNum As Int32
Dim ptrTarget As IntPtr
Dim oSingleStruct As Object
Try
fstm = New FileStream(FileName, FileMode.Open,
FileAccess.Read, FileShare.Read)
bFileOpen = True
binaryData = New Byte(fstm.Length - 1) {}
bytesRead = fstm.Read(binaryData, 0, CInt(fstm.Length))
iStructSize = Marshal.SizeOf(GetType(MyStruct))
ArrayOfStructs = New MyStruct(CInt(fstm.Length /
iStructSize) - 1) {}
'Until I can find a better bulk-copy method, we must load
one vert (8 singles) at a time).
iCurVertNum = 0
ptrTarget = Marshal.AllocHGlobal(iStructSize)
bHeapAlloc = True
For iCurVertNum = 0 To ArrayOfStructs.Length - 1

Marshal.Copy(binaryData, iCurVertNum * iStructSize,
ptrTarget, iStructSize)
oSingleStruct = Marshal.PtrToStructure(ptrTarget,
GetType(MyStruct))

ArrayOfStructs(iCurVertNum).X = oSingleStruct.X
ArrayOfStructs(iCurVertNum).Y = oSingleStruct.Y
ArrayOfStructs(iCurVertNum).Z = oSingleStruct.Z
ArrayOfStructs(iCurVertNum).tu = oSingleStruct.tu
ArrayOfStructs(iCurVertNum).tv = oSingleStruct.tv
ArrayOfStructs(iCurVertNum).NX = oSingleStruct.NX
ArrayOfStructs(iCurVertNum).NY = oSingleStruct.NY
ArrayOfStructs(iCurVertNum).NZ = oSingleStruct.NZ
Next
Marshal.FreeHGlobal(ptrTarget)
bHeapAlloc = False
Catch ex As System.OutOfMemoryException
Throw ex
Catch e As Exception
Throw e
Finally
If bFileOpen Then
fstm.Close()
End If
fstm = Nothing
If bHeapAlloc Then
Marshal.FreeHGlobal(ptrTarget)
End If
End Try
End Sub

 

[twawsico]

Does anyone at least know whether or not this is going to be possible
in VB.NET?

Thanks again.

 

[twawsico]

Ok, now realizing that this thread would prob be more appropriate for
the Interop group, I've come up with a solution that seems to work fine
and loads/copies a couple of hundred thousand structures into the array
about as fast as my C++ version (well under a second).

Here's the bulk solution, which (typically) ended up being much
shorter/easier once I got on the right track.

Public Sub ReadBinaryTerrainData(ByVal FileName As String, ByRef
ArrayOfStructs() As MyStruct)
Dim bFileOpen As Boolean = False
Dim fstm As FileStream
Dim binaryData() As Byte
Dim bytesRead As Long
Dim iStructSize As Int32
Dim prtMyStruct As IntPtr
Dim gch As GCHandle
Try
'Open the file and read the binary data into a byte array.
fstm = New FileStream(FileName, FileMode.Open,
FileAccess.Read, FileShare.Read)
bFileOpen = True
binaryData = New Byte(fstm.Length - 1) {}
bytesRead = fstm.Read(binaryData, 0, CInt(fstm.Length))
'Initialize/size our array of vertices.
iStructSize = Marshal.SizeOf(GetType(MyStruct))
ArrayOfStructs =
System.Array.CreateInstance(GetType(MyStruct), CInt(fstm.Length /
iStructSize))
'Get a handle to the vert array, pinning it to keep GC from
'zeroing it or moving it around.
gch = GCHandle.Alloc(ArrayOfStructs, GCHandleType.Pinned)
'Get a pointer to the vert array.
prtMyStruct = gch.AddrOfPinnedObject()
'Use that pointer as a destination to which to copy the
binary data
'from the byte array.
Marshal.Copy(binaryData, 0, prtMyStruct, bytesRead)
'Free the pinned handle so GC can do it's thing from here
on.
gch.Free()
Catch ex As System.OutOfMemoryException
' An exception could occur if the system is out of
' memory and the block of heap memory could not be
' set aside for you.
Throw ex
Catch e As Exception
' General exception caught, show the message and move on...
Throw e
Finally
If bFileOpen Then
fstm.Close()
End If
fstm = Nothing
If gch.IsAllocated() Then
gch.Free()
End If
End Try
End Sub