MarshalAs Problems with Arrays

[O.B.]

Below is a program that shows a test for marshaling data from a byte
array to a class structure. Unfortunately, there are two annoying
problems (bugs?) that I can't seem to get around.

The first is that if I set the Articulation array to an offset of 22
in the EntityState class instead of 24, I get the following error.
The thing is, it is properly aligned. And the amount of data to be
copied is smaller than the index of 22 anyway.

System.TypeLoadException: Could not load type 'Test.EntityState' from
assembly 'MUSE_Common, Version=7.4.0.25116, Culture=neutral,
PublicKeyToken=null' because it contains an object field at offset 22
that is incorrectly aligned or overlapped by a non-object field.

The next error is that when I set the FieldOffset to 24 for the
articulations array, it runs great ... except that only the first byte
within the "other" array in DeadReckoning is set, they rest of the
bytes are 0 in the array. If I comment out the articulations array,
all the bytes of the "other" array in DeadReckoning are copied
correctly.

Help?

-------------------------------------------------

using System;
using System.Runtime.InteropServices;

namespace MarshalTest
{
[StructLayout(LayoutKind.Explicit, Size = 16)]
public struct Articulation
{
[FieldOffset(0)]
public byte parameterTypeDesignator;

[FieldOffset(1)]
public byte change;

[FieldOffset(2)]
public ushort idAttachedTo;

[FieldOffset(4)]
public uint parameterType;

[FieldOffset(8)]
public uint parameterValue1;

[FieldOffset(12)]
public uint parameterValue2;
}

[StructLayout(LayoutKind.Explicit, Size = 20)]
unsafe public struct DeadReckoning
{
[FieldOffset(0)]
public byte header;

[FieldOffset(1)]
public fixed byte other[15];

[FieldOffset(16)]
public uint footer;
}

[StructLayout(LayoutKind.Explicit)]
public class EntityState
{
[FieldOffset(0)]
public byte mySize;

[FieldOffset(1)]
public DeadReckoning deadRecking;

[FieldOffset(21)]
public byte temp1;

// BUG: Cannot set this offset to 22 without a runtime error.
Why?
[FieldOffset(24)]
[MarshalAs(UnmanagedType.ByValArray, ArraySubType =
UnmanagedType.Struct)]
public Articulation[] articulations;

public EntityState(byte[] rawData)
{
int thisSize = rawData.Length;

IntPtr pData = Marshal.AllocHGlobal(thisSize);
Marshal.Copy(rawData, 0, pData, thisSize);
Marshal.PtrToStructure(pData, this);
Marshal.FreeHGlobal(pData);

mySize = (byte)thisSize;
}

public byte[] ToRaw()
{
byte[] byteArray = new byte[mySize];
IntPtr pointer = Marshal.AllocHGlobal(mySize);
Marshal.StructureToPtr(this, pointer, false);
Marshal.Copy(pointer, byteArray, 0, mySize);
Marshal.FreeHGlobal(pointer);
return byteArray;
}
}

class Program
{
static void Main(string[] args)
{
// Simulate raw data read from hardware
int dataSize = 22;
byte[] rawData = new byte[dataSize];
rawData[0] = (byte)dataSize;
for (int i = 0; i < dataSize; i++)
{
rawData = (byte)(i + 1);
}

// Convert raw data to class
EntityState entity = new EntityState(rawData);

// Compare original data with data stored in class
byte[] rawData2 = entity.ToRaw();

if (rawData.Length != rawData2.Length)
{
Console.WriteLine("**** ERROR *****: Data not the same
length.");
return;
}
bool dataIdentical = true;

// BUG: Offsets 3 - 16 are not identical
for (int i = 1; i < rawData.Length; i++)
{
if (rawData != rawData2)
{
Console.WriteLine(i + ":\t" + rawData +
" != " + rawData2 + "
*****");
dataIdentical = false;
}
else
{
Console.WriteLine(i + ":\t" + rawData + " == "
+ rawData2);
}
}
if (!dataIdentical)
{
Console.WriteLine("**** ERROR *****: Data not the
same.");
}
}
}
}

 

[Jeroen Mostert]

Below is a program that shows a test for marshaling data from a byte
array to a class structure. Unfortunately, there are two annoying
problems (bugs?) that I can't seem to get around.

The first is that if I set the Articulation array to an offset of 22
in the EntityState class instead of 24, I get the following error.
The thing is, it is properly aligned.

I'm afraid not. Under default packing the framework requires reference
fields to be aligned to a word boundary, that's 4 bytes in this case. So 24
bytes is OK, 22 is not. You can get around this by using SequentialLayout
and specifying a pack size of 1 (that is, no alignment padding). Using
ExplicitLayout won't work because the packing size is ignored if you do
that. I don't know why; it doesn't seem to be an entirely logical restriction.

The next error is that when I set the FieldOffset to 24 for the
articulations array, it runs great ... except that only the first byte
within the "other" array in DeadReckoning is set, they rest of the
bytes are 0 in the array. If I comment out the articulations array,
all the bytes of the "other" array in DeadReckoning are copied
correctly.

// BUG: Cannot set this offset to 22 without a runtime error.
Why?
[FieldOffset(24)]
[MarshalAs(UnmanagedType.ByValArray, ArraySubType =
UnmanagedType.Struct)]
public Articulation[] articulations;

This isn't going to work. Marshalling as .ByValArray requires that you
specify the size of the array in the SizeConst parameter. If you don't know
the size in advance, you can't use .ByValArray. So you'll have to make clear
what you're going for, here. Is "articulations" a variable-sized array or a
fixed-size array? If it's variable-sized, what indicates the number of
elements and how?

OK, snipping the rest of the code. These declarations work:

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Articulation {
public byte parameterTypeDesignator;
public byte change;
public ushort idAttachedTo;
public uint parameterType;
public uint parameterValue1;
public uint parameterValue2;
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct DeadReckoning {
public byte header;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 15)]
public byte[] other;
public uint footer;
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public class EntityState {
public byte mySize;
public DeadReckoning deadRecking;
public byte temp1;

// This is probably wrong. You'll have to marshal this array manually
if it's variable-sized.
[MarshalAs(UnmanagedType.ByValArray, ArraySubType =
UnmanagedType.Struct, SizeConst = 1)]
public Articulation[] articulations;
}

It appears fixed arrays in substructures aren't marshalled correctly, or
else I just haven't found the right way to tickle the marshaller. Using a
regular array and marshalling it as .ByValArray works, though.

 

[O.B.]

I'm afraid not. Under default packing the framework requires reference
fields to be aligned to a word boundary, that's 4 bytes in this case. So 24
bytes is OK, 22 is not. You can get around this by using SequentialLayout
and specifying a pack size of 1 (that is, no alignment padding). Using
ExplicitLayout won't work because the packing size is ignored if you do
that. I don't know why; it doesn't seem to be an entirely logical restriction.

Thanks. You are a blessing! In my real code, the offsets are not
sequential, but I can insert some "dummy" offsets to get the
sequential behavior to work.

The next error is that when I set the FieldOffset to 24 for the
articulations array, it runs great ... except that only the first byte
within the "other" array in DeadReckoning is set, they rest of the
bytes are 0 in the array. If I comment out the articulations array,
all the bytes of the "other" array in DeadReckoning are copied
correctly.

// BUG: Cannot set this offset to 22 without a runtime error.
Why?
[FieldOffset(24)]
[MarshalAs(UnmanagedType.ByValArray, ArraySubType =
UnmanagedType.Struct)]
public Articulation[] articulations;

This isn't going to work. Marshalling as .ByValArray requires that you
specify the size of the array in the SizeConst parameter. If you don't know
the size in advance, you can't use .ByValArray. So you'll have to make clear
what you're going for, here. Is "articulations" a variable-sized array or a
fixed-size array? If it's variable-sized, what indicates the number of
elements and how?

It is actually variable sized. There is another attribute in the
EntityState class that specifies the number of articulations. I have
modified the test program (below) to include this attribute and how
I'm manually marshaling the data (see the ctor of EntityState). What
is odd is that the Marshal.SizeOf operation is returning a size
smaller than the actual array size returned by ToRaw(). Am I safe to
assume that the data returned by ToRaw is legit and Marshal's SizeOf
operation cannot handle dynamic arrays? Or is the SizeOf operation
correct and I am overwriting memory I shouldn't be?

using System;
using System.Runtime.InteropServices;

namespace MarshalTest
{
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Articulation
{
public byte parameterTypeDesignator;
public byte change;
public ushort idAttachedTo;
public uint parameterType;
public uint parameterValue1;
public uint parameterValue2;
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct DeadReckoning
{
public byte header;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 15)]
public byte[] other;
public uint footer;
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public class EntityState
{
public byte mySize; // Offset 0
public DeadReckoning deadRecking; // Offset 1
public byte numArticulations; // Offset 21
[MarshalAs(UnmanagedType.ByValArray, ArraySubType =
UnmanagedType.Struct)]
public Articulation[] articulations;

public EntityState(byte[] rawData)
{
// Marshal the fixed data
int fixedSize = rawData.Length > 22 ? 22 : rawData.Length;

IntPtr pData = Marshal.AllocHGlobal(fixedSize);
Marshal.Copy(rawData, 0, pData, fixedSize);
Marshal.PtrToStructure(pData, this);
Marshal.FreeHGlobal(pData);

// Manually marshal values for each articulation
if (numArticulations > 0)
{
articulations = new Articulation[numArticulations];
byte[] tempArray = new byte[16];
for (int i = 0; i < numArticulations; i++)
{
Array.Copy(rawData, 22 + i * 16, tempArray, 0,
16);
unsafe
{
fixed (byte* pData2 = tempArray)
{
articulations =

(Articulation)Marshal.PtrToStructure((IntPtr)pData2,
typeof(Articulation));
}
}
}
}
}

public byte[] ToRaw()
{
byte[] byteArray = new byte[mySize];
IntPtr pointer = Marshal.AllocHGlobal(mySize);
Marshal.StructureToPtr(this, pointer, false);
Marshal.Copy(pointer, byteArray, 0, mySize);
Marshal.FreeHGlobal(pointer);
return byteArray;
}
}

class Program
{
static void Main(string[] args)
{
// Simulate raw data read from hardware
int dataSize = 54;
byte[] rawData = new byte[dataSize];
rawData[0] = (byte)dataSize;
for (int i = 1; i < dataSize; i++)
{
rawData = (byte)(i + 1);
}
rawData[21] = 2; // 2 articulations

// Convert raw data to class
EntityState entity = new EntityState(rawData);

// Why is size of resulting structure smaller than 54?
if (Marshal.SizeOf(entity) != dataSize)
{
Console.WriteLine("Marshal says entity size is " +
Marshal.SizeOf(entity));
}

// Compare original data with data stored in class
byte[] rawData2 = entity.ToRaw();

if (rawData.Length != rawData2.Length)
{
Console.WriteLine("**** ERROR *****: Data not the same
length.");
return;
}
bool dataIdentical = true;

for (int i = 1; i < rawData.Length; i++)
{
if (rawData != rawData2)
{
Console.WriteLine(i + ":\t" + rawData + " != "
+ rawData2 + " *****");
dataIdentical = false;
}
else
{
Console.WriteLine(i + ":\t" + rawData + " == "
+ rawData2);
}
}
if (!dataIdentical)
{
Console.WriteLine("**** ERROR *****: Data not the
same.");
}
}
}
}

 

[Jeroen Mostert]

It is actually variable sized. There is another attribute in the
EntityState class that specifies the number of articulations. I have
modified the test program (below) to include this attribute and how
I'm manually marshaling the data (see the ctor of EntityState). What
is odd is that the Marshal.SizeOf operation is returning a size
smaller than the actual array size returned by ToRaw(). Am I safe to
assume that the data returned by ToRaw is legit and Marshal's SizeOf
operation cannot handle dynamic arrays? Or is the SizeOf operation
correct and I am overwriting memory I shouldn't be?

Marshal.SizeOf will only return the amount of memory the marshaller thinks
is necessary (and also what it will use to marshal data back). The
marshaller can't handle variable-sized arrays, so it will always return the
size as if the array contained one element.

I have no idea why .ToRaw() works, actually. It shouldn't. Apparently
the marshaller is clever enough to marshal the array fully, but not clever
enough to adjust its size measurements for it. You are not overwriting
memory that's not yours, so it shouldn't be a problem.