Obtaining size of struct array-type member from fixed pack=1 struct?

[taskswap]

I'm converting an application that relies heavily on a binary network
protocol. Within this protocol are a lot of byte[] arrays of character
data, like:

[StructLayout(LayoutKind.Sequential, Pack=1)]
public unsafe struct MsgAddEntry {
public byte MsgType;
public uint Tag;
public fixed byte ID[10];
public fixed byte Val1[20];
public fixed byte Opt1[20];
public fixed byte Val2[20];
public fixed byte Opt2[20];
}

I know, it's archaic, but it works very well and has the least network
impact of any method. I've been able to work with this data using the
above method so far, but I'm having one problem. Whenever I convert
to/from these byte fields, I have to hard-code length strings. That is,
IN ADDITION to the 20 it's hard-coded to above, I have to hard-code the
length in my code when I do the conversion. The compiler won't let me
use sizeof() or .Length on the array.

This is obviously pretty inconvenient, especially since there are
hundreds of messages and almost a thousand different fields. The fields
change once every few months, so having to change two locations is
almost unmaintainable. Is there a better way to do this?

While on this topic, is there a better way to convert these byte arrays
to strings? I can't use Encoding directly because these aren't "real"
managed byte arrays. So far I've been marshaling/copying the data into
a managed byte[] and then converting from there. This works, but it's
two memory copies of the data for a single conversion - not very
efficient. If it helps, the encoding will never change; it's a legacy
system, so it's good old one-byte-per-char ASCII.

Regards,
Chad

 

[Nicholas Paldino [.NET/C# MVP]]

Chad,

It looks like you want to use the MarshalAs attribute on the fields.
You ^should^ be able to do this:

[StructLayout(LayoutKind.Sequential, Pack=1, CharSet=CharSet.Ansi)]
public struct MsgAddEntry
{
public byte MsgType;
public uint Tag;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst=10)]
public string ID[10];
[MarshalAs(UnmanagedType.ByValTStr, SizeConst=20)]
public string Val1[20];
[MarshalAs(UnmanagedType.ByValTStr, SizeConst=20)]
public string Opt1[20];
[MarshalAs(UnmanagedType.ByValTStr, SizeConst=20)]
public string Val2[20];
[MarshalAs(UnmanagedType.ByValTStr, SizeConst=20)]
public string Opt2[20];
}

Now, I assume you are using unsafe code so that you can get a pointer to
this and then pass it to some unmanaged function. While this will work, it
doesn't suit your needs, since it requires you to hard code the lengths in a
way not associated with the struct.

With this, you can indicate how the structure is marshalled, and you can
also get the size of the structure by calling the static SizeOf method on
the Marshal class.

The only difference is that you would not need to use unsafe code (and
your performance may change as a result, since you aren't fixing buffers
(which cause the GC to work around it, if necessary), but at the same time,
you are consuming more memory because you are going to allocate memory for
the marshaling, if you aren't doing it already).

Hope this helps.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

I'm converting an application that relies heavily on a binary network
protocol. Within this protocol are a lot of byte[] arrays of character
data, like:

[StructLayout(LayoutKind.Sequential, Pack=1)]
public unsafe struct MsgAddEntry {
public byte MsgType;
public uint Tag;
public fixed byte ID[10];
public fixed byte Val1[20];
public fixed byte Opt1[20];
public fixed byte Val2[20];
public fixed byte Opt2[20];
}

I know, it's archaic, but it works very well and has the least network
impact of any method. I've been able to work with this data using the
above method so far, but I'm having one problem. Whenever I convert
to/from these byte fields, I have to hard-code length strings. That is,
IN ADDITION to the 20 it's hard-coded to above, I have to hard-code the
length in my code when I do the conversion. The compiler won't let me
use sizeof() or .Length on the array.

This is obviously pretty inconvenient, especially since there are
hundreds of messages and almost a thousand different fields. The fields
change once every few months, so having to change two locations is
almost unmaintainable. Is there a better way to do this?

While on this topic, is there a better way to convert these byte arrays
to strings? I can't use Encoding directly because these aren't "real"
managed byte arrays. So far I've been marshaling/copying the data into
a managed byte[] and then converting from there. This works, but it's
two memory copies of the data for a single conversion - not very
efficient. If it helps, the encoding will never change; it's a legacy
system, so it's good old one-byte-per-char ASCII.

Regards,
Chad

 

[taskswap]

OK, thanks. The ByValTStr has helped a lot. However, I still have a
piece of unsafe code used to transfer each message into its appropriate
structure. It looks like:

fixed (void *p = &buffer[offset]) {
IntPtr ip = new IntPtr(p);
m = (MsgAddProduct) Marshal.PtrToStructure(ip,
typeof(MsgAddProduct);
}

I have a byte[] buffer that contains my message, starting at offset. I
use this code to transfer the bytes into the new structure, defined as
you've listed above.

This all works, but it still seems somewhat inefficient. What's the
overhead involved in the work above? This is going to happen thousands
of times a second, so I need it to be as efficient as possible. In C I
used to just define a structure, cast the buffer pointer to a structure
pointer, and access my fields. This works in unsafe mode in C#, but not
with strings. I haven't found any better way of accessing string data
than what you've defined above.

Next inefficiency is on send side. I need to send from byte[] buffers
but my message size is limited, so I may have to split a structure
across multiple messages. Somehow I need to get a structure divided up
into one or more byte[] buffers.

I currently do this:
public unsafe void SendObject(object x) {
Monitor.Enter(rawslot);
Marshal.StructureToPtr(x, rawslot, false);
Send((byte *) rawslot, Marshal.SizeOf(x));
Monitor.Exit(rawslot);
}

rawslot is a 16K block created with AllocHGlobal. But this seems
inefficient - I marshal the struct to unmanaged memory, then copy it
AGAIN into each message buffer? Now that the messages are managed
structures, the compiler won't let me make a fixed byte* pointer or
byte[] reference to them - it says they're already fixed. The above
code is the only thing I've found that works.

Is there a better way to do either of these? Can I possibly union a
byte array into the structure somehow? That would make things much
easier, but I haven't found the right syntax.

Thanks,
Chad

 

[Nicholas Paldino [.NET/C# MVP]]

Chad,

Short of manipulating everything in bytes yourself, there is little you
can do that will improve the performance. Because you want to work with the
fields as strings, you will have to perform some marshaling back and forth
(because strings in .NET are composed of characters, which are two bytes
each).

If you go back to using the byte arrays in the structure (which you
could do using the ByValArray value on the UnmanagedType enumeration), it is
the equivalent of what marshalling is essentially doing for you.

BTW, your locking code is incorrect. You are locking on rawslot, which
is of type IntPtr. This is a structure, and when passed as a parameter of
type object, it is boxed, with a new object instance each time. This means
that you actually lock on the buffer the first time, and never let it go
(the call to Exit will box with a different object).

I can't see how this method runs on more than one thread, considering
you never give up the lock.

On the one hand, you can eliminate the lock, and allocate the buffer as
needed each time this is called. However, this might be a little
inefficient. You could also share a buffer among all the threads, but then
you would have to lock on it.

I think the best solution is to have a buffer which is allocated
per-thread (when the threads are created). You can do this easily by
declaring a field as static, and adding the ThreadStaticAttribute to the
field. The CLR will make sure that each thread has it's own instance.

With this in place, you can assign the buffer to each thread (when the
thread starts, so you don't have to check each time), and then not have to
lock.

If you don't do a per-thread approach, and need a lock, then use the
lock statement. If your method throws an exception, your lock will be
released properly.

Also, you should have a separate static field with an instance of object
stored in it. You can use this as your lock.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

OK, thanks. The ByValTStr has helped a lot. However, I still have a
piece of unsafe code used to transfer each message into its appropriate
structure. It looks like:

fixed (void *p = &buffer[offset]) {
IntPtr ip = new IntPtr(p);
m = (MsgAddProduct) Marshal.PtrToStructure(ip,
typeof(MsgAddProduct);
}

I have a byte[] buffer that contains my message, starting at offset. I
use this code to transfer the bytes into the new structure, defined as
you've listed above.

This all works, but it still seems somewhat inefficient. What's the
overhead involved in the work above? This is going to happen thousands
of times a second, so I need it to be as efficient as possible. In C I
used to just define a structure, cast the buffer pointer to a structure
pointer, and access my fields. This works in unsafe mode in C#, but not
with strings. I haven't found any better way of accessing string data
than what you've defined above.

Next inefficiency is on send side. I need to send from byte[] buffers
but my message size is limited, so I may have to split a structure
across multiple messages. Somehow I need to get a structure divided up
into one or more byte[] buffers.

I currently do this:
public unsafe void SendObject(object x) {
Monitor.Enter(rawslot);
Marshal.StructureToPtr(x, rawslot, false);
Send((byte *) rawslot, Marshal.SizeOf(x));
Monitor.Exit(rawslot);
}

rawslot is a 16K block created with AllocHGlobal. But this seems
inefficient - I marshal the struct to unmanaged memory, then copy it
AGAIN into each message buffer? Now that the messages are managed
structures, the compiler won't let me make a fixed byte* pointer or
byte[] reference to them - it says they're already fixed. The above
code is the only thing I've found that works.

Is there a better way to do either of these? Can I possibly union a
byte array into the structure somehow? That would make things much
easier, but I haven't found the right syntax.

Thanks,
Chad

 

[taskswap]

Thanks for your response, this seems to be working well now. I switched
the lock to an object, too.

Thinking into the future, if I had more control of the protocol, what
would be a better way to do this? Should I investigate serialization?
What I'm asking, specifically, is about efficiency - what would be the
most efficient method to get a message from one system to another,
using small, binary network packets, with minimal CPU overhead in
translation? I have to allow for strings here, but there are areas
where I control both sender and receiver, so I could send 2-byte
characters if I wanted to.

Regards,
Chad