What's the best way to accumulate data?

[John]

Hi,

I have an application that receives data (floats) via a socket. My job
is to receive the bytes, byte-swap them and convert them to floats,
and make them available to the application.

The data arrives in packets. Each packet has a header telling me how
many floats will follow. So I create an array with this many elements
and then read them in to the array. There are usually 10,000 to 30,000
samples in each packet.

Sometimes the data is split between multiple packets. I need to
assemble all the data from all the packets into a single array. This
is my question. What's the best way to do this?

Here was my thought: I would still create arrays for each packet. I
would put each packet array into an array list (I don't know how many
packets there will be), then when all the packets are received (a flag
in the header tells me this), I would copy all the arrays into one
super large array (it may have 200,000 floats in it).

Does this seem reasonable? Any other ideas?

This is a real-time system and I receive a packet, or set of packets,
every 200 miliseconds.

Thanks,
John

 

[Peter Huang]

Hi John,

I think we can use the MemoryStream and the BinaryWriter to accumulate the
data.
Every time a packet arrives, we just check the header to see if this the
last packet and write payload(that the byte array that contain the data we
need) to the memorystream.

e.g.
MemoryStream ms = new MemoryStream();
BinaryWriter bw = new BinaryWriter(ms);
byte[] pb = new byte[16];
//first packet
for (int i =0; i< 16;i++)
pb=(byte)i;
//pb is what we will get from the packet
//Writer from the 4th byte
bw.Write(pb,4,pb.Length-4);
//second packet
for (int i =0; i< 16;i++)
pb=(byte)(i+16);
bw.Write(pb);
byte[] pbms=ms.ToArray();
for (int i=0;i<pbms.Length;i++)
Console.WriteLine(pbms);
bw.Flush();
bw.Close();
ms.Close();

Best regards,

Peter Huang
Microsoft Online Partner Support

Get Secure! - www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Peter Huang]

Hi John,

Did my suggestion help you that use the memorystream to accumulate the data?
If you still have any concern on this issue, please feel free to post here.

Best regards,

Peter Huang
Microsoft Online Partner Support

Get Secure! - www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[John]

Peter,

Yes, your reply did help. Thank you very much.

I'm still studying the implementation and how it impacts the
performance of my application. If I have more questions, I'll post.

Thank you!
John

 

[John]

Peter,

If I put bytes in the MemoryStream are they "boxed" or do they stay as
bytes? What about when I convert to an array?

Thanks,
John

 

[John]

Peter,

Is there a way that I can re-use the memory stream for subsequent
sweeps of data?

I think I can set the position to 0, but if the length of the second
sweep of data is less than the first, the memorystream won't contract.

I guess I have to:

ms = new MemoryStream();

before every sweep of data?

Thanks,
John

 

[Jon Skeet [C# MVP]]

If I put bytes in the MemoryStream are they "boxed" or do they stay as
bytes? What about when I convert to an array?

No, they're not bytes, because they're in an array, and an array (of
type byte[]) doesn't require boxing.

 

[Peter Huang]

Hi John,

When we call the MemoryStream.Write or MemoryStream.ToArray method, since
the parameter is byte array and the require parameter is byte, we will not
do the boxing operation. Since we are using the encapsulated class of .net
to manipulate the byte array, we will not care the concrete implement of
the MemoryStream.

If you do care the optimum space and time effiency of the code, I think you
may have to manipulate the byte array yourself.
e.g.
we can declare a bytearray of proper size and them copy the data from the
packet into the bytearray ourselves and we can manipulate the bytearray
with our will.

Also Memory has another method GetBuffer, we can get the underlying buffer
for the memorystream and manipulate the data in the memorystream.
As for reuse the memorystream, I think we can use the SetLength to gear the
size of the memorystream.

e.g.
MemoryStream ms = new MemoryStream();
BinaryWriter bw = new BinaryWriter(ms);
byte[] pb = new byte[15];
//first packet
for (int i =0; i< 15;i++)
pb=(byte)i;
bw.Write(pb);
byte[] pbms=ms.GetBuffer();
byte temp = pbms[0];
pbms[0]=pbms[1];
pbms[1]=temp;
for (int i=0;i<ms.Length ;i++)
Console.Write(pbms);
Console.WriteLine();
//second packet
ms.Position=0;
ms.SetLength(13);
Console.WriteLine(ms.Length);
pb = new byte[13];
for (int i =0; i< 13;i++)
pb=(byte)i;
bw.Write(pb);
pbms=ms.ToArray();
for (int i=0;i<pbms.Length;i++)
Console.Write(pbms);
Console.WriteLine();
bw.Close();
ms.Close();


Best regards,

Peter Huang
Microsoft Online Partner Support

Get Secure! - www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Peter Huang]

Hi John,

Do you have any concern on my suggestion that use the SetLength to adjust
the memorystream?
If you still have any concern on this issue, please feel free to post here.


Best regards,

Peter Huang
Microsoft Online Partner Support

Get Secure! - www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.