littleEndian/bigEndian

[Guest]

Hi,

as someone pulling to C# over from C++ I stumbled over something today for
which I was not able to find an answer:

HOW do I convert a byte block with a given endianess to an int32 and vice
versa?

In C++ I have this:

#pragma intrinsic(_byteswap_ushort)
#pragma intrinsic(_byteswap_ulong)

*myword = _byteswap_ulong(mybigendianword);

which results in a wonderful single assembler instruction.

The only documentation I found about endianess is on handling UTF8/16
strings, but not with binary stuff.

Can someone give me a hint how I perform such things FAST and not via my own
clumsy version?

thanks
doc

 

[Jon Skeet [C# MVP]]

as someone pulling to C# over from C++ I stumbled over something today for
which I was not able to find an answer:

HOW do I convert a byte block with a given endianess to an int32 and vice
versa?

Have a look at http://www.pobox.com/~skeet/csharp/miscutil
In particular, the EndianBinaryReader/Writer and EndianBitConverter.

Jon

 

[Guest]

Have a look at http://www.pobox.com/~skeet/csharp/miscutil
In particular, the EndianBinaryReader/Writer and EndianBitConverter.

"Every so often, someone on the newsgroup asks about something which doesn't
exist in the framework, but which I think should. Sometimes, I'll write some
code for them to plug the gap. "

Well, fits perfect I guess

I'll take a look into it, thanks.

doc

 

[GhostInAK]

Hello docschnipp,

Here's what I currently use (in VB):

Public Function ReadBigEndianInt32() As Int32

Dim tReturn As Int32 = 0
Dim tBuffer(3) As Byte

oStream.Read(tBuffer, 0, tBuffer.Length)

Array.Reverse(tBuffer)
tReturn = BitConverter.ToInt32(tBuffer, 0)

Return tReturn

End Function

And the inverse of course for Writing..

-Boo

 

[Jon Skeet [C# MVP]]

Here's what I currently use (in VB):

Public Function ReadBigEndianInt32() As Int32

Dim tReturn As Int32 = 0
Dim tBuffer(3) As Byte

oStream.Read(tBuffer, 0, tBuffer.Length)

Array.Reverse(tBuffer)
tReturn = BitConverter.ToInt32(tBuffer, 0)

Return tReturn

End Function

That's not guaranteed to work - calling Read on a stream isn't
guaranteed to read all the bytes you ask for. It's likely to, but it's
not guaranteed.

 

[Claes Bergefall]

IPAddress.NetworkToHostOrder
IPAddress.HostToNetworkOrder

(not sure why these are in the IPAddress class but they're static anyway)

/claes

 

[GhostInAK]

Hello Jon Skeet [C# MVP],

Jon,

Thanks for the comment. I became aware of that fact the other day and just
hadn't got around to updating the routine at the time of the post.

Thanks,
-Boo

 

[Guest]

IPAddress.NetworkToHostOrder
IPAddress.HostToNetworkOrder

(not sure why these are in the IPAddress class but they're static anyway)

/claes

Usually because the libs connected to sockets, tcp/ip etc. are early cross
platform libs and need all the translation of network to local endianess.

I now decided to use this variant until I find something better:

----------
UInt32 myBigEndian = 0xdeadbeef;
UInt32 mLittleEndian = ( (myBigEndian& 0xFF000000) >> 24) | ((myBigEndian&
0x00FF0000) >>8) | (( myBigEndian& 0x0000FF00) <<8 ) | ((myBigEndian&
0x000000FF) << 24);
----------

The big advantage is: it works for all scales of integers and is not too
clumsy. Unfortunately I found no source for something that allows the native
compiler to take use of the bswap mnemonic. I would really appreciate a way
to make this very fast. bswap takes 2 ( 1 cycle plus prefix ) cycles only,
the code above ~34 cycles.

I have to read a _lot_ of data, several dozends of megabytes so
investigation on this topic is a worthwhile goal for me.


thanks for input,

doc

 

[Guest]

Hi Guys,

I've made some tests, the Reverse() method seems to be the slowest of all.

The code is found below, results were:

Time 1: 3765 ( the .Reverse/BitConverter method )
Time 2: 1322 ( the manual reindexing/BitConverter method )
Time 3: 1071 ( the Bitconverter/reshift method )
(if UInt is not needed, only Int
Time 4: 191 ( the shift the bytes directly/OR method )


So I think I will stick to method 3 and 4. Does anyone knows how I can cast
a byte to uint?

Thanks for all your input.



UInt32 mLittleEndian;
int loopcount = 20000000;

s1 = Environment.TickCount;
Array.Copy(myOrg,0,myArr,0,4);
for (i = 0; i < loopcount; i++)
{
if ( BitConverter.IsLittleEndian )
{
Array.Reverse(myNew,0,4);
}
mLittleEndian = BitConverter.ToUInt32(myNew,0);
}
s2 = Environment.TickCount;
Array.Copy(myOrg,0,myArr,0,4);
for ( i = 0; i< loopcount ; i++)
{
if ( BitConverter.IsLittleEndian )
{

myNew[3] = myArr[0];
myNew[2] = myArr[1];
myNew[1] = myArr[2];
myNew[0] = myArr[3];
mLittleEndian = BitConverter.ToUInt32(myNew,0);
}
}
s3 = Environment.TickCount;

// UInt32 mLittleEndian;
for ( i = 0; i< loopcount ; i++)
{
// myBigEndian = BitConverter.ToUInt32(myNew,0);
myBigEndian = BitConverter.ToUInt32(myOrg,0);
if ( BitConverter.IsLittleEndian)
{
mLittleEndian = ( (myBigEndian & 0xFF000000) >> 24) |
((myBigEndian & 0x00FF0000) >>8) | (( myBigEndian &
0x0000FF00) <<8 ) | ((myBigEndian & 0x000000FF) << 24);
}
}
s4 = Environment.TickCount;
Int32 mSLittleEndian;
for ( i = 0; i< loopcount ; i++)
{
if ( BitConverter.IsLittleEndian)
{
mSLittleEndian = myArr[0] | (myArr[1] << 8) | (myArr[2] << 16) | (myArr[3]
<< 24);
}
}
s5 = Environment.TickCount;
s1 = s2-s1;
s2 = s3-s2;
s3 = s4-s3;
s4 = s5-s4;
Console.WriteLine("Time 1: {0} Time 2: {1} Time 3: {2} Time 4:
{3}",s1.ToString(),s2.ToString(),s3.ToString(),s4.ToString());

 

[Jeffrey Tan[MSFT]]

Hi Docschnipp,

Thanks for your feedback!

Yes, I see your concern, BSWAP instruction should be faster than manually
change the byte order. However, based on my experience, C# does not
contains the code statement to generate this specific instruction.(I think
JIT compiler takes control of the instructions generated)

To workaround this issue, I think you may use C/C++ to write a dll which
internally uses inline ASM to call BSWAP instruction. Then in C# code, you
may p/invoke the exported function of the C/C++ dll to call BSWAP
indirectly. You can also declare the C/C++ function as naked to eliminate
the function prolog and epilog.

Hope this helps!

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Jon Skeet [C# MVP]]

I have to read a _lot_ of data, several dozends of megabytes so
investigation on this topic is a worthwhile goal for me.

Are you really sure? I've just done a test with my MiscUtil library,
and I was able to read 100MB of data in BigEndian format (as Int32
values) in 2.5 seconds on my laptop. Do you really need to do it faster
than that?

 

[Guest]

Are you really sure? I've just done a test with my MiscUtil library,
and I was able to read 100MB of data in BigEndian format (as Int32
values) in 2.5 seconds on my laptop. Do you really need to do it faster
than that?

Assume you are running in a background process that is low prioritised and
the whole system is quite active due displaying graphics and controls in
various ways. I found that such threads can run for 1ms and be held by the
scheduler for 10ms, and the system load is not that high.

So you have a performance going down by factor 10.

An algorithm I did that made something in suitable time on simple testing,
was not so nice when I was using it in the final product. Therefore I just
want to investigate which version is best, and as you might have seen in my
other post, things are really different depending on how I code it.

I've taken a look at that library and it is really nice, but I just wanted
to take the opportunity to learn a bit more of C# interna than just use a
library.

thanks for your input
doc

p.s.: a nice read about my motivation:
http://www.joelonsoftware.com/Articles/fog0000000319.html

 

[Guest]

Hi Jeffrey,

Hi Docschnipp,

Yes, I see your concern, BSWAP instruction should be faster than manually
change the byte order. However, based on my experience, C# does not
contains the code statement to generate this specific instruction.(I think
JIT compiler takes control of the instructions generated)

Yes. I just thought, there is a kind of "hinting" for this, like in the C++
variant (the pragma allows the compiler to reduce the function call to a
single BSWAP instruction.

But I guess, a "byteswap" type instruction is not defined in the MSIL.

Anyway, the code I found is sufficient enough, just took the opportunity to
dive a bit into C# interna.

Thanks
doc

 

[Jon Skeet [C# MVP]]

Assume you are running in a background process that is low prioritised and
the whole system is quite active due displaying graphics and controls in
various ways. I found that such threads can run for 1ms and be held by the
scheduler for 10ms, and the system load is not that high.

If it's a background process, does it matter that much?

So you have a performance going down by factor 10.

Not really - if the system as a whole is busy, it just means that your
background process isn't disrupting the rest of the system.

An algorithm I did that made something in suitable time on simple testing,
was not so nice when I was using it in the final product. Therefore I just
want to investigate which version is best, and as you might have seen in my
other post, things are really different depending on how I code it.

That suggests that your simple testing wasn't representative. That's a
valuable lesson to learn, but you shouldn't go straight to the other
extreme and assume that every instruction counts. The lesson is to try
to *get* representative data for testing.

What's your actual situation going to be? How much data are you going
to have to process, and how long is it acceptable for it to take? Until
you know that, you won't know what your target is.

 

[Jeffrey Tan[MSFT]]

Hi Docschnipp,

Thanks for your feedback!

Yes, I see. C# is built upon the CLR, the performance is not its
specialist, C/C++/ASM will be better on this subject. So if we usually
place the performance sensitive code snippet in the C/C++ dll and use
p/invoke to invoke it.

To learn C# internals, we can divide it into 3 parts:
1. C# language
2. .Net Framework Class Library
3. .Net CLR

1. C# language is built upon the MSIL language, CLR and it leverages a
subset feature of MSIL language. "Inside Microsoft .NET IL Assembler" and
"CLR via C#, Second Edition" are 2 recommended books on this subject.
2. I think .Net Reflector is an invaluable tool to inspect the source code
of .Net FCL via different languages.
3. The 2 recommended books in #1 and "Essential .Net" have covered CLR to a
good point.

Hope this helps!

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Guest]

If it's a background process, does it matter that much?

I use it to perform things triggered by an user. So I want to keep response
to the user tight, but the background task should, of course, load the data
as fast as it can.

Not really - if the system as a whole is busy, it just means that your
background process isn't disrupting the rest of the system.

Yes, you are right. It is not 10. But it was quite puzzling to see it slow
down THAT much. The foreground work is a kind of realtime process with
elevated thread priority (15).

That suggests that your simple testing wasn't representative. That's a
valuable lesson to learn, but you shouldn't go straight to the other

Indeed The simple testing was okay for correctness but obviously not for
performance. I guess these are the things you won't learn in school and this
kind of knowledge makes you a bit of a better programmer.

extreme and assume that every instruction counts. The lesson is to try
to *get* representative data for testing.

What's your actual situation going to be? How much data are you going
to have to process, and how long is it acceptable for it to take? Until
you know that, you won't know what your target is.

This current thing shall load DICOM streams (used in medical imaging), they
can get quite large and can change their endianess within the stream. They
should be loaded in the background while the user itself works with already
loaded files (or parts of it). It also should run on mobile devices and that
is where the whole thing should at least avoid unfortunate coding style like
the (sorry GhostInAK) Array.Reverse() method.

I am surely not going to the extreme, I just wanted to explore a bit since I
am new to C# and the .NET framework. The solution I have now is totally
sufficient for the sake of 'clean' code.

Btw.: I appreciated the reading of your other articles about C#, nice writing.

best regards & thanks
doc

 

[Guest]

Hi Docschnipp,

Thanks for your feedback!

Yes, I see. C# is built upon the CLR, the performance is not its
specialist, C/C++/ASM will be better on this subject. So if we usually
place the performance sensitive code snippet in the C/C++ dll and use
p/invoke to invoke it.

1. (...)"Inside Microsoft .NET IL Assembler" and
"CLR via C#, Second Edition" are 2 recommended books on this subject.

Will take a look into it. In general I have the feeling that certain parts
of programming are kept out for the sake of cleaness and compatibility. And
maybe a (JIT-) Compiler would be clever enough to find out what I am doing
and may translate the shift/or commands into a bswap in future. This might be
really the future way, same way as inlining functions is automated by the
compiler. I was hoping there is something I can trigger in that way, but
currently I leave code optimisation out of the bill.

2. I think .Net Reflector is an invaluable tool to inspect the source code
of .Net FCL via different languages.

I took a look into the (x86) disassembly of my code which is quite
interesting, how things turn out later. But taking a look into the meta form
(MSIL) will be interesting, too.

3. The 2 recommended books in #1 and "Essential .Net" have covered CLR to a
good point.

Thanks for the book tips.

best
doc

 

[Jeffrey Tan[MSFT]]

Hi Docschnipp,

Thanks for your feedback!

Yes, I agree with that this is the key point. If JIT compiler is smart
enough to detect the running platform and the CPU instruction set, it
should be able to emit the most effective instruction to get the work done.
It seems .Net CLR compiler did not get to such degree currently.

Personally I have below suspicious: since .Net FCL does not provide a bswap
class and C# language does not provide a bswap statement, this may mean
that .Net does not include bswap function, so JIT compiler does not take
into emiting bswap instruction into plan/design. Also, this design has gone
out of my control, thanks for your understanding.

If you want to feedback this issue to our CLR/JIT team, it is a good
opporunity to submit request in the MSDN feedback center below:
http://connect.microsoft.com/Main/content/content.aspx?ContentID=2220

Our product team will receive your request and follow up with you. Hope
this helps.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Guest]

Yes, I agree with that this is the key point. If JIT compiler is smart
enough to detect the running platform and the CPU instruction set, it
should be able to emit the most effective instruction to get the work done.
It seems .Net CLR compiler did not get to such degree currently.

I imagine it is not too easy finding such an optimisation from a set of
SHIFTs and ORs

Personally I have below suspicious: since .Net FCL does not provide a bswap
class and C# language does not provide a bswap statement, this may mean
that .Net does not include bswap function, so JIT compiler does not take
into emiting bswap instruction into plan/design. Also, this design has gone
out of my control, thanks for your understanding.

Of course. Never mind.

If you want to feedback this issue to our CLR/JIT team, it is a good
opporunity to submit request in the MSDN feedback center below:
http://connect.microsoft.com/Main/content/content.aspx?ContentID=2220

Done that. I tried to consider where I would expect such a thing and where a
framework/compiler designer could possible take advantage of the hint and my
idea is now, that BitConverter should have static functions that read a given
endianess and convert it to the platform endianess.

Example:
Uint32 myHeaderSize = BitConverter.ReadUInt32BigEndian(buffer);

Before reading a file or stream one usually determines the format and would
just use this and would not need to care about the platform (which might
transition somewhere in the future [I would love to see this] ).

I know that there is the NetWorkToHost and vice versa functions in the
network namespace but this is just for 32bit integers, BitConverter should
provide it for
all multibyte integers (and maybe even floats).

Our product team will receive your request and follow up with you. Hope
this helps.

Looking forward to this.

Thanks
doc

 

[Jeffrey Tan[MSFT]]

Yes, based on my review, your feedback and statement makes sense, it is a
good idea to add a such useful method in BitConverter class.

Just as a reminder: since there is no service guarantee to this feedback
request and due to the time limit of developer team, it may cost some time
for them to give you a useful reply.

Thanks.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.