Nine Language Performance Round-up: Benchmarking Math & File I/O - not fair to VB.NET

T

Tom Gurath

http://osnews.com/story.php?news_id=5602&page=2

This benchmark tests the Math & File I/O of 9 languages/run-times.
Visual C++ (Version 7 - not managed)
Visual C#
gcc C
Visual Basic.NET
Visual J#
Java 1.3.1
Java 1.4.2
Python/Psyco
Python

In the test (code available from the site), they didn't implement the VB.NET code fairly. It is not syntactically equivalent. For the other languages, they used streaming file i/o. With VB.NET, they used FileOpen and PrintLine. With the other languages, the text written to the file was a literal string. With VB.NET, they placed the text in a String and wrote that to the file.

Here are the results they posted: (in seconds)

int
math long
math double
math
trig
I/O
TOTAL
Visual C++ 9.6 18.8 6.4 3.5 10.5 48.8
Visual C# 9.7 23.9 17.7 4.1 9.9 65.3
gcc C 9.8 28.8 9.5 14.9 10.0 73.0
Visual Basic 9.8 23.7 17.7 4.1 30.7 85.9
Visual J# 9.6 23.9 17.5 4.2 35.1 90.4
Java 1.3.1 14.5 29.6 19.0 22.1 12.3 97.6
Java 1.4.2 9.3 20.2 6.5 57.1 10.1 103.1
Python/Psyco 29.7 615.4 100.4 13.1 10.5 769.1
Python 322.4 891.9 405.7 47.1 11.9 1679.0


I modified the VB.NET code to use streaming file i/o and a literal string. Below are the results on my machine: (in milliseconds)
* I ran each test 3 times, rebooting between each language - the averages are in bold.

C++ Averages

int 6312 6265 6265 6280
dbl 4266 4266 4297 4276
lng 13172 12547 12563 12760
trig 2296 2297 2296 2296
IO 3329 3484 3219 3344
Total 29375 28859 28640 28958

C#

int 6359 6343 6343 6348
dbl 11656 11656 11656 11656
lng 16640 15921 15984 16181
trig 2750 2734 2734 2739
IO 2843 3781 2421 3015
Total 40248 40435 39138 39940

VB - Original

int 6422 6453 6453 6442
dbl 11984 12016 12000 12000
lng 16438 15797 15750 15995
trig 2797 2734 2766 2765
IO 28062 27734 27672 27822
Total 65703 64734 64641 65026

VB - Streaming IO

int 6453 6453 6438 6448
dbl 11984 11953 11953 11963
lng 16516 15719 15750 15995
trig 2766 2734 2734 2744
IO 2922 3328 2906 3052
Total 40641 40187 39781 40203

Java 1.4.2_2 (*Gave me errors - I couldn't test it.)

C:\>javac -g:none benchmark.java

C:\>java -server benchmark.class
Error: no `server' JVM at `C:\Program Files\Java\j2re1.4.2_01\bin\server\jvm.dll'.

C:\>java benchmark.class
Exception in thread "main" java.lang.NoClassDefFoundError: benchmark/class

As you can see, the VB.NET I/O is about equal to the C# I/O. Both ran faster than using C++ without the CLR. I will be e-mailing the author of the article and sending him my source code and results. I was hoping to contact someone on the VB team at Microsoft, but haven't been able to find addresses for the right individuals. Hopefully someone there will see this post.
 
O

One Handed Man [ OHM# ]

As you can see, the VB.NET I/O is about equal to the C# I/O.

//*Both* ran faster than using C++ *without* the CLR.//

Can you explain what you mean in more detail here, It can be read more than one way !

--
Best Regards - OHM

O_H_M{at}BTInternet{dot}com

http://osnews.com/story.php?news_id=5602&page=2

This benchmark tests the Math & File I/O of 9 languages/run-times.
Visual C++ (Version 7 - not managed)
Visual C#
gcc C
Visual Basic.NET
Visual J#
Java 1.3.1
Java 1.4.2
Python/Psyco
Python

In the test (code available from the site), they didn't implement the VB.NET code fairly. It is not syntactically equivalent. For the other languages, they used streaming file i/o. With VB.NET, they used FileOpen and PrintLine. With the other languages, the text written to the file was a literal string. With VB.NET, they placed the text in a String and wrote that to the file.

Here are the results they posted: (in seconds)

int
math long
math double
math
trig
I/O
TOTAL
Visual C++ 9.6 18.8 6.4 3.5 10.5 48.8
Visual C# 9.7 23.9 17.7 4.1 9.9 65.3
gcc C 9.8 28.8 9.5 14.9 10.0 73.0
Visual Basic 9.8 23.7 17.7 4.1 30.7 85.9
Visual J# 9.6 23.9 17.5 4.2 35.1 90.4
Java 1.3.1 14.5 29.6 19.0 22.1 12.3 97.6
Java 1.4.2 9.3 20.2 6.5 57.1 10.1 103.1
Python/Psyco 29.7 615.4 100.4 13.1 10.5 769.1
Python 322.4 891.9 405.7 47.1 11.9 1679.0


I modified the VB.NET code to use streaming file i/o and a literal string. Below are the results on my machine: (in milliseconds)
* I ran each test 3 times, rebooting between each language - the averages are in bold.

C++ Averages

int 6312 6265 6265 6280
dbl 4266 4266 4297 4276
lng 13172 12547 12563 12760
trig 2296 2297 2296 2296
IO 3329 3484 3219 3344
Total 29375 28859 28640 28958

C#

int 6359 6343 6343 6348
dbl 11656 11656 11656 11656
lng 16640 15921 15984 16181
trig 2750 2734 2734 2739
IO 2843 3781 2421 3015
Total 40248 40435 39138 39940

VB - Original

int 6422 6453 6453 6442
dbl 11984 12016 12000 12000
lng 16438 15797 15750 15995
trig 2797 2734 2766 2765
IO 28062 27734 27672 27822
Total 65703 64734 64641 65026

VB - Streaming IO

int 6453 6453 6438 6448
dbl 11984 11953 11953 11963
lng 16516 15719 15750 15995
trig 2766 2734 2734 2744
IO 2922 3328 2906 3052
Total 40641 40187 39781 40203

Java 1.4.2_2 (*Gave me errors - I couldn't test it.)

C:\>javac -g:none benchmark.java

C:\>java -server benchmark.class
Error: no `server' JVM at `C:\Program Files\Java\j2re1.4.2_01\bin\server\jvm.dll'.

C:\>java benchmark.class
Exception in thread "main" java.lang.NoClassDefFoundError: benchmark/class

As you can see, the VB.NET I/O is about equal to the C# I/O. Both ran faster than using C++ without the CLR. I will be e-mailing the author of the article and sending him my source code and results. I was hoping to contact someone on the VB team at Microsoft, but haven't been able to find addresses for the right individuals. Hopefully someone there will see this post.
 
H

Herfried K. Wagner [MVP]

* "Tom Gurath said:
http://osnews.com/story.php?news_id=5602&page=2
Click to expand...

Really interesting.

<quote>
About the author:
Christopher W. Cowell-Shah works in Palo Alto as a consultant for the
Accenture Technology Labs (the research & development wing of
Accenture). He has an A.B. in computer science from Harvard and a
Ph.D. in philosophy from Berkeley. Chris is especially interested in
issues in artificial intelligence, human/computer interaction and
security. His website is www.cowell-shah.com.
</quote>
 
T

Tom Gurath

Visual C++ still allows you to write non-managed code and compile that way. I'm not a C++ guy, so maybe my wording was incorrect, but I compiled without the /clr option. In fact, he specified the following as his compile options:

Visual C++: used "release" configuration, turned on "whole program optimization," set "optimization" to "maximize speed," turned on "global optimizations," turned on "enable intrinsic functions," set "favor size or speed" to "favor fast code," set "omit frame pointers" to "yes," set "optimize for processor" to "Pentium 4 and above," set "buffer security check" to "no," set "enable enhanced instruction set" to "SIMD2," and set "optimize for Windows98" to "no" within Visual Studio.

With whole program optimization turned on, you cannot compile with clr enabled. I found that out the hard way and had to go figure out how to turn it off.

In the statement about *both* being faster, I was referring to C# and VB.NET (with streaming I/O) having averaged faster results than C++ in this test.
--Tom
"One Handed Man [ OHM# ]" <O_H_M{at}BTInternet{dot}com> wrote in message As you can see, the VB.NET I/O is about equal to the C# I/O.

//*Both* ran faster than using C++ *without* the CLR.//

Can you explain what you mean in more detail here, It can be read more than one way !

--
Best Regards - OHM

O_H_M{at}BTInternet{dot}com

http://osnews.com/story.php?news_id=5602&page=2

This benchmark tests the Math & File I/O of 9 languages/run-times.
Visual C++ (Version 7 - not managed)
Visual C#
gcc C
Visual Basic.NET
Visual J#
Java 1.3.1
Java 1.4.2
Python/Psyco
Python

In the test (code available from the site), they didn't implement the VB.NET code fairly. It is not syntactically equivalent. For the other languages, they used streaming file i/o. With VB.NET, they used FileOpen and PrintLine. With the other languages, the text written to the file was a literal string. With VB.NET, they placed the text in a String and wrote that to the file.

Here are the results they posted: (in seconds)

int
math long
math double
math
trig
I/O
TOTAL
Visual C++ 9.6 18.8 6.4 3.5 10.5 48.8
Visual C# 9.7 23.9 17.7 4.1 9.9 65.3
gcc C 9.8 28.8 9.5 14.9 10.0 73.0
Visual Basic 9.8 23.7 17.7 4.1 30.7 85.9
Visual J# 9.6 23.9 17.5 4.2 35.1 90.4
Java 1.3.1 14.5 29.6 19.0 22.1 12.3 97.6
Java 1.4.2 9.3 20.2 6.5 57.1 10.1 103.1
Python/Psyco 29.7 615.4 100.4 13.1 10.5 769.1
Python 322.4 891.9 405.7 47.1 11.9 1679.0


I modified the VB.NET code to use streaming file i/o and a literal string. Below are the results on my machine: (in milliseconds)
* I ran each test 3 times, rebooting between each language - the averages are in bold.

C++ Averages

int 6312 6265 6265 6280
dbl 4266 4266 4297 4276
lng 13172 12547 12563 12760
trig 2296 2297 2296 2296
IO 3329 3484 3219 3344
Total 29375 28859 28640 28958

C#

int 6359 6343 6343 6348
dbl 11656 11656 11656 11656
lng 16640 15921 15984 16181
trig 2750 2734 2734 2739
IO 2843 3781 2421 3015
Total 40248 40435 39138 39940

VB - Original

int 6422 6453 6453 6442
dbl 11984 12016 12000 12000
lng 16438 15797 15750 15995
trig 2797 2734 2766 2765
IO 28062 27734 27672 27822
Total 65703 64734 64641 65026

VB - Streaming IO

int 6453 6453 6438 6448
dbl 11984 11953 11953 11963
lng 16516 15719 15750 15995
trig 2766 2734 2734 2744
IO 2922 3328 2906 3052
Total 40641 40187 39781 40203

Java 1.4.2_2 (*Gave me errors - I couldn't test it.)

C:\>javac -g:none benchmark.java

C:\>java -server benchmark.class
Error: no `server' JVM at `C:\Program Files\Java\j2re1.4.2_01\bin\server\jvm.dll'.

C:\>java benchmark.class
Exception in thread "main" java.lang.NoClassDefFoundError: benchmark/class

As you can see, the VB.NET I/O is about equal to the C# I/O. Both ran faster than using C++ without the CLR. I will be e-mailing the author of the article and sending him my source code and results. I was hoping to contact someone on the VB team at Microsoft, but haven't been able to find addresses for the right individuals. Hopefully someone there will see this post.
 
O

One Handed Man [ OHM# ]

Well, then I am impressed with your results.

--
Best Regards - OHM

O_H_M{at}BTInternet{dot}com

Visual C++ still allows you to write non-managed code and compile that way. I'm not a C++ guy, so maybe my wording was incorrect, but I compiled without the /clr option. In fact, he specified the following as his compile options:

Visual C++: used "release" configuration, turned on "whole program optimization," set "optimization" to "maximize speed," turned on "global optimizations," turned on "enable intrinsic functions," set "favor size or speed" to "favor fast code," set "omit frame pointers" to "yes," set "optimize for processor" to "Pentium 4 and above," set "buffer security check" to "no," set "enable enhanced instruction set" to "SIMD2," and set "optimize for Windows98" to "no" within Visual Studio.

With whole program optimization turned on, you cannot compile with clr enabled. I found that out the hard way and had to go figure out how to turn it off.

In the statement about *both* being faster, I was referring to C# and VB.NET (with streaming I/O) having averaged faster results than C++ in this test.
--Tom
"One Handed Man [ OHM# ]" <O_H_M{at}BTInternet{dot}com> wrote in message As you can see, the VB.NET I/O is about equal to the C# I/O.

//*Both* ran faster than using C++ *without* the CLR.//

Can you explain what you mean in more detail here, It can be read more than one way !

--
Best Regards - OHM

O_H_M{at}BTInternet{dot}com

http://osnews.com/story.php?news_id=5602&page=2

This benchmark tests the Math & File I/O of 9 languages/run-times.
Visual C++ (Version 7 - not managed)
Visual C#
gcc C
Visual Basic.NET
Visual J#
Java 1.3.1
Java 1.4.2
Python/Psyco
Python

In the test (code available from the site), they didn't implement the VB.NET code fairly. It is not syntactically equivalent. For the other languages, they used streaming file i/o. With VB.NET, they used FileOpen and PrintLine. With the other languages, the text written to the file was a literal string. With VB.NET, they placed the text in a String and wrote that to the file.

Here are the results they posted: (in seconds)

int
math long
math double
math
trig
I/O
TOTAL
Visual C++ 9.6 18.8 6.4 3.5 10.5 48.8
Visual C# 9.7 23.9 17.7 4.1 9.9 65.3
gcc C 9.8 28.8 9.5 14.9 10.0 73.0
Visual Basic 9.8 23.7 17.7 4.1 30.7 85.9
Visual J# 9.6 23.9 17.5 4.2 35.1 90.4
Java 1.3.1 14.5 29.6 19.0 22.1 12.3 97.6
Java 1.4.2 9.3 20.2 6.5 57.1 10.1 103.1
Python/Psyco 29.7 615.4 100.4 13.1 10.5 769.1
Python 322.4 891.9 405.7 47.1 11.9 1679.0


I modified the VB.NET code to use streaming file i/o and a literal string. Below are the results on my machine: (in milliseconds)
* I ran each test 3 times, rebooting between each language - the averages are in bold.

C++ Averages

int 6312 6265 6265 6280
dbl 4266 4266 4297 4276
lng 13172 12547 12563 12760
trig 2296 2297 2296 2296
IO 3329 3484 3219 3344
Total 29375 28859 28640 28958

C#

int 6359 6343 6343 6348
dbl 11656 11656 11656 11656
lng 16640 15921 15984 16181
trig 2750 2734 2734 2739
IO 2843 3781 2421 3015
Total 40248 40435 39138 39940

VB - Original

int 6422 6453 6453 6442
dbl 11984 12016 12000 12000
lng 16438 15797 15750 15995
trig 2797 2734 2766 2765
IO 28062 27734 27672 27822
Total 65703 64734 64641 65026

VB - Streaming IO

int 6453 6453 6438 6448
dbl 11984 11953 11953 11963
lng 16516 15719 15750 15995
trig 2766 2734 2734 2744
IO 2922 3328 2906 3052
Total 40641 40187 39781 40203

Java 1.4.2_2 (*Gave me errors - I couldn't test it.)

C:\>javac -g:none benchmark.java

C:\>java -server benchmark.class
Error: no `server' JVM at `C:\Program Files\Java\j2re1.4.2_01\bin\server\jvm.dll'.

C:\>java benchmark.class
Exception in thread "main" java.lang.NoClassDefFoundError: benchmark/class

As you can see, the VB.NET I/O is about equal to the C# I/O. Both ran faster than using C++ without the CLR. I will be e-mailing the author of the article and sending him my source code and results. I was hoping to contact someone on the VB team at Microsoft, but haven't been able to find addresses for the right individuals. Hopefully someone there will see this post.
 
T

Tom Gurath

The author of the article has since admitted his mistakes, not just in the VB.NET test but in several others as well. His comments are posted on the web site where the benchmark results are posted.
Visual C++ still allows you to write non-managed code and compile that way. I'm not a C++ guy, so maybe my wording was incorrect, but I compiled without the /clr option. In fact, he specified the following as his compile options:

Visual C++: used "release" configuration, turned on "whole program optimization," set "optimization" to "maximize speed," turned on "global optimizations," turned on "enable intrinsic functions," set "favor size or speed" to "favor fast code," set "omit frame pointers" to "yes," set "optimize for processor" to "Pentium 4 and above," set "buffer security check" to "no," set "enable enhanced instruction set" to "SIMD2," and set "optimize for Windows98" to "no" within Visual Studio.

With whole program optimization turned on, you cannot compile with clr enabled. I found that out the hard way and had to go figure out how to turn it off.

In the statement about *both* being faster, I was referring to C# and VB.NET (with streaming I/O) having averaged faster results than C++ in this test.
--Tom
"One Handed Man [ OHM# ]" <O_H_M{at}BTInternet{dot}com> wrote in message As you can see, the VB.NET I/O is about equal to the C# I/O.

//*Both* ran faster than using C++ *without* the CLR.//

Can you explain what you mean in more detail here, It can be read more than one way !

--
Best Regards - OHM

O_H_M{at}BTInternet{dot}com

http://osnews.com/story.php?news_id=5602&page=2

This benchmark tests the Math & File I/O of 9 languages/run-times.
Visual C++ (Version 7 - not managed)
Visual C#
gcc C
Visual Basic.NET
Visual J#
Java 1.3.1
Java 1.4.2
Python/Psyco
Python

In the test (code available from the site), they didn't implement the VB.NET code fairly. It is not syntactically equivalent. For the other languages, they used streaming file i/o. With VB.NET, they used FileOpen and PrintLine. With the other languages, the text written to the file was a literal string. With VB.NET, they placed the text in a String and wrote that to the file.

Here are the results they posted: (in seconds)

int
math long
math double
math
trig
I/O
TOTAL
Visual C++ 9.6 18.8 6.4 3.5 10.5 48.8
Visual C# 9.7 23.9 17.7 4.1 9.9 65.3
gcc C 9.8 28.8 9.5 14.9 10.0 73.0
Visual Basic 9.8 23.7 17.7 4.1 30.7 85.9
Visual J# 9.6 23.9 17.5 4.2 35.1 90.4
Java 1.3.1 14.5 29.6 19.0 22.1 12.3 97.6
Java 1.4.2 9.3 20.2 6.5 57.1 10.1 103.1
Python/Psyco 29.7 615.4 100.4 13.1 10.5 769.1
Python 322.4 891.9 405.7 47.1 11.9 1679.0


I modified the VB.NET code to use streaming file i/o and a literal string. Below are the results on my machine: (in milliseconds)
* I ran each test 3 times, rebooting between each language - the averages are in bold.

C++ Averages

int 6312 6265 6265 6280
dbl 4266 4266 4297 4276
lng 13172 12547 12563 12760
trig 2296 2297 2296 2296
IO 3329 3484 3219 3344
Total 29375 28859 28640 28958

C#

int 6359 6343 6343 6348
dbl 11656 11656 11656 11656
lng 16640 15921 15984 16181
trig 2750 2734 2734 2739
IO 2843 3781 2421 3015
Total 40248 40435 39138 39940

VB - Original

int 6422 6453 6453 6442
dbl 11984 12016 12000 12000
lng 16438 15797 15750 15995
trig 2797 2734 2766 2765
IO 28062 27734 27672 27822
Total 65703 64734 64641 65026

VB - Streaming IO

int 6453 6453 6438 6448
dbl 11984 11953 11953 11963
lng 16516 15719 15750 15995
trig 2766 2734 2734 2744
IO 2922 3328 2906 3052
Total 40641 40187 39781 40203

Java 1.4.2_2 (*Gave me errors - I couldn't test it.)

C:\>javac -g:none benchmark.java

C:\>java -server benchmark.class
Error: no `server' JVM at `C:\Program Files\Java\j2re1.4.2_01\bin\server\jvm.dll'.

C:\>java benchmark.class
Exception in thread "main" java.lang.NoClassDefFoundError: benchmark/class

As you can see, the VB.NET I/O is about equal to the C# I/O. Both ran faster than using C++ without the CLR. I will be e-mailing the author of the article and sending him my source code and results. I was hoping to contact someone on the VB team at Microsoft, but haven't been able to find addresses for the right individuals. Hopefully someone there will see this post.
 
Top