Numeric Casting Bug?

[Guest]

I have just found a problem in a new Dot Net program I have written. At
it's most basic, if you step into a VB application so that you can run the
command window and type:

Debug.print 1.2-1

The return value is:

0.19999999999999996

However, if you type

Debug.print cdec(1.2)-cdec(1)

the return value is:

0.2 as expected.

Has anyone else come across this, or now of why this occurs?

Any help would be appreciated!

Mike

 

[Herfried K. Wagner [MVP]]

I have just found a problem in a new Dot Net program I have written. At
it's most basic, if you step into a VB application so that you can run the
command window and type:

Debug.print 1.2-1

The return value is:

0.19999999999999996

However, if you type

Debug.print cdec(1.2)-cdec(1)

the return value is:

0.2 as expected.

Has anyone else come across this, or now of why this occurs?

That's caused by the floating point representation:

<URL:http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html>
<URL:http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html>
<URL:http://support.microsoft.com/?scid=kb;[LN];42980>

 

[Jay B. Harlow [MVP - Outlook]]

Mike,
In addition to Herfried's links, check out:
http://www.yoda.arachsys.com/csharp/floatingpoint.html

http://www.yoda.arachsys.com/csharp/decimal.html

Debug.print 1.2-1

1.2 & 1 are doubles, so you are using doubles (binary floating point) for
the calculation.

Debug.print cdec(1.2)-cdec(1)

You converted the doubles to Decimals, so you are using Decimal (decimal
floating point) for the calculation.

Instead of CDec(1.2) you could have written 1.2D, where 1.2D is a Decimal
literal, and 1.2 (or 1.2R) is a Double literal, 1.2F happens to be a Single
literal.

Hope this helps
Jay