weird problem ... Single

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Hi,

I have 2 single variable.
i.e Var1 is 6 and Var2 is 5.94, both are As Single.

but when I do Var1 - Var2, the answer give me 0.0599994.
and when I do Var1 + Var2, the answer give me 11.94

How can i make it so when Var1-Var2 should be = 0.06? how come it is giving
me 0.0599994???

i dont understand, please help, Thanks!
 
I remember when Intel stock plummeted because a scientist found a miniscule
rounding error in a pentium processor.

I tried your example, and got the error. I use an AMD processor. The same
problem occurs using Doubles.

When I changed the numbers being added (or subtracted) to Decimal type, and
left the result as Single, (or Double, or Decimal), the result was correct.

This is very strange.
 
Jon said:
I have 2 single variable.
i.e Var1 is 6 and Var2 is 5.94, both are As Single.

but when I do Var1 - Var2, the answer give me 0.0599994.
and when I do Var1 + Var2, the answer give me 11.94

How can i make it so when Var1-Var2 should be = 0.06? how come it is
giving
me 0.0599994???
Click to expand...

That's a characteristic of floating point numbers.

<URL:http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html>
<URL:http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html>
<URL:http://support.microsoft.com/?scid=kb;[LN];42980>
 
Interesting. I think you've stumbled onto a blatant bug. And I wouldn't
consider this a small round off issue. Good find Jon.
 
ocertain said:
Interesting. I think you've stumbled onto a blatant bug. And I
wouldn't consider this a small round off issue. Good find Jon.
Click to expand...

Nah, as per Herfried's post, this is perfectly normal.

Try it in VB6, you'll get the same result. I also frequently see similar
things from SQL Server. If you want fractional accuracy, use a fixed-point
storage type such as Decimal.
 
Back
Top