Fuzzy math!

  • Thread starter Thread starter schiefaw
  • Start date Start date
S

schiefaw

Hi,

I am having a slight problem with some code. I perform some additions
and subtractions on some 2 digit decimals, multiply by -1.0 and wind up
with a number with 12 digits of precision.

? (83538.18 + 0.0 + 0.0 - 40000.0 - 0.0) * -1.0
-43538.179999999993


I have replaced all the variables with their values for this example.
Does anyone know of a problem with VB's handling of decimals? And, if
so, what can I do about it (other than just rounding it off)?

Thanks,
Tony
 
Hi,

I am having a slight problem with some code. I perform some additions
and subtractions on some 2 digit decimals
Click to expand...

No, you perform some additions and subtractions on some *floating-point
numbers*. Floating point arithmetic is not exact.

, multiply by -1.0 and wind up
with a number with 12 digits of precision.

? (83538.18 + 0.0 + 0.0 - 40000.0 - 0.0) * -1.0
-43538.179999999993
Click to expand...

? (83538.18D + 0.0D + 0.0D - 40000.0D - 0.0D) * -1.0D
-43538.18D

(I use the deprecated type declaraction character 'D' here; CDec(...)
has the same effect)
I have replaced all the variables with their values for this example.
Does anyone know of a problem with VB's handling of decimals?
Click to expand...

Here's one you might like:

? 0.1 + 0.1 + 0.1 - 0.3
And, if
so, what can I do about it (other than just rounding it off)?
Click to expand...

Use fixed-point types to hold exact values.
 
I am having a slight problem with some code. I perform some additions
and subtractions on some 2 digit decimals, multiply by -1.0 and wind up
with a number with 12 digits of precision.

? (83538.18 + 0.0 + 0.0 - 40000.0 - 0.0) * -1.0
-43538.179999999993


I have replaced all the variables with their values for this example.
Does anyone know of a problem with VB's handling of decimals? And, if
so, what can I do about it (other than just rounding it off)?
Click to expand...

In addition to the other replies:

That's a characteristic of floating point numbers.

<URL:http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html>
<URL:http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html>
<URL:http://support.microsoft.com/?scid=kb;[LN];42980>
 
Back
Top