Trouble calculating exact values with double datatypes

[Lothar Behrens]

Hi,

I have found a problem in my unit tests that my code does not
calculate exact values.
My code produced 2.8000000000000003 but not 2.7000000000000003. In
both cases
I expected 2.8 / 2.7.

And when I change the calculation I get exact values. Why does this
happen?

I am using Visual Studio 2008 Professional 9.0.30729.1 SP and .NET 3.5
SP1.

Here is a sample from my watch window:

28.000000 * (100.000000 / 100.0000000) / (100.0000000 / 10.0000000)
2.8 double
28.000000 * (100.000000 / 100.0000000) * (10.0000000 / 100.0000000)
2.8000000000000003 double
27.000000 * (100.000000 / 100.0000000) * (10.0000000 / 100.0000000)
2.7 double

Thanks

Lothar

 

[Martin Honnen]

I have found a problem in my unit tests that my code does not
calculate exact values.
My code produced 2.8000000000000003 but not 2.7000000000000003. In
both cases
I expected 2.8 / 2.7.

And when I change the calculation I get exact values. Why does this
happen?

You get the precision that is possible with that number type
(http://en.wikipedia.org/wiki/Double_precision_floating-point_format)
with binary representation of numbers.
Using decimal instead of double would give you better precision. If you
want or need to use double then round as required.

 

[Lothar Behrens]

You get the precision that is possible with that number type
(http://en.wikipedia.org/wiki/Double_precision_floating-point_format)
with binary representation of numbers.
Using decimal instead of double would give you better precision. If you
want or need to use double then round as required.

I don't expect numbers like 2.8000000000000003 if I effectively divide
28 through 10.
If I divide 10 through 3, then I expect as much precision as the data
type will provide.

I have compared the code with C++ (not .NET) and there I also have
additionally compared
with using float. That would give me a more expectable result.

Rounding is an option, but only with the result of a calculation to
avoid rounding errors.
Actually I am using rounding in my unittests to get these passing.

I have to play a bit more with floating point values and
calculation
Also I have to play with the datatypes I have choosen in my DB model.
(LinqToSQL / SQLSERVER)

Thanks

Lothar

 

[Jeff Johnson]

I don't expect numbers like 2.8000000000000003 if I effectively divide
28 through 10.

Then you clearly don't understand IEEE floating-point numbers. Don't feel
bad, most people don't. Even seemingly innocuous/"obvious" calculations can
lead to these odd values. Dividing by 10 when representing numbers in binary
is no "safer" than dividing by Pi. There's a URL that people post in another
group I read to help explain IEEE numbers to people who are having the same
problems you are. If I can find it I'll post it here.

 

[Jeff Johnson]

Then you clearly don't understand IEEE floating-point numbers. Don't feel
bad, most people don't. Even seemingly innocuous/"obvious" calculations
can lead to these odd values. Dividing by 10 when representing numbers in
binary is no "safer" than dividing by Pi. There's a URL that people post
in another group I read to help explain IEEE numbers to people who are
having the same problems you are. If I can find it I'll post it here.

Found one: http://support.microsoft.com/kb/42980/EN-US/