Variable precision

[John Dann]

Just a curiosity really, but I am curious as to the explanation:

I'm trying to read an external DLL written in C++ I think using
DLLImport. The DLL version number is available from a DLL function
which returns a float variable. The value should be 2.3 but a .Net
(double) variable is returning 2.9999xxxxxx. where xxxx is a sequence
of other digits.

Obviously in retrieving the value into the .Net environment something
is ever so slightly perturbing the original value. Anyone know what
this might be?

JGD

 

[Mattias Sjögren]

John,

The value should be 2.3 but a .Net
(double) variable is returning 2.9999xxxxxx. where xxxx is a sequence
of other digits.

This is something you have to get used to when working with floating
point numbers. Not all numbers can be represented exactly. See
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems or
http://www.extremeoptimization.com/resources/Articles/FPDotNetConceptsAndFormats.aspx
for more details.


Mattias