Bug! Format statement introduces a round off

[RD]

The statements

The value passed to Amount by the function call is 111111111111111.98 and is
a double

then we 4execute the following code snippet

Dim New_amt as string
New_amt = Format(amount, "000000000000000.00")

The new_amt value returned is

"111111111111112.00"

Which is wrong.

Does any one know why and how to work around thisÉ

Thanks

Bob

 

[Tom Dacon]

It's not a bug. The value you're passing into Format has more significant
digits than the double data type can handle, so it's not unreasonable to see
roundoff with values like that. For double-precision variables you get
around fifteen or sixteen significant digits, depending on the value. If you
need more significant digits, use the Decimal type, which has (I think)
twenty-seven significant digits.

Tom Dacon
Dacon Software Consulting

 

[Herfried K. Wagner [MVP]]

The value passed to Amount by the function call is
111111111111111.98 and is a double

then we 4execute the following code snippet

Dim New_amt as string
New_amt = Format(amount, "000000000000000.00")

The new_amt value returned is

"111111111111112.00"

That's not a bug. The floating point number literal 111....1.98 cannot be
represented in a 'Double'.

<URL:http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html>
<URL:http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html>
<URL:http://support.microsoft.com/?scid=kb;[LN];42980>

 

[RD]

Can you tell me what I can use to correctly represent it, that would be very
helpfull

Thank you for your help.

Bob

That's not a bug. The floating point number literal 111....1.98 cannot be
represented in a 'Double'.

<URL:http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html>
<URL:http://support.microsoft.com/?scid=kb;[LN];42980>

 

[Jay B. Harlow [MVP - Outlook]]

RD,
As Tom suggested, a System.Decimal value.

Something like:

Dim amount As Decimal = 111111111111111.98D
Dim New_amt As String
New_amt = amount.ToString("000000000000000.00")

FWIW: The D on 111111111111111.98D indicates that it is a Decimal literal,
where as 111111111111111.98 is a Double literal and 111111111111111.98F is a
Single literal.

Hope this helps
Jay

Can you tell me what I can use to correctly represent it, that would be
very
helpfull

Thank you for your help.

Bob

That's not a bug. The floating point number literal 111....1.98 cannot
be
represented in a 'Double'.

<URL:http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html>
<URL:http://support.microsoft.com/?scid=kb;[LN];42980>