Incorrect result using Decimal data type

[Guest]

Please consider this code:

----------

Dim i As Decimal
Dim p As Decimal = 31D

i = (2D ^ (p - 1D)) * (2D ^ p - 1D)

----------

Variable "i" should be equal to "2305843008139952128" but, for some reason,
variable "i" becomes "2305843008139950000".

Am I doing something wrong, am I missing something or is this a bug?

 

[Armin Zingler]

Please consider this code:

----------

Dim i As Decimal
Dim p As Decimal = 31D

i = (2D ^ (p - 1D)) * (2D ^ p - 1D)

----------

Variable "i" should be equal to "2305843008139952128" but, for some
reason, variable "i" becomes "2305843008139950000".

Am I doing something wrong, am I missing something or is this a bug?

Another reason why Option Strict should always be On. You can enable it in
the project properties (General -> Build: Option Strict [On].

After switching it on, split your calculations into multiple steps to follow
the calculation of the final value.


Armin

 

[Jay B. Harlow [MVP - Outlook]]

xfx,
Are you using VS 2003 or VS 2005?

You might consider using a Shift operator instead of the Exponentiation
operator. Something like:

Dim i As Long
Dim p As Integer = 31

i = (1L << (p - 1)) * ((1L << p) - 1)



Remember the Exponentiation operator is defined for type Double only. So
although your operands are all Decimal, they are being implicitly being
converted to & from Double for the expression.

Hope this helps
Jay

| Please consider this code:
|
| ----------
|
| Dim i As Decimal
| Dim p As Decimal = 31D
|
| i = (2D ^ (p - 1D)) * (2D ^ p - 1D)
|
| ----------
|
| Variable "i" should be equal to "2305843008139952128" but, for some
reason,
| variable "i" becomes "2305843008139950000".
|
| Am I doing something wrong, am I missing something or is this a bug?