Arithmetic error

  • Thread starter Thread starter maxarbos
  • Start date Start date
M

maxarbos

I appoligize that this has been discussed, but I didn't see any
examples as simple as this. Three user input values are simply
subtracted: 55.45 - 52.37 - 3.08 = 0 However, excel calculates to
0.000000000000005329070518200750. They are user input values and so
are not rounded, however, I checked and verified 0's out to 30 places.

The only reason I noticed this is becuase in a matrix of results from
various inputs, all of which should have been 0 and with Tools/Options/
View set to clear '0' cells, I've got this one cell showing up.

I assume this is some sort of binary conversion error?

Thanks,
Dave
 
It is the effect of decimal to binary (IEEE 64 bit) to decimal conversion.
I'm not sure it is an error though, at least in the sense of a bug or a
mistake. It's the result of the inherent and expected imprecision in this
process.

--
Jim
|I appoligize that this has been discussed, but I didn't see any
| examples as simple as this. Three user input values are simply
| subtracted: 55.45 - 52.37 - 3.08 = 0 However, excel calculates to
| 0.000000000000005329070518200750. They are user input values and so
| are not rounded, however, I checked and verified 0's out to 30 places.
|
| The only reason I noticed this is becuase in a matrix of results from
| various inputs, all of which should have been 0 and with Tools/Options/
| View set to clear '0' cells, I've got this one cell showing up.
|
| I assume this is some sort of binary conversion error?
|
| Thanks,
| Dave
|
 
Have experienced likewise. Is the best way to "fix" to use a Rounding
function? While a year+ ago, I believe that is what I did.
 
I use the Tools, Options, Calculation, Precision as Displayed option
religiously. Of course before applying it to an existing workbook you have
to be sure that all cells have the number format that you want them rounded
to.

--
Jim
| Have experienced likewise. Is the best way to "fix" to use a Rounding
| function? While a year+ ago, I believe that is what I did.
|
|
| | > It is the effect of decimal to binary (IEEE 64 bit) to decimal
conversion.
| > I'm not sure it is an error though, at least in the sense of a bug or a
| > mistake. It's the result of the inherent and expected imprecision in
this
| > process.
| >
| > --
| > Jim
| >
| > |I appoligize that this has been discussed, but I didn't see any
| > | examples as simple as this. Three user input values are simply
| > | subtracted: 55.45 - 52.37 - 3.08 = 0 However, excel calculates to
| > | 0.000000000000005329070518200750. They are user input values and so
| > | are not rounded, however, I checked and verified 0's out to 30 places.
| > |
| > | The only reason I noticed this is becuase in a matrix of results from
| > | various inputs, all of which should have been 0 and with
Tools/Options/
| > | View set to clear '0' cells, I've got this one cell showing up.
| > |
| > | I assume this is some sort of binary conversion error?
| > |
| > | Thanks,
| > | Dave
| > |
| >
| >
|
|
 
Back
Top