miscalculation

[Jane Glover]

When I multiplied 111,111,111 x 111,111,111 in excel, I
got the answer 12,345,678,987,654,300 -- the correct
answer whould be 12,345,678,987,654,321. Does anyone have
an idea why this calculation would be incorrect?
Thanks!

 

[Bernard V Liengme]

Hi Jane,
The vast majority of computer applications use what is called the IEEE (I
triple E) convention for storing numbers: numbers are stored in binary
format with 15 digit precision. SO your answer is correct to 15 digits.

This may seem odd if you are working in pure mathematics but from a
practical point of view it is just fine - even the US national debt can be
represent with this precision and no engineering or scientific work is that
accurate.

If you really need to do this type of arithmetic there is an add on (free)
which does not use IEEE math. Let us know and someone will point you in the
right direction. Alternatively, use COBOL which can be set not to work in
binary.

Best wishes
Bernard

 

[Dan E]

Jane,

Excel will only display 15 digits of precision.
For information on this and other excel specifications
type "specifications" without quotes into the help answer
wizard. Then pick "Microsoft Excel Specifications"
from the list of results and choose "Worksheet and
workbook specifications" in the help file.

Dan E