excel calculation errors

[art]

Does anyone know (1) why excel computes the square of
111,111,111 incorrectly to 12345678987654300, and (2) if
this error suggests other errros in other calculations?

 

[Peo Sjoblom]

It's not an error really, if you look in help (excel specifications and
limits) you can se
that the number precision of 15 digits max, everything beyond will be
truncated thus
the 0 zeros you get at the end

 

[JON JON]

Art,

It was admitted by excel. Look in Help - search "limit". It was said that
number precision is limited to 15 digits.

Jon-jon

 

[Jerry W. Lewis]

"Admitted" is a bit misleading. This is a natural limit associated with
the IEEE standard for double precision numbers, and applies to almost
all computer software, regardless of platform.

Jerry

 

[Greg Lovern]

Hi Art,

You can work around this with xlPrecision. xlPrecision in Excel
returns 12,345,678,987,654,321 as the square of 111,111,111. See:

http://PrecisionCalc.com


Hope this helps,

Greg Lovern
http://PrecisionCalc.com
Get Your Numbers Right

 

[JON JON]

Pardon me! I should had said documented.

"Admitted" is a bit misleading. This is a natural limit associated with
the IEEE standard for double precision numbers, and applies to almost
all computer software, regardless of platform.

Jerry