excel calculation errors

  • Thread starter Thread starter art
  • Start date Start date
A

art

Does anyone know (1) why excel computes the square of
111,111,111 incorrectly to 12345678987654300, and (2) if
this error suggests other errros in other calculations?
 
It's not an error really, if you look in help (excel specifications and
limits) you can se
that the number precision of 15 digits max, everything beyond will be
truncated thus
the 0 zeros you get at the end
 
Art,

It was admitted by excel. Look in Help - search "limit". It was said that
number precision is limited to 15 digits.

Jon-jon
 
"Admitted" is a bit misleading. This is a natural limit associated with
the IEEE standard for double precision numbers, and applies to almost
all computer software, regardless of platform.

Jerry
 
Pardon me! I should had said documented.

Jerry W. Lewis said:
"Admitted" is a bit misleading. This is a natural limit associated with
the IEEE standard for double precision numbers, and applies to almost
all computer software, regardless of platform.

Jerry
Click to expand...
 
Back
Top