Error when multiplying 111,111,111 by 111,111,111

S

Supervisdr

Did any one notice that in excel, when 111,111,111 or 1,111,111,111 is
multiplied by the same number, the answer shown ends with few 0s. The same
multiplication on Windows calculator (Use the scientific calc- it
accommodates large numbers) the answer is correctly shown.

Funnily, Open office - CALC too makes the same mistake. !!!!!
 
J

Jerry W. Lewis

MS Excel, like most numerical software (and AFAIK all spreadsheet software)
follows the IEEE standard for double precision. Numbers in this format only
assure 15 digit accuracy. MS chose to pad with trailing zeros instead of
showing the actual impact of 53-bit binary approximation beyond the 15th
digit. The Windows calculator uses a different number storage format.

Jerry
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads


Top