PC Review


Reply
Thread Tools Rate Thread

Calculation accuracy

 
 
=?Utf-8?B?U2F0aGlzaA==?=
Guest
Posts: n/a
 
      8th Sep 2007
When i tried calculating a simple math, i found a rounding issue in excel:

I made the following computation by having each value in a cell and inputed
formula to compute the answer:

A1 = 19978.39
B1 = 21456.06
C1 = 1477.67
D1 = A1 - B1 + C1 = -0.00000000001819 (instead of ZERO).

I dont understand the reason for this type of rounding eventhough the
amounts involved are relatively round.

May i have the esteemed views of this forum??

Thanks
 
Reply With Quote
 
 
 
 
Bernard Liengme
Guest
Posts: n/a
 
      8th Sep 2007
This has to do with the way computer (not just Excel) store decimal numbers
(numbers to base 10) in binary format (numbers to base 2). For details read:
INFO: Visual Basic and Arithmetic Precision
http://support.microsoft.com/default...NoWebContent=1

(Complete) Tutorial to Understand IEEE Floating-Point Errors
http://support.microsoft.com/default...NoWebContent=1

What Every Computer Scientist Should Know About Floating Point
http://docs.sun.com/source/806-3568/ncg_goldberg.html
Go to source>>
http://www.cpearson.com/excel/rounding.htm

best wishes
--
Bernard V Liengme
Microsoft Excel MVP
www.stfx.ca/people/bliengme
remove caps from email

"Sathish" <(E-Mail Removed)> wrote in message
news:2AEE4AAF-496A-4043-A480-(E-Mail Removed)...
> When i tried calculating a simple math, i found a rounding issue in excel:
>
> I made the following computation by having each value in a cell and
> inputed
> formula to compute the answer:
>
> A1 = 19978.39
> B1 = 21456.06
> C1 = 1477.67
> D1 = A1 - B1 + C1 = -0.00000000001819 (instead of ZERO).
>
> I dont understand the reason for this type of rounding eventhough the
> amounts involved are relatively round.
>
> May i have the esteemed views of this forum??
>
> Thanks



 
Reply With Quote
Reply

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
accuracy Riju Microsoft Word New Users 7 2nd Jul 2009 05:24 PM
RTC accuracy pawihte Computer Hardware 16 12th Jun 2009 09:21 AM
calculation accuracy flavio.c Microsoft Excel Misc 3 30th Sep 2007 09:42 AM
Accuracy and CSS =?Utf-8?B?Q2hhcmxlc0E=?= Microsoft ASP .NET 5 17th Jan 2006 12:47 PM
Accuracy Vyyk Microsoft Excel Programming 1 17th Sep 2003 01:07 PM


Features
 

Advertising
 

Newsgroups
 


All times are GMT +1. The time now is 10:57 AM.