How to overcome very slight calculation error.

G

Guest

When a variable 'C' containing 0.175 is used to multiply 1325 the answer is
231.874996051192 when it should be 231.8750.
I have DIMed 'C' as double/single/variant but still get this error.

Sure a very slight error but when rounded the former goes to 231.87 which
financially is incorrect and causes reconcilliation errors.

My workaround is to use Val(C) which magically works!
1325 * Val(C) = 231.8750

What is the 'proper' way?
 
A

Albert D. Kallal

Welcome to the world of computers.

It turns out that one of the FIRST things a programmer will be taught is
that when a computer stores a "real" number (as opposed to a integer
number), the computer can only store a approximation of that number.

so, when you store a value such as 3 / 10, the resulting binary
representation of .3 is only a approximation of that result.

The following code if run in excel, word, or ms-access will produce the same
rounding errors:

Public Sub TestAdd()

Dim MyNumber As Single
Dim i As Integer

For i = 1 To 10
MyNumber = MyNumber + 1.01
Debug.Print MyNumber
Next i
End Sub

Here is the actual outpput of the above:

1.01
2.02
3.03
4.04
5.05
6.06
7.070001
8.080001
9.090001
10.1

You can see that after just 7 addtions..already rounding is occuring
and if we add the follwing line of code to the end of the above:

if MyNumber = 10.1 = True then

msgbox "the number is 10.1"

else

msgbox "the number is somthing else"

endif


The above will actuall produce:

the number is something else

So, what this means is that if you plan to use any financial calculations,
you can't store, or nor use "real" numbers, as they are a approximates. The
solution in this case is to use decimal number, or a data type of currency.
(these are stored as scaled integers).

So, if you are working with financial numbers, and don't want rounding
errors, then use currency.

further, you don't say what data type "c" is.

I just tried:


Dim c As Single

c = 0.175

Debug.Print c * 1325

the output was:

231.875

In fact, in the debug window, I just typed in:

? 0.175 * 1325
The reuslt is
231.875

However, if I type in:

? (0.175 * 1325) = (231.875 )
output is:
false

Howeer:
? ccur(0.175 * 1325.0) = ccur(231.875)
true

So, just use currency for these finical calculations. (they have a max of 4
digits after the decimal place, and are actually scaled integers).

If that data is from a table, you should convert that column to currency, or
perhaps decimal type.

I would also ensure that you always use a option explicit in your code....
 
A

Allen Browne

Try dimming C as Currency

Floating point types (such as single and double) always have these issues.

A fixed-point type like Currency should solve the problem.
 
T

Tom Lake

ThomasAJ said:
When a variable 'C' containing 0.175 is used to multiply 1325 the answer
is
231.874996051192 when it should be 231.8750.
I have DIMed 'C' as double/single/variant but still get this error.

Sure a very slight error but when rounded the former goes to 231.87 which
financially is incorrect and causes reconcilliation errors.

My workaround is to use Val(C) which magically works!
1325 * Val(C) = 231.8750

What is the 'proper' way?

What version of Access are you using? I use Access 2003 and the following
works fine for me in both double and single precision:

Sub test()
Dim c As Single ' or Double
c = 0.175
Debug.Print c * 1325
End Sub

Tom Lake
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top