J
Joshua A. Booker
Hi there,
Why is int(1.4/.05) = 27?
It should be 28.
TIA,
Josh
Why is int(1.4/.05) = 27?
It should be 28.
TIA,
Josh
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
M
Marshall Barton
Joshua said:Why is int(1.4/.05) = 27?
It should be 28.
Apparently(?), one (or both) of those number can not be
represented exactly as a binary floating point number. The
Int (and Fix) functions blindly discard any fractional part
regardless of how close the value is to an integer value.
You can avoid this by adding a small amount to the value
before the truncation:
Int(1.4/.05 + 1E-8)
If you want the value to be rounded when you convert it to
an integer, use CInt or CLng.
R
Roger Carlson
Floating point math can be off by miniscule amounts. Probably it is
returning something like 27.99999999, which the Int function converts to 27.
The ususual solution is to use Currency datatypes, which are more accurate.
Exactly how are you using this?
--
--Roger Carlson
Access Database Samples: www.rogersaccesslibrary.com
Want answers to your Access questions in your Email?
Free subscription:
http://peach.ease.lsoft.com/scripts/wa.exe?SUBED1=ACCESS-L
returning something like 27.99999999, which the Int function converts to 27.
The ususual solution is to use Currency datatypes, which are more accurate.
Exactly how are you using this?
--
--Roger Carlson
Access Database Samples: www.rogersaccesslibrary.com
Want answers to your Access questions in your Email?
Free subscription:
http://peach.ease.lsoft.com/scripts/wa.exe?SUBED1=ACCESS-L
J
Joshua A. Booker
Marsh,
I'm using the following code to round one number by another increment. I
can't get it to return the correct results when it's passed a 'whole'
(relative to the increment) number.
For example:
Rnd2Num(1.4,.05,vb_rounddown) = 1.35
In this case, because 1.4 is divisible by 0.05 I want it to return 1.4. The
line:
If CInt(Temp) = Temp Then
is supposed to catch this case.
Function Rnd2Num(Amt As Variant, RoundAmt As Variant, Direction As Integer)
As Double
On Error Resume Next
Dim Temp As Double
Temp = Amt / RoundAmt
If CInt(Temp) = Temp Then
Rnd2Num = Amt
Else
If Direction = vb_rounddown Then
Temp = Int(Temp)
Else
Temp = Int(Temp) + 1
End If
Rnd2Num = Temp * RoundAmt
End If
'Debug.Print Rnd2Num
End Function
I tried adding a small amount.
Any suggestions?
TIA,
Josh
I'm using the following code to round one number by another increment. I
can't get it to return the correct results when it's passed a 'whole'
(relative to the increment) number.
For example:
Rnd2Num(1.4,.05,vb_rounddown) = 1.35
In this case, because 1.4 is divisible by 0.05 I want it to return 1.4. The
line:
If CInt(Temp) = Temp Then
is supposed to catch this case.
Function Rnd2Num(Amt As Variant, RoundAmt As Variant, Direction As Integer)
As Double
On Error Resume Next
Dim Temp As Double
Temp = Amt / RoundAmt
If CInt(Temp) = Temp Then
Rnd2Num = Amt
Else
If Direction = vb_rounddown Then
Temp = Int(Temp)
Else
Temp = Int(Temp) + 1
End If
Rnd2Num = Temp * RoundAmt
End If
'Debug.Print Rnd2Num
End Function
I tried adding a small amount.
Any suggestions?
TIA,
Josh
M
Marshall Barton
Your routine is too sensitive to the inaccuracies of the
floating point representation of many numbers.
I Googled the newgroup archives on this topic and found a
couple of gems among the heaps of gravel. Unfortunately, my
ISP shut down before I could capture the thread's URL.
Anyway, the algorithm that seemed to be the best was Lyle
Fairfield's simple one liner:
Format(Amt / RoundAmt, "0") * RoundAmt
I'll leave it to you to work out how to fit your direction
argument into it ;-)
floating point representation of many numbers.
I Googled the newgroup archives on this topic and found a
couple of gems among the heaps of gravel. Unfortunately, my
ISP shut down before I could capture the thread's URL.
Anyway, the algorithm that seemed to be the best was Lyle
Fairfield's simple one liner:
Format(Amt / RoundAmt, "0") * RoundAmt
I'll leave it to you to work out how to fit your direction
argument into it ;-)