Int() Function

[Mark]

I am getting some odd results when using the int() function within
VBA.
Here is what is happening:

quantity = dblTransactionQuantity
quantityWithoutDecimalPlaces = dblTransactionQuantity * 1000
integerpart = Int(quantityWithoutDecimalPlaces)


When you set the dblTransactionQuantity to 1.001 intergerpart returns
1000, but it should return 1001. Is this a floating point error? If
you set the dblTransactionQuantity as 1.002 the correct answer is
returned (1002). Any suggestions?

 

[Gary Walter]

Hi Mark,

http://www.fmsinc.com/tpapers/math/index.html

example in Immediate Window:

dbl=1.001
dbl=dbl*CDec(1000)
?Int(dbl)
1001

good luck,

gary

 

[Tom van Stiphout]

On Mon, 17 Dec 2007 05:29:25 -0800 (PST), Mark <[email protected]>
wrote:

Computers work with bits, not with floating point values. They are
inherently imprecise representing floating point values. Therefore you
should never do exact math with them. Rather than saying "is 1000 *
1.001" equal to 1001?", you should say "is 1000 * 1001 close to
1001?":
Debug.Print (CDbl(1.001) * CDbl(1000.0)) - CDbl(1001.0) < 1E-10
-> True

Your use of trunctating functions like Int rather than CLng may also
exacerbate the problem.

-Tom.