Rounding errors

J

JimBlob

I have been trying to create a query in Access, that divides three numbers
For Example: 1/.165/203650. and have been unsuccessful in matching the
output to match what I get in either Excel, or in windows Calc beyond the
12th decimal place. Is this rounding error?

2.975991191066111E-05 (Access)
2.975991191066070000E-05 (Excel)
2.9759911910660744444196445178522e-5(calc)

I have the field set as double, 15 decimal places.

Thank you in advance!
Jim
 
A

Allen Browne

Some of this is rounding error, but what version of Access did you try?

Access 2007 gives these results:

a) In the Immediate Window:
? 1/.165/203650
2.97599119106607E-05

b) In the Immediate Window:
? CDec(1) / CDec(0.165) / CDec(203650)
0.0000297599119106607444441964

c) In this query:
SELECT 1/0.165/203650 AS MyResult;
0.00002975991191066074444

d) IN this query:
SELECT CDbl(1)/CDbl(0.165)/CDbl(203650) AS MyResult;
2.97599119106607E-05

(a) and (d) are clearly working with the data type Double - same as your
field type. And Excel 2007 gives the same result.

(b) and (c) are treating the data as Decimal, and so gives greater
precision.

I was actually surprised to see JET using Decimal for (c), but it reports
the query's field as type dbDecimal. It doesn't always do that. If you try:
SELECT 1/2/3 AS MyResult;
it creates a field of type Double.

If you need further description of the rounding accuracy issue inherent in
floating point numbers, see:
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems
 
J

JimBlob

Im using 2002. My original query was: SELECT
[Frequency]/[bindiff]/[SumOfFrequency] AS Expr1
FROM tblData INNER JOIN qryFrequencySum ON tblData.MeasurementID =
qryFrequencySum.MeasurementID;

Using Cdbl worked.

Thank You!
 
J

Jamie Collins

c) In this query:
SELECT 1/0.165/203650 AS MyResult;
0.00002975991191066074444

I was actually surprised to see JET using Decimal for (c), but it reports
the query's field as type dbDecimal. It doesn't always do that. If you try:
SELECT 1/2/3 AS MyResult;
it creates a field of type Double.

Allen, I was actually surprised to see you say that <g> because I've
been pointing out exactly this kind of thing in these groups literally
for years.

The basic difference between VBA and Jet SQL is that Jet uses the
DECIMAL type natively. If you think about it, this makes perfect
sense. VBA is for business logical and will benefit from hardware
support that floating point values enjoy e.g. good performance when
calculating. Jet SQL is for data management *only* and therefore
favours accuracy over performance.

And it's one of the arguments I use to counter the 'avoid Decimal in
Access' we see from (<coughs>) some regulars.

The basic rule of thumb for numeric literals in Jet:

1) Integer literal values within the range of INTEGER (Long Integer)
will be considered INTEGER (Long Integer):

SELECT TYPENAME(0), TYPENAME(1E9);

2) Integer literal values beyond the range of INTEGER (Long Integer)
but within the range of DECIMAL (Decimal) will be considered DECIMAL
(Decimal):

SELECT TYPENAME(-2147483649), TYPENAME(1E10);

3) Integer literal values beyond the range of DECIMAL (Decimal) will
be considered FLOAT (Double):

SELECT TYPENAME(79228162514264337593543950336), TYPENAME(1E29);

4) Decimal literal values within the range of DECIMAL (Decimal) will
be considered DECIMAL (Decimal):

SELECT TYPENAME(0.5), TYPENAME(-1E-28);

5) Decimal literal values beyond the range of DECIMAL (Decimal) will
be considered FLOAT (Double):

SELECT TYPENAME(123456789012345.123456789012345), TYPENAME(1E-29);

There are some exceptions to the rule (that being the nature of rules
of thumb) e.g. decimal literal values that are within the range of
INTEGER (Long Integer) will be considered INTEGER (Long Integer)
unless they are a multiple of 10 in which case they will be considered
FLOAT (Double) e.g.

SELECT TYPENAME(1001.0), TYPENAME(1000.0)

returns 'Long' and 'Double' respectively.

The next consideration is that by using / 'slash' you are using what
Microsoft calls "the floating-point division operator":

Microsoft Office XP Developer
The Currency and Decimal Data Types
http://msdn2.microsoft.com/en-us/library/aa164763(office.10).aspx

A Note About Division

Any time you use the floating-point division operator (/), you are
performing floating-point division, and your return value will be of
type Double. This is true whether your dividend and divisor are
integer, floating-point, or fixed-point values. It is true whether or
not your result has a decimal portion.

For example, running the following code from the Immediate window
prints "Double":

? TypeName(2.34/5.9)

[Unquote]

Again, in Jet the DECIMAL type exhibits different behaviour: if the
dividend or divisor is of type DECIMAL then the return value will be
of type DECIMAL:

SELECT TYPENAME(2 / 0.5), TYPENAME(3E9 / 1000)

Hopefully this should explain the differences you are seeing in the
Immediate Window.

PS If anyone is still reading, what do you think you the above?
Blowing my own trumpet here but I think it represents a fantastic
level of detail that tends to be glossed over by the average group
regular. Can anyone tell me why I'm not a more valued member of the
community? Are readers more interested in familiar answers to
straightforward questions, rather than background details and engine
fundamentals? Is it wrong to challenge fallacies and seek to correct
misstatements? Is the above too dry and boring e.g. has nobody made it
this far...?

Jamie.

--
 
A

Allen Browne

Jamie, I don't plan to make a long thread, but have offered some comments
in-line.

--
Allen Browne - Microsoft MVP. Perth, Western Australia

Reply to group, rather than allenbrowne at mvps dot org.

Jamie Collins said:
Allen, I was actually surprised to see you say that <g> because I've
been pointing out exactly this kind of thing in these groups literally
for years.

Yes: forgive me, but when I posted that reply, I did wonder if you would
bite. :)
PS If anyone is still reading, what do you think you the above?
Blowing my own trumpet here but ...

You were probably hoping for someone else to reply to this, and perhaps they
will. :)

If you were asking a serious social question (not merely a technical
comment), then I fear the answer is yes: the average Access user doesn't
really care about this level of detail. That is not to belittle your
knowledge, or your interests. It's just my perception of what people are
seeking, based on the questions in the newsgroups, the kinds of articles
that get read most at my own website, and the experience of others and the
experience of others over the years such as michka who has been known to
refer to pouring energy into investigating stuff no one esle is interested
in.

You seem to be seeking in-depth discussion, which is fine but not typical.
Most posters are seeking help to get a specific problem solved, and I enjoy
providing some help for those people, so I tend avoid threads where others
are already contributing.

All the best.
 
G

Gary Walter

I always enjoy understanding these intricasies better,
and you did an admirable job here I'd say...

of course, I'm only a few valuable neurons away
from having Asperger syndrome I suspect. :cool:

gary

Jamie Collins said:
c) In this query:
SELECT 1/0.165/203650 AS MyResult;
0.00002975991191066074444

I was actually surprised to see JET using Decimal for (c), but it reports
the query's field as type dbDecimal. It doesn't always do that. If you
try:
SELECT 1/2/3 AS MyResult;
it creates a field of type Double.

Allen, I was actually surprised to see you say that <g> because I've
been pointing out exactly this kind of thing in these groups literally
for years.

The basic difference between VBA and Jet SQL is that Jet uses the
DECIMAL type natively. If you think about it, this makes perfect
sense. VBA is for business logical and will benefit from hardware
support that floating point values enjoy e.g. good performance when
calculating. Jet SQL is for data management *only* and therefore
favours accuracy over performance.

And it's one of the arguments I use to counter the 'avoid Decimal in
Access' we see from (<coughs>) some regulars.

The basic rule of thumb for numeric literals in Jet:

1) Integer literal values within the range of INTEGER (Long Integer)
will be considered INTEGER (Long Integer):

SELECT TYPENAME(0), TYPENAME(1E9);

2) Integer literal values beyond the range of INTEGER (Long Integer)
but within the range of DECIMAL (Decimal) will be considered DECIMAL
(Decimal):

SELECT TYPENAME(-2147483649), TYPENAME(1E10);

3) Integer literal values beyond the range of DECIMAL (Decimal) will
be considered FLOAT (Double):

SELECT TYPENAME(79228162514264337593543950336), TYPENAME(1E29);

4) Decimal literal values within the range of DECIMAL (Decimal) will
be considered DECIMAL (Decimal):

SELECT TYPENAME(0.5), TYPENAME(-1E-28);

5) Decimal literal values beyond the range of DECIMAL (Decimal) will
be considered FLOAT (Double):

SELECT TYPENAME(123456789012345.123456789012345), TYPENAME(1E-29);

There are some exceptions to the rule (that being the nature of rules
of thumb) e.g. decimal literal values that are within the range of
INTEGER (Long Integer) will be considered INTEGER (Long Integer)
unless they are a multiple of 10 in which case they will be considered
FLOAT (Double) e.g.

SELECT TYPENAME(1001.0), TYPENAME(1000.0)

returns 'Long' and 'Double' respectively.

The next consideration is that by using / 'slash' you are using what
Microsoft calls "the floating-point division operator":

Microsoft Office XP Developer
The Currency and Decimal Data Types
http://msdn2.microsoft.com/en-us/library/aa164763(office.10).aspx

A Note About Division

Any time you use the floating-point division operator (/), you are
performing floating-point division, and your return value will be of
type Double. This is true whether your dividend and divisor are
integer, floating-point, or fixed-point values. It is true whether or
not your result has a decimal portion.

For example, running the following code from the Immediate window
prints "Double":

? TypeName(2.34/5.9)

[Unquote]

Again, in Jet the DECIMAL type exhibits different behaviour: if the
dividend or divisor is of type DECIMAL then the return value will be
of type DECIMAL:

SELECT TYPENAME(2 / 0.5), TYPENAME(3E9 / 1000)

Hopefully this should explain the differences you are seeing in the
Immediate Window.

PS If anyone is still reading, what do you think you the above?
Blowing my own trumpet here but I think it represents a fantastic
level of detail that tends to be glossed over by the average group
regular. Can anyone tell me why I'm not a more valued member of the
community? Are readers more interested in familiar answers to
straightforward questions, rather than background details and engine
fundamentals? Is it wrong to challenge fallacies and seek to correct
misstatements? Is the above too dry and boring e.g. has nobody made it
this far...?

Jamie.
 
M

Marshall Barton

Excellent explanation of some of the implicit data casting
that goes on in the Expression Service.
--
Marsh


Jamie said:
c) In this query:
SELECT 1/0.165/203650 AS MyResult;
0.00002975991191066074444

I was actually surprised to see JET using Decimal for (c), but it reports
the query's field as type dbDecimal. It doesn't always do that. If you try:
SELECT 1/2/3 AS MyResult;
it creates a field of type Double.

Allen, I was actually surprised to see you say that <g> because I've
been pointing out exactly this kind of thing in these groups literally
for years.

The basic difference between VBA and Jet SQL is that Jet uses the
DECIMAL type natively. If you think about it, this makes perfect
sense. VBA is for business logical and will benefit from hardware
support that floating point values enjoy e.g. good performance when
calculating. Jet SQL is for data management *only* and therefore
favours accuracy over performance.

And it's one of the arguments I use to counter the 'avoid Decimal in
Access' we see from (<coughs>) some regulars.

The basic rule of thumb for numeric literals in Jet:

1) Integer literal values within the range of INTEGER (Long Integer)
will be considered INTEGER (Long Integer):

SELECT TYPENAME(0), TYPENAME(1E9);

2) Integer literal values beyond the range of INTEGER (Long Integer)
but within the range of DECIMAL (Decimal) will be considered DECIMAL
(Decimal):

SELECT TYPENAME(-2147483649), TYPENAME(1E10);

3) Integer literal values beyond the range of DECIMAL (Decimal) will
be considered FLOAT (Double):

SELECT TYPENAME(79228162514264337593543950336), TYPENAME(1E29);

4) Decimal literal values within the range of DECIMAL (Decimal) will
be considered DECIMAL (Decimal):

SELECT TYPENAME(0.5), TYPENAME(-1E-28);

5) Decimal literal values beyond the range of DECIMAL (Decimal) will
be considered FLOAT (Double):

SELECT TYPENAME(123456789012345.123456789012345), TYPENAME(1E-29);

There are some exceptions to the rule (that being the nature of rules
of thumb) e.g. decimal literal values that are within the range of
INTEGER (Long Integer) will be considered INTEGER (Long Integer)
unless they are a multiple of 10 in which case they will be considered
FLOAT (Double) e.g.

SELECT TYPENAME(1001.0), TYPENAME(1000.0)

returns 'Long' and 'Double' respectively.

The next consideration is that by using / 'slash' you are using what
Microsoft calls "the floating-point division operator":

Microsoft Office XP Developer
The Currency and Decimal Data Types
http://msdn2.microsoft.com/en-us/library/aa164763(office.10).aspx

A Note About Division

Any time you use the floating-point division operator (/), you are
performing floating-point division, and your return value will be of
type Double. This is true whether your dividend and divisor are
integer, floating-point, or fixed-point values. It is true whether or
not your result has a decimal portion.

For example, running the following code from the Immediate window
prints "Double":

? TypeName(2.34/5.9)

[Unquote]

Again, in Jet the DECIMAL type exhibits different behaviour: if the
dividend or divisor is of type DECIMAL then the return value will be
of type DECIMAL:

SELECT TYPENAME(2 / 0.5), TYPENAME(3E9 / 1000)

Hopefully this should explain the differences you are seeing in the
Immediate Window.
 
J

Jamie Collins

the
experience of others over the years such as michka who has been known to
refer to pouring energy into investigating stuff no one esle is interested
in.

Allen, sincere thanks for your comments.

[Exiting, now with tongue firmly embedded in cheek] I'll try hooking
up with this 'michka' guy, see what he thinks of Jet these days...

Jamie.

--
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

How do you turn off automatic rounding in a query IIf statement? 2
Rounding Problem 1
Rounding numbers 3
Rounding problem 9
Rounding problem 2
Rounding question 9
Stop Rounding in Forms 2
Rounding 10

Top