float number problem

[julien]

Hi, I am using Sybase 12.5 dataserver and ASP.NET
I am calling a stored procedure from my asp.net page, in this stored
procedure, I have 2 float fields that are returned. One is directly
taken from one float column of a table, and the other one is a
substraction of 2 float columns of a table. The first field is all the
time ok, whereas the calculated field is almost all the time wrong,
that is it returns things like :
1,332294 - 1,334709 = -0,00241499999999983 or
1,334 - 1,334093 = -9,29999999998987E-5 or
1,3281 - 1,328256 = -0,000156000000000045
Sometimes it's ok like :
1,8774 - 1,864693 = 0,012707 or
103,96 - 103,8564 = 0,1036

When I launch this stored procedure with isql, which is the basic
inline tool of Sybase, I get all the time the right calculations. How
is that? What can I set in my .Net environment to do the right
calculation all the time?
Thank you for your help,
JULIEN

 

[Hans Kesting]

see http://www.yoda.arachsys.com/csharp/floatingpoint.html

 

[Kevin Spencer]

Hi Hans,

That is an excellent resource! I would recommend it to almost any
programmer.

Another possibility in this particular case might be the use of types. By
types I mean the type of the numbers being computed and the default types of
the .Net platform. In .Net operators are overloaded for different numeric
types. For example, multiplying 2 floats results in a float result.
Multiplying 2 doubles returns a double result. But what happens when you
multiply a float by a double, or an integer by a float? What data type is
returned? And what happens when you perform math on a variable of a certain
type and a numeric literal (e.g. 1.5 or 23)? If you don't cast the types
correctly and/or specify the types of literals, you can get unexpected
results, depending upon the defaults used by the .Net platform. So that
could also be an issue here.

The .Net SDK details all of the default types, the return types for
differing types in a math operation, and the types of the overloaded
operators.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
What You Seek Is What You Get.