Double+Int16

[Doug Bell]

Hi,
I am having a bit of a resolution problem.
I have a form that accepts elapsed time in various formats including 1.5 and
1:30

and converting it to minutes but I am getting a resolution problem.

Can someone explain this to me, my code:
dblMins = Val(stLoadSOs)

intHrs = Int(dblMins)

dblMins = dblMins - intHrs

intMins = intHrs * 60 + Int(dblMins * intTim)

if the input is '1:20' then the unshown code sets

stLoadSOs = "1.20" and intTim = 100

dblMins = 1.2 (correctly)

intHrs = 1 (correctly)

but then dblMins = dblMins - intHrs shows 19.999999 (incorrectly)

1.2 - 1 should equal 0.2

What is causing this resolution problem?



Thanks



Doug

 

[Jon Skeet [C# MVP]]

I am having a bit of a resolution problem.
I have a form that accepts elapsed time in various formats including 1.5 and
1:30

and converting it to minutes but I am getting a resolution problem.

Can someone explain this to me, my code:
dblMins = Val(stLoadSOs)

intHrs = Int(dblMins)

dblMins = dblMins - intHrs

intMins = intHrs * 60 + Int(dblMins * intTim)

if the input is '1:20' then the unshown code sets

stLoadSOs = "1.20" and intTim = 100

dblMins = 1.2 (correctly)

intHrs = 1 (correctly)

but then dblMins = dblMins - intHrs shows 19.999999 (incorrectly)

1.2 - 1 should equal 0.2

What is causing this resolution problem?

Your misunderstanding of floating point numbers is faulty.

dblMins isn't actually 1.2, as that isn't representable in binary
floating point. That's just how the debugger (or whatever) is showing
it.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html

 

[Doug Bell]

Thanks Jon

Your misunderstanding of floating point numbers is faulty.

dblMins isn't actually 1.2, as that isn't representable in binary
floating point. That's just how the debugger (or whatever) is showing
it.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html