Rounding Error...Please help

  • Thread starter Thread starter ywchan
  • Start date Start date
Y

ywchan

I have written the following codes:

double mean = 313.4;
int fieldValue = 321;
double result = ((double)fieldValue) - mean;
MessageBox.Show(result.ToString());
MessageBox.Show(Math.Pow(result,2).ToString());

However, for the value of result, I got 7.6000000000000227 instead of 7.6
This made my calculation wrong...

Any idea to solve it?
 
ywchan said:
I have written the following codes:

double mean = 313.4;
int fieldValue = 321;
double result = ((double)fieldValue) - mean;
MessageBox.Show(result.ToString());
MessageBox.Show(Math.Pow(result,2).ToString());

However, for the value of result, I got 7.6000000000000227 instead of 7.6
This made my calculation wrong...

Any idea to solve it?
Click to expand...

See http://www.pobox.com/~skeet/csharp/floatingpoint.html

In short, the value stored as "mean" isn't 313.4 (which a double can't
represent) - it's

313.3999999999999772626324556767940521240234375
 
I have written the following codes:
double mean = 313.4;
int fieldValue = 321;
double result = ((double)fieldValue) - mean;
MessageBox.Show(result.ToString());
MessageBox.Show(Math.Pow(result,2).ToString());

However, for the value of result, I got 7.6000000000000227 instead of 7.6
This made my calculation wrong...

Any idea to solve it?
Click to expand...

0.1 can't be binary written with final number of digits. 0.4 = 4 * 0.1, so
here is your 'error'.
 
Back
Top