Meaning of Double.Epsilon

[Daniel]

I am grappling with the idea of double.Epsilon. I have written the
following test:

[Test]
public void FuzzyDivisionTest()
{
double a = 0.33333d;
double b = 1d / 3d;

Assert.IsFalse(a == b, "Built-in == operator should not be
fuzzy");
Assert.IsTrue( a-b < double.Epsilon);
Assert.IsTrue( Math.Abs(a-b) < double.Epsilon);
Assert.IsTrue( Math.Abs(b-a) < double.Epsilon);
}

However, something is weird. The first 2 assertions pass, however the
last 2 fail. Is Math.Abs() losing my Epsilon value? Is that correct?

 

[Jon Skeet [C# MVP]]

I am grappling with the idea of double.Epsilon. I have written the
following test:

[Test]
public void FuzzyDivisionTest()
{
double a = 0.33333d;
double b = 1d / 3d;

Assert.IsFalse(a == b, "Built-in == operator should not be
fuzzy");
Assert.IsTrue( a-b < double.Epsilon);
Assert.IsTrue( Math.Abs(a-b) < double.Epsilon);
Assert.IsTrue( Math.Abs(b-a) < double.Epsilon);
}

However, something is weird. The first 2 assertions pass, however the
last 2 fail. Is Math.Abs() losing my Epsilon value? Is that correct?

Epsilon is the smallest positive non-zero value that a double can hold.
It will never be true (I *think*) that a != b and
Math.Abs(b-a) < double.Epsilon.

a-b is less than double.Epsilon because it's negative.

 

[Jon]

My interpretation of the Visual Studio help on this is that this reply is
partially correct. It is possible that
a != b
and
Math.Abs(b - a) < double.Epsilon

However the difference between a and b in your example is evidently larger
than double.Epsilon - it is 3.3333333(recurring)e-10, and double.Epsilon ==
4.94065645841247e-324

As the previous reply said, the reason the second assertion passes is that
a - b is negative.

I am grappling with the idea of double.Epsilon. I have written the
following test:

[Test]
public void FuzzyDivisionTest()
{
double a = 0.33333d;
double b = 1d / 3d;

Assert.IsFalse(a == b, "Built-in == operator should not be
fuzzy");
Assert.IsTrue( a-b < double.Epsilon);
Assert.IsTrue( Math.Abs(a-b) < double.Epsilon);
Assert.IsTrue( Math.Abs(b-a) < double.Epsilon);
}

However, something is weird. The first 2 assertions pass, however the
last 2 fail. Is Math.Abs() losing my Epsilon value? Is that correct?

Epsilon is the smallest positive non-zero value that a double can hold.
It will never be true (I *think*) that a != b and
Math.Abs(b-a) < double.Epsilon.

a-b is less than double.Epsilon because it's negative.

 

[Niki Estner]

I am grappling with the idea of double.Epsilon. I have written the
following test:

[Test]
public void FuzzyDivisionTest()
{
double a = 0.33333d;
double b = 1d / 3d;

Assert.IsFalse(a == b, "Built-in == operator should not be
fuzzy");
Assert.IsTrue( a-b < double.Epsilon);
Assert.IsTrue( Math.Abs(a-b) < double.Epsilon);
Assert.IsTrue( Math.Abs(b-a) < double.Epsilon);
}

However, something is weird. The first 2 assertions pass, however the
last 2 fail. Is Math.Abs() losing my Epsilon value? Is that correct?

Epsilon is the smallest positive non-zero value that a double can hold.
It will never be true (I *think*) that a != b and
Math.Abs(b-a) < double.Epsilon.

Yes, I think the docs about Double.Epsilon are actually wrong; Quote:
"Instead [of comparing double values with ==], determine if the two sides of
a comparison are close enough to equal for your purposes by comparing
whether the absolute value of the difference between the left and right-hand
sides is less than Epsilon."

I usually use something like "Math.Abs(a-b) < 1e-10", but I'm not sure if
this is common practice.

Niki

 

[Jon Skeet [C# MVP]]

My interpretation of the Visual Studio help on this is that this reply is
partially correct. It is possible that
a != b
and
Math.Abs(b - a) < double.Epsilon

Could you give an example? I believe that

Math.Abs(b-a) < double.Epsilon => b-a != 0 => b==a

It's the last step that I'm not sure about though - is it possible for
a and b to be different, but their difference to be so close to 0 as to
be unrepresentably small? It feels unlikely, but I know that intuition
is often horribly flawed when it comes to floating point maths.

 

[Jon Skeet [C# MVP]]

Epsilon is the smallest positive non-zero value that a double can hold.
It will never be true (I *think*) that a != b and
Math.Abs(b-a) < double.Epsilon.

Yes, I think the docs about Double.Epsilon are actually wrong; Quote:
"Instead [of comparing double values with ==], determine if the two sides of
a comparison are close enough to equal for your purposes by comparing
whether the absolute value of the difference between the left and right-hand
sides is less than Epsilon."

I usually use something like "Math.Abs(a-b) < 1e-10", but I'm not sure if
this is common practice.

Yup - that seems a much better idea. Do you want to report the problem
to MS, or shall I?

 

[Niki Estner]

Could you give an example? I believe that

Math.Abs(b-a) < double.Epsilon => b-a != 0 => b==a

It's the last step that I'm not sure about though - is it possible for
a and b to be different, but their difference to be so close to 0 as to
be unrepresentably small? It feels unlikely, but I know that intuition
is often horribly flawed when it comes to floating point maths.

I guess it's possible if a and b are stored in 80-bit FPU registers: b-a (80
bit) could still be different form 0 (using an 80-bit comparison), but the
difference would be smaller than double.Epsilon. Also Math.Abs(b-a) would
actually return 0 (if not inlined) as b-a would get casted to a (64-bit)
double value.
Sample code (compares (double.Epsilon/2) with 0):

using System;
public class MyClass
{
public static void Main()
{
double a = double.Parse("2");
Console.WriteLine(0 == (double.Epsilon/a));
Console.WriteLine((double.Epsilon/a-0) < double.Epsilon);
}
}

Niki

 

[Niki Estner]

Epsilon is the smallest positive non-zero value that a double can hold.
It will never be true (I *think*) that a != b and
Math.Abs(b-a) < double.Epsilon.

Yes, I think the docs about Double.Epsilon are actually wrong; Quote:
"Instead [of comparing double values with ==], determine if the two sides
of
a comparison are close enough to equal for your purposes by comparing
whether the absolute value of the difference between the left and
right-hand
sides is less than Epsilon."

I usually use something like "Math.Abs(a-b) < 1e-10", but I'm not sure if
this is common practice.

Yup - that seems a much better idea. Do you want to report the problem
to MS, or shall I?

You're the MVP ;-)

Niki

 

[Jon Skeet [C# MVP]]

I guess it's possible if a and b are stored in 80-bit FPU registers: b-a (80
bit) could still be different form 0 (using an 80-bit comparison), but the
difference would be smaller than double.Epsilon. Also Math.Abs(b-a) would
actually return 0 (if not inlined) as b-a would get casted to a (64-bit)
double value.

Aargh, yes. I knew there was a good reason to be hesitant

Sample code (compares (double.Epsilon/2) with 0):

<snip>

Good, thanks. Here's something interesting, based on your code:

using System;
public class MyClass
{
public static void Main()
{
double two = double.Parse("2");
double a = double.Epsilon/two;
double b = 0;
Console.WriteLine(a==b);
Console.WriteLine(Math.Abs(b-a) < double.Epsilon);
}
}

That prints out (on my box):
True
True

If you comment out the last line, however, it just prints out
False

Presumably the call to Math.Abs forces a and b to be "normal" 64-bit
values rather than just 80-bit registers, or something like that.

 

[Jon Skeet [C# MVP]]

You're the MVP ;-)

Righto. (Admittedly I didn't have any luck persuading them that
System.Decimal is a floating point type rather than a fixed point type
- although reporting it again with the online bug database for 2.0,
it's been accepted!)

 

[Niki Estner]

<snip>
Good, thanks. Here's something interesting, based on your code:

using System;
public class MyClass
{
public static void Main()
{
double two = double.Parse("2");
double a = double.Epsilon/two;
double b = 0;
Console.WriteLine(a==b);
Console.WriteLine(Math.Abs(b-a) < double.Epsilon);
}
}

That prints out (on my box):
True
True

If you comment out the last line, however, it just prints out
False

Presumably the call to Math.Abs forces a and b to be "normal" 64-bit
values rather than just 80-bit registers, or something like that.

Yes, I never really understood when variables get enregistered and when not.
And the fact that the rules change as soon as I attach a debugger doesn't
make it easier either...

Niki

 

[Jon Skeet [C# MVP]]

Yes, I never really understood when variables get enregistered and when not.
And the fact that the rules change as soon as I attach a debugger doesn't
make it easier either...

I'm glad I'm not the only one - your previous posts suggest that you
know far more about JIT optimisation than I'm ever likely to

 

[Niki Estner]

I'm glad I'm not the only one - your previous posts suggest that you
know far more about JIT optimisation than I'm ever likely to

Thanks a lot
Unfortunately I'm guessing blindly more often than I'd like to, too...

I think it would be a wise decision if MS would at least release a little
more documentation on the internals of the JIT: Many developers simply
assume it doesn't optimize at all, others try to "optimize" their code and
break real optimizations that way (register-variables are quite fragile) or
create unmaintainable code. At least a few rules of thumb (when will it
remove range checking? when will it inline a function? when will it
enregister a variable? etc.) would be very helpful here...

On the other hand, we probably wouldn't have any use for cordbg any more
then; Wouldn't that be a shame?

Niki

 

[Daniel]

Jon - thanks for the reply, I was hoping to hear from you or John
Bentley, the recognized leaders on the internet in this area!

 

[cody]

I tend to say that operators == result is undefined for floating point types
(except decimal).
You never know how variables are stored in memory. testing for equality for
a floating point variables never makes sense, even if you do:

float a = 1f;
float b = 1f;
if (a==b) { }

The Jitter may choose to place a and b in difference register types (32, 64
or 80 bits), so a==b may return true or false depending on the machine type,
framework version, debug or release settings or if a debugger is attached or
in which way the variables are used in code or wheather the method is
inlined or not.

So it would be great if the compiler would at least issue a warning if it
spots code which tests for equality with floating point values.

The only thing I could imagine is testing for NaN but in that case one
should use IsNaN() methods of struct Double and Single.

 

[cody]

Damn. Errata:

NaN cannot be used to test for is-not-a-number anyway. You *have to* use
IsNaN instead.
The same is true for NegativeInfinity and PositiveInfinity.

--
cody

Freeware Tools, Games and Humour
http://www.deutronium.de.vu || http://www.deutronium.tk

I tend to say that operators == result is undefined for floating point types
(except decimal).
You never know how variables are stored in memory. testing for equality for
a floating point variables never makes sense, even if you do:

float a = 1f;
float b = 1f;
if (a==b) { }

The Jitter may choose to place a and b in difference register types (32, 64
or 80 bits), so a==b may return true or false depending on the machine type,
framework version, debug or release settings or if a debugger is attached or
in which way the variables are used in code or wheather the method is
inlined or not.

So it would be great if the compiler would at least issue a warning if it
spots code which tests for equality with floating point values.

The only thing I could imagine is testing for NaN but in that case one
should use IsNaN() methods of struct Double and Single.

--
cody

Freeware Tools, Games and Humour
http://www.deutronium.de.vu || http://www.deutronium.tk
b-a

 

[Niki Estner]

I tend to say that operators == result is undefined for floating point
types
(except decimal).
You never know how variables are stored in memory.

Of course you do. The *memory* size is determined by the variable's type.

testing for equality for
a floating point variables never makes sense, even if you do:

float a = 1f;
float b = 1f;
if (a==b) { }

The Jitter may choose to place a and b in difference register types (32,
64
or 80 bits), so a==b may return true or false depending on the machine
type,
framework version, debug or release settings or if a debugger is attached
or
in which way the variables are used in code or wheather the method is
inlined or not.

No, I'm pretty sure (a==b) will always be true here: AFAIK it's guaranteed
that converting a 32-bit floating point value to 64 bit or 80 bit will never
cause any loss in precision; Also, you're guaranteed that a 64-bit value
will never be stored in a 32-bit or smaller register (note that intel FPU's
don't have 32-bit or 64-bit registers anyway). So, in the case of constants
the '==' operator should always work correctly.

So it would be great if the compiler would at least issue a warning if it
spots code which tests for equality with floating point values.

It's common to test a floating point value for some constant it's
initialized to (like: if (a == 0) a = GetSomeValue(). Other tests for
common values (e.g. double.Parse(str) == 1) will work, too.
A warning wouldn't hurt (if it can be turned off), but I would prefer a
special "about equal" operator with fuzzy comparison rules.

Niki

 

[Jon Skeet [C# MVP]]

It's common to test a floating point value for some constant it's
initialized to (like: if (a == 0) a = GetSomeValue(). Other tests for
common values (e.g. double.Parse(str) == 1) will work, too.
A warning wouldn't hurt (if it can be turned off), but I would prefer a
special "about equal" operator with fuzzy comparison rules.

I don't think I'd want an operator, but a utility method on Double
wouldn't hurt. It could either take the two values and the "fuzziness
factor", or there could perhaps be a default fuzziness factor which
could be changed programatically.

 

[cody]

I tend to say that operators == result is undefined for floating point

Of course you do. The *memory* size is determined by the variable's type.

Yes in memory, but you do not know which CPU registers will be used.

No, I'm pretty sure (a==b) will always be true here: AFAIK it's guaranteed
that converting a 32-bit floating point value to 64 bit or 80 bit will never
cause any loss in precision;

Maybe not precision but it will not be exactly the same value.
The following program outputs "true true false false" on my machine:

double d=0.0;
float f=0.0f;
double d1=1.0;
float f1=1.0f;
double d2=1.3;
float f2=1.3f;
float f3=float.Parse("0.2");
float f4=0.2f;
MessageBox.Show(string.Format("{0} {1} {2} {3}", d==f, d1==f1, d2==f2,
f3==f4));

Also, you're guaranteed that a 64-bit value
will never be stored in a 32-bit or smaller register

Thats clear. It it only guaranteed that a variable size 32 bit will be
stored in a register with *at least* 32 bit or more.

So, in the case of constants the '==' operator should always work
correctly.

It won't see above.

It's common to test a floating point value for some constant it's
initialized to (like: if (a == 0) a = GetSomeValue(). Other tests for
common values (e.g. double.Parse(str) == 1) will work, too.

If you are lucky it will work on some machines with some values.

A warning wouldn't hurt (if it can be turned off), but I would prefer a
special "about equal" operator with fuzzy comparison rules.

I'd vote against it. Testing equality is always bad idea anyway. Even if,
there must be a possibility to specify the tolerance value because the
fuzzyness really depends on the application. The method wouldn't do anything
else but Math.Abs(a,b)<Tolerance so there is very little point for including
such a method in the framework.

 

[Niki Estner]

...
Maybe not precision but it will not be exactly the same value.
The following program outputs "true true false false" on my machine:

double d=0.0;
float f=0.0f;
double d1=1.0;
float f1=1.0f;
double d2=1.3;
float f2=1.3f;
float f3=float.Parse("0.2");
float f4=0.2f;
MessageBox.Show(string.Format("{0} {1} {2} {3}", d==f, d1==f1, d2==f2,
f3==f4));
...

The point is that
float a=1.3f;
double b=1.3f;
Console.WriteLine(a==b);
Will always return true, because converting "1.3f" (or any other 32-bit
float-value) to double will not change it; Of course this value can actually
be different from "1.3d".

Your original example was:
float a = 1f;
float b = 1f;
if (a==b) { }

As shown above, "1f" (or any other float constant) will not be changed when
converted to a double (or 80-bit fp value), so that kind of comparison *is*
safe.

Niki