I know that Single and Double can only store values that can be stored as
x/2^y, such as 0.5 and 0.125. But when they attempt to store values such as
0.1 or 0.2 whose exact value cannot be stored, what algorithm do they use to
determine what value to store? Thanks.
--
Nathan Sokalski
(E-Mail Removed)
http://www.nathansokalski.com/

"Nathan Sokalski" <(E-Mail Removed)> schrieb
>I know that Single and Double can only store values that can be stored as
>x/2^y, such as 0.5 and 0.125. But when they attempt to store values such as
>0.1 or 0.2 whose exact value cannot be stored, what algorithm do they use
>to determine what value to store? Thanks.

As the documentation
http://msdn.microsoft.com/en-us/libr...em.single.aspx
says, the types comply with IEEE 754 which also defines rounding rules.
Search for it; you'll find wikipedia etc.



Armin

"Nathan Sokalski" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
>I know that Single and Double can only store values that can be stored as
>x/2^y, such as 0.5 and 0.125. But when they attempt to store values such as
>0.1 or 0.2 whose exact value cannot be stored, what algorithm do they use
>to determine what value to store? Thanks.

They use a mantissa and an exponent, so that a number is expressed as
(mantissa)*10^(exponent). We do this frequently in everyday engineering work
when we say for instance, that something took "1.23*10^-6 seconds". However,
when speaking about Single and Double, the "10" has to be interpreted in
base 2 (meaning that it is a "2" in base 10). The exponent and mantissa are
also in base 2. The precission of the mantissa is limited, because the
mantissa and exponent and their signs have to fit into 4 bytes (Single) or 8
bytes (Double). The numbers that can be represented exactly are those that
can be converted into base 2 within that number of bits. For instance, 0.5
(base10) is 0.1 (base 2), but some numbers that have a small number of
decimals in base 10 have an infinite number of decimal places once converted
to base 2, so they will be truncated when assigned to the mantissa, and
therefore they will not be "exact".

"Nathan Sokalski" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
>I know that Single and Double can only store values that can be stored as
>x/2^y, such as 0.5 and 0.125. But when they attempt to store values such as
>0.1 or 0.2 whose exact value cannot be stored, what algorithm do they use
>to determine what value to store? Thanks.
> --
> Nathan Sokalski
> (E-Mail Removed)
> http://www.nathansokalski.com/

What Every Computer Scientist Should Know About Floating-Point Arithmetic
http://docs.sun.com/source/806-3568/ncg_goldberg.html

Here's a PDF version
http://www.physics.ohio-state.edu/~d...point_math.pdf

Ken Halter wrote:

> What Every Computer Scientist Should Know About Floating-Point
> Arithmetic http://docs.sun.com/source/806-3568/ncg_goldberg.html
>
> Here's a PDF version
>
http://www.physics.ohio-state.edu/~d...point_math.pdf

Interesting link, thanks. I can use such a .pdf from time to time. It
is amazing that CS people know so little about it.

--
Rudy Velthuis http://rvelthuis.de

"I once heard two ladies going on and on about the pains of
childbirth and how men don't seem to know what real pain is. I
asked if either of them ever got themselves caught in a zipper."
-- Emo Philips.

I understand how they are stored, that is very simple. My question was what
algorithm they use to determine what value to store. Let me rephrase what I
meant by this. When a value is assigned to a variable that cannot be stored
as an exact value, such as the following:

Dim x As Byte = 1.2

Since the exact value cannot be stored, what algorithm is used to determine
what value will be stored? Since the value is obviously not exact, it must
be rounded up or down, so how do I know which it is?
--
Nathan Sokalski
(E-Mail Removed)
http://www.nathansokalski.com/

"Alberto Poblacion" <earthling-(E-Mail Removed)> wrote
in message news:%(E-Mail Removed)...
> "Nathan Sokalski" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
>>I know that Single and Double can only store values that can be stored as
>>x/2^y, such as 0.5 and 0.125. But when they attempt to store values such
>>as 0.1 or 0.2 whose exact value cannot be stored, what algorithm do they
>>use to determine what value to store? Thanks.
>
> They use a mantissa and an exponent, so that a number is expressed as
> (mantissa)*10^(exponent). We do this frequently in everyday engineering
> work when we say for instance, that something took "1.23*10^-6 seconds".
> However, when speaking about Single and Double, the "10" has to be
> interpreted in base 2 (meaning that it is a "2" in base 10). The exponent
> and mantissa are also in base 2. The precission of the mantissa is
> limited, because the mantissa and exponent and their signs have to fit
> into 4 bytes (Single) or 8 bytes (Double). The numbers that can be
> represented exactly are those that can be converted into base 2 within
> that number of bits. For instance, 0.5 (base10) is 0.1 (base 2), but some
> numbers that have a small number of decimals in base 10 have an infinite
> number of decimal places once converted to base 2, so they will be
> truncated when assigned to the mantissa, and therefore they will not be
> "exact".
>

> Since the exact value cannot be stored, what algorithm is used to
> determine what value will be stored? Since the value is obviously not
> exact, it must be rounded up or down, so how do I know which it is?

But you've already been told: IEEE 754.

SteveT