Floating point division

Discussion in 'Microsoft VB .NET' started by Tom C, Jan 25, 2012.

  1. Tom C

    Tom C Guest

    The following result is from the immediate window in vs2005 just to
    illustrate what I am seeing in code. Quite obviously, .08 / .1 does
    not equal 0.79999999999999993. What is going on here?

    ? 8/1
    8.0
    ? .8/1
    0.8
    ? .8/.1
    8.0
    ? .08/.1
    0.79999999999999993
    ? .08/.01
    8.0
    ? .008/.001
    8.0
    ? .008/1
    0.008
    ? .008/.01
    0.8
    ? .008/.1
    0.08
    ? .08/.1
     
    Tom C, Jan 25, 2012
    #1
    1. Advertisements

  2. Tom C

    ralph Guest

    On Wed, 25 Jan 2012 12:21:40 -0800 (PST), Tom C
    <> wrote:

    >The following result is from the immediate window in vs2005 just to
    >illustrate what I am seeing in code. Quite obviously, .08 / .1 does
    >not equal 0.79999999999999993. What is going on here?
    >


    Every beginnig developer gets burned by this sooner or later.

    http://floating-point-gui.de/

    -ralph
     
    ralph, Jan 25, 2012
    #2
    1. Advertisements

  3. Tom C

    Tom C Guest

    On Jan 25, 2:49 pm, ralph <> wrote:
    > On Wed, 25 Jan 2012 12:21:40 -0800 (PST), Tom C
    >
    > <> wrote:
    > >The following result is from the immediate window in vs2005 just to
    > >illustrate what I am seeing in code. Quite obviously, .08 / .1 does
    > >not equal 0.79999999999999993. What is going on here?

    >
    > Every beginnig developer gets burned by this sooner or later.
    >
    > http://floating-point-gui.de/
    >
    > -ralph


    I have probably been programming longer than you are old but aside
    from that, I have never seen this and I am looking for the cliff notes
    answer not a college degree in math so please don't reply with another
    antagonistic comment and link.

    thanks
     
    Tom C, Jan 25, 2012
    #3
  4. On Wed, 25 Jan 2012 13:25:33 -0800 (PST), Tom C
    <> wrote:

    >On Jan 25, 2:49 pm, ralph <> wrote:
    >> On Wed, 25 Jan 2012 12:21:40 -0800 (PST), Tom C
    >>
    >> <> wrote:
    >> >The following result is from the immediate window in vs2005 just to
    >> >illustrate what I am seeing in code. Quite obviously, .08 / .1 does
    >> >not equal 0.79999999999999993. What is going on here?

    >>
    >> Every beginnig developer gets burned by this sooner or later.
    >>
    >> http://floating-point-gui.de/
    >>
    >> -ralph

    >
    >I have probably been programming longer than you are old but aside
    >from that, I have never seen this and I am looking for the cliff notes
    >answer not a college degree in math so please don't reply with another
    >antagonistic comment and link.


    Well, now, you have something to learn about floating-point
    arithmetic. The site mentioned above appears to be a good
    introduction to the issue.

    The antagonism is actually yours. ralph's answer was very good.

    You have some reading to do.

    Sincerely,

    Gene Wirchenko
     
    Gene Wirchenko, Jan 25, 2012
    #4
  5. Tom C

    ralph Guest

    On Wed, 25 Jan 2012 13:25:33 -0800 (PST), Tom C
    <> wrote:

    >On Jan 25, 2:49 pm, ralph <> wrote:
    >> On Wed, 25 Jan 2012 12:21:40 -0800 (PST), Tom C
    >>
    >> <> wrote:
    >> >The following result is from the immediate window in vs2005 just to
    >> >illustrate what I am seeing in code. Quite obviously, .08 / .1 does
    >> >not equal 0.79999999999999993. What is going on here?

    >>
    >> Every beginnig developer gets burned by this sooner or later.
    >>
    >> http://floating-point-gui.de/
    >>
    >> -ralph

    >
    >I have probably been programming longer than you are old but aside
    >from that, I have never seen this and I am looking for the cliff notes
    >answer not a college degree in math so please don't reply with another
    >antagonistic comment and link.
    >
    >thanks


    I apologize if I ruffled your feathers. Not intended, just wanted to
    let you know this is a common issue EVERYONE runs into sooner or
    later. Surprised if you have been programming for more years than I
    (34 years) you haven't encountered this before.

    I did provide the 'Cliff Notes' answer. If you want the real
    nitty-gritty go here. (PS: Note the title.)

    "What Every Computer Scientist Should Know About Floating-Point
    Arithmetic"
    http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

    -ralph
     
    ralph, Jan 26, 2012
    #5
  6. Tom C

    Auric__ Guest

    Tom C wrote:

    > On Jan 25, 2:49 pm, ralph <> wrote:
    >> On Wed, 25 Jan 2012 12:21:40 -0800 (PST), Tom C
    >>
    >> <> wrote:
    >> >The following result is from the immediate window in vs2005 just to
    >> >illustrate what I am seeing in code. Quite obviously, .08 / .1 does
    >> >not equal 0.79999999999999993. What is going on here?

    >>
    >> Every beginnig developer gets burned by this sooner or later.
    >>
    >> http://floating-point-gui.de/
    >>
    >> -ralph

    >
    > I have probably been programming longer than you are old but aside
    > from that, I have never seen this and I am looking for the cliff notes
    > answer not a college degree in math so please don't reply with another
    > antagonistic comment and link.


    I've been programming for close to 30 years and I *still* get burned by fp
    problems on occasion.

    --
    DIVINARUM RERUM SCRIPTOR
     
    Auric__, Jan 26, 2012
    #6
  7. Tom C

    ralph Guest

    On Thu, 26 Jan 2012 02:34:52 +0000 (UTC), "Auric__"
    <> wrote:

    >Tom C wrote:
    >
    >> On Jan 25, 2:49 pm, ralph <> wrote:
    >>> On Wed, 25 Jan 2012 12:21:40 -0800 (PST), Tom C
    >>>
    >>> <> wrote:
    >>> >The following result is from the immediate window in vs2005 just to
    >>> >illustrate what I am seeing in code. Quite obviously, .08 / .1 does
    >>> >not equal 0.79999999999999993. What is going on here?
    >>>
    >>> Every beginnig developer gets burned by this sooner or later.
    >>>
    >>> http://floating-point-gui.de/
    >>>
    >>> -ralph

    >>
    >> I have probably been programming longer than you are old but aside
    >> from that, I have never seen this and I am looking for the cliff notes
    >> answer not a college degree in math so please don't reply with another
    >> antagonistic comment and link.

    >
    >I've been programming for close to 30 years and I *still* get burned by fp
    >problems on occasion.


    LOL

    Me too!
     
    ralph, Jan 27, 2012
    #7
  8. Tom C

    The Mad Ape Guest

    On 25/01/2012 9:31 PM, ralph wrote:
    > On Wed, 25 Jan 2012 13:25:33 -0800 (PST), Tom C
    > <> wrote:
    >
    >> On Jan 25, 2:49 pm, ralph<> wrote:
    >>> On Wed, 25 Jan 2012 12:21:40 -0800 (PST), Tom C
    >>>
    >>> <> wrote:
    >>>> The following result is from the immediate window in vs2005 just to
    >>>> illustrate what I am seeing in code. Quite obviously, .08 / .1 does
    >>>> not equal 0.79999999999999993. What is going on here?
    >>>
    >>> Every beginnig developer gets burned by this sooner or later.
    >>>
    >>> http://floating-point-gui.de/
    >>>
    >>> -ralph

    >>
    >> I have probably been programming longer than you are old but aside
    >>from that, I have never seen this and I am looking for the cliff notes
    >> answer not a college degree in math so please don't reply with another
    >> antagonistic comment and link.
    >>
    >> thanks

    >
    > I apologize if I ruffled your feathers. Not intended, just wanted to
    > let you know this is a common issue EVERYONE runs into sooner or
    > later. Surprised if you have been programming for more years than I
    > (34 years) you haven't encountered this before.
    >
    > I did provide the 'Cliff Notes' answer. If you want the real
    > nitty-gritty go here. (PS: Note the title.)
    >
    > "What Every Computer Scientist Should Know About Floating-Point
    > Arithmetic"
    > http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
    >
    > -ralph


    That hurt my head when I read it. The other article was more on my
    level. Fluoride in the water must have dumbed me down :)

    Good to know that I have been doing something right...just didn't knw it
    until now.

    TMA
     
    The Mad Ape, Jan 30, 2012
    #8
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. devendra pardeshi
    Replies:
    2
    Views:
    912
    Herfried K. Wagner [MVP]
    Aug 28, 2003
  2. Andreas =?ISO-8859-1?Q?M=FCller?=

    Editor is reformatting floating point values from e.g. 1.1F to 1.10000002F

    Andreas =?ISO-8859-1?Q?M=FCller?=, Sep 12, 2003, in forum: Microsoft VB .NET
    Replies:
    2
    Views:
    145
    Andreas =?ISO-8859-1?Q?M=FCller?=
    Sep 15, 2003
  3. aprivate

    how can I compare floating point numbers?

    aprivate, Jul 27, 2005, in forum: Microsoft VB .NET
    Replies:
    4
    Views:
    478
    Jay B. Harlow [MVP - Outlook]
    Jul 27, 2005
  4. Ali Chambers

    Floating point data analysis

    Ali Chambers, Aug 17, 2006, in forum: Microsoft VB .NET
    Replies:
    4
    Views:
    149
    Ali Chambers
    Aug 17, 2006
  5. Tom C

    Floating point division

    Tom C, Jan 25, 2012, in forum: Microsoft VB .NET
    Replies:
    0
    Views:
    398
    Tom C
    Jan 25, 2012
Loading...

Share This Page