Results 1 to 4 of 4

Thread: Double - Double = Wrong Answer?

  1. #1

    Thread Starter
    Stuck in the 80s The Hobo's Avatar
    Join Date
    Jul 2001
    Location
    Michigan
    Posts
    7,256

    Double - Double = Wrong Answer?

    I'm fairly new to VB.Net, but am fairly fluent in VB6. I have some code like this:

    Code:
    dblTotal = dblTwo - dblOne
    Where dblTwo = 17 and dblOne = 16.94 and dblTotal ends up being equal to "0.0599999999999987"

    huh?
    My evil laugh has a squeak in it.

    kristopherwilson.com

  2. #2
    The Devil crptcblade's Avatar
    Join Date
    Aug 2000
    Location
    Quetzalshacatenango
    Posts
    9,091
    Its a seems to be a big problem in VB6 with double precision, too.

    Try casting to a Decimal...
    VB Code:
    1. Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
    2.         Dim d1, d2, d3 As Double
    3.  
    4.         d1 = 16.94
    5.         d2 = 17
    6.  
    7.         d3 = CType(d2, Decimal) - CType(d1, Decimal)
    8.  
    9.         MessageBox.Show(d3)
    10.  
    11.     End Sub
    Laugh, and the world laughs with you. Cry, and you just water down your vodka.


    Take credit, not responsibility

  3. #3
    Frenzied Member
    Join Date
    Feb 2003
    Location
    Argentina
    Posts
    1,950
    That's a well known bug ("feature" in MS lingo) of floating point numbers in binary. It's the reason you don't check floating point numbers for exact equality, just for an arbitrarily small difference. Like crpptcblade says, you'll get the same problem in VB6 as well, not to mention C, C++, Java, Pascal, etc.

  4. #4

    Thread Starter
    Stuck in the 80s The Hobo's Avatar
    Join Date
    Jul 2001
    Location
    Michigan
    Posts
    7,256
    I switched to using the Decimal datatype and it works fine.
    My evil laugh has a squeak in it.

    kristopherwilson.com

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width