|
-
Jan 20th, 2004, 09:50 PM
#1
Thread Starter
Stuck in the 80s
Double - Double = Wrong Answer?
I'm fairly new to VB.Net, but am fairly fluent in VB6. I have some code like this:
Code:
dblTotal = dblTwo - dblOne
Where dblTwo = 17 and dblOne = 16.94 and dblTotal ends up being equal to "0.0599999999999987"
huh?
-
Jan 20th, 2004, 10:03 PM
#2
Its a seems to be a big problem in VB6 with double precision, too.
Try casting to a Decimal...
VB Code:
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim d1, d2, d3 As Double
d1 = 16.94
d2 = 17
d3 = CType(d2, Decimal) - CType(d1, Decimal)
MessageBox.Show(d3)
End Sub
Laugh, and the world laughs with you. Cry, and you just water down your vodka.
Take credit, not responsibility
-
Jan 20th, 2004, 10:53 PM
#3
Frenzied Member
That's a well known bug ("feature" in MS lingo) of floating point numbers in binary. It's the reason you don't check floating point numbers for exact equality, just for an arbitrarily small difference. Like crpptcblade says, you'll get the same problem in VB6 as well, not to mention C, C++, Java, Pascal, etc.
-
Jan 20th, 2004, 11:04 PM
#4
Thread Starter
Stuck in the 80s
I switched to using the Decimal datatype and it works fine.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|