Numbers funny business - please help
Could anyone explain the following for me because I am completely baffled:
int(2.01*100) returns 200
int(2.05*100) returns 204
but as expected
int(2.02*100) returns 202
int(2.06*100) returns 206
int(2.09*100) returns 209
and
int(201) returns 201
int(205) returns 205
Before you ask why do I need to use Int it is because the first number, which in my code is a variable, may have more than two decimal places.
And another baffler which probably is behind the results above
Dim Number As Double = 2.01 * 100 'returns 200.99999999999997
Dim Number As Double = 2.05 * 100 'returns 204.99999999999997
Please help me I am going crazy here.
Re: Numbers funny business - please help
Welcome to the wonderful world of floating-point numbers. This phenomenon is well documented. You should do some reading on floating-point numbers to get a full understanding.
Maybe you should use Math.Round instead of Int.
Re: Numbers funny business - please help
I can't use Math.Round because I want to round down always. Is there no solution to this phenomenon
Re: Numbers funny business - please help
Try using Decimal instead of Double, which suffers far less from such errors. That's why Decimal is used for financial calculations. Decimal is bigger and slower than Double though, hence the reason that Double is used wherever it can without causing issues.
Re: Numbers funny business - please help
Thanks for that. Decimal works fine.
This is a mighty big error. Why hasn't it been addressed?
Re: Numbers funny business - please help
Because floating points are imprecise... it's always been that way. Mostly has to do with the fact that decimals don't translate into binary very well sometimes.
And it has been addressed. That's why we have decimal type.
-tg
Re: Numbers funny business - please help
Quote:
Originally Posted by
natrap
This is a mighty big error. Why hasn't it been addressed?
You haven't done any reading on floating-point numbers, have you? If you do, then you'll understand.
Re: Numbers funny business - please help
Quote:
Originally Posted by
natrap
I can't use Math.Round because I want to round down always. Is there no solution to this phenomenon
Use Math.Floor then (but I realize this won't solve your main issue).
Floating point manipulation can be frustrating - there are entire university course dedicated to numerical analysis which cover this and more.
It can seem unintuitive to realize that a $5 calculator (or Windows 'Calculator') can perform rings around the output that we get for simple floating point calculations, but there it is. I've often thought that programming languages might benefit from having a 'calculator' mode - that is, perform as well as a calculator without having to have a math degree to make it happen.
Re: Numbers funny business - please help
That's because calculators don't use floating point processors... they only have to deal with the first 8-10 significant digits, and therefore use what would be the equivelent of a decimal type.
-tg
Re: Numbers funny business - please help
Calculator mode = Decimal.
But Decimal = relatively slow
Therefore Calculator = relatively slow (but your fingers are far slower, so you never notice).