Abs(Int(Val(37.3) * 100))=?
Deal Experts,
I am stuck in a small questions. I just wanted to know about the result of the below function
Code:
Public Sub test()
MsgBox Abs(Int(Val(37.3) * 100))
End Sub
After running this, result is 3729 but this is not corrrect. It should be 3730.
Can anyone help me
Re: Abs(Int(Val(37.3) * 100))=?
Try now
vb Code:
Sub Test()
MsgBox Abs(Val(37.3) * 100)
End Sub
or
vb Code:
Sub aaa()
MsgBox Abs(37.3 * 100)
End Sub
or
vb Code:
Sub aaa()
MsgBox Abs(CInt(Val(37.3) * 100))
End Sub
Re: Abs(Int(Val(37.3) * 100))=?
Thanks for the solution.
but my intention is to know the reason why its returing 3729?
Is this a bug?
Re: Abs(Int(Val(37.3) * 100))=?
You're probably familiar with the Int() function, which returns the integer portion of a value. Similarly, the CInt() function does much the same thing. The main difference between the two functions is the data type of the value it returns:
- Int() returns the same data type as the value it's passed. - CInt() always returns an Integer data type.
In addition, CInt rounds the value given to it, while Int truncates the decimal portion.
Re: Abs(Int(Val(37.3) * 100))=?
- Int() returns the same data type as the value it's passed. - CInt() always returns an Integer data type.
Then it should return like this
Code:
ABS(INT(VAL(37.3)*100))
STEP 1 : VAL(37.3)*100)=3730
STEP 2 : INT(3730)
STEP 3 : ABS(3730)
STEP 4 : 3730
but its returning 3729. its funny :) Please try it and then suggest.
Re: Abs(Int(Val(37.3) * 100))=?
Quote:
Originally Posted by vksingh24
but my intention is to know the reason why its returning 3729?
Just to explain a little more what you are seeing...
Floating point numbers store the value in a very different way to which the value is displayed.
For example 11 is stored accurately as 2^3*(1+(1/2^2)+(1/2^3)). The red bits show what is stored
37.3 cannot be stored accurately in the same way, 2^5*(1+(0.165625)) where the 0.165625 portion cannot be accurately represented in a binary fraction form.
The nearest a single can get is 2^5*(1+(1/2^3)+(1/2^5)+(1/2^7)+(1/2^10)+(1/2^11)+(1/2^14)+(1/2^15)+(1/2^18)+(1/2^19)+(1/2^22)+(1/2^23)) ~ 37.2999992
A Double gets closer but it still cannot store it perfectly.
More info at Wikipedia
Edit: if you used the Currency or Decimal data types you would not see this phenomenon.