|
-
Nov 27th, 2008, 04:52 AM
#1
Thread Starter
Addicted Member
Abs(Int(Val(37.3) * 100))=?
Deal Experts,
I am stuck in a small questions. I just wanted to know about the result of the below function
Code:
Public Sub test()
MsgBox Abs(Int(Val(37.3) * 100))
End Sub
After running this, result is 3729 but this is not corrrect. It should be 3730.
Can anyone help me
Do Good. Be Good. The World is yours.
-
Nov 27th, 2008, 05:04 AM
#2
Re: Abs(Int(Val(37.3) * 100))=?
Try now
vb Code:
Sub Test()
MsgBox Abs(Val(37.3) * 100)
End Sub
or
vb Code:
Sub aaa()
MsgBox Abs(37.3 * 100)
End Sub
or
vb Code:
Sub aaa()
MsgBox Abs(CInt(Val(37.3) * 100))
End Sub
A good exercise for the Heart is to bend down and help another up...
Please Mark your Thread " Resolved", if the query is solved
MyGear:
★ CPU ★ Ryzen 5 5800X
★ GPU ★ NVIDIA GeForce RTX 3080 TI Founder Edition
★ RAM ★ G. Skill Trident Z RGB 32GB 3600MHz
★ MB ★ ASUS TUF GAMING X570 (WI-FI) ATX Gaming
★ Storage ★ SSD SB-ROCKET-1TB + SEAGATE 2TB Barracuda IHD
★ Cooling ★ NOCTUA NH-D15 CHROMAX BLACK 140mm + 10 of Noctua NF-F12 PWM
★ PSU ★ ANTEC HCG-1000-EXTREME 1000 Watt 80 Plus Gold Fully Modular PSU
★ Case ★ LIAN LI PC-O11 DYNAMIC XL ROG (BLACK) (G99.O11DXL-X)
★ Monitor ★ LG Ultragear 27" 240Hz Gaming Monitor
★ Keyboard ★ TVS Electronics Gold Keyboard
★ Mouse ★ Logitech G502 Hero
-
Nov 27th, 2008, 05:27 AM
#3
Thread Starter
Addicted Member
Re: Abs(Int(Val(37.3) * 100))=?
Thanks for the solution.
but my intention is to know the reason why its returing 3729?
Is this a bug?
Do Good. Be Good. The World is yours.
-
Nov 27th, 2008, 06:11 AM
#4
Re: Abs(Int(Val(37.3) * 100))=?
You're probably familiar with the Int() function, which returns the integer portion of a value. Similarly, the CInt() function does much the same thing. The main difference between the two functions is the data type of the value it returns:
- Int() returns the same data type as the value it's passed. - CInt() always returns an Integer data type.
In addition, CInt rounds the value given to it, while Int truncates the decimal portion.
A good exercise for the Heart is to bend down and help another up...
Please Mark your Thread " Resolved", if the query is solved
MyGear:
★ CPU ★ Ryzen 5 5800X
★ GPU ★ NVIDIA GeForce RTX 3080 TI Founder Edition
★ RAM ★ G. Skill Trident Z RGB 32GB 3600MHz
★ MB ★ ASUS TUF GAMING X570 (WI-FI) ATX Gaming
★ Storage ★ SSD SB-ROCKET-1TB + SEAGATE 2TB Barracuda IHD
★ Cooling ★ NOCTUA NH-D15 CHROMAX BLACK 140mm + 10 of Noctua NF-F12 PWM
★ PSU ★ ANTEC HCG-1000-EXTREME 1000 Watt 80 Plus Gold Fully Modular PSU
★ Case ★ LIAN LI PC-O11 DYNAMIC XL ROG (BLACK) (G99.O11DXL-X)
★ Monitor ★ LG Ultragear 27" 240Hz Gaming Monitor
★ Keyboard ★ TVS Electronics Gold Keyboard
★ Mouse ★ Logitech G502 Hero
-
Nov 27th, 2008, 07:26 AM
#5
Thread Starter
Addicted Member
Re: Abs(Int(Val(37.3) * 100))=?
- Int() returns the same data type as the value it's passed. - CInt() always returns an Integer data type.
Then it should return like this
Code:
ABS(INT(VAL(37.3)*100))
STEP 1 : VAL(37.3)*100)=3730
STEP 2 : INT(3730)
STEP 3 : ABS(3730)
STEP 4 : 3730
but its returning 3729. its funny Please try it and then suggest.
Do Good. Be Good. The World is yours.
-
Nov 27th, 2008, 07:35 AM
#6
Re: Abs(Int(Val(37.3) * 100))=?
 Originally Posted by vksingh24
but my intention is to know the reason why its returning 3729?
Just to explain a little more what you are seeing...
Floating point numbers store the value in a very different way to which the value is displayed.
For example 11 is stored accurately as 2^3*(1+(1/2^2)+(1/2^3)). The red bits show what is stored
37.3 cannot be stored accurately in the same way, 2^5*(1+(0.165625)) where the 0.165625 portion cannot be accurately represented in a binary fraction form.
The nearest a single can get is 2^5*(1+(1/2^3)+(1/2^5)+(1/2^7)+(1/2^10)+(1/2^11)+(1/2^14)+(1/2^15)+(1/2^18)+(1/2^19)+(1/2^22)+(1/2^23)) ~ 37.2999992
A Double gets closer but it still cannot store it perfectly.
More info at Wikipedia
Edit: if you used the Currency or Decimal data types you would not see this phenomenon.
Last edited by Milk; Nov 27th, 2008 at 07:39 AM.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|