Results 1 to 6 of 6

Thread: Abs(Int(Val(37.3) * 100))=?

  1. #1

    Thread Starter
    Addicted Member
    Join Date
    Aug 2007
    Location
    India
    Posts
    141

    Abs(Int(Val(37.3) * 100))=?

    Deal Experts,

    I am stuck in a small questions. I just wanted to know about the result of the below function

    Code:
    Public Sub test()    
        MsgBox Abs(Int(Val(37.3) * 100))
    End Sub
    After running this, result is 3729 but this is not corrrect. It should be 3730.

    Can anyone help me
    Do Good. Be Good. The World is yours.

  2. #2
    Discovering Life Siddharth Rout's Avatar
    Join Date
    Feb 2005
    Location
    Mumbai, India
    Posts
    12,001

    Re: Abs(Int(Val(37.3) * 100))=?

    Try now

    vb Code:
    1. Sub Test()
    2.     MsgBox Abs(Val(37.3) * 100)
    3. End Sub

    or

    vb Code:
    1. Sub aaa()
    2.     MsgBox Abs(37.3 * 100)
    3. End Sub

    or

    vb Code:
    1. Sub aaa()
    2.      MsgBox Abs(CInt(Val(37.3) * 100))
    3. End Sub
    A good exercise for the Heart is to bend down and help another up...
    Please Mark your Thread "Resolved", if the query is solved


    MyGear:
    ★ CPU ★ Ryzen 5 5800X
    ★ GPU ★ NVIDIA GeForce RTX 3080 TI Founder Edition
    ★ RAM ★ G. Skill Trident Z RGB 32GB 3600MHz
    ★ MB ★ ASUS TUF GAMING X570 (WI-FI) ATX Gaming
    ★ Storage ★ SSD SB-ROCKET-1TB + SEAGATE 2TB Barracuda IHD
    ★ Cooling ★ NOCTUA NH-D15 CHROMAX BLACK 140mm + 10 of Noctua NF-F12 PWM
    ★ PSU ★ ANTEC HCG-1000-EXTREME 1000 Watt 80 Plus Gold Fully Modular PSU
    ★ Case ★ LIAN LI PC-O11 DYNAMIC XL ROG (BLACK) (G99.O11DXL-X)
    ★ Monitor ★ LG Ultragear 27" 240Hz Gaming Monitor
    ★ Keyboard ★ TVS Electronics Gold Keyboard
    ★ Mouse ★ Logitech G502 Hero

  3. #3

    Thread Starter
    Addicted Member
    Join Date
    Aug 2007
    Location
    India
    Posts
    141

    Re: Abs(Int(Val(37.3) * 100))=?

    Thanks for the solution.

    but my intention is to know the reason why its returing 3729?

    Is this a bug?
    Do Good. Be Good. The World is yours.

  4. #4
    Discovering Life Siddharth Rout's Avatar
    Join Date
    Feb 2005
    Location
    Mumbai, India
    Posts
    12,001

    Re: Abs(Int(Val(37.3) * 100))=?

    You're probably familiar with the Int() function, which returns the integer portion of a value. Similarly, the CInt() function does much the same thing. The main difference between the two functions is the data type of the value it returns:

    - Int() returns the same data type as the value it's passed. - CInt() always returns an Integer data type.

    In addition, CInt rounds the value given to it, while Int truncates the decimal portion.
    A good exercise for the Heart is to bend down and help another up...
    Please Mark your Thread "Resolved", if the query is solved


    MyGear:
    ★ CPU ★ Ryzen 5 5800X
    ★ GPU ★ NVIDIA GeForce RTX 3080 TI Founder Edition
    ★ RAM ★ G. Skill Trident Z RGB 32GB 3600MHz
    ★ MB ★ ASUS TUF GAMING X570 (WI-FI) ATX Gaming
    ★ Storage ★ SSD SB-ROCKET-1TB + SEAGATE 2TB Barracuda IHD
    ★ Cooling ★ NOCTUA NH-D15 CHROMAX BLACK 140mm + 10 of Noctua NF-F12 PWM
    ★ PSU ★ ANTEC HCG-1000-EXTREME 1000 Watt 80 Plus Gold Fully Modular PSU
    ★ Case ★ LIAN LI PC-O11 DYNAMIC XL ROG (BLACK) (G99.O11DXL-X)
    ★ Monitor ★ LG Ultragear 27" 240Hz Gaming Monitor
    ★ Keyboard ★ TVS Electronics Gold Keyboard
    ★ Mouse ★ Logitech G502 Hero

  5. #5

    Thread Starter
    Addicted Member
    Join Date
    Aug 2007
    Location
    India
    Posts
    141

    Re: Abs(Int(Val(37.3) * 100))=?

    - Int() returns the same data type as the value it's passed. - CInt() always returns an Integer data type.

    Then it should return like this

    Code:
    ABS(INT(VAL(37.3)*100))
    
    STEP 1 :  VAL(37.3)*100)=3730
    STEP 2 :  INT(3730)
    STEP 3 :  ABS(3730)
    STEP 4 :  3730
    but its returning 3729. its funny Please try it and then suggest.
    Do Good. Be Good. The World is yours.

  6. #6
    Cumbrian Milk's Avatar
    Join Date
    Jan 2007
    Location
    0xDEADBEEF
    Posts
    2,448

    Re: Abs(Int(Val(37.3) * 100))=?

    Quote Originally Posted by vksingh24
    but my intention is to know the reason why its returning 3729?
    Just to explain a little more what you are seeing...
    Floating point numbers store the value in a very different way to which the value is displayed.
    For example 11 is stored accurately as 2^3*(1+(1/2^2)+(1/2^3)). The red bits show what is stored
    37.3 cannot be stored accurately in the same way, 2^5*(1+(0.165625)) where the 0.165625 portion cannot be accurately represented in a binary fraction form.
    The nearest a single can get is 2^5*(1+(1/2^3)+(1/2^5)+(1/2^7)+(1/2^10)+(1/2^11)+(1/2^14)+(1/2^15)+(1/2^18)+(1/2^19)+(1/2^22)+(1/2^23)) ~ 37.2999992
    A Double gets closer but it still cannot store it perfectly.

    More info at Wikipedia

    Edit: if you used the Currency or Decimal data types you would not see this phenomenon.
    Last edited by Milk; Nov 27th, 2008 at 07:39 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width