Results 1 to 4 of 4

Thread: Preserve decimal precision in Bitwise Operation

  1. #1

    Thread Starter
    Member
    Join Date
    Apr 2012
    Posts
    37

    Preserve decimal precision in Bitwise Operation

    I have a number that is a decimal , fractional number, less than 1000 in an array, EG: 518.25
    Adding the value 1024 "Flags" this number as unique among the rest. To get to the original number I "And" it with 1023.
    However, I lose the ".25" and get 518.

    Code is as follows:

    Dim N1 As Long
    N1 = 518.25
    MsgBox 1023& And (N1 + 1024&)

    Produces "518"
    Is there a way to retrieve the precision and get 518.25?
    Thanks

  2. #2
    Sinecure devotee
    Join Date
    Aug 2013
    Location
    Southern Tier NY
    Posts
    6,598

    Re: Preserve decimal precision in Bitwise Operation

    Dim N1 As Long
    N1 is an integer type, so can't hold fractions, so...
    N1 = 518.25

    N1 is now equal to 518, it rounded down, in this case.

    You could multiply the number by 100 when you assign it so you know it has 2 decimal points (essentially Fixed Point representation).
    You'll need to use a larger mask and flag bit so you don't interfere with your number, I would say (128 * 1024 - 1) so your mask would be 131071, and your flag would be 131072. (these things are usually easier in Hex, so your mask would be &H1FFFF and your flag &H20000).
    Divide by 100 after masking to recover your two decimal digits.
    Binary operations only work on integer type numbers.
    Last edited by passel; Jan 13th, 2015 at 01:17 PM.

  3. #3

    Thread Starter
    Member
    Join Date
    Apr 2012
    Posts
    37

    Re: Preserve decimal precision in Bitwise Operation

    Thanks passel,
    The following now works:
    Dim N1 as single 'Long wont work
    N1=518.25
    N1=N1 * 100
    MsgBox (&H1FFFF And (N1 + &H20000)) / 100

    Produces 518.25

    Thanks for the quick reply

  4. #4
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    19,541

    Re: Preserve decimal precision in Bitwise Operation

    If multiplying your number by 100 will make it exceed max Long value, another option is to remove & then add the decimal portion
    Code:
    X = (N1 - Int(N1)) + (Int(N1) And 1023)
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width