Page 1 of 2 12 LastLast
Results 1 to 40 of 44

Thread: [RESOLVED] Fix

  1. #1

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Resolved [RESOLVED] Fix

    can anyone explain this:

    fix(30 * 0.3) = 8
    fix(csng(30 * 0.3)) = 9

  2. #2
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,302

    Re: Fix

    Did you actually look to see what values were being passed to Fix in each case? Fix truncates a number to the nearest integer so clearly the first expression is less that 9 while the second is greater than or equal to 9. That's just the nature of floating-point numbers.

  3. #3
    PowerPoster
    Join Date
    Nov 2017
    Posts
    3,116

    Re: Fix

    See if this is enlightening:

    Msgbox(9 - 30*0.3).

  4. #4
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    19,541

    Re: Fix

    Per MSDN, Fix(number) is equivalent to Sgn(number) * Int(Abs(number))

    So, let's look at that last part of the evaluation: Int(Abs(Number)) and remove the Abs() portion...

    Int(30 * 0.3) = 8 where Int() is testing a Double
    Int(30! * 0.3!) = 9 where Int() is testing a Single
    However, 30*.3 = 9 & Int(9#) = Int(9!)

    Just more head-scratching where doubles are concerned
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

  5. #5

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: Fix

    even so, its still inconsistent.

    10 - 30 * 0.3 = 1
    while
    9 - 30 * 0.3 = 0,???...

    I get it that fix truncates. but the math is not consistent. if 10-x=1 why is 9-x=not 0? or if 9-x=not 0, shouldn't 10-x be not 1?

    edit:
    thx LaVolpe
    so that means, if I write Fix(30! * 0.3!) it will give me 9

    in my operation I have: (byte is a byte data type 0-255), now byte = 30

    Fix(byte * 0.3) = 8

    but if I do

    Fix(byte * 0.3!) = 9

    this solved my problems!
    Last edited by baka; May 7th, 2020 at 11:40 AM.

  6. #6
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    19,541

    Re: Fix

    Solved your problem, but I didn't realize Int() would be affected in this way. Just makes one wonder about one's usage of Int() in their programs now? Rhetorical

    baka, in your case, that should work because a Byte*Single has result as Single not double. VB auto-interprets untyped numbers using fractions as doubles
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

  7. #7

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: Fix

    I learned that I need to specify the datatype for all the numbers I use in my different formulas/algorithms.
    before I could write Fix(l / 2) and that worked well, but now I will include the datatype, so Fix(l / 2&) (l=long)
    and I will avoid using doubles. single is good enough precision!

  8. #8
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    19,541

    Re: [RESOLVED] Fix

    Just a matter of curiosity. I can replicate 8 being returned when using doubles, but it's wild

    30 * 0.3 = 0.3 / (1/30), so...

    cdbl(csng(1/30)) = 3.33333350718021E-02
    cdbl(.3!) = 0.300000011920929
    0.300000011920929 / 3.33333350718021E-02 = 8.99999988824131
    And Int(8.99999988824131) = 8
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

  9. #9
    PowerPoster
    Join Date
    Feb 2017
    Posts
    4,997

    Re: [RESOLVED] Fix

    BTW, since the subject seems to be already cleared and resolved, an OT:

    did you know that 0.9999(period 9) = 1?

  10. #10

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    also very strange that:
    .3! = 0.300000011920929

    also
    CDbl(30# * 0.3) = 9
    CDbl(1# / 30#) = 0,33333333333333

    and..
    30 * 0.3 is actually 30# * 0.3#, or 0.3# / (1#/30#) that should give us:
    0.3# / 0,3333333333333333333333333333
    and that result in 0,9000000000000000....9

    but it seems that it will do a conversion from any datatype to double and creates those extras, as we see in .3! = 0.300000011920929
    and that for me is flawed

  11. #11
    Sinecure devotee
    Join Date
    Aug 2013
    Location
    Southern Tier NY
    Posts
    6,582

    Re: [RESOLVED] Fix

    Doubles are binary numbers, so based on Base 2.

    Your thinking is based on Base 10.

    There are numbers that can't be represented in Base 10 with a limited number of digits, i.e. 1/3 for instance.
    But of course in Base 3, that would be a single digit.

    Since the number of bits available in any binary floating point number can only approximate the value if it is not a Base 2 multiple, just as in base 10 if you limit yourself to 20 digits, then 1/3 will be "way off", missing an infinite number of 3's in the lower part of the number.
    So, 1/3 represented as a decimal number of 20 digits and added three times would never equal 1, it would equal .999... for 20 digits.

    Since a Double has 54 or so binary digits (bits) available for the mantissa, that is where the value will be cut off when it is approximating a decimal number. If the decimal number is a sum of Base 2 exponent values, and fit in the bits available it will be exact.

    But on the other hand, an example like 0.10 can be represented exactly in Base 10, but is an infinite series of binary digits in binary, so is cut off at 54 bits.
    In your first case, since you have a Double value with 54 bits in the mantissa, and you convert it to a Single, with only 24-bits (including the implied msb), the 25th bit of your double value must have been a 1, so that rounded up when you converted to a Single, so rather than being slightly below 9 (by about 2^-53), it ended up being slightly above, by about (2^-24).

    If you really need 0.1 to be 0.1 exactly, then they have a type for that, i.e the Decimal type. Floating point will always have to choose a value that is close to 0.1, i.e. within about 2^-24 for Single and within 2^-53 or so for Double.
    Last edited by passel; May 7th, 2020 at 10:01 PM.
    "Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

  12. #12

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    no matter how it stores in memory,
    when data type conversion between singles and doubles makes .3! = 0.300000011920929 its still bad and it will affect any calculations.

    0.3! should be 0.3# or 0.300000000000000# and not 0.300000011920929#

  13. #13
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,302

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    no matter how it stores in memory,
    when data type conversion between singles and doubles makes .3! = 0.300000011920929 its still bad and it will affect any calculations.

    0.3! should be 0.3# or 0.300000000000000# and not 0.300000011920929#
    Then you had better invent a new type of computer that can do that. You are pretty much ignoring the concept of significant figures. Computers that behave this way are used all over the world in science, engineering and many other fields without issue. One wonders why this is an issue for you and not anyone else. I'm sure that everyone else is wrong.

  14. #14
    Sinecure devotee
    Join Date
    Aug 2013
    Location
    Southern Tier NY
    Posts
    6,582

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    no matter how it stores in memory,
    when data type conversion between singles and doubles makes .3! = 0.300000011920929 its still bad and it will affect any calculations.

    0.3! should be 0.3# or 0.300000000000000# and not 0.300000011920929#
    Well, as I said 0.3 is not a base 2 number so would be an endless stream of bits.
    When you set a Single to that value, i.e. 0.3!, then you are cutting off the value at 24-bits of significance. (2^-23 shifted for 0.3)

    At that point it is a number that is close to 0.3, but not 0.3.
    Also at that point, the computer has no idea that the number is suppose to be 0.3 because it has been converted to a number that is as close to 0.3 as possible with the given floating point representation. So, the value is a value that is near 0.3 and that is that.

    If you have the computer convert the single to a string, then it may produce a string 0.3, because it may round the number for its string representation, so it is rounding 0.300000011920929 to 0.3000000 by rounding the last few bits to the closest decimal value.

    That 0.000000011920929 represents the value of the smallest bit (2^-23) shifted , when representing .3 in a 32-bit float.
    Microsoft's Win10 calculator in Scientific mode says the value is actually 0.300000011920928955078125.

    So, .3 as a single is .3 with an error of being 0.000000011920928955078125 above that actual value.

    That is the value. That is as close as it can get.

    If you then ask the compiler to convert that single (whose true value if the string conversion would print it is .300000011920928955078125) to a double, which has a lot more bits to work with, then it won't throw away the .0000000119209... value in the single (i.e. arbitrarily throw away or set the last bit of the Single value to 0) when converting to a double.
    When converting from a Single to a double you want to keep every bit of the mantissa that is available, i.e. 24-bits (including the implied bit). All the conversion is doing when going from Single to double, as for the significance, is adding a bunch of 0's to the end of the number.
    Since it doesn't know the value represents .3 it can't set the last bit to 0 and add all the additional 0's and 1's to the bits beyond the 24 bits given to make the value closer to .3. How is it to know that .3 is what you want at that point?

    Bottom line .3! and .3# are two different numbers, both of which are close to .3, but neither of them are .3. The second is closer than the first obviously because it has a lot more bits to represent the number so the error will be around a billion times smaller, but it won't be .3.

    When you convert a single that was set to .3 to a double, then you get to see what the majority of the error was (limited by the precision of a double, and subject to the string conversion to decimal representation) between the value .3 and the Single's approximation of .3.

    p.s. Just for extra info,
    Just like 1/3 end up being an endless .3333333...
    .3 decimal in binary just so happens to coincidentally end up being an endless series of 0011001100110011

    The single, for its 24 bits of mantissa ends up cutting the series off between two 1's, i.e. ...00110011001<cut>1001100110011, so, the value is rounded up since the value below the cut is greater than .5 (relatively).
    In Hex, the Single value would be 3E99 9999 9999 9999 9999 ... if allowed to exceed 32-bits
    But, since the number has to be cut off at 3E99 9999 to fit in a 32-bit single, the number becomes 3E99 999A,
    i.e
    3E99 9999 9999 9999... is the number you'd like.
    3E99 9999 0000 0000... is choice 1, truncate to 32 bits
    3E99 999A 0000 0000... is choice 2, round to 32 bits
    Which is closer to the number you want, choice 1 or choice 2. The computer chooses choice 2, of course.

    For a double you have the same repeating binary pattern for the mantissa of course, alternating pairs of ones and zeros, 110011001100...

    But because of the alignment difference caused by the difference in size of the exponent part of the floating point number, the hex value are different, but the bit pattern that makes up the digits is the same, just shifted by two bits.
    3FD3 3333 3333 3333
    Of course the real value for .3 would continue indefinitely with 333333... in the double representation, if extended.

    So, you have a Single value whose value is 3E99 999A and when you convert it to a double, it would have to be extended to 64 bits, so that would be done by adding 0's to the end, there is no other way to convert a 32-bit value to a 64-bit value and add extra significance to the original value.

    So, the first step, assuming the exponent field didn't change in size would be.
    3E99 999A 0000 0000 which gives you a 64-bit Single. But we don't have 64-bit singles, so the mantissa has to shift to the right to make room for the expanded Double exponent, so you end up with
    3FD3 3333 4000 0000.
    3FD3 3333 3333 3333 would be the result if you converted .3 into a double directly, but since you first put it into a single, you lost a lot of bits during that conversion, and then converting that values with all the lost bits, can't magically reproduce the missing bits from nowhere.
    Last edited by passel; May 8th, 2020 at 02:27 AM.
    "Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

  15. #15

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    I get all that but the issue is still the fix function converting everything to double.

    fix(30 * 0.3) = 8
    fix(30 * 0.30000000000001) = 9
    fix(30 * 0.300000011920929) = 9
    fix(30 * 0.299999999999999) = 8
    fix(30! * 0.3!) = 9

    I can't even write 0.30000000000000#
    this means I need to "think" in doubles or always assign the datatype.
    no matter computers or whatever, or reason, it doesnt matter.
    is like saying humans will always rape, its part of their behavior but doesn't make it right.

  16. #16
    PowerPoster
    Join Date
    Feb 2017
    Posts
    4,997

    Re: [RESOLVED] Fix

    An alternative is to use Round instead of Fix, but I don't know what you need to do.

  17. #17
    Sinecure devotee
    Join Date
    Aug 2013
    Location
    Southern Tier NY
    Posts
    6,582

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    I get all that but the issue is still the fix function converting everything to double.

    fix(30 * 0.3) = 8
    fix(30 * 0.30000000000001) = 9
    fix(30 * 0.300000011920929) = 9
    fix(30 * 0.299999999999999) = 8
    fix(30! * 0.3!) = 9

    I can't even write 0.30000000000000#
    this means I need to "think" in doubles or always assign the datatype.
    no matter computers or whatever, or reason, it doesnt matter.
    is like saying humans will always rape, its part of their behavior but doesn't make it right.
    I'm not sure what you're saying there.
    This is not a Single vs Double issue, and you're only talking about a test case of one value, i.e. 0.3 .

    The first four examples you are passing literal values, so those will all be Doubles and that has nothing to do with the Fix function itself.
    That last example is multiplying two Singles, so a Single will be passed to fix.

    I guess you're saying that the correct answer is the ones that return 9?
    That fact that 30! * 0.3! happens to be slightly larger than 9, so gets truncated to 9 is just the particular case, and isn't really a Single vs Double issue in the general case.

    It is only because the .3 value happens to end with the bit 33 of the value being a 1 with the exponent size of a float 32 representation, so the 32-bit floating point value is rounded up so is slightly above 0.3. whereas the 65th bit of the value with the float 64 exponent is 0, so the value doesn't round up so the floating point value is slightly less than 0.3 .

    But it isn't guaranteed that the 32-bit float will always round up. I guess statistically, it should round up around 50% of the time, so 50% of the time, the Single's value will be slightly less than the decimal value it is set to (assuming not a sum of powers of 2), and the Fix function would return the -1 value just like the double did in this case. It really isn't a double versus Single thing, in the general case.

    Using any floating point numbers (Single or Double) with values after the decimal point, the actual value will be slightly high or slightly low, probably in equal measure.

    You could always wrap the Fix function with your own version that adds a slight bias to the value to have it always come out on the high side, if the result is close to being an Integer value, if that is what you prefer Fix to do.
    Last edited by passel; May 8th, 2020 at 09:59 AM.
    "Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

  18. #18

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    maybe not, but Single at least make it right, at least as close possible.
    while doubles is all wrong.

    30 * 0.3 = 9 and nothing else, if its wrong because how computers works, it is still wrong,
    I didnt input 0.299995235923523 or 0.30000124120410241240124012 but 0.3 and for me 0.3 is 0.3 and not something else.
    no matter how it works, or the limitations or whatever. for me 0.3 is 0.3 and thats it.

    if you think 30 * 0.3 = 8 is correct, Im not sure what to say.

    also, I use Fix, because its not always 30, it can be any number.

  19. #19
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: [RESOLVED] Fix

    And what's 1/3 or 22/7 or Pi?
    There are more numbers which can not be represented in either base 10 or 2 then numbers which can exactly be represented.
    And you are ignoring the fact that there are only 32 (or 64) bits to store values.

    Otherwise you need to switch to integer calculations only.

  20. #20
    Sinecure devotee
    Join Date
    Aug 2013
    Location
    Southern Tier NY
    Posts
    6,582

    Re: [RESOLVED] Fix

    I'm saying that being a Single and being .3 works for .3, in this situation.

    If you were using .2 or .4 or .6 or .7, then it may not work. Since the case where it works and the case where it doesn't work should be about 50/50, the fact that it "works" for .3 is just because .3 when converted to a Single is slightly above .3.

    If your use case only requires you to use .3 (and because it is a Single it is really .300000011920928955078125), then it will work for you, so is a "fix".
    But, technically you are only fooling yourself if you thing you are using .3, because you aren't. It is impossible to use .3 when you're using a floating point type.

    When you set a double to .3, I'm not sure what it is offhand, but I assume it is in the neighborhood of .29999999999998xxx
    Last edited by passel; May 8th, 2020 at 10:29 AM.
    "Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

  21. #21

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    again, I get the computer part, but Im not talking about that and actually I dont care.
    for me the result is the importance. if I input 0.3 I dont want or care if the computer is changing it 0.30001 or 0.29998 what I want is the math to be as "accurate" possible.

    also, I dont get why u "defend" it so much, its like you "believe" in the limitation and incorrectness of how floating numbers works.
    I will always have a critical mind and even if I know the limitations I dont need to like it or defend it.

    0.3 is 0.3 and thats final. you can not change my mind saying its 0.300000011920928955078125, because in my mind 0.3 is equal 0.3.
    what you believe and understand about all of this, is not doing anything for me, and actually you are not helping at all.
    sure you explain how it works inside the computer, how the cpu is handling it, bit-wise.
    but Im not a bit-created entity, Im a human, and in my Mind, Im not using floating points.

  22. #22
    PowerPoster PlausiblyDamp's Avatar
    Join Date
    Dec 2016
    Location
    Pontypool, Wales
    Posts
    2,458

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    again, I get the computer part, but Im not talking about that and actually I dont care.
    for me the result is the importance. if I input 0.3 I dont want or care if the computer is changing it 0.30001 or 0.29998 what I want is the math to be as "accurate" possible.
    Unless you can invent an new form of number storage for computers then it is as accurate as possible using current technology. The fact it isn't as accurate for some numbers as we would like is frustrating, but it is the way these things work.

    Quote Originally Posted by baka View Post
    also, I dont get why u "defend" it so much, its like you "believe" in the limitation and incorrectness of how floating numbers works.
    I will always have a critical mind and even if I know the limitations I dont need to like it or defend it.
    This isn't people "beleiving" in this limitation, the limitation is a fact of how computers work. Liking or defending it doesn't come into it - this is how it works and you need to take that into consideration when writing code, the same as every other person alive using modern computer architectures

    Quote Originally Posted by baka View Post
    0.3 is 0.3 and thats final. you can not change my mind saying its 0.300000011920928955078125, because in my mind 0.3 is equal 0.3.
    what you believe and understand about all of this, is not doing anything for me, and actually you are not helping at all.
    sure you explain how it works inside the computer, how the cpu is handling it, bit-wise.
    but Im not a bit-created entity, Im a human, and in my Mind, Im not using floating points.
    It doesn't matter how your mind works or what numerical base you are working with, computers are using base 2 for numbers and there is no way to store 0.3 exactly as 0.3 in base 2. If you are using a computer that works on base 2, and you are using one that does, then you need to understand the limitations of base 2 and any numbers that are stored using it.


    https://en.wikipedia.org/wiki/IEEE_754 is a pretty decent reference if you want to read more about the topic.
    Last edited by PlausiblyDamp; May 8th, 2020 at 11:24 AM.

  23. #23
    PowerPoster
    Join Date
    Nov 2017
    Posts
    3,116

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    again, I get the computer part, but Im not talking about that and actually I dont care.
    for me the result is the importance. if I input 0.3 I dont want or care if the computer is changing it 0.30001 or 0.29998 what I want is the math to be as "accurate" possible.

    also, I dont get why u "defend" it so much, its like you "believe" in the limitation and incorrectness of how floating numbers works.
    I will always have a critical mind and even if I know the limitations I dont need to like it or defend it.

    0.3 is 0.3 and thats final. you can not change my mind saying its 0.300000011920928955078125, because in my mind 0.3 is equal 0.3.
    what you believe and understand about all of this, is not doing anything for me, and actually you are not helping at all.
    sure you explain how it works inside the computer, how the cpu is handling it, bit-wise.
    but Im not a bit-created entity, Im a human, and in my Mind, Im not using floating points.
    Ok, so your initial question:

    Quote Originally Posted by baka View Post
    can anyone explain this:

    fix(30 * 0.3) = 8
    fix(csng(30 * 0.3)) = 9
    That was code you were running through your brain and wondering why you were getting those results???

    Oh yeah, no. You were running that code on a computer. And its been explained to you over and over why the computer is producing those results.

    No one is saying 30*0.3 = 8. No one. Get over it.

  24. #24

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    I agree PlausiblyDamp, I understand the base 2 limitations. but as I explained, for me that doesnt matter, its only the result that matters, and if 30! * 0.3! helped me, thanx to LaVolpe explanations, thats the solution, even if its not perfect, its close enough. 30 * 0.3 gave the "wrong" expected result, and that for me is wrong, no matter why. and I dont care the specifics, it will not help me, no matter how much you explain about it. I need a "fix", and LaVolpe gave me that, and Im happy for that.

    -----
    well, why do u keep commenting. I got the answer from LaVolpe and I changed to "resolved".
    but u keep posting and arguing and trying to be smart, for what? to win a competition? did I ask for you answers?
    I got mine and Im done, but you try to tell me how I need to think. I already answered, I get it what u say, but my own perspective is different, no matter right or wrong. so try to get it AND get over it.
    Last edited by baka; May 8th, 2020 at 01:30 PM.

  25. #25
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    I need a "fix", and LaVolpe gave me that, and Im happy for that.
    But now you're going to apply it wrongly (in your App-Code).

    When you write (in your post #7):
    "...before I could write Fix(l / 2) and that worked well,
    but now I will include the datatype, so Fix(l / 2&) (l=long)"


    Then this is clearly a mis-use of the Fix-Function (which does 2 additional Runtime-calls under the covers).

    Better and faster would be:
    SomeResult = SomeLong \ 2 '<- using the Integer-Div-Operator

    And all this happened, because nobody asked you at the beginning:
    - "Why do you have to use the Fix()-function?"
    - "Can you give some more background, what you really want to do?"

    And in case these questions would have been properly answered by you
    (without you making any fuss of "hijacking my thread"),
    then you'd have been supplied with better matching answers.

    (Like e.g. CLng(30 * 0.3) or CCur(30 * 0.3) or CDec(30 * 0.3)...)

    In the state this thread currently is, some other inexperienced users who stumble over it,
    might think that: Fix(l / 2&) (l=Long) is a good idea... (which it is not).

    Olaf

  26. #26
    PowerPoster
    Join Date
    Feb 2017
    Posts
    4,997

    Re: [RESOLVED] Fix

    Quote Originally Posted by Schmidt View Post
    And all this happened, because nobody asked you at the beginning:
    - "Why do you have to use the Fix()-function?"
    - "Can you give some more background, what you really want to do?"
    That was more or less my idea, when I posted:

    Quote Originally Posted by Eduardo- View Post
    An alternative is to use Round instead of Fix, but I don't know what you need to do.

  27. #27
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: [RESOLVED] Fix

    Quote Originally Posted by Eduardo- View Post
    That was more or less my idea, when I posted:
    An alternative is to use Round instead of Fix, but I don't know what you need to do.
    Sorry, overlooked that ... just wanted to make a statement against the current "trend",
    that "original posters" these days somehow feel "entitled" (or at least "encouraged"),
    to make demands about what kind of answers they deem worthy in their "plea for help".

    "It's my thread"... seems to over-rule the common sense (what public-forums are for) by now.

    Olaf

  28. #28
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    9,853

    Re: [RESOLVED] Fix

    WOW, this got a lot of discussion, and I admittedly didn't read every word.

    However, in my mind, it all comes down to the fact that 0.3 doesn't have an exact representation in either IEEE Single or IEEE Double.

    Therefore, it comes down to whether the 0.3 binary representation is going to be ever so slightly larger than 0.3 or ever so slightly smaller than 0.3.

    Without even testing, I feel certain that an IEEE Single is going to be slightly smaller (resulting in a product slightly smaller than 10), and an IEEE Double is going to be slightly larger (resulting in a product slightly larger than 10).

    These decimal-to-binary (and vice-versa) conversion anomalies are well known.

    EDIT: Ok, I got that backwards. Single is slightly LARGER than 0.3 and Double is slightly SMALLER than 0.3.

    After putting a 0.3 in a Single, and then converting back to decimal (and calculating all the way out until we get an exact decimal representation), the representation is:
    0.300000011920928955078125

    And, after putting 0.3 in a Double, and then converting back to decimal (and calculating all the way out until we get an exact decimal representation), the representation is:
    0.299999999999999988897769753748434595763683319091796875

    So, we can see that 0.3 in a Single is slightly larger, and 0.3 in a Double is slightly smaller. Hence: Fix(30! * 0.3!) = 9, and Fix(30# * 0.3#) = 8

    EDIT2: Here's a webpage that will help with figuring that out. Request both Single and Double for best illustration.
    Last edited by Elroy; May 8th, 2020 at 03:04 PM.
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  29. #29
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: [RESOLVED] Fix

    again, I get the computer part, but Im not talking about that and actually I dont care.
    for me the result is the importance. if I input 0.3 I dont want or care if the computer is changing it 0.30001 or 0.29998 what I want is the math to be as "accurate" possible.

    also, I dont get why u "defend" it so much, its like you "believe" in the limitation and incorrectness of how floating numbers works.
    I will always have a critical mind and even if I know the limitations I dont need to like it or defend it.
    You sound like a normal end user instead of a programmer.
    As a programmer you need to learn about how cpu’s work and how programming languages behave.
    Then as a programmer you have to deal with it and either create solutions or accept the current behavior is.

  30. #30

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    Olaf,

    your comment to use \ instead of / is valid, and I use it, but sometimes I forget, its a good reminder and I appreciate that.
    I appreciate members like LaVolpe, Elroy, Eduardo-, passel and PlausiblyDamp that are explaining, giving examples and even trying to understand my view. even if they would think Im wrong from their p.o.v.

    what I dont appreciate are the others, they believe they are better and with condescending attitude. I can be without that. so please just stop and "help" someone else.
    Last edited by baka; May 8th, 2020 at 11:22 PM.

  31. #31
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    9,853

    Re: [RESOLVED] Fix

    Another alternative to get what you're after is to use the Decimal type:

    Fix(CDec(30!) * CDec(0.3!)) = 9

    Even...

    Fix(CDec(30#) * CDec(0.3#)) = 9

    It's too bad that VB6 doesn't provide a way to specify Decimal literal constants.

    But, one thing it does do is some strange rounding when we convert types. For instance, the Decimal type has precisely 28 "decimal digits" of precision. So, when doing CDec(0.3#), we might wonder why it doesn't put 0.299999999999999988897769754 in the Decimal type. However, it doesn't. It puts 0.300000000000000000000000000 in the Decimal type. And then, in the above multiplication, it does it correctly, multiplying 30 by 0.3, resulting in precisely 9. And then, the Fix essentially does nothing.

    So, even if we start with Singles or Doubles, if we convert them to Decimals before we do our math, things will work out as we'd expect (intuitively, without worrying about binary-to-decimal and decimal-to-binary conversion).

    Note, in the above, I capitalize Decimal when I'm talking about the Decimal type ... and I leave decimal lower case when I'm just talking about base-10 numbers.

    -----------

    EDIT: Just to add a bit more, the Decimal type essentially stores numbers as base-10 numbers. It's a bit of a hybrid in that it stores the mantissa as binary. But, the mantissa is just an integer, and integers are the same regardless of the base. What's most important is that the exponent is indicated as the decimal-point's position in a base-10 number. Therefore, Decimal types store base-10 numbers exactly as we think of them, and don't do any floating-point decimal-to-binary (and vice-versa) conversions, which is what caused the anomaly in the OP.
    Last edited by Elroy; May 8th, 2020 at 06:48 PM.
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  32. #32
    Sinecure devotee
    Join Date
    Aug 2013
    Location
    Southern Tier NY
    Posts
    6,582

    Re: [RESOLVED] Fix

    Quote Originally Posted by Elroy View Post
    ...
    Without even testing, I feel certain that an IEEE Single is going to be slightly smaller (resulting in a product slightly smaller than 10), and an IEEE Double is going to be slightly larger (resulting in a product slightly larger than 10).
    ...
    EDIT: Ok, I got that backwards. Single is slightly LARGER than 0.3 and Double is slightly SMALLER than 0.3.
    ...
    This is just a FYI.
    The above was one of my points. Whether the Floating Point number is slightly larger or slightly smaller than the desired value (or exact in the case of sums of powers of 2), is not depended on the type, but dependent on the bit pattern of the number and how it rounds so is not easily predicted just by looking at the number.

    So, while 0.5 is exact (a power of 2), the following table shows that larger or smaller isn't depended on Single vs Double representation.
    Code:
    Value :  Single   Double
    -------------------------
    0.1      High     High
    0.2      High     High
    0.3      High     Low
    0.4      High     High
    0.5      Exact    Exact
    0.6      High     Low
    0.7      Low      Low
    0.8      High     High
    0.9      Low      High
    So, if for some reason your code had to use .7 or .9 percent in its calculation instead of .3, the current "fix" would fail.
    Also, if you had happened to be using any of the values that are "High" for the Double in the table above instead of .3, you wouldn't have even realized you had a problem.

    Being aware of these realities is not a bad thing to know for future work.
    "Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

  33. #33
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,121

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    0.3 is 0.3 and thats final. you can not change my mind saying its 0.300000011920928955078125, because in my mind 0.3 is equal 0.3.
    As passel already told before, you have to work with Decimal datatype for this to be true.

    It is as simple as this
    Code:
    Option Explicit
    
    Private Sub Form_Load()
        Dim aThird As Variant
        
        aThird = CDec(3) / 10
        Debug.Print 30 * aThird
        Debug.Print 25 * aThird
        Debug.Print 20 * aThird
        Debug.Print 15 * aThird
        Debug.Print 10 * aThird
    End Sub
    There are no "hairs" left over from the calculations and Fix will work as expected.

    Decimal datatype is like working with Long and Integers but shifted to the right of the floating-point, so your 0.3 is represented like 3 * 10^-1 in memory and 0.123 is stored as 123 * 10^-3.

    This way no "wrong" calculations can happen but the numbers are actually *less* precise than Double/Single and I leave the explanation why floating-point numbers are more precise than fixed-point ones to someone else :-))

    cheers,
    </wqw>

  34. #34
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    9,853

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    0.3 is 0.3 and thats final.
    Yeah, I've got to chime in as well, and say that that's just NOT true. Many of us are so steeped in thinking in base-10, that it's difficult for us to "think" in any other base (such as base-2). And, just to keep it short, the take-home message is: What can be perfectly represented with floating-point representation in one base (say, base-10) can't be perfectly represented with floating-point in all other bases (such as base-2, for example).

    And both IEEE Singles and IEEE Doubles represent both their mantissa and exponent as base-2. And as passel first pointed out, and I mentioned as well, only the Decimal-type (within VB6) perfectly represents floating point base-10 numbers.

    This is all an extremely well known problem with a great deal of documentation on it. And it's been a problem for many decades.

    I remember a language back in the 80s called C-Basic. It had a number representation called binary-coded-decimal (BCD). (This was also done in COBOL.) It was a bit more wasteful than the VB6 Decimal-type, but it took each nibble (four-bits) and coded a 0 thru 9 into it. This is only 10 values, and a nibble can support up to 16 values, and this is why it was a bit wasteful. However, with the addition of an exponent portion (i.e., where to put the decimal point), this BCD number encoding method could perfectly represent base-10 numbers (as does the Decimal-type in VB6).

    BCD was intuitively desirable. As opposed to the Decimal-type, you could look at a BCD number in memory and rather easily figure out exactly what the number was. Here's the mantissa encoding of BCD:

    Name:  bcd.png
Views: 204
Size:  7.3 KB

    So, again, to say that "0.3 is 0.3 and that's final" just doesn't appreciate the finer points of how computers store floating-point numbers and how they do math with them.

    That brings up one final point ... for the last 30 years (or maybe a bit longer), we've been using floating-point-processors (FPPs) to do most of our floating-point math. And that's true of VB6 as well when dealing with Singles and Doubles. It's just MUCH faster than doing this math with software. And, these FPPs do math as base-2, not base-10. So, for those who've been at it a while, we understand that there are occasionally base-2-to-base-10 (and vice-versa) conversion issues.

    And, if we wish to completely circumvent these issues, we must use an encoding that doesn't use base-2, such as VB6's Decimal-type.
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  35. #35

  36. #36

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    yeah, as I have explained numerous times, I get the computer base-2 part, but in my mind 0.3 is equal 0.3 and nothing else.
    so, depending on the perspective.
    I prioritize my mind, not the computer. so, in my mind 0.3 is 0.3.

    what to do to make it as close possible to what I want to achieve?
    and in this case,
    a number between 1 to 30 multiply with 0.3, using fix since I don't want any rounding.
    LaVolpe helped me with the "single" usage, that "solved" my problem.
    BUT of course it will not solve ALL problems. and I get that.

    next time maybe I need to do something like:
    fix(10 * 0.7) = 6
    fix(10 * 0.7!) = 6
    fix(10 * 0.7000001!) = 7

    using 0.7000001! if I want to make it 7, and that is a "possible" solution to my problem, not a universal fix for everything.

    all of you that helped me understand the base-2, I appreciate it, but you failed to understand my p.o.v. since you want me to think like you,
    but Im not you, my mind works differently and that how I do my programming. but that doesnt mean I can't learn, but I learn my way.
    and I learned about the issues base-2 can make when working in a base-10 plane. and I appreciate it, but I don't appreciate when you tell me how I need to think. thats my problem to handle.

  37. #37
    PowerPoster
    Join Date
    Nov 2017
    Posts
    3,116

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    yeah, as I have explained numerous times, I get the computer base-2 part, but in my mind 0.3 is equal 0.3 and nothing else.
    so, depending on the perspective.
    I prioritize my mind, not the computer. so, in my mind 0.3 is 0.3.

    what to do to make it as close possible to what I want to achieve?
    and in this case,
    a number between 1 to 30 multiply with 0.3, using fix since I don't want any rounding.
    LaVolpe helped me with the "single" usage, that "solved" my problem.
    BUT of course it will not solve ALL problems. and I get that.

    next time maybe I need to do something like:
    fix(10 * 0.7) = 6
    fix(10 * 0.7!) = 6
    fix(10 * 0.7000001!) = 7

    using 0.7000001! if I want to make it 7, and that is a "possible" solution to my problem, not a universal fix for everything.

    all of you that helped me understand the base-2, I appreciate it, but you failed to understand my p.o.v. since you want me to think like you,
    but Im not you, my mind works differently and that how I do my programming. but that doesnt mean I can't learn, but I learn my way.
    and I learned about the issues base-2 can make when working in a base-10 plane. and I appreciate it, but I don't appreciate when you tell me how I need to think. thats my problem to handle.
    This is a programming forum. You asked a computer programming related question, you got computer programming related answers. If that upsets you then perhaps that is something you need to address outside of this forum.

  38. #38
    PowerPoster
    Join Date
    Feb 2017
    Posts
    4,997

    Re: [RESOLVED] Fix

    Quote Originally Posted by baka View Post
    what to do to make it as close possible to what I want to achieve?
    and in this case,
    a number between 1 to 30 multiply with 0.3, using fix since I don't want any rounding.
    If you don't want any rounding then why are you using Fix?

    30 * 0.3 gives the correct result, unlike Fix(30 * 0.3):

    Code:
        Debug.Print 30 * 0.3
        Debug.Print Fix(30 * 0.3)
    The question of the whole thread: what do you want to achieve?

  39. #39

    Thread Starter
    The Idiot
    Join Date
    Dec 2014
    Posts
    2,721

    Re: [RESOLVED] Fix

    you are right, not all cases I need Fix, but its not always 30,
    the OP example I needed it to work for a number between 1 to 30, I dont want any rounding, as x.000 to x.999 need to be x, that's why I use Fix.
    without fix, 29 * 0.3 would give me 9 not 8 (as I wrote in #18 and #36)

  40. #40
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    19,541

    Re: [RESOLVED] Fix

    Well, if you do this in 2 statements, you should always get what you want. Not sure that's any better & doesn't address the "why" of the initial problem.
    Code:
    Dim vResult As Double ' or Single as desired
    vResult = 30 * 0.3: MsgBox Fix(vResult)
    However, others have posted workarounds too, i.e., Fix(CDec(30 * 0.3))
    At least, you have choices and understand the "why"
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width