
May 7th, 2020, 11:07 AM
#1
Thread Starter
Frenzied Member
[RESOLVED] Fix
can anyone explain this:
fix(30 * 0.3) = 8
fix(csng(30 * 0.3)) = 9

May 7th, 2020, 11:18 AM
#2
Re: Fix
Did you actually look to see what values were being passed to Fix in each case? Fix truncates a number to the nearest integer so clearly the first expression is less that 9 while the second is greater than or equal to 9. That's just the nature of floatingpoint numbers.

May 7th, 2020, 11:18 AM
#3
Re: Fix
See if this is enlightening:
Msgbox(9  30*0.3).

May 7th, 2020, 11:28 AM
#4
Re: Fix
Per MSDN, Fix(number) is equivalent to Sgn(number) * Int(Abs(number))
So, let's look at that last part of the evaluation: Int(Abs(Number)) and remove the Abs() portion...
Int(30 * 0.3) = 8 where Int() is testing a Double
Int(30! * 0.3!) = 9 where Int() is testing a Single
However, 30*.3 = 9 & Int(9#) = Int(9!)
Just more headscratching where doubles are concerned

May 7th, 2020, 11:35 AM
#5
Thread Starter
Frenzied Member
Re: Fix
even so, its still inconsistent.
10  30 * 0.3 = 1
while
9  30 * 0.3 = 0,???...
I get it that fix truncates. but the math is not consistent. if 10x=1 why is 9x=not 0? or if 9x=not 0, shouldn't 10x be not 1?
edit:
thx LaVolpe
so that means, if I write Fix(30! * 0.3!) it will give me 9
in my operation I have: (byte is a byte data type 0255), now byte = 30
Fix(byte * 0.3) = 8
but if I do
Fix(byte * 0.3!) = 9
this solved my problems!
Last edited by baka; May 7th, 2020 at 11:40 AM.

May 7th, 2020, 11:45 AM
#6
Re: Fix
Solved your problem, but I didn't realize Int() would be affected in this way. Just makes one wonder about one's usage of Int() in their programs now? Rhetorical
baka, in your case, that should work because a Byte*Single has result as Single not double. VB autointerprets untyped numbers using fractions as doubles

May 7th, 2020, 12:00 PM
#7
Thread Starter
Frenzied Member
Re: Fix
I learned that I need to specify the datatype for all the numbers I use in my different formulas/algorithms.
before I could write Fix(l / 2) and that worked well, but now I will include the datatype, so Fix(l / 2&) (l=long)
and I will avoid using doubles. single is good enough precision!

May 7th, 2020, 12:12 PM
#8
Re: [RESOLVED] Fix
Just a matter of curiosity. I can replicate 8 being returned when using doubles, but it's wild
30 * 0.3 = 0.3 / (1/30), so...
cdbl(csng(1/30)) = 3.33333350718021E02
cdbl(.3!) = 0.300000011920929
0.300000011920929 / 3.33333350718021E02 = 8.99999988824131
And Int(8.99999988824131) = 8

May 7th, 2020, 12:19 PM
#9
Re: [RESOLVED] Fix
BTW, since the subject seems to be already cleared and resolved, an OT:
did you know that 0.9999(period 9) = 1?

May 7th, 2020, 12:25 PM
#10
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
also very strange that:
.3! = 0.300000011920929
also
CDbl(30# * 0.3) = 9
CDbl(1# / 30#) = 0,33333333333333
and..
30 * 0.3 is actually 30# * 0.3#, or 0.3# / (1#/30#) that should give us:
0.3# / 0,3333333333333333333333333333
and that result in 0,9000000000000000....9
but it seems that it will do a conversion from any datatype to double and creates those extras, as we see in .3! = 0.300000011920929
and that for me is flawed

May 7th, 2020, 09:57 PM
#11
Re: [RESOLVED] Fix
Doubles are binary numbers, so based on Base 2.
Your thinking is based on Base 10.
There are numbers that can't be represented in Base 10 with a limited number of digits, i.e. 1/3 for instance.
But of course in Base 3, that would be a single digit.
Since the number of bits available in any binary floating point number can only approximate the value if it is not a Base 2 multiple, just as in base 10 if you limit yourself to 20 digits, then 1/3 will be "way off", missing an infinite number of 3's in the lower part of the number.
So, 1/3 represented as a decimal number of 20 digits and added three times would never equal 1, it would equal .999... for 20 digits.
Since a Double has 54 or so binary digits (bits) available for the mantissa, that is where the value will be cut off when it is approximating a decimal number. If the decimal number is a sum of Base 2 exponent values, and fit in the bits available it will be exact.
But on the other hand, an example like 0.10 can be represented exactly in Base 10, but is an infinite series of binary digits in binary, so is cut off at 54 bits.
In your first case, since you have a Double value with 54 bits in the mantissa, and you convert it to a Single, with only 24bits (including the implied msb), the 25th bit of your double value must have been a 1, so that rounded up when you converted to a Single, so rather than being slightly below 9 (by about 2^53), it ended up being slightly above, by about (2^24).
If you really need 0.1 to be 0.1 exactly, then they have a type for that, i.e the Decimal type. Floating point will always have to choose a value that is close to 0.1, i.e. within about 2^24 for Single and within 2^53 or so for Double.
Last edited by passel; May 7th, 2020 at 10:01 PM.
"Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

May 7th, 2020, 11:06 PM
#12
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
no matter how it stores in memory,
when data type conversion between singles and doubles makes .3! = 0.300000011920929 its still bad and it will affect any calculations.
0.3! should be 0.3# or 0.300000000000000# and not 0.300000011920929#

May 7th, 2020, 11:25 PM
#13
Re: [RESOLVED] Fix
Originally Posted by baka
no matter how it stores in memory,
when data type conversion between singles and doubles makes .3! = 0.300000011920929 its still bad and it will affect any calculations.
0.3! should be 0.3# or 0.300000000000000# and not 0.300000011920929#
Then you had better invent a new type of computer that can do that. You are pretty much ignoring the concept of significant figures. Computers that behave this way are used all over the world in science, engineering and many other fields without issue. One wonders why this is an issue for you and not anyone else. I'm sure that everyone else is wrong.
Last edited by jmcilhinney; May 8th, 2020 at 12:24 AM.

May 8th, 2020, 12:33 AM
#14
Re: [RESOLVED] Fix
Originally Posted by baka
no matter how it stores in memory,
when data type conversion between singles and doubles makes .3! = 0.300000011920929 its still bad and it will affect any calculations.
0.3! should be 0.3# or 0.300000000000000# and not 0.300000011920929#
Well, as I said 0.3 is not a base 2 number so would be an endless stream of bits.
When you set a Single to that value, i.e. 0.3!, then you are cutting off the value at 24bits of significance. (2^23 shifted for 0.3)
At that point it is a number that is close to 0.3, but not 0.3.
Also at that point, the computer has no idea that the number is suppose to be 0.3 because it has been converted to a number that is as close to 0.3 as possible with the given floating point representation. So, the value is a value that is near 0.3 and that is that.
If you have the computer convert the single to a string, then it may produce a string 0.3, because it may round the number for its string representation, so it is rounding 0.300000011920929 to 0.3000000 by rounding the last few bits to the closest decimal value.
That 0.000000011920929 represents the value of the smallest bit (2^23) shifted , when representing .3 in a 32bit float.
Microsoft's Win10 calculator in Scientific mode says the value is actually 0.300000011920928955078125.
So, .3 as a single is .3 with an error of being 0.000000011920928955078125 above that actual value.
That is the value. That is as close as it can get.
If you then ask the compiler to convert that single (whose true value if the string conversion would print it is .300000011920928955078125) to a double, which has a lot more bits to work with, then it won't throw away the .0000000119209... value in the single (i.e. arbitrarily throw away or set the last bit of the Single value to 0) when converting to a double.
When converting from a Single to a double you want to keep every bit of the mantissa that is available, i.e. 24bits (including the implied bit). All the conversion is doing when going from Single to double, as for the significance, is adding a bunch of 0's to the end of the number.
Since it doesn't know the value represents .3 it can't set the last bit to 0 and add all the additional 0's and 1's to the bits beyond the 24 bits given to make the value closer to .3. How is it to know that .3 is what you want at that point?
Bottom line .3! and .3# are two different numbers, both of which are close to .3, but neither of them are .3. The second is closer than the first obviously because it has a lot more bits to represent the number so the error will be around a billion times smaller, but it won't be .3.
When you convert a single that was set to .3 to a double, then you get to see what the majority of the error was (limited by the precision of a double, and subject to the string conversion to decimal representation) between the value .3 and the Single's approximation of .3.
p.s. Just for extra info,
Just like 1/3 end up being an endless .3333333...
.3 decimal in binary just so happens to coincidentally end up being an endless series of 0011001100110011
The single, for its 24 bits of mantissa ends up cutting the series off between two 1's, i.e. ...00110011001<cut>1001100110011, so, the value is rounded up since the value below the cut is greater than .5 (relatively).
In Hex, the Single value would be 3E99 9999 9999 9999 9999 ... if allowed to exceed 32bits
But, since the number has to be cut off at 3E99 9999 to fit in a 32bit single, the number becomes 3E99 999A,
i.e
3E99 9999 9999 9999... is the number you'd like.
3E99 9999 0000 0000... is choice 1, truncate to 32 bits
3E99 999A 0000 0000... is choice 2, round to 32 bits
Which is closer to the number you want, choice 1 or choice 2. The computer chooses choice 2, of course.
For a double you have the same repeating binary pattern for the mantissa of course, alternating pairs of ones and zeros, 110011001100...
But because of the alignment difference caused by the difference in size of the exponent part of the floating point number, the hex value are different, but the bit pattern that makes up the digits is the same, just shifted by two bits.
3FD3 3333 3333 3333
Of course the real value for .3 would continue indefinitely with 333333... in the double representation, if extended.
So, you have a Single value whose value is 3E99 999A and when you convert it to a double, it would have to be extended to 64 bits, so that would be done by adding 0's to the end, there is no other way to convert a 32bit value to a 64bit value and add extra significance to the original value.
So, the first step, assuming the exponent field didn't change in size would be.
3E99 999A 0000 0000 which gives you a 64bit Single. But we don't have 64bit singles, so the mantissa has to shift to the right to make room for the expanded Double exponent, so you end up with
3FD3 3333 4000 0000.
3FD3 3333 3333 3333 would be the result if you converted .3 into a double directly, but since you first put it into a single, you lost a lot of bits during that conversion, and then converting that values with all the lost bits, can't magically reproduce the missing bits from nowhere.
Last edited by passel; May 8th, 2020 at 02:27 AM.
"Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

May 8th, 2020, 09:05 AM
#15
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
I get all that but the issue is still the fix function converting everything to double.
fix(30 * 0.3) = 8
fix(30 * 0.30000000000001) = 9
fix(30 * 0.300000011920929) = 9
fix(30 * 0.299999999999999) = 8
fix(30! * 0.3!) = 9
I can't even write 0.30000000000000#
this means I need to "think" in doubles or always assign the datatype.
no matter computers or whatever, or reason, it doesnt matter.
is like saying humans will always rape, its part of their behavior but doesn't make it right.

May 8th, 2020, 09:29 AM
#16
Re: [RESOLVED] Fix
An alternative is to use Round instead of Fix, but I don't know what you need to do.

May 8th, 2020, 09:55 AM
#17
Re: [RESOLVED] Fix
Originally Posted by baka
I get all that but the issue is still the fix function converting everything to double.
fix(30 * 0.3) = 8
fix(30 * 0.30000000000001) = 9
fix(30 * 0.300000011920929) = 9
fix(30 * 0.299999999999999) = 8
fix(30! * 0.3!) = 9
I can't even write 0.30000000000000#
this means I need to "think" in doubles or always assign the datatype.
no matter computers or whatever, or reason, it doesnt matter.
is like saying humans will always rape, its part of their behavior but doesn't make it right.
I'm not sure what you're saying there.
This is not a Single vs Double issue, and you're only talking about a test case of one value, i.e. 0.3 .
The first four examples you are passing literal values, so those will all be Doubles and that has nothing to do with the Fix function itself.
That last example is multiplying two Singles, so a Single will be passed to fix.
I guess you're saying that the correct answer is the ones that return 9?
That fact that 30! * 0.3! happens to be slightly larger than 9, so gets truncated to 9 is just the particular case, and isn't really a Single vs Double issue in the general case.
It is only because the .3 value happens to end with the bit 33 of the value being a 1 with the exponent size of a float 32 representation, so the 32bit floating point value is rounded up so is slightly above 0.3. whereas the 65th bit of the value with the float 64 exponent is 0, so the value doesn't round up so the floating point value is slightly less than 0.3 .
But it isn't guaranteed that the 32bit float will always round up. I guess statistically, it should round up around 50% of the time, so 50% of the time, the Single's value will be slightly less than the decimal value it is set to (assuming not a sum of powers of 2), and the Fix function would return the 1 value just like the double did in this case. It really isn't a double versus Single thing, in the general case.
Using any floating point numbers (Single or Double) with values after the decimal point, the actual value will be slightly high or slightly low, probably in equal measure.
You could always wrap the Fix function with your own version that adds a slight bias to the value to have it always come out on the high side, if the result is close to being an Integer value, if that is what you prefer Fix to do.
Last edited by passel; May 8th, 2020 at 09:59 AM.
"Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

May 8th, 2020, 10:13 AM
#18
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
maybe not, but Single at least make it right, at least as close possible.
while doubles is all wrong.
30 * 0.3 = 9 and nothing else, if its wrong because how computers works, it is still wrong,
I didnt input 0.299995235923523 or 0.30000124120410241240124012 but 0.3 and for me 0.3 is 0.3 and not something else.
no matter how it works, or the limitations or whatever. for me 0.3 is 0.3 and thats it.
if you think 30 * 0.3 = 8 is correct, Im not sure what to say.
also, I use Fix, because its not always 30, it can be any number.

May 8th, 2020, 10:16 AM
#19
Re: [RESOLVED] Fix
And what's 1/3 or 22/7 or Pi?
There are more numbers which can not be represented in either base 10 or 2 then numbers which can exactly be represented.
And you are ignoring the fact that there are only 32 (or 64) bits to store values.
Otherwise you need to switch to integer calculations only.

May 8th, 2020, 10:24 AM
#20
Re: [RESOLVED] Fix
I'm saying that being a Single and being .3 works for .3, in this situation.
If you were using .2 or .4 or .6 or .7, then it may not work. Since the case where it works and the case where it doesn't work should be about 50/50, the fact that it "works" for .3 is just because .3 when converted to a Single is slightly above .3.
If your use case only requires you to use .3 (and because it is a Single it is really .300000011920928955078125), then it will work for you, so is a "fix".
But, technically you are only fooling yourself if you thing you are using .3, because you aren't. It is impossible to use .3 when you're using a floating point type.
When you set a double to .3, I'm not sure what it is offhand, but I assume it is in the neighborhood of .29999999999998xxx
Last edited by passel; May 8th, 2020 at 10:29 AM.
"Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

May 8th, 2020, 10:52 AM
#21
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
again, I get the computer part, but Im not talking about that and actually I dont care.
for me the result is the importance. if I input 0.3 I dont want or care if the computer is changing it 0.30001 or 0.29998 what I want is the math to be as "accurate" possible.
also, I dont get why u "defend" it so much, its like you "believe" in the limitation and incorrectness of how floating numbers works.
I will always have a critical mind and even if I know the limitations I dont need to like it or defend it.
0.3 is 0.3 and thats final. you can not change my mind saying its 0.300000011920928955078125, because in my mind 0.3 is equal 0.3.
what you believe and understand about all of this, is not doing anything for me, and actually you are not helping at all.
sure you explain how it works inside the computer, how the cpu is handling it, bitwise.
but Im not a bitcreated entity, Im a human, and in my Mind, Im not using floating points.

May 8th, 2020, 11:18 AM
#22
Re: [RESOLVED] Fix
Originally Posted by baka
again, I get the computer part, but Im not talking about that and actually I dont care.
for me the result is the importance. if I input 0.3 I dont want or care if the computer is changing it 0.30001 or 0.29998 what I want is the math to be as "accurate" possible.
Unless you can invent an new form of number storage for computers then it is as accurate as possible using current technology. The fact it isn't as accurate for some numbers as we would like is frustrating, but it is the way these things work.
Originally Posted by baka
also, I dont get why u "defend" it so much, its like you "believe" in the limitation and incorrectness of how floating numbers works.
I will always have a critical mind and even if I know the limitations I dont need to like it or defend it.
This isn't people "beleiving" in this limitation, the limitation is a fact of how computers work. Liking or defending it doesn't come into it  this is how it works and you need to take that into consideration when writing code, the same as every other person alive using modern computer architectures
Originally Posted by baka
0.3 is 0.3 and thats final. you can not change my mind saying its 0.300000011920928955078125, because in my mind 0.3 is equal 0.3.
what you believe and understand about all of this, is not doing anything for me, and actually you are not helping at all.
sure you explain how it works inside the computer, how the cpu is handling it, bitwise.
but Im not a bitcreated entity, Im a human, and in my Mind, Im not using floating points.
It doesn't matter how your mind works or what numerical base you are working with, computers are using base 2 for numbers and there is no way to store 0.3 exactly as 0.3 in base 2. If you are using a computer that works on base 2, and you are using one that does, then you need to understand the limitations of base 2 and any numbers that are stored using it.
https://en.wikipedia.org/wiki/IEEE_754 is a pretty decent reference if you want to read more about the topic.
Last edited by PlausiblyDamp; May 8th, 2020 at 11:24 AM.

May 8th, 2020, 11:22 AM
#23
Re: [RESOLVED] Fix
Originally Posted by baka
again, I get the computer part, but Im not talking about that and actually I dont care.
for me the result is the importance. if I input 0.3 I dont want or care if the computer is changing it 0.30001 or 0.29998 what I want is the math to be as "accurate" possible.
also, I dont get why u "defend" it so much, its like you "believe" in the limitation and incorrectness of how floating numbers works.
I will always have a critical mind and even if I know the limitations I dont need to like it or defend it.
0.3 is 0.3 and thats final. you can not change my mind saying its 0.300000011920928955078125, because in my mind 0.3 is equal 0.3.
what you believe and understand about all of this, is not doing anything for me, and actually you are not helping at all.
sure you explain how it works inside the computer, how the cpu is handling it, bitwise.
but Im not a bitcreated entity, Im a human, and in my Mind, Im not using floating points.
Ok, so your initial question:
Originally Posted by baka
can anyone explain this:
fix(30 * 0.3) = 8
fix(csng(30 * 0.3)) = 9
That was code you were running through your brain and wondering why you were getting those results???
Oh yeah, no. You were running that code on a computer. And its been explained to you over and over why the computer is producing those results.
No one is saying 30*0.3 = 8. No one. Get over it.

May 8th, 2020, 01:18 PM
#24
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
I agree PlausiblyDamp, I understand the base 2 limitations. but as I explained, for me that doesnt matter, its only the result that matters, and if 30! * 0.3! helped me, thanx to LaVolpe explanations, thats the solution, even if its not perfect, its close enough. 30 * 0.3 gave the "wrong" expected result, and that for me is wrong, no matter why. and I dont care the specifics, it will not help me, no matter how much you explain about it. I need a "fix", and LaVolpe gave me that, and Im happy for that.

well, why do u keep commenting. I got the answer from LaVolpe and I changed to "resolved".
but u keep posting and arguing and trying to be smart, for what? to win a competition? did I ask for you answers?
I got mine and Im done, but you try to tell me how I need to think. I already answered, I get it what u say, but my own perspective is different, no matter right or wrong. so try to get it AND get over it.
Last edited by baka; May 8th, 2020 at 01:30 PM.

May 8th, 2020, 01:58 PM
#25
Re: [RESOLVED] Fix
Originally Posted by baka
I need a "fix", and LaVolpe gave me that, and Im happy for that.
But now you're going to apply it wrongly (in your AppCode).
When you write (in your post #7):
"...before I could write Fix(l / 2) and that worked well,
but now I will include the datatype, so Fix(l / 2&) (l=long)"
Then this is clearly a misuse of the FixFunction (which does 2 additional Runtimecalls under the covers).
Better and faster would be:
SomeResult = SomeLong \ 2 '< using the IntegerDivOperator
And all this happened, because nobody asked you at the beginning:
 "Why do you have to use the Fix()function?"
 "Can you give some more background, what you really want to do?"
And in case these questions would have been properly answered by you
(without you making any fuss of "hijacking my thread"),
then you'd have been supplied with better matching answers.
(Like e.g. CLng(30 * 0.3) or CCur(30 * 0.3) or CDec(30 * 0.3)...)
In the state this thread currently is, some other inexperienced users who stumble over it,
might think that: Fix(l / 2&) (l=Long) is a good idea... (which it is not).
Olaf

May 8th, 2020, 02:21 PM
#26
Re: [RESOLVED] Fix
Originally Posted by Schmidt
And all this happened, because nobody asked you at the beginning:
 "Why do you have to use the Fix()function?"
 "Can you give some more background, what you really want to do?"
That was more or less my idea, when I posted:
Originally Posted by Eduardo
An alternative is to use Round instead of Fix, but I don't know what you need to do.

May 8th, 2020, 02:41 PM
#27
Re: [RESOLVED] Fix
Originally Posted by Eduardo
That was more or less my idea, when I posted:
An alternative is to use Round instead of Fix, but I don't know what you need to do.
Sorry, overlooked that ... just wanted to make a statement against the current "trend",
that "original posters" these days somehow feel "entitled" (or at least "encouraged"),
to make demands about what kind of answers they deem worthy in their "plea for help".
"It's my thread"... seems to overrule the common sense (what publicforums are for) by now.
Olaf

May 8th, 2020, 02:46 PM
#28
Re: [RESOLVED] Fix
WOW, this got a lot of discussion, and I admittedly didn't read every word.
However, in my mind, it all comes down to the fact that 0.3 doesn't have an exact representation in either IEEE Single or IEEE Double.
Therefore, it comes down to whether the 0.3 binary representation is going to be ever so slightly larger than 0.3 or ever so slightly smaller than 0.3.
Without even testing, I feel certain that an IEEE Single is going to be slightly smaller (resulting in a product slightly smaller than 10), and an IEEE Double is going to be slightly larger (resulting in a product slightly larger than 10).
These decimaltobinary (and viceversa) conversion anomalies are well known.
EDIT: Ok, I got that backwards. Single is slightly LARGER than 0.3 and Double is slightly SMALLER than 0.3.
After putting a 0.3 in a Single, and then converting back to decimal (and calculating all the way out until we get an exact decimal representation), the representation is:
0.300000011920928955078125
And, after putting 0.3 in a Double, and then converting back to decimal (and calculating all the way out until we get an exact decimal representation), the representation is:
0.299999999999999988897769753748434595763683319091796875
So, we can see that 0.3 in a Single is slightly larger, and 0.3 in a Double is slightly smaller. Hence: Fix(30! * 0.3!) = 9, and Fix(30# * 0.3#) = 8
EDIT2: Here's a webpage that will help with figuring that out. Request both Single and Double for best illustration.
Last edited by Elroy; May 8th, 2020 at 03:04 PM.
Any software I post in these forums written by me is provided “AS IS” without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. Please understand that I’ve been programming since the mid1970s and still have some of that code. My contemporary VB6 project is approaching 1,000 modules. In addition, I have a “VB6 random code folder” that is overflowing. I’ve been at this long enough to truly not know with absolute certainty from whence every single line of my code has come, with much of it coming from programmers under my employ who signed intellectual property transfers. I have not deliberately attempted to remove any licenses and/or attributions from any software. If someone finds that I have inadvertently done so, I sincerely apologize, and, upon notice and reasonable proof, will reattach those licenses and/or attributions. To all, peace and happiness.

May 8th, 2020, 02:56 PM
#29
Re: [RESOLVED] Fix
again, I get the computer part, but Im not talking about that and actually I dont care.
for me the result is the importance. if I input 0.3 I dont want or care if the computer is changing it 0.30001 or 0.29998 what I want is the math to be as "accurate" possible.
also, I dont get why u "defend" it so much, its like you "believe" in the limitation and incorrectness of how floating numbers works.
I will always have a critical mind and even if I know the limitations I dont need to like it or defend it.
You sound like a normal end user instead of a programmer.
As a programmer you need to learn about how cpu’s work and how programming languages behave.
Then as a programmer you have to deal with it and either create solutions or accept the current behavior is.

May 8th, 2020, 04:48 PM
#30
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
Olaf,
your comment to use \ instead of / is valid, and I use it, but sometimes I forget, its a good reminder and I appreciate that.
I appreciate members like LaVolpe, Elroy, Eduardo, passel and PlausiblyDamp that are explaining, giving examples and even trying to understand my view. even if they would think Im wrong from their p.o.v.
what I dont appreciate are the others, they believe they are better and with condescending attitude. I can be without that. so please just stop and "help" someone else.
Last edited by baka; May 8th, 2020 at 11:22 PM.

May 8th, 2020, 06:26 PM
#31
Re: [RESOLVED] Fix
Another alternative to get what you're after is to use the Decimal type:
Fix(CDec(30!) * CDec(0.3!)) = 9
Even...
Fix(CDec(30#) * CDec(0.3#)) = 9
It's too bad that VB6 doesn't provide a way to specify Decimal literal constants.
But, one thing it does do is some strange rounding when we convert types. For instance, the Decimal type has precisely 28 "decimal digits" of precision. So, when doing CDec(0.3#), we might wonder why it doesn't put 0.299999999999999988897769754 in the Decimal type. However, it doesn't. It puts 0.300000000000000000000000000 in the Decimal type. And then, in the above multiplication, it does it correctly, multiplying 30 by 0.3, resulting in precisely 9. And then, the Fix essentially does nothing.
So, even if we start with Singles or Doubles, if we convert them to Decimals before we do our math, things will work out as we'd expect (intuitively, without worrying about binarytodecimal and decimaltobinary conversion).
Note, in the above, I capitalize Decimal when I'm talking about the Decimal type ... and I leave decimal lower case when I'm just talking about base10 numbers.

EDIT: Just to add a bit more, the Decimal type essentially stores numbers as base10 numbers. It's a bit of a hybrid in that it stores the mantissa as binary. But, the mantissa is just an integer, and integers are the same regardless of the base. What's most important is that the exponent is indicated as the decimalpoint's position in a base10 number. Therefore, Decimal types store base10 numbers exactly as we think of them, and don't do any floatingpoint decimaltobinary (and viceversa) conversions, which is what caused the anomaly in the OP.
Last edited by Elroy; May 8th, 2020 at 06:48 PM.
Any software I post in these forums written by me is provided “AS IS” without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. Please understand that I’ve been programming since the mid1970s and still have some of that code. My contemporary VB6 project is approaching 1,000 modules. In addition, I have a “VB6 random code folder” that is overflowing. I’ve been at this long enough to truly not know with absolute certainty from whence every single line of my code has come, with much of it coming from programmers under my employ who signed intellectual property transfers. I have not deliberately attempted to remove any licenses and/or attributions from any software. If someone finds that I have inadvertently done so, I sincerely apologize, and, upon notice and reasonable proof, will reattach those licenses and/or attributions. To all, peace and happiness.

May 9th, 2020, 03:36 AM
#32
Re: [RESOLVED] Fix
Originally Posted by Elroy
...
Without even testing, I feel certain that an IEEE Single is going to be slightly smaller (resulting in a product slightly smaller than 10), and an IEEE Double is going to be slightly larger (resulting in a product slightly larger than 10).
...
EDIT: Ok, I got that backwards. Single is slightly LARGER than 0.3 and Double is slightly SMALLER than 0.3.
...
This is just a FYI.
The above was one of my points. Whether the Floating Point number is slightly larger or slightly smaller than the desired value (or exact in the case of sums of powers of 2), is not depended on the type, but dependent on the bit pattern of the number and how it rounds so is not easily predicted just by looking at the number.
So, while 0.5 is exact (a power of 2), the following table shows that larger or smaller isn't depended on Single vs Double representation.
Code:
Value : Single Double

0.1 High High
0.2 High High
0.3 High Low
0.4 High High
0.5 Exact Exact
0.6 High Low
0.7 Low Low
0.8 High High
0.9 Low High
So, if for some reason your code had to use .7 or .9 percent in its calculation instead of .3, the current "fix" would fail.
Also, if you had happened to be using any of the values that are "High" for the Double in the table above instead of .3, you wouldn't have even realized you had a problem.
Being aware of these realities is not a bad thing to know for future work.
"Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930

May 9th, 2020, 03:54 AM
#33
Re: [RESOLVED] Fix
Originally Posted by baka
0.3 is 0.3 and thats final. you can not change my mind saying its 0.300000011920928955078125, because in my mind 0.3 is equal 0.3.
As passel already told before, you have to work with Decimal datatype for this to be true.
It is as simple as this
Code:
Option Explicit
Private Sub Form_Load()
Dim aThird As Variant
aThird = CDec(3) / 10
Debug.Print 30 * aThird
Debug.Print 25 * aThird
Debug.Print 20 * aThird
Debug.Print 15 * aThird
Debug.Print 10 * aThird
End Sub
There are no "hairs" left over from the calculations and Fix will work as expected.
Decimal datatype is like working with Long and Integers but shifted to the right of the floatingpoint, so your 0.3 is represented like 3 * 10^1 in memory and 0.123 is stored as 123 * 10^3.
This way no "wrong" calculations can happen but the numbers are actually *less* precise than Double/Single and I leave the explanation why floatingpoint numbers are more precise than fixedpoint ones to someone else :))
cheers,
</wqw>

May 9th, 2020, 09:00 AM
#34
Re: [RESOLVED] Fix
Originally Posted by baka
0.3 is 0.3 and thats final.
Yeah, I've got to chime in as well, and say that that's just NOT true. Many of us are so steeped in thinking in base10, that it's difficult for us to "think" in any other base (such as base2). And, just to keep it short, the takehome message is: What can be perfectly represented with floatingpoint representation in one base (say, base10) can't be perfectly represented with floatingpoint in all other bases (such as base2, for example).
And both IEEE Singles and IEEE Doubles represent both their mantissa and exponent as base2. And as passel first pointed out, and I mentioned as well, only the Decimaltype (within VB6) perfectly represents floating point base10 numbers.
This is all an extremely well known problem with a great deal of documentation on it. And it's been a problem for many decades.
I remember a language back in the 80s called CBasic. It had a number representation called binarycodeddecimal (BCD). (This was also done in COBOL.) It was a bit more wasteful than the VB6 Decimaltype, but it took each nibble (fourbits) and coded a 0 thru 9 into it. This is only 10 values, and a nibble can support up to 16 values, and this is why it was a bit wasteful. However, with the addition of an exponent portion (i.e., where to put the decimal point), this BCD number encoding method could perfectly represent base10 numbers (as does the Decimaltype in VB6).
BCD was intuitively desirable. As opposed to the Decimaltype, you could look at a BCD number in memory and rather easily figure out exactly what the number was. Here's the mantissa encoding of BCD:
So, again, to say that "0.3 is 0.3 and that's final" just doesn't appreciate the finer points of how computers store floatingpoint numbers and how they do math with them.
That brings up one final point ... for the last 30 years (or maybe a bit longer), we've been using floatingpointprocessors (FPPs) to do most of our floatingpoint math. And that's true of VB6 as well when dealing with Singles and Doubles. It's just MUCH faster than doing this math with software. And, these FPPs do math as base2, not base10. So, for those who've been at it a while, we understand that there are occasionally base2tobase10 (and viceversa) conversion issues.
And, if we wish to completely circumvent these issues, we must use an encoding that doesn't use base2, such as VB6's Decimaltype.
Any software I post in these forums written by me is provided “AS IS” without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. Please understand that I’ve been programming since the mid1970s and still have some of that code. My contemporary VB6 project is approaching 1,000 modules. In addition, I have a “VB6 random code folder” that is overflowing. I’ve been at this long enough to truly not know with absolute certainty from whence every single line of my code has come, with much of it coming from programmers under my employ who signed intellectual property transfers. I have not deliberately attempted to remove any licenses and/or attributions from any software. If someone finds that I have inadvertently done so, I sincerely apologize, and, upon notice and reasonable proof, will reattach those licenses and/or attributions. To all, peace and happiness.

May 9th, 2020, 09:42 AM
#35
Re: [RESOLVED] Fix
Ooops, I got Decimal and Currency datatypes all mixed up above. . . Anyway
cheers,
</wqw>

May 9th, 2020, 10:49 AM
#36
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
yeah, as I have explained numerous times, I get the computer base2 part, but in my mind 0.3 is equal 0.3 and nothing else.
so, depending on the perspective.
I prioritize my mind, not the computer. so, in my mind 0.3 is 0.3.
what to do to make it as close possible to what I want to achieve?
and in this case,
a number between 1 to 30 multiply with 0.3, using fix since I don't want any rounding.
LaVolpe helped me with the "single" usage, that "solved" my problem.
BUT of course it will not solve ALL problems. and I get that.
next time maybe I need to do something like:
fix(10 * 0.7) = 6
fix(10 * 0.7!) = 6
fix(10 * 0.7000001!) = 7
using 0.7000001! if I want to make it 7, and that is a "possible" solution to my problem, not a universal fix for everything.
all of you that helped me understand the base2, I appreciate it, but you failed to understand my p.o.v. since you want me to think like you,
but Im not you, my mind works differently and that how I do my programming. but that doesnt mean I can't learn, but I learn my way.
and I learned about the issues base2 can make when working in a base10 plane. and I appreciate it, but I don't appreciate when you tell me how I need to think. thats my problem to handle.

May 9th, 2020, 11:07 AM
#37
Re: [RESOLVED] Fix
Originally Posted by baka
yeah, as I have explained numerous times, I get the computer base2 part, but in my mind 0.3 is equal 0.3 and nothing else.
so, depending on the perspective.
I prioritize my mind, not the computer. so, in my mind 0.3 is 0.3.
what to do to make it as close possible to what I want to achieve?
and in this case,
a number between 1 to 30 multiply with 0.3, using fix since I don't want any rounding.
LaVolpe helped me with the "single" usage, that "solved" my problem.
BUT of course it will not solve ALL problems. and I get that.
next time maybe I need to do something like:
fix(10 * 0.7) = 6
fix(10 * 0.7!) = 6
fix(10 * 0.7000001!) = 7
using 0.7000001! if I want to make it 7, and that is a "possible" solution to my problem, not a universal fix for everything.
all of you that helped me understand the base2, I appreciate it, but you failed to understand my p.o.v. since you want me to think like you,
but Im not you, my mind works differently and that how I do my programming. but that doesnt mean I can't learn, but I learn my way.
and I learned about the issues base2 can make when working in a base10 plane. and I appreciate it, but I don't appreciate when you tell me how I need to think. thats my problem to handle.
This is a programming forum. You asked a computer programming related question, you got computer programming related answers. If that upsets you then perhaps that is something you need to address outside of this forum.

May 9th, 2020, 11:47 AM
#38
Re: [RESOLVED] Fix
Originally Posted by baka
what to do to make it as close possible to what I want to achieve?
and in this case,
a number between 1 to 30 multiply with 0.3, using fix since I don't want any rounding.
If you don't want any rounding then why are you using Fix?
30 * 0.3 gives the correct result, unlike Fix(30 * 0.3):
Code:
Debug.Print 30 * 0.3
Debug.Print Fix(30 * 0.3)
The question of the whole thread: what do you want to achieve?

May 9th, 2020, 12:02 PM
#39
Thread Starter
Frenzied Member
Re: [RESOLVED] Fix
you are right, not all cases I need Fix, but its not always 30,
the OP example I needed it to work for a number between 1 to 30, I dont want any rounding, as x.000 to x.999 need to be x, that's why I use Fix.
without fix, 29 * 0.3 would give me 9 not 8 (as I wrote in #18 and #36)

May 9th, 2020, 12:09 PM
#40
Re: [RESOLVED] Fix
Well, if you do this in 2 statements, you should always get what you want. Not sure that's any better & doesn't address the "why" of the initial problem.
Code:
Dim vResult As Double ' or Single as desired
vResult = 30 * 0.3: MsgBox Fix(vResult)
However, others have posted workarounds too, i.e., Fix(CDec(30 * 0.3))
At least, you have choices and understand the "why"
Posting Permissions
 You may not post new threads
 You may not post replies
 You may not post attachments
 You may not edit your posts

Forum Rules

Click Here to Expand Forum to Full Width
