-
[RESOLVED] Number Precision: How to explain?
Can you explain the stuff below?
I am not sure about internal storage of these numbers.
(Noted that Rnd() returns a Single data type number.)
Code:
Sub Test()
Dim c As Variant
Dim d As Double
Dim v As Variant
Dim w As Variant
v = Rnd
Debug.Print TypeName(v), "v = "; Format(v, "0.000000000000000"), "Len(v)="; Len(v)
d = v
Debug.Print TypeName(d), "d = "; Format(d, "0.000000000000000"), "Len(d)="; Len(d)
Debug.Print "v=d : "; (v = d)
c = CDec(v)
Debug.Print TypeName(c), "c = "; Format(c, "0.000000000000000"), "Len(c)="; Len(c)
w = CLng(v * 10 ^ 7) / 10 ^ 7
Debug.Print TypeName(w), "w = "; Format(w, "0.000000000000000"), "Len(w)="; Len(w)
Debug.Print "w=v : "; (w = v)
Debug.Print
End Sub
Code:
Single v = 0.234101400000000 Len(v)= 9
Double d = 0.234101355075836 Len(d)= 8
v=d : True
Decimal c = 0.234101400000000 Len(c)= 9
Double w = 0.234101400000000 Len(w)= 9
w=v : False
-
Re: Number Precision: How to explain?
Where are you, all forums' experts?
-
Re: Number Precision: How to explain?
Well - first of all - len() is a function that returns a string's length [edit] or the amount of memory needed to store a variable [/edit]
And I got TRUE on that last test
Code:
Single v = 0.705547500000000 Len(v)= 9
Double d = 0.705547511577606 Len(d)= 8
v=d : True
Decimal c = 0.705547500000000 Len(c)= 9
Double w = 0.705547500000000 Len(w)= 9
w=v : True
What you have stumbled upon is the imprecision of floating point. It's designed to store large values - but it does so with imprecision based on how the value is stored (only so many digits are stored along with an exponent value to determine the decimal placement).
What exactly is your need here?
-
Re: Number Precision: How to explain?
Hmm.. I don't remember seeing this one, I guess I must have accidentally skipped it.
Anyhoo, as Single and Double are floating-point (as opposed to scaled integer like Decimal is), there are "rounding errors" after a few decimal places - and as they have different ranges, they have different degrees of that issue.
I can't remember for sure how they are actually stored, but I think it is something like the reciprocal of the decimal places (ie: 1/0.7055...), but with Single and Double having different numerators (hence why d has a different value).
To find out, have a search for what is mentioned in the help (Double: "are stored as IEEE 64-bit (8-byte) floating-point numbers", Single: "are stored as IEEE 32-bit (4-byte) floating-point numbers").
-
Re: Number Precision: How to explain?
@szlamany, I understand what you say, but as you see:
v=d is always TRUE. I think v was converted to Double before compared with d? Look at the digits of v and d. Hard to explain.
w=v can be vary as seen: FALSE (my case) or TRUE (your case). But all digits of w match with those of v.
In your case, with v=d and w=v, that implies d=w: Both d and w are Double and look at the values of d and w. How to explain?
The picture will be clearer if someone can explain how these numbers are stored in memory bit-by-bit and how they are compared in v=d and w=v.
Edit: @si, forget about c as it is a Decimal.
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by anhn
...The picture will be clearer if someone can explain how these numbers are stored in memory bit-by-bit and how they are compared in v=d and w=v.
You must be kidding ;)
Quote:
F_floating format (single precision floating), DEC VAX, 32 bits, the first bit (high order bit in a register, first bit in memory) is the sign magnitude bit (one=negative, zero=positive or zero), followed by 15 bits of an excess 128 binary exponent, followed by a normalized 24-bit fraction with the redundant most significant fraction bit not represented. Zero is represented by all bits being zero (allowing the use of a longword CLR to set a F_floating number to zero). Exponent values of 1 through 255 indicate true binary exponents of -127 through 127. An exponent value of zero together with a sign of zero indicate a zero value. An exponent value of zero together with a sign bit of one is taken as reserved (which produces a reserved operand fault if used as an operand for a floating point instruction). The magnitude is an approximate range of .29*10-38 through 1.7*1038. The precision of an F_floating datum is approximately one part in 223, or approximately seven (7) decimal digits).
32 bit floating format (single precision floating), AT&T DSP32C, 32 bits, the first bit (high order bit in a register, first bit in memory) is the sign magnitude bit (one=negative, zero=positive or zero), followed by 23 bits of a normalized two’s complement fractional part of the mantissa, followed by an eight bit exponent. The magnitude of the mantissa is always normalized to lie between 1 and 2. The floating point value with exponent equal to zero is reserved to represent the number zero (the sign and mantissa bits must also be zero; a zero exponent with a nonzero sign and/or mantissa is called a “dirty zero” and is never generated by hardware; if a dirty zero is an operand, it is treated as a zero). The range of nonzero positive floating point numbers is N = [1 * 2-127, [2-2-23] * 2127] inclusive. The range of nonzero negative floating point numbers is N = [-[1 + 2-23] * 2-127, -2 * 2127] inclusive.
40 bit floating format (extended single precision floating), AT&T DSP32C, 40 bits, the first bit (high order bit in a register, first bit in memory) is the sign magnitude bit (one=negative, zero=positive or zero), followed by 31 bits of a normalized two’s complement fractional part of the mantissa, followed by an eight bit exponent. This is an internal format used by the floating point adder, accumulators, and certain DAU units. This format includes an additional eight guard bits to increase accuracy of intermediate results.
D_floating format (double precision floating), DEC VAX, 64 bits, the first bit (high order bit in a register, first bit in memory) is the sign magnitude bit (one=negative, zero=positive or zero), followed by 15 bits of an excess 128 binary exponent, followed by a normalized 48-bit fraction with the redundant most significant fraction bit not represented. Zero is represented by all bits being zero (allowing the use of a quadword CLR to set a D_floating number to zero). Exponent values of 1 through 255 indicate true binary exponents of -127 through 127. An exponent value of zero together with a sign of zero indicate a zero value. An exponent value of zero together with a sign bit of one is taken as reserved (which produces a reserved operand fault if used as an operand for a floating point instruction). The magnitude is an approximate range of .29*10-38 through 1.7*1038. The precision of an D_floating datum is approximately one part in 255, or approximately 16 decimal digits).
Go digest this link
http://en.wikipedia.org/wiki/IEEE_754
Bottom line is that double only stores 16 digits - single stores 8. The storage format also allows for the "exponent" placement. All other digits are zero.
When the number is displayed it's not the true figure.
You might have float 3.99999999 and it's displayed as 4!!!!You cannot look at the value visually - it's not going to tell the truth!
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by szlamany
You must be kidding ;)
Bottom line is that double only stores 16 digits - single stores 8. The storage format also allows for the "exponent" placement. All other digits are zero.
When the number is displayed it's not the true figure.
You might have float 3.99999999 and it's displayed as 4!!!!You cannot look at the value visually - it's not going to tell the truth!
I have no way to understand the lengthy paragraphs you quoted. But I fully aware of your own comments above.
I know about 16 digits so I formatted all numbers up to that (if I was not wrong on counting).
I know very well the case of 3.99999999... and 4. But 2 Double values with the same display format, if they are equal, all display digits must be matched.
Am I wrong? Look again at d and w in your case.
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by anhn
...But 2 Double values with the same display format, if they are equal, all display digits must be matched.
Am I wrong? Look again at d and w in your case.
No that is not true - that's the whole problem.
What you display has been formatted and has lost it's original "content".
The = operator is comparing the values on a more internal level.
Remember that you might have 3.9999999 in the double - in memory - but the display of it will be 4.
That fact alone means display equality does not mean value equality with floating point.
Floating point is meant to send rockets into space - what are you trying to do with it??
-
Re: Number Precision: How to explain?
Yep, a typical implicit CStr going on - but less obvious than with things like Dates.
Quote:
Originally Posted by anhn
I think v was converted to Double before compared with d?
Same here - due to the precision, it wouldn't make much sense the other way around.
Quote:
Edit: @si, forget about c as it is a Decimal.
I did - I just briefly mentioned Decimal to point out (as I'm sure you knew already) that floating-point numbers aren't as accurate as the alternatives.
Quote:
I have no way to understand the lengthy paragraphs you quoted.
Try the link instead - it is much clearer.
-
Re: Number Precision: How to explain?
Haha! Just accidentally found out last week.
Still cannot understand.
Code:
Single v = 0.705547500000000 Len(v)= 9
Double d = 0.705547511577606 Len(d)= 8
v=d : True
Double w = 0.705547500000000 Len(w)= 9
w=v : True
w=v AND v=d ===> w=d ===> 0.705547511577606 = 0.705547500000000 :confused: :ehh: :confused:
-
Re: Number Precision: How to explain?
d is being derived from v... consider comparing the mantissas rather than the debug.print outputs... if they are not the same then unexpected behavior is platform specific, e.g. quirks of the language rather than processor more often than not... they implement their own arithmetic and as long as precision is close enough whose gonna complain? That's what floating point arithmetic is all about.
-
Re: Number Precision: How to explain?
@anhn - please look at this example. I put a real lot of effort into it.
Code:
Private Sub Form_Load()
Dim x As Long
Dim i As Single, j As Single, k As Single, z As Single
i = 50
j = 15
k = i / j
Debug.Print "50 / 15 = "; k
For x = 1 To j
z = z + k
Debug.Print "Iteration "; x; "brings z up to "; z
Next x
End Sub
Simple - right? It takes 50 and divides by 15 - which we all know is 3.333333.
Then it loops 15 times to add up all those 3.333333's back again.
You would imagine that it would never arrive back at 50 - right?
Well - here is the output!
Code:
50 / 15 = 3.333333
Iteration 1 brings z up to 3.333333
Iteration 2 brings z up to 6.666667
Iteration 3 brings z up to 10
Iteration 4 brings z up to 13.33333
Iteration 5 brings z up to 16.66667
Iteration 6 brings z up to 20
Iteration 7 brings z up to 23.33333
Iteration 8 brings z up to 26.66667
Iteration 9 brings z up to 30
Iteration 10 brings z up to 33.33334
Iteration 11 brings z up to 36.66667
Iteration 12 brings z up to 40
Iteration 13 brings z up to 43.33333
Iteration 14 brings z up to 46.66666
Iteration 15 brings z up to 50
So you can see that the "displayed" value of 3.333333 was never true. The very second iteration made 6.666667 - how could 3.333333 + 3.333333 = 6.666667?
See how the display of the value has absolutely nothing to do with what is going on internally with the math on a floating point value?
Look at the inconsistency of iteration 14? It's not 46.66667 - it's 46.66666!
-
Re: Number Precision: How to explain?
@szlamany,
As I mentioned earlier I fully aware of what you say in the example above (at least since 20 years ago) and I know that very very well.
What I said is "2 Double values with the same display format, if they are equal, all display digits must be matched."
One of your example is 3.9999999 and 4, even if they are internally equal, they do not have the same format on display.
You may misunderstand me. What you try to explain is not what I want. However, thank you for your information.
-
Re: Number Precision: How to explain?
Have a think about what I posted earlier:
Quote:
Originally Posted by me
Yep, a typical implicit CStr going on - but less obvious than with things like Dates.
You know that the way Date values get displayed (in the Watch window etc, Debug, MsgBox, ...) is not related to the way they are stored - and the effect here is very similar (albeit harder to find out about).
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by leinad31
d is being derived from v... consider comparing the mantissas rather than the debug.print outputs... if they are not the same then unexpected behavior is platform specific, e.g. quirks of the language rather than processor more often than not... they implement their own arithmetic and as long as precision is close enough whose gonna complain? That's what floating point arithmetic is all about.
A good precise comment. Haha! You are right, that is the nature of floating point numbers in computing. A minor mistake may blow up the Challenger in the sky.
-
Re: Number Precision: How to explain?
Ok - I've enhanced my example - I'm determined to make some point here :D
Code:
Private Sub Form_Load()
Dim x As Long
Dim i As Single, j As Single, k As Single, z As Single
i = 50
j = 15
k = i / j
Debug.Print "50 / 15 = "; k
For x = 1 To j
z = z + k
Debug.Print "Iteration "; x; "brings z up to "; z
Select Case x
Case 1
If z = 3.333333 Then Debug.Print "Equal to 3.333333"
If z <> 3.333333 Then Debug.Print "Not Equal to 3.333333"
Case 2
If z = 6.666667 Then Debug.Print "Equal to 6.666667"
If z <> 6.666667 Then Debug.Print "Not Equal to 6.666667"
Case 3
If z = 10 Then Debug.Print "Equal to 10"
If z <> 10 Then Debug.Print "Not Equal to 10"
Case 4
If z = 13.33333 Then Debug.Print "Equal to 13.33333"
If z <> 13.33333 Then Debug.Print "Not Equal to 13.33333"
Case 5
If z = 16.66667 Then Debug.Print "Equal to 16.66667"
If z <> 16.66667 Then Debug.Print "Not Equal to 16.66667"
Case 6
If z = 20 Then Debug.Print "Equal to 20"
If z <> 20 Then Debug.Print "Not Equal to 20"
Case 7
If z = 23.33333 Then Debug.Print "Equal to 23.33333"
If z <> 23.33333 Then Debug.Print "Not Equal to 23.33333"
Case 8
If z = 26.66667 Then Debug.Print "Equal to 26.66667"
If z <> 26.66667 Then Debug.Print "Not Equal to 26.66667"
Case 9
If z = 30 Then Debug.Print "Equal to 30"
If z <> 30 Then Debug.Print "Not Equal to 30"
Case 10
If z = 33.33334 Then Debug.Print "Equal to 33.33334"
If z <> 33.33334 Then Debug.Print "Not Equal to 33.33334"
Case 11
If z = 36.66667 Then Debug.Print "Equal to 36.66667"
If z <> 36.66667 Then Debug.Print "Not Equal to 36.66667"
Case 12
If z = 40 Then Debug.Print "Equal to 40"
If z <> 40 Then Debug.Print "Not Equal to 40"
Case 13
If z = 43.33333 Then Debug.Print "Equal to 43.33333"
If z <> 43.33333 Then Debug.Print "Not Equal to 43.33333"
Case 14
If z = 46.66666 Then Debug.Print "Equal to 46.66666"
If z <> 46.66666 Then Debug.Print "Not Equal to 46.66666"
Case 15
If z = 50 Then Debug.Print "Equal to 50"
If z <> 50 Then Debug.Print "Not Equal to 50"
End Select
Next x
End Sub
Returns in the immediate window
Code:
50 / 15 = 3.333333
Iteration 1 brings z up to 3.333333
Not Equal to 3.333333
Iteration 2 brings z up to 6.666667
Not Equal to 6.666667
Iteration 3 brings z up to 10
Equal to 10
Iteration 4 brings z up to 13.33333
Not Equal to 13.33333
Iteration 5 brings z up to 16.66667
Not Equal to 16.66667
Iteration 6 brings z up to 20
Equal to 20
Iteration 7 brings z up to 23.33333
Not Equal to 23.33333
Iteration 8 brings z up to 26.66667
Not Equal to 26.66667
Iteration 9 brings z up to 30
Not Equal to 30
Iteration 10 brings z up to 33.33334
Not Equal to 33.33334
Iteration 11 brings z up to 36.66667
Not Equal to 36.66667
Iteration 12 brings z up to 40
Equal to 40
Iteration 13 brings z up to 43.33333
Not Equal to 43.33333
Iteration 14 brings z up to 46.66666
Not Equal to 46.66666
Iteration 15 brings z up to 50
Not Equal to 50
A series of mostly not equals - but equal on every 10th iterations.
Except for the final one - it's not actually equal to 50!! But it displays as 50.
The link I gave actually discussed how two floating point memory addresses are compared for equal to.
-
Re: Number Precision: How to explain?
I fixed a typo in that code that made the results less readable - I've fixed both the code and the immediate window results...
My point is that after 15 loops it displays that Z = 50
But the IF statement says Z <> 50.
Isn't that the whole issue you are discussing here?
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by si_the_geek
Have a think about what I posted earlier:
You know that the way Date values get displayed (in the Watch window etc, Debug, MsgBox, ...) is not related to the way they are stored - and the effect here is very similar (albeit harder to find out about).
Date is not the case of this. I know that 2 different Date values can have the same display value even with time. What I say is :
If a and b have the same number data type, and a = b, and if we use the same format string such as "0.0000000000000000" then
Format(a, "0.0000000000000000") and Format(b, "0.0000000000000000") must give the same output string.
Quote:
Originally Posted by szlamany
Isn't that the whole issue you are discussing here?
No! That is not what I want to discuss.
Now, I tell you why I came up with that test:
Code:
Dim v As Variant
v = Rnd
Debug.Print TypeName(v), "v = "; Format(v, "0.000000000000000"), "Len(v)="; Len(v)
'-- v is a Single value return from Rnd, it has maximum of 7 digits excluding leading 0's.
Dim d As Double
d = v
'-- at this point, I thought d must have exactly display digits as v with all digits after
'-- the 8th digit are 0, but that is not the case.
'-- more than that Len(v) = 9 but Len(d) = 8, but this is not an important point.
Debug.Print TypeName(d), "d = "; Format(d, "0.000000000000000"), "Len(d)="; Len(d)
That is all about.
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by anhn
A good precise comment. Haha! You are right, that is the nature of floating point numbers in computing. A minor mistake may blow up the Challenger in the sky.
Have you checked the mantissas? Languages simply have to conform to the IEEE requirements in terms of memory storage in which the rightmost bit (or 23 + 1 implied significant binary digits) of which (note, still subject to further exponentiation) is 1/(2^23). Have you checked how many digits 1/(2^23) will give you? 0.00000011920928955078125.
But wait! Debug.Print of 1/(2^23) in a Single outputs 1.192093E-07 (or seven significant decimal digits) while the Double of that Single is 1.19209289550781E-07 (or 15 significant decimal digits)? What mystery is this that may cause the Challenger to blow up in the sky?!?
Pretend that I made a language Graphical Basic or GB for short and this language is IEEE Single/Double compliant... pretend that it not only complies to the memory model but also religiously displays all digits of the decimal representation of the binary mantissa-exponent... because it religiously displays all digits in order to assure developers that their computations/comparisons are correct (checked via Debug.Print) and that Challenger will not blow up, the language can only Debug.Print say 1,000 (pretend value) floating point numbers per second...
But when I tried GB to output only 15 digits it would be able to Debug.Print 10,000 floating point values per second! Dang those Strings and their slowness! The floating point number is stored in memory anyway in compliance to IEEE and further arithmetic will still be correct in the floating point sense... so as maker of GB I decree it will only output 15 digits instead of 23, after all a language that can make apps that run "faster" will sell better.
But wait! What if I have it output only seven significant decimal digits, binary 1/(2^23) is still represented in memory anyway and CSng(1/2^23) still equals CDbl(Csng(1/2^23))... besides, that fool developer should have used an appropriate unit instead of kilometers (expecting 23 significant decimal digits?) as the unit of measure anyway! So now GB will show just 7 decimals (remember, internally in memory 1/2^23 is still there) and be able to Debug.Print 100,000 floating point numbers per second. And to again assure developers that Doubles really support more digits, will have that output more significant digits compared to Float... after all they'll be none the wiser that it really should be up to 1/2^52 digits due to size of mantissa... the E notation should suffice. I am the maker of GB after all, my language is entitled to such quirks as I please as long as it conforms to the standard.
That was the gist of it... technically it should have been 1+ 1/2^23 (debug.print displays 1) in order for exponent part of floating point number to remain 0 and for bit 23 in mantissa to be on.
I hope that little story has cleared up the difference between internal storage and output which everyone has been trying to tell you, unfortunately you have been as stubborn as ever. Did you even bother to check the binary mantissa as suggested? In that light, I hope that this post will help you develop your sarcasm skills to a level that is informative and artistic and not a nuisance.
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by leinad31
...unfortunately you have been as stubborn as ever.
Yes, I am very stubborn indeed. That might be my problem (or your?) because I cannot make you guys fully understand what I concern.
What I only concern here is with
Code:
Single v = 0.705547500000000 Len(v)= 9
Double d = 0.705547511577606 Len(d)= 8
v=d : True
Double w = 0.705547500000000 Len(w)= 9
w=v : True
We have w=d (because w=v and v=d), but d has 15 decimal digits while w has only 7.
Both of them are Double, their displays used the same format string "0.000000000000000".
That's it.
We can stop discuss this topic here. That's enough. No gain, no loss.
-
Re: Number Precision: How to explain?
w=v depends on the values, it can be True or False. Again it is a quirk of the platform. Consider expressions Cdbl("0.7055475") = Csng("0.7055475") and msgbox Cdbl("0.7055475") - Csng("0.7055475")... there's a difference because Csng is more of an approximation compared to the double which can use more significant binary bits in the conversion... and yet they are consider by VB as equal. We will never know how religiously VB processes the small significant binary digits... it may have processed only up to 23rd bit (or even less) and considered that as a good enough match (equal) to save processing time.
Same with conversion from none floating point memory locations, which can be considered as the sum of series 1/2^1 + 1/2^2 + ... + 1/2^n for bits that are turned on... it may consider tracing up to 1/2^15 as a good enough converted/approximate value... it may try to trace (find closest match) up to 1/2^23 or even more... the only way to check would be to extract the mantissa and exponent and derive the value using other means as using Debug.Print or MsgBox directly will not give you the actual value stored in memory.
So once again. Check the binary (mantissa-exponent) and not the Debug.Print outputs.
So once again. Check the binary (mantissa-exponent) and not the Debug.Print outputs.
So once again. Check the binary (mantissa-exponent) and not the Debug.Print outputs.
So once again. Check the binary (mantissa-exponent) and not the Debug.Print outputs.
-
Re: Number Precision: How to explain?
Quote:
w=v depends on the values, it can be True or False.
That is what I mentioned in post #5. And I only concern when it is True.
So once again, forget it! I don't care about mantissa-exponent here.
You cannot answer (or cannot understand) what I concern.
That is not a hard coded number like "0.7055475". With a hard coded number it will be easy to explain:
Absolutley you will never have (0.705547511577606 = 0.705547500000000) = True.
That is a value came from memory via Rnd() with some conversions, because not sure exactly what happens so I try to find an answer.
What I guess now is because of the declarations:
Code:
Dim d As Double
Dim v As Variant
Dim w As Variant
d is a real Double but w is a Variant with subtype Double lately.
They are not strictly the same data type.
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by anhn
I don't care about mantissa-exponent here.
You cannot answer (or cannot understand) what I concern.
Unfortunately your are essentially comparing string outputs, while VB and the CPU compares the bits... so your approach will really lead to nowhere nor give you clues on what to check next.
Quote:
Originally Posted by anhn
Absolutley you will never have (0.705547511577606 = 0.705547500000000) = True.
Your original code involves conversions, statement 0.705547511577606 = 0.705547500000000 is not True is not representative of what you were originally trying to do. cdbl(0.705547511577606) = csng(0.705547511577606) is True and cdbl(0.705547511577606) - csng(0.705547511577606) is not zero are more representative of what you were trying to do and gives light to the quirks of the platform. It should also be noted, or rather emphasized, that all your test values are derived from a Single.
EDIT: Found reference that in VB6 when comparing Single and Double, the Double is converted to Single (smaller data type prevails) which explains the unexpected behavior. Unsurprisingly, its platform related as already mentioned in post #11. Is that the answer you were looking for? Wait, let me rephrase that... Is that the answer you want to read?
-
Re: Number Precision: How to explain?
All this talk of 15 decimal digits vs 7 decimal digits!
I showed how 50 <> 50!
Quote:
Originally Posted by szlamany
Code:
.
.
.
Iteration 15 brings z up to 50
Not Equal to 50
Except for the final one - it's not actually equal to 50!! But it displays as 50.
@anhn - leinad31 has a really good point here.
You are formatting and displaying for DEBUG.PRINT values that are stored very differently then integer or string.
Then based on what you "see" of these formatted values you are making assumptions about equality - and when TRUE or FALSE doesn't follow your eyeball assumption you are asking why.
Is that essentially your question?
Well - I showed for display two values the DOUBLE VARIABLE that was mathematically aggregated back up to 50 - and that value being FALSE when compared to 50!
Exact same thing you had - just less digits.
Wouldn't you be interested in knowing how VB6 and/or the cpu actually compares floating point values for parity? Obviously it's not going on "every digit in the mantissa" - or this thread would not exist.
I guess leinad31 found the answer in the end - it converts a double to a single when comparing a double with another single. And that makes perfect sense - since floating point values are approximations anyway!
Which is why for 25 years I've had to force people coding for me to not use FLOAT for financial calculations! I had an old business partner that did an entire pension calculaton with floating point variables!
-
Re: Number Precision: How to explain?
Quote:
Originally Posted by anhn
Date is not the case of this. I know that 2 different Date values can have the same display value even with time.
That isn't what I meant, what I meant is that Date values are not displayed (via Watch window, etc) in the same way as they are stored - there is a completely hidden conversion to a String.
All I was saying is that the same kind of thing happens for other data-types too, but as most (such as Integer) are precise values, the conversion for them shows an accurate value - but the same is not true for floating point values (as szlamany's example proved).
Using Format doesn't help that much as it just adds formatting to the conversion process, it doesn't make the conversion itself more accurate.
-
Re: Number Precision: How to explain?
Guys, I understand what you mentioned. But that is not what I wanted to prove. You did a good job but proving the reverse way.
You proved 2 numbers with the same display but not equal to each other.
I accepted that and I also had a lot of examples of this kind. I fully aware of that. One of this kind is as an exercise:
How to build an accurate arrays of time slots during the day with interval of 10, 15 or 30 minutes using loop.
I want to show 2 equal numbers of the same data type (compared in memory: x=y) but when display (using the same format string) they show 2 different strings. That must not be the case. There is something else happened during comparison that I haven't find out. We might not understand each other.
I think it will be better if we can use CopyMemory to see exactly what is the value of each byte in memory for each number.
Anyway, thanks all for discussion.
-
Re: [RESOLVED] Number Precision: How to explain?
Quote:
Originally Posted by szlamany
I guess leinad31 found the answer in the end - it converts a double to a single when comparing a double with another single. And that makes perfect sense - since floating point values are approximations anyway!
I found the answer. There were some wrong assumptions.
1. On comparing a Single with a Double in v=d, the Double was converted to Single first
(matched with what szlamany said above, not the other way around as I thought and confirmed by si_the_geek).
2. With w=v and v=d we may not have w=d
(here w and d are Double and v is Single, because again both w and d were converted to Single first)
3. With w=v, we may not have w-v=0:
On subtracting, the Single was converted to Double first.
Code:
Single v = 0.705547500000000
Double d = 0.705547511577606
v=d : True
v-d=0 : True
Double w = 0.705547500000000
w=v : True
w-v=0 : False
w=d : False
w-d=0 : False
It is no longer surprise, with w=d is False, so the display strings of w and d are different.
(My strong point was if w=d is True then their display strings must be the same).
