Can anyone explain this? These values were taken from my Immediate window, and reflect what my program is doing ((ue) is a Double data type):
Whaaaaat?!Code:?ue
2
?str(ue)
2
?int(ue)
1
Printable View
Can anyone explain this? These values were taken from my Immediate window, and reflect what my program is doing ((ue) is a Double data type):
Whaaaaat?!Code:?ue
2
?str(ue)
2
?int(ue)
1
This misbehavior is probably a bug related to newer Windows version(s), as they're rare but known (because Microsoft tends from time to time to break something). Which one you're using?
what value is actually assigned or calculated into ue?
Seems very unlikely.Quote:
Originally Posted by MikiSoft
Since we haven't seen the code for all we know you have overridden VBA.Int() with your own Int() function in the current namespace.
here is the contents of my debug window
Whaaaaaaat ?????Code:?1+1
123
?2+2
234
?"treddie is a joker"
well,if you say so
System is Windows 7 Ultimate, 6.1.7600 Build 7600
The equation to get value (ue) is:
It then goes into:Code:ue = (bx / vx)
All variables are declared as Double.Code:qx = (Int(ue)) + 1
This problem is intermittent...Multiple passes come through this same code and the result is unpredictable. I thought maybe it has to do with vb showing ue = 2, when maybe it is actually extremely close to "2", as in "1.999999...9". Seems like I have seen this before, but it is rare. Now, in this program, it is happening a lot.
That was my assumption from the start.
Your current program must just be calculating things that are nearly a whole number, but not quite.
This is to be expected when using floating point numbers.
Code:'a is declared as a Double.
'in the immediate window
a = 1.999999999999995
?a
1.99999999999999
a = 1.999999999999996
?a
2
?int(a)
1
Passel...I think your test pretty much sums it up. The bummer is that Microsoft would let that ambiguity slip through. So vb is is displaying a rounded version of what it is actually using internally? Why not show the frickin float in its entirety?! That sounds like a vb bug.Quote:
Originally Posted by passel
Blame Intel...
Problem is that floats are by their very nature imprecise because digital computers cna't handle them well enough through binary representation... So what you're seeing is the computer's best guess at how it can store the number. It's the Pentium Approximation Syndrome.
And yet it's really only a problem depending on your scale.... further out it goes, the worse it gets... and then when you hit a repeating decimal like 2/3 ... it has to stop after some point, it can't keep recording decimal places infinitely.
There's a couple of sayings:
2+2=5 for extremely large values of 2
We are Pentium of Intel, rounding is futile, you will be approximated.
-tg
Lol!
I refuse to be approximated. "I am a man! Not a number!" -quote by #6, I believe.
Well, it ain't necessarily elegant, but I wrote this fault-tolerant Int() function, to address this vb issue:
Code:'Fault tolerant Int():
Dim ResultFlag As Byte
Dim TempLoop As Integer
Dim ue_str As String
Dim CharStr As String
Dim ue_int As Integer
ue_str = Str(ue)
ResultFlag = 0
For TempLoop = 1 To Len(ue_str)
CharStr = Mid(ue_str, TempLoop, 1)
If CharStr = "." Then 'Life is good...True float value is intact.
ResultFlag = 1
GoTo jumpover
End If
Next TempLoop
jumpover:
If ResultFlag = 0 Then 'True float value may be compromized.
ue_int = Val(ue_str)
ElseIf ResultFlag = 1 Then 'True float value is intact.
ue_int = Int(ue)
End If
Use Exit For instead of Goto jumpover and remove the label jumpover:
Good point. Thanks!
or the entire loop can be dismissed:
wether it is fault tolerant or not, i dare not sayCode:If InStr(ue_str, ".") Then
'contains "."
Else
'does not contain "."
End If
after all, you suppose a locale specific decimal separator (.)
where does that double come from ?
from a textbox filled in by a user ?
Great point. Much better solution.Quote:
or the entire loop can be dismissed:
Either from a text file that gets numerical data from an Adobe Illustrator file, or from textboxes that are protected from human errors by a checker function.Quote:
wether it is fault tolerant or not, i dare not say
after all, you suppose a locale specific decimal separator (.)
where does that double come from ?
from a textbox filled in by a user ?
Hm...This seems to be much better. Forget everything else, just use:
For more user control over where the accuracy "trigger" point should lie:Code:xx = Int(Val(Str(x)))
where (r) is the desired decimal place to round at.Code:xx = Int(Round(Val(Str(x)),r))
Without rounding, it appears that the trigger decimal place is different for numbers that have more digits in the integer portion of the float.
Not really good advice here...Quote:
Originally Posted by treddie
The line:
is truly horrible ... why not use Round directly on x?Code:xx = Int(Round(Val(Str(x)),r))
or in case x is within Int32-Range:Code:xx = Round(x)
or if xx is of type Long then implicit type-conversion might suffice as well:Code:xx = CLng(x) 'CLng() does rounding as well
OlafCode:xx = x 'also the implicit conversions (in case xx is an Integer-Type) do a rounding
True. I just added that as a mod to the original line. Personally, I don't think I would have any use for the second version, since the whole point is to leave the number alone at however many decimal places it contains (keep it untouched), unless it reaches the point of ambiguity where the result of Int (x) becomes unreliable.Quote:
is truly horrible ... why not use Round directly on x?
Thanks everyone for your help. I am marking this thread as resolved.