[RESOLVED] Whaaaat?! Int(x) doing something weird.

[treddie]

Can anyone explain this? These values were taken from my Immediate window, and reflect what my program is doing ((ue) is a Double data type):

?ue
 2 
?str(ue)
 2
?int(ue)
 1

Whaaaaat?!

[MikiSoft]

This misbehavior is probably a bug related to newer Windows version(s), as they're rare but known (because Microsoft tends from time to time to break something). Which one you're using?

[westconn1]

what value is actually assigned or calculated into ue?

[dilettante]

This misbehavior is probably a bug related to newer Windows version(s), as they're rare but known (because Microsoft tends from time to time to break something). Which one you're using?

Seems very unlikely.


Since we haven't seen the code for all we know you have overridden VBA.Int() with your own Int() function in the current namespace.

[IkkeEnGij]

here is the contents of my debug window

?1+1
 123 
?2+2
 234 
?"treddie is a joker"
well,if you say so

Whaaaaaaat ?????

[treddie]

System is Windows 7 Ultimate, 6.1.7600 Build 7600

The equation to get value (ue) is:

ue = (bx / vx)

It then goes into:

qx = (Int(ue)) + 1

All variables are declared as Double.

This problem is intermittent...Multiple passes come through this same code and the result is unpredictable. I thought maybe it has to do with vb showing ue = 2, when maybe it is actually extremely close to "2", as in "1.999999...9". Seems like I have seen this before, but it is rare. Now, in this program, it is happening a lot.

[passel]

That was my assumption from the start.
Your current program must just be calculating things that are nearly a whole number, but not quite.
This is to be expected when using floating point numbers.

'a is declared as a Double.

'in the immediate window

a = 1.999999999999995
?a
 1.99999999999999 

a = 1.999999999999996
?a
 2 
?int(a)
 1

[treddie]

That was my assumption from the start.
Your current program must just be calculating things that are nearly a whole number, but not quite.
This is to be expected when using floating point numbers.

'a is declared as a Double.

'in the immediate window

a = 1.999999999999995
?a
 1.99999999999999 

a = 1.999999999999996
?a
 2 
?int(a)
 1

Passel...I think your test pretty much sums it up. The bummer is that Microsoft would let that ambiguity slip through. So vb is is displaying a rounded version of what it is actually using internally? Why not show the frickin float in its entirety?! That sounds like a vb bug.

[techgnome]

Blame Intel...

Problem is that floats are by their very nature imprecise because digital computers cna't handle them well enough through binary representation... So what you're seeing is the computer's best guess at how it can store the number. It's the Pentium Approximation Syndrome.

And yet it's really only a problem depending on your scale.... further out it goes, the worse it gets... and then when you hit a repeating decimal like 2/3 ... it has to stop after some point, it can't keep recording decimal places infinitely.

There's a couple of sayings:
2+2=5 for extremely large values of 2


We are Pentium of Intel, rounding is futile, you will be approximated.


-tg

[treddie]

Lol!
I refuse to be approximated. "I am a man! Not a number!" -quote by #6, I believe.

[treddie]

Well, it ain't necessarily elegant, but I wrote this fault-tolerant Int() function, to address this vb issue:

  'Fault tolerant Int():
  
  Dim ResultFlag As Byte
  Dim TempLoop As Integer
  Dim ue_str As String
  Dim CharStr As String
  Dim ue_int As Integer
  
  ue_str = Str(ue)
  
  ResultFlag = 0
  
  For TempLoop = 1 To Len(ue_str)
    CharStr = Mid(ue_str, TempLoop, 1)
    
    If CharStr = "." Then    'Life is good...True float value is intact.
      ResultFlag = 1
      GoTo jumpover
    End If

  Next TempLoop
  
jumpover:

  If ResultFlag = 0 Then  'True float value may be compromized.
    ue_int = Val(ue_str)
  ElseIf ResultFlag = 1 Then  'True float value is intact.
    ue_int = Int(ue)
  End If

[jmsrickland]

Use Exit For instead of Goto jumpover and remove the label jumpover:

[treddie]

Good point. Thanks!

[IkkeEnGij]

or the entire loop can be dismissed:

  If InStr(ue_str, ".") Then
     'contains "."
  Else
     'does not contain "."
  End If

wether it is fault tolerant or not, i dare not say
after all, you suppose a locale specific decimal separator (.)
where does that double come from ?
from a textbox filled in by a user ?

[treddie]

or the entire loop can be dismissed:

Great point. Much better solution.

wether it is fault tolerant or not, i dare not say
after all, you suppose a locale specific decimal separator (.)
where does that double come from ?
from a textbox filled in by a user ?

Either from a text file that gets numerical data from an Adobe Illustrator file, or from textboxes that are protected from human errors by a checker function.

[treddie]

Hm...This seems to be much better. Forget everything else, just use:

xx = Int(Val(Str(x)))

For more user control over where the accuracy "trigger" point should lie:

xx = Int(Round(Val(Str(x)),r))

where (r) is the desired decimal place to round at.

Without rounding, it appears that the trigger decimal place is different for numbers that have more digits in the integer portion of the float.

[Schmidt]

Hm...This seems to be much better. Forget everything else, just use:

xx = Int(Val(Str(x)))

For more user control over where the accuracy "trigger" point should lie:

xx = Int(Round(Val(Str(x)),r))

Not really good advice here...

The line:

  xx = Int(Round(Val(Str(x)),r))

is truly horrible ... why not use Round directly on x?

  xx = Round(x)

or in case x is within Int32-Range:

  xx = CLng(x) 'CLng() does rounding as well

or if xx is of type Long then implicit type-conversion might suffice as well:

  xx = x 'also the implicit conversions (in case xx is an Integer-Type) do a rounding

Olaf

[treddie]

is truly horrible ... why not use Round directly on x?

True. I just added that as a mod to the original line. Personally, I don't think I would have any use for the second version, since the whole point is to leave the number alone at however many decimal places it contains (keep it untouched), unless it reaches the point of ambiguity where the result of Int (x) becomes unreliable.

[treddie]

Thanks everyone for your help. I am marking this thread as resolved.