[RESOLVED] Interesting: 65535 and &HFFFF

[anhn]

Can anyone say something about this behaviour in VB:
Hex(65535) = "FFFF" but &HFFFF = -1

Option Explicit

Sub Test()
   Dim lngX As Long
   Dim lngY As Long
   
   lngX = 65535
   lngY = &HFFFF
   
   Debug.Print lngX, Hex(lngX)
   Debug.Print lngY, Hex(lngY)
End Sub
------------------------------
 65535        FFFF
-1            FFFFFFFF

[LaVolpe]

Try this: Debug.Print &HFFFF, &HFFFF&
One is an Integer and one is a Long, depending on how you declare it. Remember that VB will treat any value within the range of an Integer as an Integer if it is not explicity delcared as some other variable type.

Likewise: Hex(-1&) & Hex(-1) will produce different results as one is long and one is integer. -1 Long is &HFFFFFFFF and -1 Integer is &HFFFF

[anhn]

LaVolpe,
That is the answer as I expected.
Although lngY was declared as Long but &HFFFF is not.

[LaVolpe]

LaVolpe,
That is the answer as I expected.
Although lngY was declared as Long but &HFFFF is not.

Yes, but you were printing Hex(lngY) which is long. When you set lngY=&HFFFF, VB converted the 2-byte integer &HFFFF to a Long of &HFFFFFFFF, 4-bytes. The result is -1 either way.

[anhn]

What I mean is for safety I should write:

lngY = &HFFFF&

if I want to say 65535.

[LaVolpe]

Bingo. Keep this in mind when you use APIs too. As most API constants out there are hex, depending on where you get them, they may or may not be "typed" correctly. A common example: &H8000 and &H8000& (2 diff values). Which is really the constant's value? Annoying -- research may be needed to determine that.

[Lord Orwell]

I personally think it's extremely annoying how a positive hex number can be interpreted as a negative decimal. But that's vb for you. I originally came across this issue designing a program to encode data inside a 24-bit bitmap by replacing the low bit on every byte with a bit of data. (basically a 1MB image could hold 125k of data with no noticable visual difference in quality)

[Merri]

VB appears to go for the smallest datatype after math:

&HFFFF0000 \ &H10000

Result? Integer -1

Thus to workaround:

(&HFFFF0000 \ &H10000) And &HFFFF&

And we're getting the expected result, 65535.

Option Explicit

Private Sub Form_Load()
    Dim lngVal As Long, int1 As Integer, int2 As Integer, lng1 As Long, lng2 As Long
    
    lngVal = &HAABBCCDD
    
    ' To split unsigned Long to signed Integer
    int1 = (lngVal \ &H10000)
    int2 = (lngVal And &H7FFF) Or (((lngVal And &H8000) <> 0) * 32768)
    
    ' To split unsigned Long to unsigned Integer (= Long)
    lng1 = (lngVal \ &H10000) And &HFFFF&
    lng2 = lngVal And &HFFFF&
    
    ' output
    Debug.Print int1, int2, Hex$(int1), Hex$(int2)
    Debug.Print lng1, lng2, Hex$(lng1), Hex$(lng2)
End Sub

Complex enough to follow? If you'd do anything in other way with int2, you'd get an overflow error. This is because VB doesn't always go for the smallest datatype, so you have to work around the problem in creative ways.

[LaVolpe]

VB appears to go for the smallest datatype after math:

&HFFFF0000 \ &H10000

Result? Integer -1

Merri, I got a Long return type. When doing straight math (no conversion functions used, ie, CInt, etc), I believe VB will return the value as the VarType of the largest VarType used in the calculation.

To test my idea, I used the following

? (VarType(&HFFFF0000 \ &H10000)=vbLong)
' ^^ 2 longs 
? (VarType(&HFFFF0000 \ &H100)=vbLong)
' ^^ 1 long, 1 vb integer (2 bytes)

Edited: Not true? What logic VB uses is anyone's guess. Take a look at this:

? (VarType(&HFFFF * 1#)=vbDouble), (VarType(&HFFFF \ 1#)=vbLong)
' 1st calc: Int * Double = Double
' 2nd calc: Int \ Double = Long 
' ^^ I expected Integer since Int division used & numerator is Int, 
' but maybe VB is converting Double to Long because of Int division?

[leinad31]

Pre-.Net int div and mod operators are limited to long returning expressions as operands... there's no overloading in VB6.

[Merri]

LaVolpe: MsgBox &HFFFF0000 \ &H10000 gives you -1, which implies the result has been, at some point, coerced into an Integer. Of course, VarType gives you different results, it handles the end value before any coercion has occurred.

However, what is really interesting is that if you store the result into a Long... lngResult = &HFFFF0000 \ &H10000 - a coercion occurs! It goes from Long to Integer and back to Long. Using CLng doesn't help, the coercion happens when storing the value to Long. What I don't understand here is that it doesn't always do it for every possible value below 65536. Does to coercion occur because of certain math was being used? Or is there some logic that I fail to figure out?

[westconn1]

my 2 penneth

?.07*100=7
False

[anhn]

@westconn1
Your example is a different problem, that relates to calculation of floating point numbers.
This error is probably caused by hardward (CPU) and/or software (VB) due to the difficulty of conversion back and fort between decimal numbers and binary numbers (only the laters can be understanded by arithmetic processing unit).
There is no way to represent exactly 0.7 in binary system. With a certain number of bits used, it must have an error up to a known limit.

? (0.07 * 100)=7
False
? CInt(0.07 * 100)=7
True
? CLng(0.07 * 100)=7
True
? CSng(0.07 * 100)=7
True
? CDbl(0.07 * 100)=7
False
? CDec(0.07 * 100)=7
True

[Merri]

To find out how much difference there actually is:

MsgBox Format$((0.07 * 100) - 7, "0.000000000000000000000000000000")

Of course even that value is prone to floating point math errors. Just tells you how important it is to handle data always in the correct datatype.

[Milk]

&HFFFF0000 \ &H10000

Hold on a minute am I missing something? an answer of -1 implies they are both processed as long... &HFFFF0000= -65536, &H10000= 65536, &H10000 is beyond the range of an integer.

[Merri]

You're not missing anything, I guess. Both are Long and you can do that kind of math with hex numbers: they are actually being processed unsigned.

Lets review this thing step by step...

1) lngValue = &HFFFF0000 \ &H10000
2) Expected result: &HFFFF&
3) Result in lngValue: -1& = &HFFFFFFFF

4) lngValue = (&HFFFF0000 \ &H10000) And &HFFFF&
5) Expected result: &HFFFF&
6) Result in lngValue: 65535 = &HFFFF&

[Milk]

I still don't get what you mean (I am very slow at times) as far as I can tell the processing is done signed but the hex displays literal bits. I would expect &HFFFF0000 \ &H10000 to equal &HFFFFFFFF and not &HFFFF& because I know any value of &H80000000 and above to be -2147483648 plus the previous 31 bits.

This is actually quite relevant for me at the moment because I'm trying to find the way of performing unsigned 32bit integer division using the signed long data type, effectively manually dividing the last bit. It's easy using a double but I'm trying to avoid coercion.

incidentally figure this...
I can use A = &H80000000, no problem but as soon as I try A = -2147483648& it won't have it and wants it as a double even though it represents the same number.

[anhn]

Haha! Haha! It seems to be funny: I found a post in another forum here, he said he wants &HFFFF equal to -1 but why that gives him the value of 65535. Another way around here but this case is in VB.NET environment.

Thank you all guys for your participations in this dicussion.

At first, I created this thread to raise an awareness of the important of explicitly type declaration even with constants. If we don't understand what really happens, sometimes we won't get an error but we may get wrong results, who knows, it may lead to a disaster such as to blow up a shuttle.

Microsoft support site has this KB article, everyone should read:
&H8000 to &HFFFF Hex = -32,768 to -1, Affects LONG Bit Masking

By googling "&HFFF &HFFFF&" you may see about 1300 results.

I mark this thread Resolved now.