Overflow error into a Real variable (***RESOLVED***)

Code:

---

  Dim tst_Str As Byte, int_Temp As Double
  Dim T1 As Byte, T2 As Byte, OutVal As Double

  Open "a:\6-29 R2 after chelex LB=1Hz" For Binary Access Read Lock Read As #1
  int_Temp = 1
 
  While Not EOF(1)
     
      Get #1, int_Temp, T1
      int_Temp = int_Temp + 1
     
      OutVal = (256 * (T1 Xor 15)) + (T1 And 15)      <=====***
     
      Get #1, int_Temp, T2
      int_Temp = int_Temp + 1
   
  Wend

---

when T1 = 170 then OutVal is going to equal (256*160) + 10.
or 40970.

Why am I getting an Overflow error? I tried setting the type to
Long, same thing.

I know I'm doing something stupid, but what?

Because you have declared T1 variable as Byte which can hold only from 0 to 255

I ran into a similar problem several years ago. I don't reacall how to duplicate it though, but in your case I think it has to do with the fact that T1 is a Byte and only has a range of 0 to 255. It is not the Double OutVal that is getting the OverFlow error. Try doing the (T1 Xor 15)) + (T1 And 15) math first on seperate lines and storing them in an integer or a long. Then use that int or long in the OutVal equation.

Greg

That is true serge.. but run this command:

VB Code:

MsgBox (256 * (170 Xor 15)) + (170 And 15)

not dealing with bytes, just straigh forward math.. it still overflows

-mcd

From DerFarm:

Quote:

---

when T1 = 170 then OutVal is going to equal (256*160) + 10.
or 40970.

---

I replaced T1 with 170 in my above code.. 170 is not > 255.. so why is this happening

-mcd

Ok, I think this is it:

when I perform T1 AND 240 (assuming T1 to be a value < 256) I
get a byte result.

msgbox and the immdiate window both are trying to perform the
256*(T1 and 240) into an integer value.

the following code works perfectly:
OutVal = 205.1 * (T1 And 240) [32816]

This does not:
OutVal = 205 * (T1 And 240) [32800]

But This DOES:
OutVal = 204 * (T1 And 240) [32640]

Go figure.

When you make an operation between two integers, the value is limited to -32768 to 32767 (16 bit integer).

You can use

VB Code:

dim a as long
a=256
OutVal = (a * (T1 Xor 15)) + (T1 And 15)

It happens because VB is evaluating to the smallest data type.

Code:

---

Debug.Print (CDbl(256) * CDbl((170 Xor 15))) + (170 And 15)

---

Quote:

---

It happens because VB is evaluating to the smallest data type.

---

This is what I was getting at, I just couldn't remember the details. It happened to me about 5 years ago, and to be honest, I'm surprised the issue does not surface more often.

Greg