-
May 11th, 2015, 02:28 PM
#1
[RESOLVED] Pointer Arithmetic
While we have convenient "undocumented" hidden functions like VarPtr(), StrPtr(), and ObjPtr() that return a Long... sometimes you need to add or subtract offsets.
Since these are really ULong values, we get away with simple Long arithmetic up until the point where some of the pointers go negative. My question is:
Has the VB6 community has ever arrived at a best practice for handling pointer offset arithmetic properly?
"Properly" obviously means dealing with the "bit 31 on" or "negative" issue. But I'd probably go further and want it to be fast and minimal coding as well.
I can't think of any API call that does this, but an obvious possibility involves a pair of UDTs that split up a Currency variable along with LSet statements to move data in and out. The old MS KB article How To Convert Between Signed and Unsigned Numbers suggests running the data through Double typed variables, and extrapolating from the code there a simple "PointerAdd" function could be written:
Code:
Function UnsignedAdd(ByVal V1 As Long, ByVal V2 As Long) As Long
Const OFFSET_4 = 4294967296#
Const MAXINT_4 = 2147483647
Dim D As Double
If V1 >= 0 Then D = V1 Else D = V1 + OFFSET_4
If V2 >= 0 Then D = D + V2 Else D = D + (V2 + OFFSET_4)
If D < 0 Or D >= OFFSET_4 Then Error 6 'Overflow.
If D <= MAXINT_4 Then UnsignedAdd = D Else UnsignedAdd = D - OFFSET_4
End Function
Still, that looks awkward, and a lot more code than you'd want to use inline rather than as a function. But do we have better alternatives?
Is there any advantage to using Currency instead of Double as the intermediate type?
-
May 11th, 2015, 02:58 PM
#2
Re: Pointer Arithmetic
For small offsets, the following inline, unsigned pointer addition/subtraction works pretty well:
Code:
Public Const SIGN_BIT As Long = &H80000000
? "&H"; Hex$((&H7FFFFFFF Xor SIGN_BIT) + 1& Xor SIGN_BIT)
&H80000000
? "&H"; Hex$((&H80000000 Xor SIGN_BIT) - 1& Xor SIGN_BIT)
&H7FFFFFFF
Unfortunately though, it fails when adding/subtracting large offsets (I haven't bothered determining exact value where it starts to fail).
On Local Error Resume Next: If Not Empty Is Nothing Then Do While Null: ReDim i(True To False) As Currency: Loop: Else Debug.Assert CCur(CLng(CInt(CBool(False Imp True Xor False Eqv True)))): Stop: On Local Error GoTo 0
Declare Sub CrashVB Lib "msvbvm60" (Optional DontPassMe As Any)
-
May 11th, 2015, 03:09 PM
#3
Re: Pointer Arithmetic
That looks promising, any guesses regarding what "large offsets" might be? Examples that fail?
-
May 11th, 2015, 03:25 PM
#4
Re: Pointer Arithmetic
Just a dumb question? Has anyone seen a pointer that 'goes negative'? I know it is theoretically possible.
FYI. I've cheated in the past when I felt maybe this might be an issue. The cheat involved overlaying an array (SafeArray Structure) of n-bytes over the pointer. Works well enough at a small overhead cost IMO. Likely not suitable in all scenarios.
Edited: The assumption of course, is that VB knows how to access the data once the high bit is needed. However, you would be referencing the data at pointer xyz as array item 0 (or whatever LBound you chose).
Last edited by LaVolpe; May 11th, 2015 at 03:42 PM.
-
May 11th, 2015, 04:24 PM
#5
Re: Pointer Arithmetic
Originally Posted by dilettante
... any guesses regarding what "large offsets" might be? Examples that fail?
Here's a couple of extreme examples:
Code:
? "&H"; Hex$((&H1 Xor SIGN_BIT) + &HFFFFFFFE Xor SIGN_BIT) '<-- Throws RTE 6: Overflow
'Result should've been &HFFFFFFFF
? "&H"; Hex$((&HFFFFFFFF Xor SIGN_BIT) - &HFFFFFFFE Xor SIGN_BIT) '<-- Throws RTE 6: Overflow
'Result should've been &H1&
I have never encountered a need to add/subtract huge offsets from pointer values so I've never really bothered to figure out the exact range of bad values.
Originally Posted by LaVolpe
Has anyone seen a pointer that 'goes negative'? I know it is theoretically possible.
It would be possible if the executable was marked /LARGEADDRESSAWARE and either 4-Gigabyte Tuning was enabled or the 32-bit app is running under a 64-bit Windows OS.
On Local Error Resume Next: If Not Empty Is Nothing Then Do While Null: ReDim i(True To False) As Currency: Loop: Else Debug.Assert CCur(CLng(CInt(CBool(False Imp True Xor False Eqv True)))): Stop: On Local Error GoTo 0
Declare Sub CrashVB Lib "msvbvm60" (Optional DontPassMe As Any)
-
May 11th, 2015, 04:29 PM
#6
Re: Pointer Arithmetic
Originally Posted by Bonnie West
It would be possible if the executable was marked /LARGEADDRESSAWARE and either 4-Gigabyte Tuning was enabled or the 32-bit app is running under a 64-bit Windows OS.
Ah, makes sense. Thanx
-
May 11th, 2015, 06:40 PM
#7
Re: Pointer Arithmetic
Yeah, that's where I ran into it just the other day (/LARGEADDRESSAWARE).
Thanks for the input!
-
May 11th, 2015, 07:43 PM
#8
Re: Pointer Arithmetic
I place Pointer-Arithmetic usually only in compiled Dlls.
Dlls, which I always compile to Native-Code (with all extended Options checked,
except the first one when I deal with SafeArrays in my Dlls).
One of the checked-in Options include (besides other speed-relevant ones):
- "No checking for Integer-Overflow"
- ...
and thus one is safe with directly applied Pointer-Additions, Subtractions in this case
of native compiled Binaries.
Here's a small snippet, so that you can try it out yourselves.
Code:
Option Explicit
Private Sub Form_Click()
Dim BasePtr As Long
Cls
BasePtr = &H7FFFFFFF
Print "BasePtr", "Hex: "; Hex(BasePtr); ", Dec: "; BasePtr
BasePtr = BasePtr + 1 'increment by one
Print "Incr. by 1", "Hex: "; Hex(BasePtr); ", Dec: "; BasePtr
BasePtr = BasePtr + 1 'increment by one
Print "Incr. by 1", "Hex: "; Hex(BasePtr); ", Dec: "; BasePtr; " ... a.s.o."
Print
BasePtr = BasePtr - 1 'decrement by one
Print "Decr. by 1", "Hex: "; Hex(BasePtr); ", Dec: "; BasePtr
BasePtr = BasePtr - 1 'decrement by one
Print "Decr. by 1", "Hex: "; Hex(BasePtr); ", Dec: "; BasePtr
BasePtr = BasePtr - 1 'decrement by one
Print "Decr. by 1", "Hex: "; Hex(BasePtr); ", Dec: "; BasePtr; " ... a.s.o."
End Sub
As said, works only natively compiled (including at least the "no check for Integer-Overflow"-Option)...
Running the compiled executable and clicking the Form will then produce the following output:
Olaf
-
May 12th, 2015, 01:33 AM
#9
Re: [RESOLVED] Pointer Arithmetic
You should use this approach: In IDE use currency for INT64 arithmetic, and in the compiled file (only Native, Ignore integer overflow check) this:
VB Code:
Option Explicit Private Type int64cx d As Currency End Type Private Type int64lng l As Long h As Long End Type Private Sub Form_Load() Dim inIde As Boolean Dim index As Long Dim ret As Long Debug.Assert MakeTrue(inIde) For index = 0 To 100 If inIde Then Dim cx As int64cx Dim l As int64lng l.h = 0 LSet cx = l cx.d = cx.d * 137687 + 12.3467@ LSet l = cx ret = l.l Else ' Only this code compiled ret = ret * 137687 + 123467 End If Me.Print ret Next End Sub Private Function MakeTrue(value As Boolean) As Boolean MakeTrue = True value = True End Function
Result very fast:
-
May 12th, 2015, 05:46 AM
#10
Re: Pointer Arithmetic
Originally Posted by dilettante
Yeah, that's where I ran into it just the other day (/LARGEADDRESSAWARE).
Thanks for the input!
I was bitten by this some time ago when I got Overflow errors in our log files. Had to painfully edit every place where I previously blissfully used `lPtr + 3` or `VarPtr(vArray) + 8` for instance to call `UnsignedAdd(VarPtr(vArray), 8)`
I had a custom implementation of `UnsignedAdd` that used `Currency` or `On Error Goto ...` or both but then couple of moths ago stole Krool's implementation from this forum
Code:
Public Function UnsignedAdd(ByVal Start As Long, ByVal Incr As Long) As Long
UnsignedAdd = ((Start Xor &H80000000) + Incr) Xor &H80000000
End Function
Works for negative `Incr` too. Obviously `Incr` cannot be above 2GB while `Start` can and is treated as unsigned ptr.
cheers,
</wqw>
-
Sep 14th, 2015, 05:54 PM
#11
Re: Pointer Arithmetic
Originally Posted by Bonnie West
Here's a couple of extreme examples:
Code:
? "&H"; Hex$((&H1 Xor SIGN_BIT) + &HFFFFFFFE Xor SIGN_BIT) '<-- Throws RTE 6: Overflow
'Result should've been &HFFFFFFFF
? "&H"; Hex$((&HFFFFFFFF Xor SIGN_BIT) - &HFFFFFFFE Xor SIGN_BIT) '<-- Throws RTE 6: Overflow
'Result should've been &H1&
I have never encountered a need to add/subtract huge offsets from pointer values so I've never really bothered to figure out the exact range of bad values.
FYI. Curiosity brought this thread to mind. I think I have a safe solution that can add/subtract, using any/all of the 32 bits without the overflow errors...
Code:
Const SIGN_BIT = &H80000000
Private Function pvSafePointerAdd(thePointer As Long, AmountToAdjust As Long) As Long
Dim xAdj As Long
If AmountToAdjust < 0& Then xAdj = SIGN_BIT
pvSafePointerAdd = ((thePointer Xor SIGN_BIT) + (AmountToAdjust Xor xAdj)) Xor (SIGN_BIT Xor xAdj)
End Function
Private Function pvSafePointerSubtract(thePointer As Long, AmountToAdjust As Long) As Long
Dim xAdj As Long
If AmountToAdjust < 0& Then xAdj = SIGN_BIT
pvSafePointerSubtract = ((thePointer Xor SIGN_BIT) - (AmountToAdjust Xor xAdj)) Xor (SIGN_BIT Xor xAdj)
End Function
Edited: of course error handling still needed should some math exceed 32 bits, i.e., value > 4,294,967,295
Last edited by LaVolpe; Sep 14th, 2015 at 06:07 PM.
-
Sep 14th, 2015, 07:31 PM
#12
Frenzied Member
Re: Pointer Arithmetic
Originally Posted by LaVolpe
FYI. Curiosity brought this thread to mind. I think I have a safe solution that can add/subtract, using any/all of the 32 bits without the overflow errors...
Code:
Const SIGN_BIT = &H80000000
Private Function pvSafePointerAdd(thePointer As Long, AmountToAdjust As Long) As Long
Dim xAdj As Long
If AmountToAdjust < 0& Then xAdj = SIGN_BIT
pvSafePointerAdd = ((thePointer Xor SIGN_BIT) + (AmountToAdjust Xor xAdj)) Xor (SIGN_BIT Xor xAdj)
End Function
Private Function pvSafePointerSubtract(thePointer As Long, AmountToAdjust As Long) As Long
Dim xAdj As Long
If AmountToAdjust < 0& Then xAdj = SIGN_BIT
pvSafePointerSubtract = ((thePointer Xor SIGN_BIT) - (AmountToAdjust Xor xAdj)) Xor (SIGN_BIT Xor xAdj)
End Function
Edited: of course error handling still needed should some math exceed 32 bits, i.e., value > 4,294,967,295
Previously I used your code to allocate new memory,what is the difference between above code and this:
Code:
Private Function SafeOffset(ByVal ptr As Long, Offset As Long) As Long
' ref http://support.microsoft.com/kb/q189323/ ' unsigned math
' Purpose: Provide a valid/safe pointer offset. Primarily for use with CopyMemory
' If a pointer +/- the offset wraps around the high bit of a long, the
' pointer needs to change from positive to negative or vice versa.
' A return of zero indicates the offset exceeds the min/max unsigned long bounds
' Zero is an invalid memory address
Const MAXINT_4NEG As Long = -2147483648# ' min value of a Long
Const MAXINT_4POS As Long = 2147483647 ' max value of a Long
If Not ptr = 0& Then ' zero is an invalid pointer
If Offset < 0& Then ' subtracting from pointer
If ptr < MAXINT_4NEG - Offset Then
' wraps around high bit (backwards) & changes to Positive from Negative
SafeOffset = MAXINT_4POS - ((MAXINT_4NEG - ptr) - Offset - 1&)
ElseIf ptr > 0 Then ' verify pointer does not wrap around 0 bit
If ptr > -Offset Then SafeOffset = ptr + Offset
Else
SafeOffset = ptr + Offset
End If
Else ' Adding to pointer
If ptr > MAXINT_4POS - Offset Then
' wraps around high bit (forward) & changes to Negative from Positive
SafeOffset = MAXINT_4NEG + (Offset - (MAXINT_4POS - ptr) - 1&)
ElseIf ptr < 0& Then ' verify pointer does not wrap around 0 bit
If ptr < -Offset Then SafeOffset = ptr + Offset
Else
SafeOffset = ptr + Offset
End If
End If
End If
End Function
Suddenly pop up in my mind: Can we use BigInteger for computing?
-
Sep 14th, 2015, 07:39 PM
#13
Re: [RESOLVED] Pointer Arithmetic
Jonney be worth testing with the values that Bonnie posted earlier that fail using her logic (and others that use similar logic). If the code I posted years ago work, great. Then it's just a matter of CPU cycles I'd imagine. Faster/safest code should always be used for that type of function IMO, as it is likely to be used in a loop.
Edited: One major difference is that that SafeOffset doesn't allow passing POSITIVE offsets that include the high bit, i.e., an offset of +2147483648 and greater since then the offset would be treated as a negative.
Last edited by LaVolpe; Sep 14th, 2015 at 07:43 PM.
-
Sep 14th, 2015, 08:13 PM
#14
Frenzied Member
Re: [RESOLVED] Pointer Arithmetic
Originally Posted by LaVolpe
Jonney be worth testing with the values that Bonnie posted earlier that fail using her logic (and others that use similar logic). If the code I posted years ago work, great. Then it's just a matter of CPU cycles I'd imagine. Faster/safest code should always be used for that type of function IMO, as it is likely to be used in a loop.
Edited: One major difference is that that SafeOffset doesn't allow passing POSITIVE offsets that include the high bit, i.e., an offset of +2147483648 and greater since then the offset would be treated as a negative.
I see.
I haven't yet tested at particular situation. I set the Number of Rows to 10000000 and Cols = 5, Both code are OK without failure. In Task Management, Memory occupied only 287MB. But my ReDimPreserve2DArray failed with insufficient Memory at ReDim dummy(nNewFirstUBound, nNewLastUBound), Headache is that VB6 not allow to Redim Preserve for First Dim of 2D array:
Code:
Private Sub ReDimPreserve2DArray(ArrayToPreserve() As Long, nNewFirstUBound As Long, nNewLastUBound As Long)
'When you use Preserve to redim an dynamic multidimensional array you can only resize the last dimension
'and can not change the lower bounds. If you try to will generate a runtime error 9: subscript out of range.
'For example:
'Dim Data() As Long,ReDim Data(4,6),...,ReDim Data(24,6) will generate runtime error 9
'But If you resize the last dimension such as ReDim Data(4,26), it is OK.
' http://www.vbforums.com/showthread.php?782809-ReDim-Preserve-to-a-2D-(Multi-Dimensional)-Array-in-Visual-Basic-6
If Not ArrayIsNotEmpty(ArrayToPreserve) Then Exit Sub
Dim dummy() As Long
ReDim dummy(nNewFirstUBound, nNewLastUBound)
Dim i As Long
Dim cLen As Long
'(That is because the the leftmost index values will be together in memory, so will be essentially a linear copy of memory located together.)
cLen = (UBound(ArrayToPreserve, 1) + 1) * 4
For i = 0 To UBound(ArrayToPreserve, 2)
CopyMemory dummy(0, i), ArrayToPreserve(0, i), cLen
Next i
ArrayToPreserve = dummy
End Sub
Edited: go back to my old question, what is the best work around for Redim Preserve for 2D array? If we can get rid of dummy() array, the insufficient memory may not trigger.
Last edited by Jonney; Sep 14th, 2015 at 08:16 PM.
-
Sep 14th, 2015, 08:59 PM
#15
Re: [RESOLVED] Pointer Arithmetic
You can't redim the 1st dimension of a 2D array because of the way the memory is written to memory. Much like an Excel sheet. The 1st dimension are the columns and the 2nd dimension are the rows. If you have 100 rows and 20 columns, then each of the column values are written in contiguous memory for the 1st row, next 20 columns for next row and so on. Adding another column requires inserting a new memory block in front of rows 2-100 rows and one after row 100. Hope that gives a good visual.
This is off topic and if you ant to pursue workarounds and/or different methods of storing arrayed data, might be worthy of your own thread.
Last edited by LaVolpe; Sep 14th, 2015 at 09:23 PM.
-
Sep 14th, 2015, 09:17 PM
#16
Frenzied Member
Re: [RESOLVED] Pointer Arithmetic
Originally Posted by LaVolpe
You can't redim the 1st dimension of a 2D array because of the way the memory is written to memory. Much like an Excel sheet. The 1st dimension are the columns and the 2nd dimension are the rows. If you have 100 rows and 20 columns, then each of the column values are written in contiguous memory for the 1st row, next 20 columns for next row and so on. Adding another column requires inserting a new memory block in front of each of the 19 rows and one after the last row. Hope that gives a good visual.
This is off topic and if you ant to pursue workarounds and/or different methods of storing arrayed data, might be worthy of your own thread.
OK. I opened a new thread.
-
Sep 15th, 2015, 01:32 AM
#17
Re: Pointer Arithmetic
Originally Posted by LaVolpe
I think I have a safe solution that can add/subtract, using any/all of the 32 bits without the overflow errors...
Looks good! Thanks for that!
On Local Error Resume Next: If Not Empty Is Nothing Then Do While Null: ReDim i(True To False) As Currency: Loop: Else Debug.Assert CCur(CLng(CInt(CBool(False Imp True Xor False Eqv True)))): Stop: On Local Error GoTo 0
Declare Sub CrashVB Lib "msvbvm60" (Optional DontPassMe As Any)
-
Oct 8th, 2024, 03:35 AM
#18
Re: Pointer Arithmetic
Originally Posted by Schmidt
Dlls, which I always compile to Native-Code (with all extended Options checked,
except the first one when I deal with SafeArrays in my Dlls).
I know this is an old thread but this is rather intriguing. Could explain how the first optimization option (Assume No Aliasing) affects SafeArray operations?
-
Oct 8th, 2024, 07:48 AM
#19
Re: Pointer Arithmetic
Originally Posted by VanGoghGaming
I know this is an old thread but this is rather intriguing. Could explain how the first optimization option (Assume No Aliasing) affects SafeArray operations?
Assume you have (in a Class) the following Helper-Vars for "array-mapping" (defined at Class-Level):
Private WChars() As Integer, saWChars As SafeArray1D
Then you can (in Class-Initialize) already ensure all the "Binding-necessities"
(properly setting all the saWChars-Members and also already bind the WChars-Array).
This will make your real routines faster, because you then only have to set:
saWChars.pvData = StrPtr(SomeString)
...and your (already bound) WChars-Array will reflect the right Char-Values.
Now, if you change the saWChars.pvData-member to different Strings(different StringPointers - e.g. when looping):
- "Assume No Aliasing" left unchecked will always work properly
.. (just by changing the .pvData-Member - WChars() will reflect the new String-Data)
- "Assume No Aliasing" checked, will often not work
.. (it often still reflects the previous Data in the WChars()-Array)
Only observable when native compiled of course (Aliasing-optimizations are not available in IDE/PCode).
Olaf
-
Oct 9th, 2024, 07:44 AM
#20
Re: [RESOLVED] Pointer Arithmetic
I wasn't able to reproduce this, it works as expected every time. I guess you would need some complex conditions to match in order to reproduce this issue (like passing the pvData member as a parameter to a function or something but that still works as expected so I don't know).
-
Oct 9th, 2024, 08:15 AM
#21
Re: [RESOLVED] Pointer Arithmetic
Originally Posted by VanGoghGaming
I wasn't able to reproduce this, it works as expected every time. I guess you would need some complex conditions to match in order to reproduce this issue (like passing the pvData member as a parameter to a function or something but that still works as expected so I don't know).
What did you test?
There must be two String parameters. You must conditionally set saWChars.pvData = StrPtr(SomeString) between these two *and* modify one of the Strings, not both, so the compiler thinks second one is not touched and reuses StrPtr(SecondString) from a register or similar.
Finally at callsite for both String actual arguments must use the same variable for aliasing to happen.
cheers,
</wqw>
-
Oct 9th, 2024, 09:00 AM
#22
Re: [RESOLVED] Pointer Arithmetic
I think I kinda get what you're saying but I find all this aliasing concept very confusing to wrap my head around... The MSDN page about it doesn't help either, it shows a useless example where a function is called with a global parameter and then refers it using both the global name and the parameter name, nobody does that in practice...
-
Oct 9th, 2024, 12:51 PM
#23
Re: [RESOLVED] Pointer Arithmetic
Every ByRef param is susceptible to this optimization. Say you have a ByRef String and an UDT (always ByRef). When you call the procedure you can pass an instance of the UDT and again a String from this UDT i.e. Call MyProc(info.Name, info). This way you can access the same BSTR in memory using two symbols inside the procedure. This prevents the compiler from optimizing the access to actual data by each one of these symbols using a temp pointer or a CPU register unless it assumes there is no aliasing from such ByRef params. Only the developer can guarantee/fix there is no aliasing so the compiler can emit optimal codegen.
-
Oct 9th, 2024, 01:53 PM
#24
Re: [RESOLVED] Pointer Arithmetic
Okay but from Olaf's reply I understood that this behavior is unpredictable, sometimes it works and other times it doesn't. I'm sure Olaf, as the developer, could have taken appropriate measures to ensure aliasing doesn't happen. So the conclusion would be that this optimization is unreliable even when we are careful about what ByRef parameters are passed to our functions.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|