Array optimization - String or Integer?

My app is currently running pretty good, but I'm trying to figure out if my current setup can be furthermore optimized.

I have a database table containing only 2 columns, an X and an Y coordinate. At the beginning of my application, I use a recordset to transfer all those coordinates in an array. I didn't feel like simply using the recordset because I felt that dealing with a matrix/array would make things MUCH quicker. (Ain't I right?) I'll be browsing that huge array back and forth many times.

That database/array might end up containing a million records. YES it'll take a while at startup to fill the array - but that's not a big problem for me.. The app will afterward work with it for 2-3 days until I ask it to stop ( or unless I figure out I made a programming mistake and the memory-ressources gets eaten up too quick :o )

(please feel free to comment any of these techniques - I'm open to all good suggestions)

Another concern would be the array itself: it is declared as a STRING array. Now, if we nevermind the "Why use string if it only contains X,Y integer data???" - Will using a string array make things slower than using an integer array? I've been suggested this and I'd like to have some clarifications. Maybe does it only apply to string variables and such?

Thank you guys for the feedback.

Quote:

---

Will using a string array make things slower than using an integer array?

---

You bet it will!

The conversion from string to integer every time you make use of a value will take forever. If they aren't already integers in the database, convert the columns to a numeric data type and save tons of time. At the very least convert them when you read them in.

As you are using the data repeatedly, using an array (rather than a recordset) is a good idea - it should be much quicker.

As to it containing a million records, you may find that it uses up all of your spare physical memory, at which point your computer will become dramatically slower. Whether this is an issue or not depends on the size of the data, and the physical memory you have spare at any point when the program runs.


As to the data type of the array.. think what you would use if you had lots of separate variables instead, as an array is basically a group of variables.

To hold numeric data, Strings will use more space than the proper data types (any Integer takes 2 bytes, but a string takes 10 bytes plus the length of the string - so "99" will take 12 bytes [10 extra], and "32000" will take 15 [13 extra]). Assuming a million rows of just two values, that would be between 20MB and 30MB as string data, but only 4MB as numeric data.

Strings in VB are generally very slow, and if you are performing any type of mathematical calculations on them, this will be even worse - as they will get automatically converted to numbers (possibly Integers) before the calculation, and the final result converted back to a string.


You should always try to use the correct data type for the data, as it not only removes the extra conversions, but also dramatically reduces the chances of conversion/concatenation errors too.

si_the_geek, bogie: GREAT FEEDBACK... thank you!

My database is already in INTEGER type. The reason why I used a string array is that this array contains information from many sources - it's like having 500 arrays into one. So at the end of each "sub-array", I have some string-info in the array, something like "END-OF-ARRAY;INFO-BLABLA" to know where I'm at, in the array.

I guess I'll reconfigure how it works to avoid having that huge array as a string array - I guess I could manage to do that easily.

My machine has 3GB of physical memory. I'm planning on having a standalone machine(s) to deal with this - and I'll add all the required memory if needed.

si_the_beek, thanks for the brief but very informative tips about byte-usage of those variables.

:thumb:

Another question comes to mind:

To make sure my app doesn't crash, I have some checkups here and there on the "UBound(matMyMatrix, 1)" to avoid going further.

Should I save the UBound result in a variable and use this for the checkup instead? Or will UBound offer me an as blazing-fast result?

Rather than having the extra string info in the array, I would add a third 'column' to the array which points to another (string) array which contains that data. You would then be able to know from any row of the original data which 'sub-array' the data comes from.


As for storing the result of UBound to a variable, that could be a good idea (especially if you are calling it within a loop), as calling functions (even built in ones) takes time, as does actually running them. In this particular case the difference isn't huge, but if you also make a similar sort of change for other function calls you will get a noticeable gain.

If you are resizing the array at any point (which is very slow, especially with Preserve), using this variable allows you to use a much faster method.


I would recommend reading the article How do I optimize my code? from our Classic VB FAQs (in the FAQ forum, which is shown near the top of our home page), as it contains several useful tips.

si_the_geek strikes again.

Good stuff.. I was already thinking about that 3rd row idea. But I think I will store info in that 3rd row only at the beginning of a "sub-array" since I don't need to check it up at any row. When I reach the beginning of a sub-array, I will simply keep its info in another variable and loop until the 3rd columns contains something again. I guess that'll lighten my array, too.

I do have to redim the array with PRESERVE.. But I'll put it on a 5-10 minutes timer so it won't be growing THAT much. Or anyway not that often.

I'll make sure I read that optimize-your-code thread. I think I print it once to help me pass time in the plane, but I didn't read it and prolly lost it since.

Thank again for the tips.

Long is the fastest data type to process since it is native to the processor.

Rule of thumb:
When in doubt use integer, long if necessary (>32768). Integer processing will always outrun strings and single precision.

I recall writing code once upon a time that converted single precision to integers by removing the decimal, performing the integer calculations, tracking the decimal point movement, and then displaying the results as a string by repositioning the decimal after the integer computations finished.

That "idiotic" procedure was faster than letting the computer perform single-precision manipulations with the same values.:eek:

I might have to benchmark both solutions. If randem is right, it sounds like I should go LONG. I will NEVER deal with decimals.

If your values will never exceed the range of Integer, -32768 to 32767, then stick with the Integer data type. They are half the size of Long, which could be significant given the size of your array, and you will never notice the difference in speed.

randem's generalized point about Long being more efficient is correct, but Logo's point about using an extra 4mb of memory is the bottom line for speed in this case.

Quote:

---

Originally Posted by Krass

My database is already in INTEGER type. The reason why I used a string array is that this array contains information from many sources - it's like having 500 arrays into one. So at the end of each "sub-array", I have some string-info in the array, something like "END-OF-ARRAY;INFO-BLABLA" to know where I'm at, in the array.

---

Sounds like it could be done better as a UDT array. A UDT array would also give you a more logical structure. Something like:

vb Code:

Private Type CoordinateType
    x As Integer
    y As Integer
End Type
 
Private Type MainType
    Info As String
    Coords() As CoordinateType
    Length As Long ' Stores the UBound() of Coords()
End Type
 
Private mtyp() As MainType
 
Public Sub InitArray()
    Dim i As Long
    Dim j As Long
 
    ReDim mtyp(500)
    For i = 0 To 500
        With mtyp(i)
            .Info = "Description goes here"
            ReDim .Coords(2000)
            .Length = 2000
            For j = 0 To 2000
                With .Coords(j)
                    .x = rst!x
                    .y = rst!y
                End With
            Next
        End With
    Next
End Sub

That's the basic idea, anyway.

Getting back to the database most large database engines use the integer type but it is really a long data type... Unless it is defined as short integer or small integer or something like that.

Besides I have never heard of a resultset being a million records and I have done some of the most busy financial systems and no where near that many records are used at one time...

That's a much better idea Ellis... I knew there was something that could be done for the organisational side of things, but didn't think of that even tho I use it regularly myself. :blush:

Quote:

---

Originally Posted by randem

Getting back to the database most large database engines use the integer type but it is really a long data type... Unless it is defined as short integer or small integer or something like that.

---

Very true, but as long as the values are always within the valid range for VB's Integer, that wont be a problem - the coercion will be fine.

I definitely agree with Logophobic's comments on the use of Integer for this - the Long data type is likely to cause more harm than good, unless it is required due to the scale of the numbers.

Quote:

---

Besides I have never heard of a resultset being a million records and I have done some of the most busy financial systems and no where near that many records are used at one time...

---

That's a fair point.. this amount of data is rather extreme (but not unheard of), and there may well be a much better/faster way to cope with this data - but it depends on what Krass is actually trying to achieve.

Hah. The usefulness of those forums will always amaze me.

I never played with UDT. At least not for such large jobs. I'll dig into that and that'll probably be my next posts topic!

A million records is something I never had to play with, too. Those X,Y coordinates are actually the coordinates of each "given-color"-pixels in many different pictures, which I'll have to compare with another smaller array (another picture).

Thanks for the feedback.

From a brief first reading of Ellis Dee's idea of using a User-Defined Type Array, the following questions pops up:

Will I be able to redim that array with PRESERVE just like normal arrays?

the .info (declared as string) will be filled with data 0.005% of the time. I guess it would be a waste of memory (speed) having that string declaration in there? I mean, even if it was integer! I'll need to know that the "sub-array" I'm browsing right now is, let's say, "252", but I'll only note that down when I'm reaching a new sub-array, I won't be using its content all the time.

Also I don't think the UBound result should be stored in the UDT, since it's pretty static and an outside variable would do the job as well.

Just my first 2 cents.. I'll work a bit on this and see how it goes.

Thanks

You will be able to ReDim in the usual way, as the UDT's (MainType and CoordinateType) are treated the same way as other data types (Integer/String/..).

Your concerns about the .Info and .Length items aren't valid, as there is only one copy for each set of data, and the actual X/Y data is in the Coords() array (which itself is based on a UDT!).


It would probably be a good idea to create a quick test program with a similar data structure, and step through it (including adding items to the watch window) to help you understand it.

si_the_geek, thanks for clearing things up.

I understand that whole UDT thing very good now, well.... how Ellis Dee used it. - I must say that looks perfect. Nicely done, Ellis Dee, your sample code will probably look a lot like my final code ;)

While playing Guitar Hero 2 and Settlers of Catan tonight I was thinking of keeping it the way it is (but manage to avoid strings). But that new method sure looks sweet!

You guys should play guitar hero 2..

Thanks for the help again
(gh2)

Quote:

---

Originally Posted by Krass

Also I don't think the UBound result should be stored in the UDT, since it's pretty static and an outside variable would do the job as well.

---

Either way works, but in the code sample above the UBound() effectively is "outside", because it's not a part of the coordinates array.

Is each subarray the same length? If so, then you're right, it's pointless waste to maintain 500 copies of the same number in the UDT.

For the text descriptions, though, regardless of how infrequently you need to reference them the way it is shown above is the best way to go if you have a different description for each subarray.

Ellis, each subarray will indeed be of a diffrent length. So your method is good.

That makes me think: some of my arrays are going to be 300-long, some 900-long, but the average will be about 500. I guess it would be a good idea to redim Coords() to 1000 right at the beginning of my procedure (when I fill the X,Y) .. and, at the end, do a last negative redim to remove all unused rows?

That sounds like a good idea to me, because I used to do a loop and do a redim preserve to add +1 each time I need to add a new X,Y.. So that's a lot or redim PRESERVE.

My only worry is: Are the memory/ressources gonna get free'd nicely without hassle? Would you recommend this approach?

(... I *just* realised : if this approach is good, alright.. it is good knowledge for me. But that would be optimizing the part of filling my arrays - the part that is not critical for me, optimizingly speaking!.. oh well)

Thanks.

Quote:

---

Originally Posted by Krass

That makes me think: some of my arrays are going to be 300-long, some 900-long, but the average will be about 500. I guess it would be a good idea to redim Coords() to 1000 right at the beginning of my procedure (when I fill the X,Y) .. and, at the end, do a last negative redim to remove all unused rows?

---

That'd work fine. Memory will be freed nicely when you ReDim Preserve it down to the final size after each subarray loop finishes.

You don't have any way to identify how many coordinate pairs are in a given subarray before you start populating it?

Just out of interest, you say this is about comparing arrays (representing particular pixels) between many different images. Would you mind telling us a little more about that? There are some blindingly fast ways to work with pixel data which might help if appropriate.

Each sub-arrays are MASKS of a word (for some reason I couldn't go by letter). Comparing all those sub-arrays (masks) against one other main picture will allow me to extract info from that image as it if was text (some player stats info from an online game: that has been done before, it's something like it, but in a more personal field) - anyway you get the idea.

My loops are checking up blue pixels, as it's all I need. Sometimes yellow, for numbers - in case I need that to be done.

I know I've been proven wrong before (!), but I'm pretty sure the part of storing X,Ys in a database, to then fill an array/udt is the way to go. Those sub-arrays are actually created by me, dynamically, and it would slow down the whole process to have it deal with "finding blue pixels" everytime in my "huge" loop. - those are already found out and in a DB!

That method was primarily used to avoid using the very slow Pic1.point method which would slow down the process a lot. (because my sub arrays are X,Y coords retrieved from a picturebox, loaded up with each of my sub-arrays one after the other).

That might be confusing.

One optimization that will need to be done is a technique I heard in one of my last posts on which I'll have to give more attention when my main "looping" code is up and running, optimized. It consists in turning the picturebox content in an array instantly - it sounded like a nice technique, but on the first read I didn't quite understood - will get back to it later.

Ellis Dee: You are right. Once the X,Ys are in my DB, I could easily know how long I must create the coordinate array - avoiding redims..... Like I said, I don't need much optimization for THAT application portion but - it'll be done that way!

Thank you's!

The Point method (and similar ones) are painfully slow... it sounds like your future method is DIB Sections, which (once you've got your head around them!) will make a huge improvement to the speed of reading/writing pixels.

Here's a quick way to get those pixels into an array using GetDIBits. There is a slight caveat :D extacting the data this way inverts the Y axis and alters the colours (they become BGRA as opposed to RGBA). I've included a conversion function to toggle between RGBA and BGRA. There is also an option to produce a two Dimensional array but you will find the single dimensional array performs better. If you want to use this with the two dimensional array you will have to invert the Y coordinates in your Database, much better would be to then convert each pair of X and inverted Y integers into a single Long and use a one dimensional array, this is easier than it sounds and can be done quickly.

Also seeing as you are only interested in a few colours it's worth converting the image data from a long array and into a byte array that just holds the colours you are interested in, see the retByteData function.

vb Code:

'Module code
Option Explicit
 
Private Declare Function GetObjectA Lib "GDI32" (ByVal hObject As Long, ByVal nCount As Long, ByRef lpObject As Any) As Long
Private Declare Function GetDIBits Lib "GDI32" (ByVal hDC As Long, ByVal hBitmap As Long, ByVal nStartScan As Long, ByVal nNumScans As Long, lpBits As Any, lpBI As BITMAPINFO, ByVal wUsage As Long) As Long
Private Declare Sub RtlMoveMemory Lib "kernel32" (Destination As Any, Source As Any, ByVal Length As Long)
 
Private Type BITMAP '24 bytes
 bmType As Long
 bmWidth As Long
 bmHeight As Long
 bmWidthBytes As Long
 bmPlanes As Integer
 bmBitsPixel As Integer
 bmBits As Long
End Type
 
Private Type RGBQUAD
 Blue As Byte
 Green As Byte
 Red As Byte
 Alpha As Byte
End Type
 
Private Type BITMAPINFOHEADER '40 bytes
 bmSize As Long
 bmWidth As Long
 bmHeight As Long
 bmPlanes As Integer
 bmBitCount As Integer
 bmCompression As Long
 bmSizeImage As Long
 bmXPelsPerMeter As Long
 bmYPelsPerMeter As Long
 bmClrUsed As Long
 bmClrImportant As Long
End Type
 
Private Type BITMAPINFO
 bmHeader As BITMAPINFOHEADER
 bmColors(0 To 255) As RGBQUAD
End Type
 
Public Function RetDIBDataLong(PictureObject As Object, Optional Return2DArray As Boolean) As Long()
Dim BM As BITMAP, bmi As BITMAPINFO, lngImageData() As Long
   'The PictureObject must be an object with both a HDC and Image property
   'such as a PictureBox, UserControl or Form
   
   GetObjectA PictureObject.Image, 24, BM
   bmi.bmHeader.bmWidth = BM.bmWidth
   bmi.bmHeader.bmHeight = BM.bmHeight
   bmi.bmHeader.bmSize = 40
   bmi.bmHeader.bmPlanes = 1
   bmi.bmHeader.bmBitCount = 32
   
   If Return2DArray Then
     ReDim lngImageData(BM.bmWidth - 1, BM.bmHeight - 1)
     GetDIBits PictureObject.hDC, PictureObject.Image.Handle, 0, BM.bmHeight, lngImageData(0, 0), bmi, 0
   Else
     ReDim lngImageData(BM.bmWidth * BM.bmHeight - 1)
     GetDIBits PictureObject.hDC, PictureObject.Image.Handle, 0, BM.bmHeight, lngImageData(0), bmi, 0
   End If
   
   RetDIBDataLong = lngImageData
End Function
 
Public Function retByteData(PictureObject As Object, SearchPalette() As Long) as Byte()
'The ImageObject must be an object with both a HDC and Image property
'such as a PictureBox, UserControl or Form
 
'The searchpalette should contain no more than 255 colours and these
'colours must be in the form of DIBSection colours i.e. BGRA not RGBA see
'ConvertColour function below
 
 
Dim i As Long, ii As Long, lPalLen As Long, lngImageData() As Long, bytImageData() As Byte
  lPalLen = UBound(SearchPalette)
  If lPalLen > 254 Then lPalLen = 254
  
  lngImageData = RetDIBDataLong(PictureObject)
  i = UBound(lngImageData)
  ReDim bytImageData(i)
  
  For i = 0 To i
    For ii = 0 To lPalLen
      If lngImageData(i) = SearchPalette(ii) Then
        bytImageData(i) = ii + 1
        Exit For
      End If
    Next ii
  Next i
  
 'If you only want a couple of colours they might be better HardCoded here rather
 'than in the passed SearchPalette, example loop would then be...
  'For i = 0 To i
      'Select Case lngImageData(i)
        'Case Colour1
          'bytImageData(i) = 1
        'Case Colour2
          'bytImageData(i) = 2
      'End Select
  'Next i
  
  retByteData = bytImageData
End Function
 
Public Function ConvertColour(ByVal Colour As Long) As Long
Dim uRGBA As RGBQUAD, Swp As Byte
  RtlMoveMemory uRGBA, Colour, 4&
  With uRGBA
     Swp = .Red
    .Red = .Blue
    .Blue = Swp
  End With
  RtlMoveMemory Colour, uRGBA, 4&
  ConvertColour = Colour
End Function

vb Code:

'Example calling code for a form with two pictureboxes and two command buttons
Option Explicit
Private Declare Function SetPixelV Lib "GDI32" (ByVal hDC As Long, ByVal X As Long, ByVal Y As Long, ByVal crColor As Long) As Byte
 
Private Sub Command1_Click()
Dim PicData() As Byte, pal() As Long, i As Long, W As Long
  ReDim pal(3)
  'First fill the pal array with the colours we want to capture, up to 255
  'RGB can be used to generate 32bpp DIB Colours by swaping Red for Blue and visa versa
  pal(0) = RGB(0, 3, 1) 'same as ConvertColour(RGB(1, 3, 0))
  pal(1) = RGB(3, 0, 15)
  pal(2) = RGB(33, 0, 206)
  pal(3) = RGB(249, 252, 250)
  PicData = retByteData(Picture1, pal)
  
  'This next bit is slow, it is just to show the data we have just gained visualy
  With Picture1
    W = Picture1.ScaleX(.ScaleWidth, .ScaleMode, vbPixels)
  End With
  
  Picture2.Cls
  For i = 0 To UBound(PicData)
    If PicData(i) Then SetPixelV Picture2.hDC, i Mod W, i \ W, QBColor(PicData(i))
  Next i
  'As you can see the Y axis has been inverted
End Sub
 
 
Private Sub Command2_Click()
 
Dim PicData() As Long, X As Long, Y As Long, UBY As Long
  'Get the pixel data very quickly
  PicData = RetDIBDataLong(Picture1, True)
  
  'Then a slow method to show how the Y axis is inverted and the Red and Blue bytes are swapped
  Picture2.Cls
  UBY = UBound(PicData, 2)
  For X = 0 To UBound(PicData, 1)
    For Y = 0 To UBY
      SetPixelV Picture2.hDC, X, Y, PicData(X, Y)
    Next Y
  Next X
End Sub

To copy the code, quote the post and copy from there.
Attached is the image I tested this with.

Hello.

I think I'm done optimizing my main loop. Using the method you guys suggested (user defined types) - and avoiding strings - significantly speeds up the process.

I don't think the code can be done better, below is an idea of what it looks like... It's designed to find matching pixels (same X,Y from my sub-array (from 1 millions record array - about 500 records each times) as the X,Y from the small image I'm comparing it to).

Code:

---

Do While xSmall <= intSmallUBound - 1 And xHuge <= HugeType(i).Length - 1
    If HugeType(i).Coords(xHuge).x < (arrSmallSubArray(0, xSmall) + xAdd) Then
        xHuge = xHuge + 1
    ElseIf HugeType(i).Coords(xHuge).x > (arrSmallSubArray(0, xSmall) + xAdd) Then
        xSmall = xSmall + 1
    ElseIf HugeType(i).Coords(xHuge).y < (arrSmallSubArray(1, xSmall) + yAdd) Then
        xHuge = xHuge + 1
    ElseIf HugeType(i).Coords(xHuge).y > (arrSmallSubArray(1, xSmall) + yAdd) Then
        xSmall = xSmall + 1
    Else
        xSmall = xSmall + 1
        xHuge = xHuge + 1
        MatchCount = MatchCount + 1
    End If
Loop

---

My next step is to read, understand and implement Milk's piece of code. Btw thank you, that seems to be pretty complete. That's the last portion slowing down my application with those .pset and .point calls. I'm hoping to have time testing this next week.

I've got a (maybe a little off-topic) question. When populating a recordset, why must I use .movelast to be able to retrieve .recordcount? If I just populate it and check the .recordcount it will always show "0". Not a big problem, but I find it weird to have to use .movelast and .movefirst to retrieve that info.

Thank you!

Quote:

---

Originally Posted by Krass

When populating a recordset, why must I use .movelast to be able to retrieve .recordcount? If I just populate it and check the .recordcount it will always show "0". Not a big problem, but I find it weird to have to use .movelast and .movefirst to retrieve that info.

---

It's an optimization; the server doesn't send the entire recordset until it has to.

If you have some time, I've always been curious if the With operator speeds up UDT array access. If you change the code you posted to the following, is it any faster?

Code:

---

With HugeType(i)
    Do While xSmall <= intSmallUBound - 1 And xHuge <= .Length - 1
        If .Coords(xHuge).x < (arrSmallSubArray(0, xSmall) + xAdd) Then
            xHuge = xHuge + 1
        ElseIf .Coords(xHuge).x > (arrSmallSubArray(0, xSmall) + xAdd) Then
            xSmall = xSmall + 1
        ElseIf .Coords(xHuge).y < (arrSmallSubArray(1, xSmall) + yAdd) Then
            xHuge = xHuge + 1
        ElseIf .Coords(xHuge).y > (arrSmallSubArray(1, xSmall) + yAdd) Then
            xSmall = xSmall + 1
        Else
            xSmall = xSmall + 1
            xHuge = xHuge + 1
            MatchCount = MatchCount + 1
        End If
    Loop
End With

---

Also, I see couple tweaks you could do to squeeze a little more speed out of the posted code.

Code:

---

With HugeType(i)
    Do While xSmall < intSmallUBound And xHuge < .Length ' removing the two subtractions should help a bit
        Select Case arrSmallSubArray(0, xSmall) + xAdd
            Case Is > .Coords(xHuge).x: xHuge = xHuge + 1
            Case Is < .Coords(xHuge).x: xSmall = xSmall + 1
            Case Else
                Select Case arrSmallSubArray(1, xSmall) + yAdd
                    Case Is > .Coords(xHuge).y: xHuge = xHuge + 1
                    Case Is < .Coords(xHuge).y: xSmall = xSmall + 1
                    Case Else
                        xSmall = xSmall + 1
                        xHuge = xHuge + 1
                        MatchCount = MatchCount + 1
                End Select
        End Select
    Loop
End With

---

While I concede that this logic is much less readable, it should be perceptibly more efficient. Every iteration of the Do...Loop has at least 2 fewer arithmetic operations, and each match saves another two operations. Given the huge number of passes this loop is making, this could add up to a fairly decent speed boost.

I don't quite understand what the code is supposed to achieve, but there are a few things you could do that may improve the speed a bit.

The With block as Ellis suggested should give an improvement, as should the changes on the Do line.


Over the life of the loop you are repeatedly checking the same values which involve array access (slightly slower than reading a variable), and some of them also involve calculations too (which again take time).

I would recommend storing the result of (arrSmallSubArray(0, xSmall) + xAdd) (and the Y equivalent into variables and using them instead - of course re-calculating the value whenever xSmall changes.

You could do similar for HugeType(i).Coords(xHuge) , as that also takes time (two array accesses, and one member lookup), resetting it whenever xHuge changes. If you use a With block, this may not get much benefit.

Ellis's use of Select Case is a reasonable alternative to these variables, but I think the variables will be faster... although the Select Case method can be improved by using Sgn function, eg:

Code:

---

        Select Case Sgn((arrSmallSubArray(0, xSmall) + xAdd) - (.Coords(xHuge).x))
            Case 1: xHuge = xHuge + 1
            Case -1: xSmall = xSmall + 1
            case Else

---

..whether this beats the variables depends on various factors, so would need to be tested.


There is another possible improvement which is highly dependent on the data, and may not make a big difference.. if the final Else condition is a regular occurrence, you would find speed gains by re-ordering the If's (as an = comparison is faster than a < one, and it would mean a maximum of 3 If's instead of 4).


As to the recordset issue, you can get a RecordCount without a MoveLast if set the CursorLocation to clientside, and change the CursorType. But.. that is slow, and for speed you should run a "SELECT Count(*)" type query instead.

Quote:

---

Originally Posted by si_the_geek

... although the Select Case method can be improved by using Sgn function, eg:

Code:

---

        Select Case Sgn((arrSmallSubArray(0, xSmall) + xAdd) - (.Coords(xHuge).x))
            Case 1: xHuge = xHuge + 1
            Case -1: xSmall = xSmall + 1
            case Else

---

---

This benchmarks much slower on my machine:

vb Code:

Private Sub Sample()
    Const Iterations = 10000000
    Dim sngStart As Single
    Dim i As Long
    Dim x As Long
    Dim y As Long
    
    x = 32
    y = 57
    sngStart = Timer
    For i = 1 To Iterations
        Select Case x
            Case Is < y
        End Select
    Next
    MsgBox Format(Timer - sngStart, "0.000")
    sngStart = Timer
    For i = 1 To Iterations
        Select Case Sgn(x - y)
            Case -1
        End Select
    Next
End Sub

I get 0.062 and 0.230 respectively. It effectively adds a function call and an operation, though four times slower is more extreme than I expected.

Your other suggestions will almost certainly speed things up a bit despite the code sprawl and extra assignments:

Code:

---

Dim lngHugeX As Long
Dim lngHugeY As Long
Dim lngSmallX As Long
Dim lngSmallY As Long
Dim lngLength As Long

With HugeType(i)
    lngLength = .Length
    With .Coords(xHuge)
        lngHugeX = .x
        lngHugeY = .y
    End With
    lngSmallX = arrSmallSubArray(0, xSmall) + xAdd
    lngSmallY = arrSmallSubArray(1, xSmall) + yAdd
    Do While xSmall < intSmallUBound And xHuge < lngLength
        Select Case lngHugeX
            Case Is < lngSmallX
                xHuge = xHuge + 1
                With .Coords(xHuge)
                    lngHugeX = .x
                    lngHugeY = .y
                End With
            Case Is > lngSmallX
                xSmall = xSmall + 1
                lngSmallX = arrSmallSubArray(0, xSmall) + xAdd
                lngSmallY = arrSmallSubArray(1, xSmall) + yAdd
            Case Else
                Select Case lngHugeY
                    Case Is < lngSmallY
                        xHuge = xHuge + 1
                        With .Coords(xHuge)
                            lngHugeX = .x
                            lngHugeY = .y
                        End With
                    Case Is > lngSmallY
                        xSmall = xSmall + 1
                        lngSmallX = arrSmallSubArray(0, xSmall) + xAdd
                        lngSmallY = arrSmallSubArray(1, xSmall) + yAdd
                    Case Else
                        xSmall = xSmall + 1
                        xHuge = xHuge + 1
                        MatchCount = MatchCount + 1
                        With .Coords(xHuge)
                            lngHugeX = .x
                            lngHugeY = .y
                        End With
                        lngSmallX = arrSmallSubArray(0, xSmall) + xAdd
                        lngSmallY = arrSmallSubArray(1, xSmall) + yAdd
                End Select
        End Select
    Loop
End With

---

If you are comparing arrSmallSubArray() to all 500 HugeType() subarrays, then you could definitely optimize that by adding xAdd and yAdd to the values in a single loop once before you begin the million comparisons. Even if you then went through and subtracted the offsets back out at the end, the speed savings should be significant.

Of course, doing that would reduce the speed improvement of the above code, since there would be fewer operations that get avoided.

Now that I'm thinking about it, I think it will be as fast or faster to read an array value twice than it is to read it once and assign it to another variable, so I'm not sure how effective the above code will be. I'll craft yet another version and put it in a new post.

Code:

---

For i = 0 To UBound(arrSmallSubArray, 2)
    arrSmallSubArray(0, i) = arrSmallSubArray(0, i) + xAdd
    arrSmallSubArray(1, i) = arrSmallSubArray(1, i) + yAdd
Next
'....code you didn't post goes here
For i = 0 To UBound(HugeType)
    '....code you didn't post goes here
    With HugeType(i)
        Do While xSmall < intSmallUBound And xHuge < .Length ' removing the two subtractions should help a bit
            With .Coords(xHuge)
                Select Case arrSmallSubArray(0, xSmall)
                    Case Is > .x: xHuge = xHuge + 1
                    Case Is < .x: xSmall = xSmall + 1
                    Case Else
                        Select Case arrSmallSubArray(1, xSmall)
                            Case Is > .y: xHuge = xHuge + 1
                            Case Is < .y: xSmall = xSmall + 1
                            Case Else
                                xSmall = xSmall + 1
                                xHuge = xHuge + 1
                                MatchCount = MatchCount + 1
                        End Select
                End Select
            End With
        Loop
    End With
    '....code you didn't post goes here
Next
'....code you didn't post goes here
For i = 0 To UBound(arrSmallSubArray, 2)
    arrSmallSubArray(0, i) = arrSmallSubArray(0, i) - xAdd
    arrSmallSubArray(1, i) = arrSmallSubArray(1, i) - yAdd
Next

---

Note the highlighted interior With block. That should make the interior loop as fast or faster than the method of assigning the values to local variables, which will help keep the number of local variables down to a manageable level.

EDIT: I'm waffling. While I stand by the code in this post, I think si's technique may help if applied to just the arrSmallSubArray values. By how much I can't even guess.

Hello. I was more than happy to benchmark some of your solutions, and here are the results (in seconds - this was done over a small set of records - of course using exactly the same for all tests, executed 3 times each):

Quote:

---

Exactly as I first posted it : 29, 29, 29
Exactly as I first posted it (no -1 in do) : 29, 29, 29
After Ellis's post 28 (without "with") : 29, 28, 29
After Ellis's post 28 (with "with") : 25, 26, 26
After geek's post 29 (without variable - using 2 sgn calls) : 26, 25, 26

---

I am about to test the code using variable instead. My tests stops are post #29.

Si_the_geek: "=" won't be the most regular occurence... Those < and > operators are used a lot to move my pointer in the arrays UNTIL they are "=".

And also I was wondering if I should use the SELECT count(*) instead of using .movelast/.movefirst. Because after all, I need all the populated info in the recordset and loop it, so using count(*) would mean populating ANOTHER recordset just to know the recordcount.

You guys are really into it. Devoted help - I like.

Fair point on the sgn etc Ellis.. but your test was biased, as you only compared to one outcome rather than two, and there weren't array/property accesses on the cases either.

Of course, the effectiveness of my method would depend on the actual data in use - and if there is a preference towards the first outcome then the extra function and subtraction (which I admittedly didn't account for fully) will make it slower.


It's all academic tho, as your latest version using the extra variables eliminates the problem that the method was trying to solve!

Quote:

---

Originally Posted by si_the_geek

Fair point on the sgn etc Ellis.. but your test was biased, as you only compared to one outcome rather than two, and there weren't array/property accesses on the cases either.

---

Conceded.

Quote:

---

Originally Posted by Krass

Hello. I was more than happy to benchmark some of your solutions, and here are the results (in seconds - this was done over a small set of records - of course using exactly the same for all tests, executed 3 times each):

---

Compiled exe, right? IDE rsults aren't meaningful...

As you may have guessed, I (and several others on the boards) enjoy optimizing code for speed, so if there are other sections to your processing loop, I'd be happy to take a look at the whole routine.

No that was not on compiled exe. Only by executing it. Shouldn't that be meaningful if they were ALL tried that way? (conceded it may in fact run faster after compilation)...

Strangely enough, using the same code (from my last test - 26, 25, 26) with an "interior with" have dropped me back to 29, 30, 29. (still using sgn)

Based on those tests, the 2 sgn calls seems to give the same speed as not using them. I guess I can just use them anyway?..

What seemed to really give a performance boost was the first global "with", without "interior with", clocking at "25, 25, 26"....

I still havn't tested using a variable at any point (except for those already in my first post, of course)

Testing speed in the IDE is virtually meaningless I'm afraid - code not only runs slower, but also in a different way completely.

In the past I have seen two similar pieces of code that were compared, one of them was about twice as fast as the other in the IDE, but then several times slower than it when compiled!

Quote:

---

Originally Posted by Krass

No that was not on compiled exe. Only by executing it. Shouldn't that be meaningful if they were ALL tried that way? (conceded it may in fact run faster after compilation)...

---

No, the IDE will be slower on some things but not others. IDE results have no predictive value on compiled speed when comparing different methods.

If you were comparing the same method with different data, the relative speeds in the IDE may give you rough idea, but even then the ratios will be all wrong. (28, 26 and 30 show much less variation than 7, 5, 6, for example.)

That thread will have learned me a couple of things.

I think I'll be short on time today but hopefully I'll be able to re-do those timechecks with compiled IDE soon. And will post the results.

I actually became very curious as to how it will differ!..

I guess the compiled version will show a difference between using sgn calls or not. In IDE, it changed absolutely nothing.

Also, that "interior with" was bad, adding an average of 5 seconds to the process. Do you think it has chances of improving when compiled? Only one way to know, I guess!

Quote:

---

Originally Posted by Krass

Also, that "interior with" was bad, adding an average of 5 seconds to the process. Do you think it has chances of improving when compiled?

---

Almost certainly.

I was able to re-do those tests with compiled version and here are the results:

Note that I still didn't even try to use more variables than there was in my original post. I guess that would be the next test.

Quote:

---

Test#1: Exactly as I first posted it : 14, 14, 13
Test#2: Exactly as I first posted it (no -1 in do) : 14, 14, 14
Test#3: After Ellis's post 28 (without "with") : 13, 13, 13
Test#4: After Ellis's post 28 (with "with") : 13, 12, 13
Test#5: After geek's post 29 (without variable - using 2 sgn calls) : 13, 12, 12
Test#6: Added interior with : 14, 14, 13

---

...Looks to me that the best solutions is to use those sgn calls and avoid the "interior with" (Test #5).

I am reposting what the code now looks like:

Code:

---

Do While xSmall < intSmallUBound And xHuge < .Length ' removing the two subtractions should help a bit
    Select Case Sgn((arrSmallSubArray(0, xSmall) + xAdd) - (.Coords(xHuge).x))
        Case 1: xHuge = xHuge + 1
        Case -1: xSmall = xSmall + 1
        Case Else
            Select Case Sgn((arrSmallSubArray(1, xSmall) + yAdd) - (.Coords(xHuge).y))
                Case 1: xHuge = xHuge + 1
                Case -1: xSmall = xSmall + 1
                Case Else
                    xSmall = xSmall + 1
                    xHuge = xHuge + 1
                    MatchCount = MatchCount + 1
            End Select
    End Select
Loop

---

That looks good, doesn't it?

Yeah, that looks pretty tight.

Well, that was an overall EXCELLENT optimization, I'd say. Your tricks did help a lot.

On a 14 seconds test, I think about 10 or 12 are wasted manipulating an image in a picturebox using .pset and .point. I can't wait to have Milk's DIB Section solution up and running..

I was skeptic at first, knowing I'd end up with a million records to browse but this is veeery quick, even tho that test was done with only 36,000 records, not a million.

...My code is to be executed at a precise time/date, and must end within about 40 hours of execution. I think that'll work perfectly. I will always be able to setup more than 1 machine to do the task anyway.

Thanks again, optimization masters.

If I understand this problem correctly there is a way to improve this enormously, not that there is anything wrong with the above routines, just that as far as I can tell you are comparing an array of particular pixels to many arrays of particular pixels.

A much better approach would be to have an array of all the pixels in the target image which you compare to the coordinates... i.e.

Code:

---

If ImageData2D(HugeType(i).Coords(xHuge).x, HugeType(i).Coords(xHuge).y) = tgtColour Then
  MatchCount = MatchCount + 1
End If

'or simpler

If ImageData2D(.x, .y) = tgtColour Then MatchCount = MatchCount + 1

---

even better convert your two X and Y integer coords into a single Long index

Code:

---

If ImageData1D(HugeType(i).Index(ii)) = tgtColour Then MatchCount = MatchCount + 1

---

ATM :confused: I'm not sure what XAdd and YAdd are for? Are your search coordinates from a smaller image than the database coordinates, or are you searching all over the image?

Hi Milk..

Your approach seems pretty good. I should have thought about such a simple method, even tho I'll have to think further more if this can be adapted to suit my needs. It might be better than looping my arrays to find matches.

My huge array is filled with tons of sub-arrays. Those tons sub-arrays will always (or almost always?) contains more pixels than the target image it's been compared to.

The target image has been modified/played with (by code) before comparing it to my tons of sub-arrays. While doing those pre-checkup modifications, the whole image might move some pixels in any direction.. (example: in some cases, the top of the image will be deleted, and it'll be 'cropped/moved' to be at the top-left of the picturebox)... That is the main reason why I am using xAdd and yAdd.......:

The image has moved a couple of pixels, so the sub-arrays won't contain the same exact X and Ys. So, for each target image that I'll compare to many sub-arrays, it'll actually do a loop for xAdd = -5 to 5 (same for Y), as an offset.

Some of my tests shows a > 90% pixel match at offset 0, -1, while some at 3, 2.. and so on.

You'll be asking why not just remembering how many pixels you moved it left, top or else.. well, before modifying the image by code, I couldn't know where it actually really starts on the left or at the top. My code modifying the image is sometimes taking off the image important parts of its content, but this I have to deal with. A big-enough part is remaining so I can use it to see in which sub-arrays it "fits" best. Reason of the checking method with an offset.

Makes a lot more demanding process - but so far, tests have shown pretty good results, based on both accuracy and speed. Your method might be a considerable help.. Will have to think about that.

My present method advances in the arrays until X and Y are the same. I believe I could just use your method but that'll make me look for match on every X,Y contained in the sub-arrays. Might still be quicker than dealing movements in both arrays.

Thanks!

Milk, I've played with your DIb Sections functions and it sure looks sweet.

I, for now, will focus only on black pixels. I got rid of the SearchPalette and ended up with that code:

Code:

---

Public Function retByteData(PictureObject As Object)
    Dim i As Long, ii As Long, lngImageData() As Long, bytImageData() As Byte
   
    lngImageData = RetDIBDataLong(PictureObject)
    i = UBound(lngImageData)
    ReDim bytImageData(i)
   
    'Checking only black pixels (Case 0) (avoiding SearchPalette)
    For i = 0 To i
        If lngImageData(i) = 0 Then bytImageData(i) = 1
    Next i
   
    retByteData = bytImageData
End Function

---

...if the color in lngImageDate(i) = black, then I write 1 in the byte array, ... but later on, how will I know the X&Ys if I only have "1" and "0" in the byte array? Unless "i" = X and Y converted to long. That'd be clever. But then what to do with my xAdd and yAdd? I'd have to reconvert "i" to different X and Y coords, add xAdd and yAdd and reconvert it to "i"?

I'm not quite sure I understand the way it works.

In your previous post, you suggested the following:

Code:

---

If ImageData1D(HugeType(i).Index(ii)) = tgtColour Then MatchCount = MatchCount + 1

---

In this example, I believe ImageData1D = lngImageData used in your previous code?

..you suggested to "convert your two X and Y integer coords into a single Long index". If I got it right, the hugetype will contain a Index array, let's say of ~700 records, some having "1" and some having "0", right? (considering I get rid of all colors BUT black).

Exemple: HugeType(52).Index(523) = 142525. "142525" would be my X and my Y, mixed together and saved as long? Sounds good to me if that works well. To do that, I'd need to know how to invert my Ys, and how to transfer my X,Y coord to long correctly.

Quote:

---

Originally Posted by Krass

<snip>...if the color in lngImageDate(i) = black, then I write 1 in the byte array, ... but later on, how will I know the X&Ys if I only have "1" and "0" in the byte array? Unless "i" = X and Y converted to long. That'd be clever. But then what to do with my xAdd and yAdd? I'd have to reconvert "i" to different X and Y coords, add xAdd and yAdd and reconvert it to "i"?

---

Actually this is why I asked what they were for, I think because your doing lots different searches with different xAdd and yAdd values using a 2D array will be easier.

The retByteData can be easily adapted to a return 2D array.

Code:

---

Public Function retByteData2D(PictureObject As Object, SearchColour As Long) As Byte()
Dim x As Long, y As Long, uby As Long, lngImageData() As Long, bytImageData() As Byte
'The PictureObject must be an object with both a HDC and Image property
'such as a PictureBox, UserControl or Form

  lngImageData = RetDIBDataLong(PictureObject, True)
  x = UBound(lngImageData, 1)
  uby = UBound(lngImageData, 2)
  ReDim bytImageData(x, uby)
 
  For x = 0 To x
    For y = 0 To uby
      If lngImageData(x, y) = SearchColour Then
        bytImageData(x, y) = 1
      End If
    Next y
  Next x
 
  retByteData2D = bytImageData
End Function

---

Regarding two Integer coordinates as one Long...

There are two main ways to do this, consider a small image, width 4, height 3.

2D Dib indexes would look like
0,2 1,2 2,2 3,2 <-- 11 Mod Width, 11 \ Width <--edited
0,1 1,1 2,1 3,1
0,0 1,0 2,0 3,0

1D Dib indexes would look like
08 09 10 11 <-- 3 + 2 * Width
04 05 06 07
00 01 02 03

You could also convert the above 2D coordinates to longs like this
131072 131073 131074 131075 <-- 3 + 2 * 65536
065536 065537 065538 065539
000000 000001 000002 000003

Does that make sense? Basically if you need to use many different yAdd and Xadd values I would stick to 2D.

Edit: :blush: I made a mistake in the code in post #26

Code:

---

#
Public Function retByteData(PictureObject As Object, SearchPalette() As Long) as byte()

---

Ok, I will stick with 2d arrays.

I may be stupid :cry: but I don't understand those 2:

2D Dib indexes would look like
0,2 1,2 2,2 3,2 <--11 \ Width + 11 Mod Width
0,1 1,1 2,1 3,1
0,0 1,0 2,0 3,0

131072 131073 131074 131075 <-- 3 + 2 * 65536
065536 065537 065538 065539
000000 000001 000002 000003

In the first example, where does 11 comes from? And the * 65536 on the 2and example?..... I'm totally lost :ehh:

If you have some spare time, would you take me by the hand?...

:blush: oops 11 \ Width + 11 Mod Width should be 11 Mod Width, 11 \ Width The 11 is the 1D index as shown in the next example.

I = X + Y * Width
X = I Mod Width
Y = I \ Width

And the other one...

An Integer has the range -32768 to 32767 and a Long, -2147483648 to 2147483647

-32768 * 65536 = -2147483648
32767 * 65536 = 2147483647

P = X + Y * 65536
X = P Mod 65536
Y = P \ 65536

Hi Milk,

Thanks for clarifying.

After such modification, the Command1_Click would require some modifications. I tried something like that:

Code:

---

   
Dim PicData() As Byte, i As Long, y As Long, W As Long
   
    PicData = retByteData2D(Picture1, 0)
   
    'This next bit is slow, it is just to show the data we have just gained visualy
    With Picture1
      W = Picture1.ScaleX(.ScaleWidth, .ScaleMode, vbPixels)
    End With

    Picture2.Cls
    For i = 0 To UBound(PicData, 1)
        For y = 0 To UBound(PicData, 2)
            If PicData(i, y) Then SetPixelV Picture2.hDC, i Mod W, i \ W, QBColor(PicData(i))
        Next y
    Next i

---

... but that still gives me a "subscript out of range" at the SetPixelV line. I don't feel very comfortable with this, maybe would it be easy for you to edit your first posted procedure so it can now work with a 2D array?

Thank you Milk for you time and dealing with a newbie...

Nevermind the last post, Milk... after scratching my head quite a couple of times I ended up with that code. Please correct me if it is incorrect in any way, but I'd doubt it since it does show your "Felix the cat", upside down, just like your original code.

Code:

---

Private Sub Command1_Click()
    Dim PicData() As Byte, i As Long, y As Long, W As Long
   
    PicData = retByteData2D(Picture1, 0)
   
    'This next bit is slow, it is just to show the data we have just gained visualy
    With Picture1
      W = Picture1.ScaleX(.ScaleWidth, .ScaleMode, vbPixels)
    End With

    Picture2.Cls
    For i = 0 To UBound(PicData, 1)
        For y = 0 To UBound(PicData, 2)
            If PicData(i, y) Then SetPixelV Picture2.hDC, i, y, 0
        Next y
    Next i
End Sub

---

...notice how I've changed the SetPixelV call.

Now my PicData() is filled with 0's and 1's. All 1's are black pixels and that's all I'm interested into. That array upper bounds will always be approx 280,78 (can't quite remember - but I think that's not important). Let's just take the top left 9 x 9.. it would look like this:

1 0 0 0 0 0 0 0 1 [ 0 to 280 ...]
0 1 0 0 0 0 0 1 0
0 0 1 0 0 0 1 0 0
0 0 0 1 0 1 0 0 0
0 0 0 0 1 0 0 0 0
0 0 0 1 0 1 0 0 0
0 0 1 0 0 0 1 0 0
0 1 0 0 0 0 0 1 0
1 0 0 0 0 0 0 0 1
[ 0 to 78 ...]

So, as an example, the top-left part of my picture actually looks like a "X" sign. We will agree that the very most top-left pixel is a black pixel at 0,0 (yes, on a picture box, 0,0 is top-left, and not bottom-left). After using dib sections, as you mentionned, the Y gets inverted so that this 0,0 pixel is now located at 0,77.

Since I am now using a 2d array, do I still need to convert my database X,Y coords to long? If so, how would I now compare that "long data" to the PicData array? To me it sounds like I could only convert the Y part of my database coord?.. and then compare it to the PicData array. But maybe that was some processing time you were trying to help me avoid. In clear, it's the following line of code that I'd like to run succesfully:

Code:

---

If PicData(.x, .y) = 1 Then MatchCount = MatchCount + 1

---

...where .x and .y are coords from my database.

So, instead of inverting the Y in my database on every loop instance, shouldn't it be easier to flip the array and end up with good original coords? My 0,77 black pixels would get back to 0,0 and I'd be on my way..

Any tip VERY welcome.

Edit:

Quote:

---

I just thought that "flipping" the array back in good order would make life easier for me.. My current code, which modifies the target images in various ways, would still be applicable without much modifications... I'll still wait for your input. Thanks.

---

Either flip the Y axis as you extract it (like below) or flip the Y axis in your database and re-save the database.

Code:

---

  For x = 0 To x
    For y = 0 To uby
      If lngImageData(x, y) = SearchColour Then
        bytImageData(x, uby - y) = 1
      End If
    Next y
  Next x

---

As to converting to Longs, don't! Get it working with 2D Integer coordinates then later maybe try a Long version. The Long version would work very well if your database coords and the extraction image coords were taken from images with the same width, which I'm not sure they are.

Code:

---

If PicData(.Coords(xHuge).x, .Coords(xHuge).y) = 1 Then MatchCount = MatchCount + 1

'or perhaps...

If PicData(.Coords(xHuge).x, .Coords(xHuge).y) <> 1 Then MissCount = MissCount + 1
If MissCount > 100 then Exit For

---

All those techniques have been applied and tested. Works like a charm.

I removed the following code from Command1_Click which became useless (so it seems):

Code:

---

'This next bit is slow, it is just to show the data we have just gained visualy
With Picture1
  W = pic1.ScaleX(.ScaleWidth, .ScaleMode, vbPixels)
End With

---

I will also get rid of the Command2, as this was just for demonstration purpose.

I will need your expertise (again) to achieve a little task. My PicData is now perfectly filled with black pixels AFTER all my picture modifications were applied. My last step was to move the content TOP-LEFT in the picturebox (getting rid of completely whited-out lines or columns).

I was using such code:

Code:

---

pic1.PaintPicture pic1.Picture, 0, -10

---

I believe this technique is yet another 'slow' picturebox technique. Let's say I know there are 6 vertical lines (on the left) and 10 horizontal lines (on the top) to get rid off, would there be a way to apply this straight to my PicData array? I believe there will be some ReDim tricks to be done here.

Also, I have brought final modification to the retByteArray2D function:

Code:

---

Public Function retByteData2D(PictureObject As Object) As Byte() ', SearchColour As Long) As Byte()
    Dim x As Long, y As Long, uby As Long, lngImageData() As Long, bytImageData() As Byte
    Dim lngColor As Long, intRed As Integer, intGreen As Integer, intBlue As Integer

    lngImageData = RetDIBDataLong(PictureObject, True)
    x = UBound(lngImageData, 1)
    uby = UBound(lngImageData, 2)
    ReDim bytImageData(x, uby)
 
    For x = 0 To x
        For y = 0 To uby
            lngColor = lngImageData(x, y)
            intRed = lngColor And 255
            intGreen = lngColor \ 256 And 255
            intBlue = lngColor \ 65536 And 255

            If (intRed + intGreen + intBlue) / 3 > 30 Then
                'Pixel NOT black enough - do nothing
            Else
                'Pixel black or almost black - save pixel to bytImageData
                bytImageData(x, uby - y) = 1
            End If
        Next y
    Next x
   
    retByteData2D = bytImageData
End Function

---

...Yes, I won't just check for black pixels, after second thought. As you can see in the above code, I'll keep any pixels that is actually black or ALMOST black. I don't think the SearchPalette shown earlier would have helped here, right? If you have any speed-concern about my code, feel free to correct me!

Thanks!.. My app is taking form and I am happy.

Your IF statement is far from ideal in terms of speed.

Division is particularly slow (especially with / rather than \ ), and it is faster to multiply instead - so it would be quicker to multiply the opposite side of the comparison:

Code:

---

            If (intRed + intGreen + intBlue) > 30 * 3 Then

---

..however, as that is a constant value, just change the constant instead:

Code:

---

            If (intRed + intGreen + intBlue) > 90 Then

---

I don't think PaintPicture is particularly slow in comparison to the alternative methods, but it may be extra work that you don't need - it sounds as if you could just start your loops in a different place instead (eg: rather than start your X loop at 0, start at 6 to ignore the first 6 columns)

Si_the_geek... Modification has been made. Thanks.

Starting the loop at 6 instead of 0 would be a good way to ignore first 6 columns.... but I'm wondering if it'd still be better to move my picture at the beginning of the huge loop... else the code will end up something like that:

Code:

---

If PicData(.Coords(xHuge).x + xAdd + intAvoidedCols, .Coords(xHuge).y + yAdd + intAvoidedRows) = 1 Then MatchCount = MatchCount + 1

---

...see, that adds an operator to my huge loop, which will of course be the more ressource/time demanding (the huge loop).

What do you think?

Moreover, I lied. I wasn't using that line of code:

Code:

---

pic1.PaintPicture pic1.Picture, 0, -10

'...but THAT one!
for i = 0 to 10
    pic1.PaintPicture pic1.Picture, 0, -1
next i

---

For some odd reasons, doing -10 at a time was keeping the old result in place, duplicating the picture.. (maybe was there some pic1.Picture = pic1.image missing here and there)....

You can alter RetDIBDataLong to return the RGBQUAD type like this. Remember to alter all the variable names throughout the function.

Code:

---

Public Function RetDIBDataRGBA(PictureObject As Object, Optional Return2DArray As Boolean) As RGBQUAD()
Dim BM As BITMAP, bmi As BITMAPINFO, RGBAImageData() As RGBQUAD

---

Then you don't have to extract the bytes out of the long....

Code:

---

Public Function retByteData2D(PictureObject As Object) As Byte()
    Dim x As Long, y As Long, uby As Long, RGBAImageData() As RGBQUAD, bytImageData() As Byte
   
    RGBAImageData = RetDIBDataRGBA(PictureObject, True)
    x = UBound(RGBAImageData, 1)
    uby = UBound(RGBAImageData, 2)
    ReDim bytImageData(x, uby)
 
    For x = 0 To x
        For y = 0 To uby
            with RGBAImageData(x,y)
              if clng(.Red) + .Green + .Blue < 90 then bytImageData(x, uby - y) = 1
            end with
        Next y
    Next x

    retByteData2D = bytImageData
End Function

---

Away from IDE, Untested warning!

There's no need for using intAvoidedCols/intAvoidedRows inside the loop - as you could simply modify the xAdd/yAdd values before the loop instead.

Milk, your code using RGBQUAD was good. Good work.

I've been able to test my application again, now using dib sections. My main huge loop is executing the following on each target image:

Code:

---

For i = 0 To intHugeUBound
    For xAdd = -5 To 5
        intOffsetX = xAdd + intModifOffsetX
        For yAdd = -5 To 5
            MatchCount = 0
            xHuge = 0
            intOffsetY = yAdd + intModifOffsetY
            '289,79
            'HugeType(i).Length
           
            Do While xHuge < HugeType(i).Length
                If HugeType(i).Coords(xHuge).x + intOffsetX < 290 And HugeType(i).Coords(xHuge).y + intOffsetY < 80 Then
                    If HugeType(i).Coords(xHuge).x + intOffsetX > -1 And HugeType(i).Coords(xHuge).y + intOffsetY > -1 Then
                        If PicData(HugeType(i).Coords(xHuge).x + intOffsetX, HugeType(i).Coords(xHuge).y + intOffsetY) = 1 Then
                            MatchCount = MatchCount + 1
                            'Picture2.PSet (.Coords(xHuge).x + intOffsetX, .Coords(xHuge).y + intOffsetY), vbBlue
                            DoEvents
                        End If
                    End If
                End If
                xHuge = xHuge + 1
            Loop
        Next yAdd
    Next xAdd
    'MsgBox MatchCount
   
    'Debug.Print "hello"
   
    'If (MatchCount / intSmallUBound) * 100 > 80 Then
    '    MsgBox HugeType(i).Info
    'End If
   
Next i

---

Sure, those both "double-if" lines (< 290 and > -1) are probably slowing things down a bit. Also, that example is not using any "with". I am getting horrible results compared to my old technique and it wouldn't be MUCH better by fixing it by using a with or optimizing those "double-if" lines.

The part of modifying my target image, using dib sections instead of .pset and .point, is unbelievably quicker than my previous technique. And that was the part slowing down my old technique too much. It's the "comparing pixels" that's now taking forever.

Why is it slow: I think it's because EVERY pixels from my huge coords array (from the DB) are checked against the target image array. And that huge array will ALWAYS have much more coords than actually needed (in many cases). Using my old technique, I was browsing both arrays until we have a match, and quitting the process when the huge arrays coords are bigger than the biggest coord from the target image, hence skipping a lot of useless processing. That old technique was from post 29 from Si the geek...

Yes I am going to use a "intMissCount" after the work is done to speed up the process even more, but results are too slow right now to even think of optimizing the actual technique.

What I'm thinking of doing is keep the following DIB sections routines, but instead of keeping that array filled with 0s and 1s, I'll make a loop to find 1's and build back an array like I did before, and I'll test my old technique. I should get the exact same 'good' results I had, but better since it's now using dib sections.

Also, it is true that moving ('cropping') my target image was useless. The matching-pixel counts are the same good results and it works pretty good that way.

Sorry for making such a long post - I'm trying to make sure things are clear. When all of this is finished, my app will run much more quickly than I would have thought - and that's because of you guys.. *thanks*

Edit:

Quote:

---

Btw, for your information, the old technique was clocking at 12 seconds (compiled), while my actual new technique is clocking at ~30 seconds, which is comparable to my initial technique without using the dib sections.

---

Edit:

Quote:

---

Never mind the following post, I am now using this method:

If intMissCount = 60 Then
MatchCount = 0
Exit Do
End If

I might end up chanding "60" for a pre-calculated % of the total number of pixels in the target image.

---

Right after my last (long) post, I've started to work to modify my code, exactly as I mentionned it. The results are very satisfying and I've never had my application running that fast.

Si_the_geek, I was able to witness how awful "/" is compared to "\". That's good knowledge, thank you.

Here is a routine I am using right now. This code is at the bottom-most of my loop and is executed quadrillions of times...

Code:

---

If (xSmall * 20) > intSmallUBound Then
    If ((MatchCount \ (intSmallUBound \ 20)) * 100) < 80 Then
        MatchCount = 0
        Exit Do
    End If
End If

'First line used to be:
If xSmall > (intSmallUBound \ 20) Then
'...but thanks to your advice, I am using a multiplication instead of a division

---

In clear (as if you needed clarification on such basic code), when I have compared the first 5% of the target image pixels and didn't reach 80% of match, I'll quit the loop, since there are almost no chance we're going to end up on a good match % at the very end.

I am pretty sure the line with 2 divisions and 1 multiplication can be optimized. Would you have an idea? I could have tried something on my own, but then I thought you may end up suggesting a built-in-%-function ...

Assuming that intSmallUBound is set before the loop, there are improvements you can make to the speed of each of those lines, simply by storing the results to variables.

You could store (intSmallUBound \ 20) into a different variable (before the loop), then use it like this:

Code:

---

If xSmall > intSmallUBoundD20 Then

---

..it only removes a single multiplication or division, but as it is running so many times it should make a difference.

Similar can be done for the second line, but before we do that a bit of re-organisation is in order, so that we can get MatchCount on it's own (I presume it is the only variable in that line which actually changes inside the loop). Here are my steps:

Code:

---

    If ((MatchCount \ (intSmallUBound \ 20)) * 100) < 80 Then
    If (MatchCount \ (intSmallUBound \ 20)) < (80 / 100) Then
    If MatchCount < (intSmallUBound \ 20) * (80 / 100) Then

---

So, by setting up variable before the loop which contains (intSmallUBound \ 20) * (80 / 100) , you can do a simple comparison of the two variables instead.