Comparing JPG image to an Image on a specified window

[GRPsuper9]

Hey guys,

I have a specific image that I want my application to look for on a specific window. I've done something similar to this along time ago with VB6, but it was slow, and it was checking for specific points in the picture rather than the whole image itself (for speed reasons). I named this image "Image1.jpg" and I want to compare it to the window with the handle "001C08F2" and class "SunAwtCanvas"

It has been a while since I have programmed and I'm trying to get into it again.

Anyone have any ideas on how to accomplish this?

[.paul.]

have a look at this:

http://www.vbforums.com/showpost.php...97&postcount=3

[GRPsuper9]

Thanks for your help.

Is that assuming both images are the same width and height though? Because I have a smaller image I want to compare to an image on the webbrowser of my form, so I would have to have it scan what it sees on the webbrowser and then check to see if the image exists. (Note: It would scan the webbrowser without scrolling up and down. This is assuming the scroll bars are not enabled.)

[.paul.]

no. in that example there's a larger image + the code loops through each pixel in the larger image + looks for the smaller image contained within that larger image

[GRPsuper9]

Ah. Alright. Thats assuming you have two pictures though, correct? What about if I have one picture and I want to scan the WebBrowser1 control for the same pixels?

[.paul.]

you'd have to capture an image of the webbrowser contents

[GRPsuper9]

I'm still having trouble seeing how I can compare both my pictures.

I have been able to code my program so the user can load a picture into userPictureBox, then when they click the button "Find on Screen", my program takes a snapshot of the screen and puts it into screenPictureBox. How could I make your code compatible to mine? I've been having trouble switching up the imgs and such in your code.

[.paul.]

ok. here it is simplified:

Private Function getImageRect(ByVal largerImage As Bitmap, ByVal imageToFind As Bitmap) As Rectangle
    For x As Integer = 0 To largerImage.Width - 1
        For y As Integer = 0 To largerImage.Height - 1
            If largerImage.GetPixel(x, y) = imageToFind.GetPixel(0, 0) Then
                Dim dontMatch As Boolean = False
                Dim doesMatch As Integer = 0
                For x2 As Integer = 0 To imageToFind.Width - 1
                    For y2 As Integer = 0 To imageToFind.Height - 1
                        If largerImage.GetPixel(x + x2, y + y2) <> imageToFind.GetPixel(x2, y2) Then
                            dontMatch = True
                            Exit For
                        Else
                            doesMatch += 1
                        End If
                    Next
                    If dontMatch Then Exit For
                Next
                If dontMatch = False AndAlso doesMatch = (imageToFind.Width * imageToFind.Height) Then Return New Rectangle(x, y, imageToFind.Width, imageToFind.Height)
            End If
        Next
    Next
End Function

[GRPsuper9]

Thanks for helping!

I got 1 problem though. Since it still wasn't working, I took away the function and I put it straight to the button that executes the code. I replaced everything properly and now where the GetPixel codes come up, I'm getting an error. For example:

If PictureBox2.GetPixel(x, y) = PictureBox1.GetPixel(0, 0) Then

and

If PictureBox2.GetPixel(x + x2, y + y2) <> PictureBox1.GetPixel(x2, y2) Then

The error I'm getting is: 'GetPixel' is not a member of 'System.Windows.Forms.PictureBox'.

Any idea how to clear that up?

[.paul.]

getPixel is a member of bitmap:

Dim image1 As Bitmap = DirectCast(PictureBox1.Image, Bitmap)
Dim image2 As Bitmap = DirectCast(PictureBox2.Image, Bitmap)
If image2.GetPixel(x, y) = image1.GetPixel(0, 0) Then
 
End If

[GRPsuper9]

Ugh, I hate to keep bothering you about this, but I must be doing something wrong.

This is the code I currently have:

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        getImageRect()
    End Sub
 
    Private Function getImageRect() As Rectangle
        Dim imageToFind As Bitmap = DirectCast(PictureBox1.Image, Bitmap)
        Dim largerImage As Bitmap = DirectCast(PictureBox2.Image, Bitmap)
        For x As Integer = 0 To largerImage.Width - 1
            For y As Integer = 0 To largerImage.Height - 1
                If largerImage.GetPixel(x, y) = imageToFind.GetPixel(0, 0) Then
                    Dim dontMatch As Boolean = False
                    Dim doesMatch As Integer = 0
                    For x2 As Integer = 0 To imageToFind.Width - 1
                        For y2 As Integer = 0 To imageToFind.Height - 1
                            If largerImage.GetPixel(x + x2, y + y2) <> imageToFind.GetPixel(x2, y2) Then
                                dontMatch = True
                                Exit For
                            Else
                                doesMatch += 1
                            End If
                        Next
                        If dontMatch Then Exit For
                    Next
                    If dontMatch = False AndAlso doesMatch = (imageToFind.Width * imageToFind.Height) Then Return New Rectangle(x, y, imageToFind.Width, imageToFind.Height)
                End If
            Next
        Next
    End Function

The images I have are actually screenshots of my code. PictureBox1 (imageToFind) is a little image with just the two words of "DirectCast" in it. The PictureBox2 (largerImage) is a picture of all of the code. For some reason it cannot find the DirectCast within the entire largerImage :/. Did you try this out and have luck?

[.paul.]

as i explained in the link i posted, any deterioration in either of the images will cause the function to fail

[GRPsuper9]

These are the two images I'm using though. The pixels shouldn't be any different because they are from the same image.

[.paul.]

i didn't try it, but there are 256^3 colors used in windows + color.black looks exactly like color.fromargb(0,0,1) but vb doesn't see it that way...

[GRPsuper9]

So would it not be possible for me to use your method with what I am attempting to do? :/

[.paul.]

it is possible but as you can see it's limited.

[.paul.]

did you try:

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
    dim r as rectangle = getImageRect()  
    msgbox(r isnot nothing)  
End Sub

???

dunno how i missed that

[GRPsuper9]

Still a no go with it working. I'll try and see what I can find tomorrow and work on it a little more. Gonna go to sleep, got work in the morning :|. Haha.

Thanks though! Let me know if you find out about other methods to do this or ways to fix this.

[GRPsuper9]

Haven't had any success using google. Anyone else got one that might possibly work with what I'm trying to do?

[boops boops]

The important thing is to allow some margin of error in your comparison. It's easier to do if you first convert the images or the pixels to grayscale. A simple method is to add up the red, green and blue bytes of each pixel and dividing by 3. (There are also more sophisticated grayscale formulas that weight the RGB bytes according to the eye's sensitivity to different primary colors, but keep it simple for the moment.) If you are just comparing two pixels, you don't even need to divide by 3 because that factor is the same on both sides of the comparison.

Another technique is to only require a certain percent -- say 70% -- of the pixels to match on a first pass. That means keeping a count of the pixels that match, instead of exiting the For loop as soon as you find a mismatch. If you do get an approximate match this way, you could do a second pass with a higher percent or 100% to see if the images really are identical. This can be combined with the grayscale+comparison margin I described above.

To sum up, the loop part of your code might look like this. I haven't checked it so it's just to give you the idea:

Dim matches As Integer
Dim matchPercent As Integer = 80 'Try different values! 
Dim margin As Integer = 30 'Try different values!
Dim totalPixels As Integer = imageToFind.Width * imageToFind.Height
Dim doesMatch As Boolean = False
 
For x2 As Integer = 0 To imageToFind.Width - 1
       For y2 As Integer = 0 To imageToFind.Height - 1
               Dim clr1 = largerImage.GetPixel(x + x2, y + y2)
               Dim clr2 = imageToFind.GetPixel(x2, y2)
 
               'Compare the grayscale values with a margin:
               Dim gray1 As Integer = clr1.R + clr1.G + clr1.B
               Dim gray2 As Integer = clr2.R + clr2.G + clr2.B
               If gray1 - gray2 <= margin AndAlso gray1 - gray2 >= - margin Then
                      matches +=1
                      If CInt(matches * 100 / totalPixels) >= matchPercent Then
                              doesMatch = True
                              Exit For  
                      End If
              End If
        Next
        If doesMatch Then Exit For
Next

Note that you don't need dontMatch in this method.

EDIT: This method just allows for possible color differences. If one of the images is stretched, rotated or otherwise distorted compared to the other it still won't work.

BB