[RESOLVED] Extracting text

[akhileshbc]

Hi...

I'm trying to create a small program, which I think will be helpful to others(in here).

My aim is to convert the English text shown in screen to Malayalam. English-to-Malayalam conversion is not a problem. But the problem is, how to extract the text from the screen.
In most of the converter/translator, the text needs to be copied or written down in the translator tool. But my program will cut off that part. That is, simply move the program window (having a rectangle hollow portion) to the English word that needs to be converted. And upon pressing a button (or autodetect), the English word is converted to Malayalam.

So, my question is how to extract the English text(or words) from the screen (which is seen through the hollow portion of the window.

Here's a screenshot of what I'm trying to do.



I hope I'll get some ideas from you..

Thank you...

[akhileshbc]

After doing some research (using Google), I found that OCR is what I need to use for the extraction.

Can you tell me how to create a hole on the form ?

[akhileshbc]

I just figured it myself

In the form, set a color for TransparencyKey. Then the same color is set as the BackColor for a Picturebox. It becomes hollow..

Now my question is how to capture the image of that hollow portion ? Any ideas ?

[jmcilhinney]

Graphics.FromImage and Graphics.CopyFromScreen

[akhileshbc]

Graphics.FromImage and Graphics.CopyFromScreen

Thanks jmcilhinney...
I'll check that.

[akhileshbc]

I found some code from MSDN. But for testing it, the way I try to show it on PictureBox seems to be not correct:

Imports System
Imports System.Drawing
 
Public Class Form1
 
    Dim memoryImage As Bitmap
 
    Private Sub CaptureScreen()
        Dim myGraphics As Graphics = Me.CreateGraphics()
        Dim s As Size = Me.Size
        memoryImage = New Bitmap(s.Width, s.Height, myGraphics)
        Dim memoryGraphics As Graphics = Graphics.FromImage(memoryImage)
        memoryGraphics.CopyFromScreen(Me.Location.X, Me.Location.Y, 0, 0, s)
 
        PictureBox2.BackgroundImage = myGraphics '~~~ Not the correct way ?
    End Sub
 
 
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        CaptureScreen()
    End Sub
End Class

[akhileshbc]

I found it:

 PictureBox2.Image = memoryImage

...

[jmcilhinney]

Make sure that you dispose any Graphics objects you create. For short-lived, disposable objects like that you should always use a Using block.

[akhileshbc]

Thanks I'll keep that in mind...

But I'm having some problems with the positioning:

Imports System
Imports System.Drawing
 
Public Class Form1
 
    Dim memoryImage As Bitmap
 
    Private Sub CaptureScreen()
        Dim myGraphics As Graphics = Me.CreateGraphics()
        Dim s As Size = PictureBox1.Size ' Me.Size
        memoryImage = New Bitmap(s.Width, s.Height, myGraphics)
        Dim memoryGraphics As Graphics = Graphics.FromImage(memoryImage)
        memoryGraphics.CopyFromScreen(Me.Location.X + Me.PictureBox1.Location.X, Me.Location.Y + Me.PictureBox1.Location.Y, 0, 0, s) '(Me.Location.X, Me.Location.Y, 0, 0, s)
 
        PictureBox2.Image = memoryImage
    End Sub
 
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        CaptureScreen()
    End Sub
 
End Class

See there's some change in the X and Y position of the captured image



The top picturebox is the hollow portion and the bottom one shows the captured image (AutoSize set to TRUE)
...

[J-Deezy]

It's to do with the location of where you're copying from screen.

This is your current code is it not?

memoryGraphics.CopyFromScreen(Me.Location.X, Me.Location.Y, 0, 0, s)

Well, this is going to get the graphics starting in the location of the form not the picture box isn't it. Try switching it to

memoryGraphics.CopyFromScreen((Me.Location.X + PictureBox1.Location.X), (Me.Location.Y + PictureBox1.Location.Y), 0, 0, s)

Seems to me that should work.

EDIT, sorry had a mental blank.
EDIT2, I went to test it and I found that the y axis needs a little more adjusting due to the top border not being included in Me.Location.Y and also the border at the bottom not being equal in width to the top border. I did a few little jiggles and got it working nicely, here's the modified Capture code:

Private Sub CaptureScreen()
 
        Dim BorderWidth As Integer = ((Me.Width - Me.ClientSize.Width) / 2) 'this gets the outer border widths on the left and right sides, coincidentally they are identical in size to the bottom border
        Dim TopBorder As Integer = ((Me.Height - Me.ClientSize.Height) - BorderWidth) 'now that we obtained the height of the bottom border, we can simply calculate the height of the top border, which is what we needed all along
        
        Dim myGraphics As Graphics = Me.CreateGraphics()
 
        Dim s As Size = Me.Size
 
        memoryImage = New Bitmap(s.Width, s.Height, myGraphics)
 
        Dim memoryGraphics As Graphics = Graphics.FromImage(memoryImage)
 
        memoryGraphics.CopyFromScreen((Me.Location.X + PictureBox1.Location.X), (Me.Location.Y + PictureBox1.Location.Y + TopBorder), 0, 0, s)
 
 
        PictureBox2.Image = memoryImage
 
    End Sub

Enjoy

[akhileshbc]

Hi JD...

No. That doesn't work.

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        TextBox1.Text = "Form(X,Y)=" & Me.Location.X & "," & Me.Location.Y & " :: PicBox1(X,Y)=" & PictureBox1.Location.X & "," & PictureBox1.Location.Y
    End Sub

Output on each run:

Form(X,Y)=22,29 :: PicBox1(X,Y)=20,25
Form(X,Y)=44,58 :: PicBox1(X,Y)=20,25
Form(X,Y)=110,145 :: PicBox1(X,Y)=20,25
etc...

X and Y of form changes to the position where it is shown during runtime. But PicBox's position is constant and it's origin is based on form's (gray area) top-left point. Whereas the Form's origin is based on the screen's top-left point !!!

[akhileshbc]

Thanks J-Deezy...
I didn't see that you were editing your post.

That worked. I added BorderWidth to the X position too.
Now the code is:

Imports System
Imports System.Drawing
 
Public Class Form1
 
    Dim memoryImage As Bitmap
 
    Private Sub CaptureScreen()
        Dim BorderWidth As Integer = ((Me.Width - Me.ClientSize.Width) / 2) 'this gets the outer border widths on the left and right sides, coincidentally they are identical in size to the bottom border
        Dim TopBorder As Integer = ((Me.Height - Me.ClientSize.Height) - BorderWidth) 'now that we obtained the height of the bottom border, we can simply calculate the height of the top border, which is what we needed all along
 
        Dim myGraphics As Graphics = Me.CreateGraphics()
        Dim s As Size = PictureBox1.Size ' Me.Size
        memoryImage = New Bitmap(s.Width, s.Height, myGraphics)
        Dim memoryGraphics As Graphics = Graphics.FromImage(memoryImage)
        memoryGraphics.CopyFromScreen((Me.Location.X + PictureBox1.Location.X + BorderWidth), (Me.Location.Y + PictureBox1.Location.Y + TopBorder), 0, 0, s) '(Me.Location.X + Me.PictureBox1.Location.X, Me.Location.Y + Me.PictureBox1.Location.Y, 0, 0, s) '(Me.Location.X, Me.Location.Y, 0, 0, s) 
 
        PictureBox2.Image = memoryImage
    End Sub
 
 
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        CaptureScreen()
    End Sub
End Class

Now do you have any suggestions on character/word reading. Any tutorial specialized for VB.Net ?

Edit:

Added rep

[J-Deezy]

Thanks J-Deezy...
I didn't see that you were editing your post.

That worked. I added BorderWidth to the X position too.
Now the code is:

Imports System
Imports System.Drawing
 
Public Class Form1
 
    Dim memoryImage As Bitmap
 
    Private Sub CaptureScreen()
        Dim BorderWidth As Integer = ((Me.Width - Me.ClientSize.Width) / 2) 'this gets the outer border widths on the left and right sides, coincidentally they are identical in size to the bottom border
        Dim TopBorder As Integer = ((Me.Height - Me.ClientSize.Height) - BorderWidth) 'now that we obtained the height of the bottom border, we can simply calculate the height of the top border, which is what we needed all along
 
        Dim myGraphics As Graphics = Me.CreateGraphics()
        Dim s As Size = PictureBox1.Size ' Me.Size
        memoryImage = New Bitmap(s.Width, s.Height, myGraphics)
        Dim memoryGraphics As Graphics = Graphics.FromImage(memoryImage)
        memoryGraphics.CopyFromScreen((Me.Location.X + PictureBox1.Location.X + BorderWidth), (Me.Location.Y + PictureBox1.Location.Y + TopBorder), 0, 0, s) '(Me.Location.X + Me.PictureBox1.Location.X, Me.Location.Y + Me.PictureBox1.Location.Y, 0, 0, s) '(Me.Location.X, Me.Location.Y, 0, 0, s) 
 
        PictureBox2.Image = memoryImage
    End Sub
 
 
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        CaptureScreen()
    End Sub
End Class

Now do you have any suggestions on character/word reading. Any tutorial specialized for VB.Net ?

Edit:

Added rep

Oops, yeah I forgot about the borderwidth for the X axis, my mistake. Sorry, I can't help you with the text extraction, I am but a noob

[jmcilhinney]

It should look like this:

Dim bounds = Me.RectangleToScreen(Me.Panel1.Bounds)
Dim copy As New Bitmap(bounds.Width, bounds.Height)
 
Using g = Graphics.FromImage(copy)
    g.CopyFromScreen(bounds.Location, Point.Empty, bounds.Size)
End Using
 
Me.PictureBox1.Image = copy

Note the use of a Panel for the cut-out. No point using a PictureBox when it won't contain a picture.

[akhileshbc]

Thanks jmcilhinney ...

That's great and only few lines of code!..

May I ask you one more question ? Do you have any suggestions for a free OCR SDK ? I did some searching, but was not successful in finding any free one.

[jmcilhinney]

No idea I'm afraid.

[akhileshbc]

No idea I'm afraid.

Sorry, I can't help you with the text extraction, I am but a noob

No problem..

I'll try searching deeper.

[MarMan]

OCR code is no trivial task. Do you have a planned target for this? If so it is easier, but not easy. If you want it to work everywhere, good luck. OCR code that does not work everywhere goes for big buck and takes many months with a team of many people.

If you are aiming at a particular font you can zoom in and inspect it. Fonts are made up of multiple colors now, even if you choose 'black'. One way to do it would be to change the contents of your window to black and white. You will have to experiment to see what works best, usually low values can be black and higher values can be white, but from my limited experience it works best with different cutoff values for red, blue and green. When I get home I will see if I still have what I started with, although it is far from complete.

[akhileshbc]

Hi MarMan...

I'm developing a small kind of converter/translator program (I have mentioned the details in first post). So, it is targeted on whatever text in the computer screen. And I'm not focusing on stylized fonts. Only normal fonts are taken into consideration.

The idea you have mentioned is little different for me to catch up with, because I don't have much experience doing these things.

When I get home I will see if I still have what I started with, although it is far from complete.

That would be very helpful.

Thanks...

[J-Deezy]

It should look like this:

Dim bounds = Me.RectangleToScreen(Me.Panel1.Bounds)
Dim copy As New Bitmap(bounds.Width, bounds.Height)
 
Using g = Graphics.FromImage(copy)
    g.CopyFromScreen(bounds.Location, Point.Empty, bounds.Size)
End Using
 
Me.PictureBox1.Image = copy

Note the use of a Panel for the cut-out. No point using a PictureBox when it won't contain a picture.

Ahh, so much simpler than my version. At least I learned something new today!

[MarMan]

The code that I started with is VB6. Do you have access to VB6, or would it be useless to post. I will have to change it because right now it demos with sensitive data I would have to replace with something else.

Going quickly through the code I found one comment that stated the fount I was trying to read displays a lower case "L" identicle to an upper case "I". That is an example of the types of problems you face with this type of task. Another is the pixels that get lit up for a character can be slightly different depending on the preceding and following character.

If you would be interested in the VB6 source post here or PM me and give we a url of an example you would like to use and I will see if I can make something do something,. At least give you a starting point.

[MarMan]

Sorry for the delay getting back to you.

Here is an example of OCR. I ripped it out of a program I was working on so it may sem a bit odd, but it is for informational purposes anyway. I had to blank out the grid because it contained persona information, but it works on the headers.

The biggest flaws to this approach:

1. Needs to store any font you wish to use in its own format
2. Doesn't work on text represented in graphics that do not match a screen font
3. Limitied in font size

I will quickly address these.
1. This is not bad. All you would have to do is write some code to loop through each screen font, loop through some sizes, print one letter to a picturebox, convert it to black & white, gather the information and write it to a file.
2. If you want to read any text you will have to use fuzzy logic. Plus the code will be much more difficult. There is no such thing as a small program for OCR. Using this approach will go from large to huge.
3. This method stores font information as binary data, which breaks down with large font sizes. If you wish to use large fonts or medium (depending on what you call small) you will have to modify it.

There is a benefit to limiting it to screen fonts. You can acheive 99% accuracy with a smaller code base. I will help if I can.

Good luck!

[akhileshbc]

That's super ...

But I have paused the development of this program because of it's complexity. Initially, I thought about using third party controls. But none of them offered accuracy.

And your solution is much more complex (but it seems to be more accurate), so that I might not be able to handle it at this moment. But I'll keep your sample for further studying