Captured images now streaming to memory. Now how to access this stream with ffmpeg?

[Peter Porter]

I now have captured images streaming to memory. I just have to figure out how to get ffmpeg to access that memory so it could convert everything into a movie file. Removing the Picturebox code from the project will improve on the frame capture. The picturebox is just to prove that the memory stream works.

The updated project is attached at the bottom of this post.



The Capture Timer:

Private Sub VidREC_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles VidREC.Tick
        Dim ct As Integer = CInt(ComboBox1.SelectedItem)
        Dim Frames As Integer = Convert.ToInt32(Label3.Text)

        VidREC.Interval = ct
        Label7.Text = "Recording"

        Dim BMP As New System.Drawing.Bitmap(MonitorWidth, MonitorHeight)
        Dim Cap As System.Drawing.Graphics = System.Drawing.Graphics.FromImage(BMP)
        Dim imgStream As MemoryStream = New MemoryStream()

        Cap.CopyFromScreen(New Point(0, 0), New Point(0, 0), ScreenSize)
        BMP.Save(imgStream, System.Drawing.Imaging.ImageFormat.Jpeg)
        PictureBox1.Image = Image.FromStream(imgStream)

        Frames = Frames + 1
        Label3.Text = Convert.ToString(Frames)

        If Label1.Text = "stop" Then
            VidREC.Enabled = False
            ComboBox1.SelectedIndex = 3
        End If

        BMP.Dispose()
        Cap.Dispose()

    End Sub

[Shaggy Hiker]

So, what error message do you get on the crash?

You are creating an array with 100 million bitmaps. Each bitmap is the size of the screen. Therefore, if you had a fairly modest 1024x768 resolution with 32-bit depth, then each image would be 3.14 million bytes in size (which is easy as pie). That would mean that the array would take up a bit more than 3 terabytes of memory. Do you have that much RAM, or will you be swapping images to disk?

[Peter Porter]

Hi Shaggy,

I changed the array to store 1000 bitmaps, but I still get the same problem. No images display in the picturebox.

What I plan on doing is capturing the images to the array while at the same time saving them to disc via multithreading, but first I want to make sure they's being stored in the array by playing them in the picture box when I hit play. Multithreading comes later.

[Shaggy Hiker]

No crash, but nothing shows? That's not too bad....yet. Put a breakpoint on this line:

PictureBox1.Image = imagevault(counter)

When execution stops on the breakpoint, take a look at imagevault. You won't be able to see all that much about the images, but you'll be able to see whether or not there is something in the slots, especially the one at counter. I forget whether you can also see some useful attributes about the items, such as the length and width, or even just the size, but you may. Of course, if the array is empty, then that's why nothing is displaying and the investigation will shift to the recording phase.

[Peter Porter]

It does crash. I get "ArgumentException was unhandled" with "Parameter is not Valid" for PictureBox1.Image = imagevault(counter) highlighted in tan.

[Shaggy Hiker]

Do as I suggested with the breakpoint and I think you will find that imagevalue(counter) is Nothing. I believe that would result in exactly the exception you are seeing. That would be pretty telling, too, because it would indicate that the array is NOT being filled.

[Peter Porter]

How can I check the imagevault array slots after the program crashes?

[dday9]

This actually sounds like a job better suited for XNA. Are you able to use XNA? If so I wouldn't mind working up an example real quick.

[Shaggy Hiker]

Are you familiar with breakpoints? If not, now's the time to learn, as no tool is more valuable. You would put a breakpoint on the line that causes the crash, then execution will stop at the breakpoint just before the crash and allow you to look at the items.

However, at the time of the crash, the IDE should take you to the line that caused the crash. At that time, you can also look at the contents of the variable by putting the mouse in the variable and pressing Shift+F9.

[Peter Porter]

Hi dday9,

I never used XNA before. I had a working version of this program to capture into arrays, but that was months ago. Don't remember how I did it, but what I do remember is that it crashed after a few hundred images stating "Out of Memory". If I can get to the way it was months ago, I'll fix the out of memory problem with multithreading... while capturing, saving the images as jpegs, then disposing of the saved image array. This is to get high frame rates, because currently it doesn't record enough frames because saving the images slows it down.

[Peter Porter]

Thanks Shaggy. I did try the Breakpoint, but I added it while the program was running when I should have added it before.

The Breakpoint has been added correctly, and from what I'm reading for the imagevault array, it states for everything "Parameter is not Valid".
For the picturebox it states "Nothing" for Image.

Not sure how to fix this. It should be storing them into this array.

[Shaggy Hiker]

It doesn't matter whether you add a breakpoint is added while running or before. Either way, execution will stop when the breakpoint is reached (unless you add it after the exception has already halted execution). If the breakpoint is on this line:

PictureBox1.Image = imagevault(counter)

then PictureBox1.Image will certainly be Nothing, because the line hasn't actually executed, yet, so no image has yet been set. At that time, select imagevault (if you just click on it, you will probably get imagevalue(counter), which isn't the same thing) and press Shift+F9. That will bring up a window that allows you to see the array and everything in it. If that doesn't work, then you are doing something wrong, though I can't say what. You might also hover the mouse over counter, or select counter and press Shift+F9, but if counter was incorrect you would be getting an Index Out of Range exception rather than Parameter is not valid. What you need to see is the array in that window that comes up when you press Shift+F9, so that you can expand the array and see the contents. If the contents is a whole bunch of Nothing, then that's the problem.

[Peter Porter]

The QuickWatch window opened and it showed nothing under Name, Value or Type.

Why aren't they being stored into the array? I know the program is capturing, because it's saving the images to a folder. What I'm trying to do is get a higher frame rate of capture by storing the images to an array before saving via multithreading later.

[Peter Porter]

Ok. I figured out the problem, but the fix creates another problem. When disposing BMP, it also disposes the arrays within the imagevault array. Removing BMP.Dispose(), imagevault only stores up to 174 images before the program crashes. If I capture less than 174 images, I can replay them in the picturebox, so I know the imagevault array is storing images.

If I could figure out a way to keep BMP.Dispose(), but prevent it from disposing the images copied to the imagevault array.

[Peter Porter]

Ok, I discovered another way to record my screen using FFmpeg and DirectShow.

As soon as I have a working project, I'll update this thread and post it in the Code Bank.

[Shaggy Hiker]

When did I mention QuickWatch? You seemed to do everything OTHER than what I had suggested. I'm not a fan of QuickWatch, though I have used it occasionally.

You aren't copying images to the array. Since a bitmap is a reference type, the variable is nothing more than a reference to the actual bitmap, which is located somewhere in memory. So, when you put BMP into the array, all you are doing is copying the reference into the array. Another way of looking at it is that BMP holds the address of the bitmap in memory. What you are putting in the array is just a copy of that address. You aren't copying the bitmap into the array, so you just have two copies of the address, not two copies of the bitmap. It's like having a house with the address written on two scraps of paper. If you burn down the house, then both addresses point to the same burned house. Making more copies of the address doesn't change that.

[Peter Porter]

Shaggy, you didn't mention QuickWatch. That's what popped up when I followed your instructions, also I have a computer keyboard outside of my language. I'm not ignoring your instructions on puprose.

I made some changes to the code where the picturebox clones the BMP array, and with save disabled, the picturebox redisplays the images smoothly. I also have the garbage collector running to prevent the program from crashing. I did away with disposing of BMP because with it the Picturebox would be blank with a large red X.

The only thing now is if I could stream the images to a resuable memory block of a virtual buffer that ffmpeg could read and merge to a video flie. I tried Filestreaming, but it slows capturing to 23 frames a second. Do you know how I could stream to memory, or straight to ffmpeg?

[Peter Porter]

I now have captured images streaming to memory. I just have to figure out how to get ffmpeg to access that memory so it could convert everything into a movie file.