[RESOLVED] Layer rendering (image program)

[bergerkiller]

Hello all

I'm working on my second project; a layered image displayer.
It works like this:
- The user adds a new "image layer"
- The user can set the texture, image, size, rotation and color
- This all can be done using a selection square and some dialogs

It all works fine, but I found that there is one thing still unclear to me:
- Can you render layers without totally starting over (refresh) on every change?

As example, I have 200 image layers. I change one image layer somewhere in between them, and the program has to refresh all those images. (1 - 200)
This takes awfully long.

Now I found a small solution; I gave all layers a "finalimage" so it can render a single layer. Next I used, in the program, a back, active and fore layer.

This way the program only has to render 3 images when chaging the "active" layer. When you select another layer, all 3 layers will refresh. (with all those 200 layers).

here is the code to get an idea:

    Dim OLDINDEX As Integer = -1
    Dim BackLayer As New Bitmap(800, 600)
    Dim ActiveLayer As New Bitmap(800, 600)
    Dim FrontLayer As New Bitmap(800, 600)
    Public CurrentIndex As Integer = -1
    Public PerformGlobalUpdate As Boolean = False
    Public Sub Render(ByRef g As Graphics)
        'Get the layer modified types
        Dim UpdateActiveLayer As Boolean = False
        Dim UpdateBackLayer As Boolean = False
        Dim UpdateFrontLayer As Boolean = False
        If OLDINDEX = CurrentIndex And PerformGlobalUpdate = False Then
            For i As Integer = 0 To Layers.Count - 1
                If Layers(i).Changed = True And i < CurrentIndex Then UpdateBackLayer = True
                If Layers(i).Changed = True And i > CurrentIndex Then UpdateFrontLayer = True
                If Layers(i).Changed = True And i = CurrentIndex Then UpdateActiveLayer = True
            Next
        Else
            PerformGlobalUpdate = False
            UpdateBackLayer = True
            UpdateActiveLayer = True
            UpdateFrontLayer = True
        End If
        If UpdateActiveLayer = True Then ActiveLayer = New Bitmap(800, 600)
        If UpdateFrontLayer = True Then FrontLayer = New Bitmap(800, 600)
        If UpdateBackLayer = True Then BackLayer = New Bitmap(800, 600)
        If UpdateBackLayer = True Or UpdateFrontLayer = True Or UpdateActiveLayer = True Then
            Dim e As Integer = Layers.Count - 1
            Dim s As Integer = 1
            If UpdateBackLayer = False Then s = CurrentIndex
            If UpdateFrontLayer = False Then e = CurrentIndex
            For i As Integer = s To e
                If i < Layers.Count And i > 0 And ((UpdateBackLayer = True And i < CurrentIndex) Or (UpdateFrontLayer = True And i > CurrentIndex) Or (UpdateActiveLayer = True And i = CurrentIndex)) Then
                    If i < CurrentIndex Then Layers(i).ApplyToImage(BackLayer)
                    If i > CurrentIndex Then Layers(i).ApplyToImage(FrontLayer)
                    If i = CurrentIndex Then Layers(i).ApplyToImage(ActiveLayer)
                End If
            Next
        End If
        'Render all 3 the layers
        If CurrentIndex <> 0 Then g.DrawImage(BackLayer, New Point(0, 0))
        If CurrentIndex <> -1 Then g.DrawImage(ActiveLayer, New Point(0, 0))
        If CurrentIndex <> Layers.Count - 1 Then g.DrawImage(FrontLayer, New Point(0, 0))
        OLDINDEX = CurrentIndex
    End Sub

It all works fine, but the only problem now is that it takes ages to select a different layer.

Anyone knows of an alternative solution to render different layers?

[NickThissen]

I've no idea what you are doing, but I'll try to explain how I would approach a layered image program.

I suppose you have a class that represents one image on a layer? It would store the image, the location, size, etc. Let's call this class ImageItem (because Image is already taken, perhaps you can come up with a better name):

Public Class ImageItem
  
    Public Image As Image
    Public Location As Point
    Public Size As Size
    
    Public Sub Draw(ByVal g As Graphics)
       g.DrawImage(Me.Image, Me.Location, Me.Size) 'or something like this
    End Sub
 
End Class

Generally you would use properties but I used fields so I could type it quicker


I would then make a class Layer that holds a collection of these image classes. You can give it a name too.

Public Class Layer
 
   Public Name As String
   Public Selected As Boolean
   Public ImageItems As List(Of ImageItem)
 
   Public Sub Draw(ByVal g As Graphics)
      For Each img As ImageItem In Me.ImageItems
         img.Draw(g)
      Next
   End Sub
 
End Class

Finally, on your form (or your 'canvas', picturebox, whatever), you keep a list of these Layers, and you can draw them when required. Now you can simply call the Draw method of one layer, and only the images in that layer will be drawn. They will be drawn on top of the existing image in the picturebox (or whatever you're drawing on) though so you probably need transparent images.


You could add images and layers using something like this

Dim l1 As New Layer()
l1.ImageItems.Add(New ImageItem() With {.Image = My.Resources.Image1, _
                    .Location = New Point(25, 100), _
                    .Size = New Size(50, 50)}
l1.ImageItems.Add(New ImageItem() With {.Image = My.Resources.Image2, _
                    .Location = New Point(15, 30), _
                    .Size = New Size(50, 20)}
l1.ImageItems.Add(New ImageItem() With {.Image = My.Resources.Image3, _
                    .Location = New Point(45, 300), _
                    .Size = New Size(50, 30)}
 
Dim l2 As New Layer()
l2.ImageItems.Add(New ImageItem() With {.Image = My.Resources.Image4, _
                    .Location = New Point(125, 300), _
                    .Size = New Size(50, 150)}
l2.ImageItems.Add(New ImageItem() With {.Image = My.Resources.Image5, _
                    .Location = New Point(153, 50), _
                    .Size = New Size(501, 230)}
l2.ImageItems.Add(New ImageItem() With {.Image = My.Resources.Image6, _
                    .Location = New Point(415, 10), _
                    .Size = New Size(510, 301)}
 
Me.Layers = New List(Of Layer)
Me.Layers.Add(l1)
Me.Layers.Add(l2)

You would generally handle the Paint event of your picturebox or drawing surface, iterate through the Layers list, and draw them all in turn. To control the order in which they are drawn, you just switch around the order of the layers in the list. You could also add some kind of Index property to your Layer and then sort the Layers list by that before iterating through it.

Private Sub PictureBox1_Paint(ByVal sender As Object, ByVal e As EventArgs) Handles PictureBox1.Paint
   For Each l As Layer In Me.Layers
      l.Draw(e.Graphics)
   Next
End Sub

[bergerkiller]

Yeah I already did that before, but then you get this problem:

For Each l As Layer In Me.Layers
      l.Draw(e.Graphics)
   Next

If I have like 200 layers, it will take ages to render.
That is what I meant with "refreshing", it renders every layer all over again.

Not an answer to my question, sorry if I have been unclear.

I want an alternative way of rendering layers without rendering everything all over again. Basically rendering one picture in a long stack of pictures, and end up with a see-through-able stack.

(bet this is impossible, but so many programs do this so there must be a way.)

EDIT:

What you posted is almost similar to my code example, all 200 images are converted in 3 layers:
- background, active and foreground layers
But then, if the active layer shifts to the background layer, you'll have to re-draw everything. (long wait on selecting new layer, because then the active layer changes).

Simplified: stack 200 picturebox-like controls ontop of eachother and change one picturebox in this stack.
My solution was to divide these 200 in 3 groups, then you only need to draw 3 times. Problem with this is, that if the layer-category changes these 3 layers have to be refreshed with all those 200 again. Not a real good solution...

And yep, got everything packed in classes. Inside these classes also a render function to render the image to a suitable 800x600 bitmap.

[bergerkiller]

Bumping it. :P

I found that you could fasten-up the rendering by giving every layer a separate updater thread.

It's like rendering all 200 layers simultaneously.

Now the only problem is binding it all together.

I tried using event triggers, to handle the rendering finish of a single layer.
The problem still remains unfortunately; all 200 layers must be rendered from 1 to 200.

My final thought is using a tree of layers. Every root has 2 nodes connected, and every node/root holds the finished rendering data.

You'll need 8 roots to render 256 layers. So to render a single layer, it updates 8 times the image of 2 nodes.

Second, I figured out it is VERY WRONG to use 800 x 600 bitmaps to hold the data.
It takes 2 ms to generate a new bitmap, so for 200 nodes it takes around 400 ms. Big lag.
Drawing the bitmap takes another 3 ms. 5 ms for every change.
Guess I'll try to store the graphics data.

Can't get any further than this, is there no reference or documentation about layered rendering in GDI? Or should I use directx 9.0 for this kind of stuff? And if so, how?

[NickThissen]

I don't know how bitmaps came into the picture. You're not drawing the layers to a bitmap and then drawing that to a picturebox or something, are you? You can just draw each layer directly to the graphics surface.

Threaded rendering doesn't really make sense, cause you're going to have to draw everything on the same surface in the end. I'm pretty sure multiple threads cannot draw on one object simultaneously.

As for saving the layers, you could just serialize the list of layers. I'm pretty sure an Image is serializable, so that would mean about 3-4 lines of code to save everything, and another 2-3 to load it back.

I'm afraid I can't help you much further. I've done quite a lot of GDI+ drawing, mainly to draw custom controls and such, but I've never run into the problem of having too much to draw. If you are really stuck you might want to check out the source of Paint.NET. It's a photoshop-like program written in C# (easily converted to VB unless it uses pointers or unsafe code, which now that I think about it it probably does...) and I'm sure it uses layers as well. I don't think the source is publicly available anymore but with a bit of googling you can the source of older versions easily.

[bergerkiller]

Actually your post did help a little.

I was storing the layers as bitmaps, and updating these bitmaps if the layer changes. The update thread checks on a rate of 20 fps if a layer has changed, and if so drawing all bitmaps to the form.

But now I found the main problem. Bitmap generation is slow, even slower than the actual layer graphics generation. You know of some way to store the graphics data of a layer and then applying it to an existing graphics?

Sorta like:
1. Layer changes and generates it's own graphics.
2. layer sends a modified call to the parent layer
3. Parent layer loads all his child layers' graphics
4. parent layer calls HIS parent layer.
5. All so forth until the parent is the root of everything and it updates the form

Basically I'm trying to find the fastest way of storing the layer. Bitmaps is a NOT.

EDIT:
I'm afraid I can't use a graphics object to store the data...you need to creategraphics of some sort to be able to use it.
First that was a bitmap, but those are slow...

[NickThissen]

You store the layers just like I said: in a Layer class that holds a collection of ImageItems, where one ImageItem represents on image in your layer.

Here, I've made you a simple example. It creates three layers with each three ImageItems (with a bunch of random images). You can select a layer and move it around. You could build on this example or implement it into your existing application. I would assume users would have the ability to move around individual items in a layer, and also resizing. It sounds like you have all that already though, but if you don't know how to do that you could check out my Shape Editor example (C# only though). It is very similar to the example I made you now except that there is no concept of layers (well, technically each Shape is a layer with one image only).

By the way, instead of calling Invalidate() with no arguments to update the picturebox, I call Invalidate with a Rectangle as its argument, which means only everything in that rectangle is re-drawn. That might make it a bit faster.

And oh yeah, the bounds of a layer are defined by the union of the bounds of all ImageItems in the layer. That seemed appropriate.


Let me know if I'm on the right track with this. Also please note that I cannot guarantee it will work fast with 200+ layers. It probably won't, but it sure as hell will beat storing each layer as a new bitmap each time.


Finally, I made this in VS2010 and manually converted it back to VS2008. I could open it in VS2008 but if you can't let me know.

[bergerkiller]

Wow great example, thank you

So you didn't refresh the entire screen, but only parts of it...seems so logical to think of it

And yeah, I already made all parts like the selection square, rotation graphic functions etc. Quite funny that you had the "getlayeratpoint", I made it too. (with a lot more functions like current selected layer first so you can move layers that are behind other larger layers)

Now to find out how you combined it all...

Owww. that's a downside

In my program there are layers with alpha channels. I see you invalidated a small region and painted the new image onto it.
The invalidation also invalidates the image data of other (overlapping) layers, causing it to disappear.

So the only solution to that is invalidating all regions that contain layers OR invalidating everything.

On the other hand, wow it's smooth!
Added 1000 layers using a for loop and still quite smooth! This must be some solution to me.

EDIT

Ah I get how you did that. You made the "canvas" redraw itself when an item is moved. No bitmap was used; you passed the onpaint graphics object through all layers and made them apply changes to it.

Thanks you so much, it helped a lot. And from the Netherlands like me XD
Time to change everything so the layers will modify a given graphics object instead of drawing it's own image over it.
Final question; is it possible to save the graphics data of the layers so the next time the loop passes the layer it can set that? Would make it a lot faster if large "imageitems" were used:

I mean this part:

    Public Sub Draw(ByVal g As Graphics)
        'Loop items if modified
        if Me.Modified = True then
             'Draw each item
             For Each item As ImageItem In Me.ImageItems
                 item.Draw(g)
             Next
             SaveGraphics()
        Else
             DrawLoaded(g)
        End if
        'Draw a selection border if selected
        If Me.Selected Then
            g.DrawRectangle(Pens.Red, Me.Bounds)
        End If
    End Sub

Of course I need to make the code for "SaveGraphics" and "DrawLoaded(Byval g as Graphics)"
The GraphicsState feature doesn't work, because it doesn't hold the drawn images.

[bergerkiller]

Wow I really can't understand how it can render 50 256 x 256 images so smooth!

Yep the onpaint and passing it's graphics solution was the solution I needed.

Now only need to figure out how I can store a graphics object of a single layer and applying it to the given graphics object, if that is even possible.
I'll just stick with re-generating all layers w/o storing it.

Thanks for the help.

[NickThissen]

I'm not sure what you mean by storing the graphics object of a single layer and applying it. The Graphics object is just an instance of the Graphics class used to paint on the picturebox. The picturebox itself provides it (by its Paint event, or OnPaint override in this case, same thing) and I simply pass it along to the layer, and then to the ImageItems, so they can draw themselves.

What would you want to do in the DrawLoaded method then? It would still have to draw everything, wouldn't it? So how is that any different than the loop above it? You can't magically draw a whole bunch of images in one go.

The point of the Paint event (or OnPaint override) is that the PictureBox (Canvas) decides when it wants to repaint itself (or you can force it by calling Invalidate). When that happens, it is completely cleared, and you have to draw EVERYTHING again manually. You can't choose to leave out one layer because that layer will then not be visible anymore.

[bergerkiller]

Well the point of storing it is, for example, if you have a layer with 5 2048 x 2048 images.

The point of storing it is to prevent the useless re-drawing of all these images. Why generate them again if you can use a previously-generated image?

Or does it work differently, and doesn't it draw all 5 images again?

[NickThissen]

You misunderstand. When the PictureBox needs to be repainted (when it raises its Paint event and calls its OnPaint method), it is completely blank and you NEED to paint EVERYTHING. Anything you don't paint in the Paint event is not visible.

If you have 5 2048x2048 images, you will have to draw them. There is only one way: using the DrawImage method.

The whole point of using the Paint event is that this only happens when required. People new to this often use a timer or something and draw it every x seconds, but if you just let your window sit and do nothing, it doesn't draw anything as it just keeps displaying the same image. Only when you resize the window, or drag something else on top of it (I think) does it need to refresh, and at that point you have no choice but to draw the images again. If you don't, they aren't there when the picturebox is updated.

[bergerkiller]

I know, previously I used a thread (sort of liek the timer) to refresh, and only made it refresh when needed. (checked if updatescreen = true and if so, set it to false and begin updating the screen). If it was false, it waited for 50 milliseconds for the next check (to reduce cpu usage).

It was quite smooth, but not when there were 20+ 800 x 600 layers.

So yeah, what does the invalidate region do then? You said it draws everything all over again, does that apply the same for the drawing to the form itself?

Kinda confused now.

EDIT

And is it time consuming to draw 5 2048 x 2048 images? If so I need some way of storing it...

[NickThissen]

Try this piece of code for example:

    Private _AlreadyPainted As Boolean = False

    Private Sub PictureBox1_Paint(ByVal sender As System.Object, ByVal e As System.Windows.Forms.PaintEventArgs) Handles PictureBox1.Paint
        If Not _AlreadyPainted Then
            e.Graphics.FillRectangle(Brushes.Blue, PictureBox1.ClientRectangle)
        End If

        _AlreadyPainted = True
    End Sub

It will fill a PictureBox with a blue color initially, but it will only do that once. As soon as the Paint event is raised, the AlreadyPainted boolean is set to True and the paint code does not run again.

Run this code. The PictureBox is blue. Now resize the form so that part of the picturebox is outside of the boundaries of the form (invisible). Then, make the form larger again. You will see that the picturebox does not repaint that part!

This happens because, when the form is resized, it tells the PictureBox to re-paint that part which has become visible. Due to our boolean check, it does not paint the blue color anymore. However, it does still paint the rest of the picturebox blue (the part that remained visible), because it has not refreshed that.


If you Invalidate the entire PictureBox (either by calling Invalidate without arguments or by calling Refresh), then it will draw the entire picturebox. If you call Invalidate with a Rectangle argument or a Region argument, it will only draw that part. It will still run the code that draws the picturebox (in the Paint event), but that code does not result in the screen changing. The code running doesn't take very much time (neglegible), but the actual painting does, so you should not do that when it's not required.

In my first example, when you move a layer, you only need to refresh the previous bounds (so it clears the screen there) and refresh the new bounds (so it draws the layer at the new location). If the layer is small, that's a huge part of the screen that doesn't need to redraw.


It is probably time consuming to draw 5 images that large, but I haven't tried. Unfortunately, there is no way to 'store' it and draw it faster, unless you simply don't draw them again. if they have changed, or moved, you have to draw them again, but if they haven't just leave them out of the invalidated region and they won't be redrawn.

[bergerkiller]

Aaah that helps.

So you don't necessarily have to re-draw everything. You only need to draw the images that are inside the invalidated regions?

And yeah I noticed how it works. Added 5 layers with a 2048 x 2048 image. If 4 are not visible the 5th renders smooth. Does this mean the drawimage only draws on the regions visible and not everywhere?

[NickThissen]

No, DrawImage tells 'windows' to draw that image on to the picturebox. 'Windows' will only do that however when the PictureBox refreshes that part of the picturebox. If you refresh a different part, you still run the DrawImage code, but you won't see the image because 'windows' has not drawn that part of the picturebox (yet).

[bergerkiller]

Ok. Graphics tell what is where; the control/window tells what should be painted using the graphics object.

I made an app for myself using the onpaint methode, but now I found one annoying bug I experienced before.

While the form is redrawing the old and new regions, the screen control background color shines through the new image.

How can I prevent the form from redrawing it's own control color all the time? I tried to override the onpaint, but no success so far.

EDIT

nvm; forgot to add Me.DoubleBuffered = True XD

[NickThissen]

What do you want to draw instead of the back color? Just draw that. You can use FillRectangle to fill a rectangle with a solid color, a gradient, whatever...

[bergerkiller]

Nah I'm testing at the moment, trying and finding out what could be supported using this methode.

One interesting thing is you can give the layer "Children" of type label. Then you can add labels to a label, generating a hierarchy. The label draw sub then also executes the draw sub on all children.

So this stuff is interesting

Ah I get it now.
Just set an invalidated region for the old and new region of the image when the region changes. This way you can just change the position and the screen updates it automatically.

EDIT

Yep it all works. Thanks for your help.
Now making all sorts of (overriding) classes, such as a new color class with new features like:
- to/from ole db
- to/from decimal text
- changed property
- to/from drawing color
- to imageatribute

It is a lot smoother now I used the onpaint methode, so it definately helps.

[bergerkiller]

Yep my conclusion is completely certain now:

- Do not store layers as bitmaps, bitmap generation is slow
- Just re-draw everything all the time
- Only store bitmaps if generation takes longer than 20 ms (e.g. resizing images).
- To change the color of an image, use an imageattribute and apply it using the graphics.drawimage.
- Using this methode I managed to draw 100 layers with a trail effect. Awesome.

Thanks nick, you example improved the render speed a lot.

[boops boops]

I think I have something to add if you do decide to store layers as bitmaps: store them in 32bppPARGB (premultiplied) format. In some tests with 100 layers images of 600x600 random A, R, G and B pixels, I found that compositing them in 32bppPARGB format was consistently 2.2 x as fast as when stored in 32bppARGB format.

Some other observations:

1. Compositing the layers to a memory bitmap (instead of painting them one after another in OnPaint or the Paint Sub) was about 30% slower with ARGB images, but about 2% faster with PARGB images. This is a paradoxical result but I can't see a fault in my testing logic. In any case the difference is not enough to get excited about, so just use PARGB.

2. Using OnPaint instead of handling the Paint event makes no difference whatsoever with the same graphics code. It's what I would expect since MyBase.OnPaint simply fires the Paint event.

3. Using Graphics.CompositingQuality = HighQuality takes about 7x as long as the default (which corresponds to CompositingQuality = HighSpeed). Other graphics quality settings have no influence on compositing, which is what I would expect.

4. I am prepared to be proved wrong on this, but I suspect that it will be possible to composite multiple layers much faster by storing each layer as an integer or byte array (as obtained by the LockBits method) and compositing per pixel. Assuming each pixel is on average 50% opaque, a given pixel composite should reach 100% opacity after on average 8 layers. So in theory the average number of layers to be composited per pixel should be on average 8, even if there are hundreds of layers. It does not seem that Graphics.DrawImage uses this approach, since the processing time is roughly proportional to the number of layers.

BB

[bergerkiller]

Well actually, the only time I want to know is:
- How long does it take to generate a 800 x 600 bitmap (3 ms in my tests)
- How long does it take to draw a 800 x 600 bitmap (2 ms in my tests)

[NickThissen]

What's the difference between generating a bitmap and drawing a bitmap? I don't understand what you mean by generating one... If you mean creating one, surely you have to draw it too? You use bmp.CreateGraphics to get the graphics object, and then the DrawImage method on that graphics object to draw it to the bitmap. Of course it is going to be slower then just DrawImage by itself.

[boops boops]

Well actually, the only time I want to know is:
- How long does it take to generate a 800 x 600 bitmap (3 ms in my tests)
- How long does it take to draw a 800 x 600 bitmap (2 ms in my tests)

What method are you using to generate the bitmaps? And how are you measuring the timing? BB

[bergerkiller]

To generate bitmaps:

        Dim bmp As New System.Drawing.Bitmap(800, 600)

To test drawing:

        Dim dbmp As New System.Drawing.Bitmap(800, 600)
        Dim bmp As New System.Drawing.Bitmap(800, 600)
        Dim g As System.Drawing.Graphics = System.Drawing.Graphics.FromImage(bmp)
        g.DrawImage(dbmp, New System.Drawing.Point(0, 0))
        g.Dispose()

I used a StopWatch object to calculate the time.
Example of calculating the different steps:

        Dim S As New Stopwatch
        S.Start()
        Dim dbmp As New System.Drawing.Bitmap(800, 600)
        Dim bmp As New System.Drawing.Bitmap(800, 600)
        Dim bitmapgenerationtime As Long = S.ElapsedMilliseconds * 0.5
        S.Reset()
        S.Start()
        Dim g As System.Drawing.Graphics = System.Drawing.Graphics.FromImage(bmp)
        Dim graphicscreatetime As Long = S.ElapsedMilliseconds
        S.Reset()
        S.Start()
        g.DrawImage(dbmp, New System.Drawing.Point(0, 0))
        Dim graphicsdrawtime As Long = S.ElapsedMilliseconds
        g.Dispose()
        S = Nothing
        MsgBox("Bitmapgeneration: " & bitmapgenerationtime & " ms" & vbCrLf & "Graphicsgeneration: " & graphicscreatetime & " ms" & vbCrLf & "Draw time: " & graphicsdrawtime & " ms")

What's the difference between generating a bitmap and drawing a bitmap?

The difference is:
- generating is:

Dim bmp as new bitmap(800, 600)

- drawing is:

g.DrawImage(bmp, New Point(0, 0))

Both takes time to do, and the larger the bitmap, the longer.

EDIT:

There is a second factor involved; the clip of the graphics object. This is the region the graphics object will draw on the surface. If this is small, drawing will be quick. Large (like total) will take longer.

[NickThissen]

Sorry, I have no idea what you're doing.

First, you create a new Bitmap. Then you create another Bitmap. Then you create a Graphics object from the second bitmap, and draw the first bitmap. So you are drawing a blank bitmap onto a blank bitmap. The result? Nothing at all. A blank bitmap, which you already had after your first line of code...

[bergerkiller]

then you don't understand the purpose; this is a duration test. The actual code used textures of course, which I added on layer creation.

I had to find out how long it took to generate (and draw) the 800 x 600 bitmap I used before. (to store the layers).
The layer draw code did the following:
- Layer changed?
+ yes
Generate and use new bitmap
+ no
use previously generated bitmap

Basically needed if it takes long to re-generate the layer. Then you can store the layers to speed-up the drawing system.
Unfortunately it takes long to "store" the layer, which made it useless. instead I removed the "changed" part and just made it re-draw all the time. (which is quicker, because usually the separate images are smaller than the 800 x 600 stored bitmap).

In short terms, previously I stored the layers as 800 x 600 bitmaps and made them draw to the form in an updater thread. It checked every 50 ms if the screen needs updating. (program itself sends a message it needs to update).

Now a difficult part (first thread I posted), it checks of all layers if they are part of the background, active or foreground layers.
If a layer in the background changes, it updates the entire background bitmap. If a layer in the active layer was changed, it updates the active layer. Same for foreground layer. At last, it draws the 3 layers to the form.

The benefit: it draws only 3 times the layers and it only needs to re-draw the active layer.
The downside: it needs to update all 3 layers and all child layers if the order changes.

Bet you don't follow me anymore, because it is hard to understand myself. Took me a day of testing to script it. xD

Here is the actual drawing to form part:

        If CurrentIndex <> 0 Then g.DrawImage(BackLayer, New Point(0, 0)) 'dont draw if there is no backlayer
        If CurrentIndex <> -1 Then g.DrawImage(ActiveLayer, New Point(0, 0)) 'dont draw if the activelayer is empty
        If CurrentIndex <> Layers.Count - 1 Then g.DrawImage(FrontLayer, New Point(0, 0)) 'Dont draw if there is no frontlayer

[NickThissen]

Even with actual textures you wouldn't create two bitmaps and then draw one on to the other. If you already have a bitmap, why draw it to another bitmap?

[bergerkiller]

well I did, to store the layers.

Had to do the second 3-layered storage because else it has to draw all those 800 x 600 bitmaps of all the layers to the form, even if nothing had changed.
The 3-layer-system allowed me to change the 100+x drawing to a maximum of 3x drawing if only the selected layer (activelayer) changed.

And yeah, it was kinda useless to store them. At the time it seemed a reasonable solution.

Why draw it to a new bitmap?
because the new bitmap also contains data of previous layers. It has to add the layer to it.

This was how the layer class looked like (almost):

    Public Class Layer
        Public name As String
        Public images As List(Of Texture)
        Public Changed As Boolean
        Public Finalimage As Bitmap
        Public Sub Draw(ByRef g As Graphics)
            If Me.Changed = True Then
                Me.Changed = False
                Finalimage = New Bitmap(800, 600)
                Dim g2 As Graphics = Graphics.FromImage(Finalimage)
                g2.TranslateTransform(Texture.X, Texture.Y)
                g2.DrawImage(Texture.image, New Point(0, 0))
                g2.Dispose()
                'in the actual code there was some resizing, recoloring and mutliple images of type texture were drawn.
            End If
            g.DrawImage(Me.Finalimage, New Point(0, 0))
        End Sub
    End Class

The updater did nothing more than check every layer if it was changed, if so, add it to the update queue of one of the 3 sublayers.

The idea was to balance out the time it takes to draw to the layer finalimage and the time to draw it to the form. Unfortunately, it takes longer to store the layer than to draw to the form, making it useless.

[boops boops]

Hi Bergerkiller, it's possible that you are not getting useful timing information by bracketing the Graphics.DrawImage command with the stopwatch reading. If I measure Graphics.DrawImage in a similar way with a 800x600 PARGB image I get a reading of 1 ms:

Private Sub PictureBoxX_Paint(ByVal sender As Object, ByVal e As System.Windows.Forms.PaintEventArgs) Handles Me.Paint
      Dim sw As Stopwatch = Stopwatch.StartNew
      e.Graphics.DrawImage(layer, Point.Empty)
      Console.WriteLine(sw.ElapsedMilliseconds.ToString)
End Sub

With an ARGB image it takes 8 ms! I have a normal onboard graphics chip, not a fast graphics card. However, I do not trust this way of measuring the drawing time.

Instead I have been measuring the time it takes to return from a Refresh instruction. This just has the effect of raising a Paint event with the above handler, so you might thing the timing would be identical.

Dim sw As Stopwatch = Stopwatch.StartNew
PictureBoxX.Refresh()
Console.WriteLine(sw.ElapsedMilliseconds.ToString)

But this gives a time of about 11 ms. for a PARGB bitmap and 16 ms. for an ARGB.

Using an empty bitmap instead of a bitmap full of random pixels does not seem make a big difference. I get a return time from Refresh of about 10 ms. for an empty PARGB 600 x 800 bitmap as opposed to 11 ms. for a random PARGB bitmap. ARGB bitmaps take 16 ms, whether empty or full.

I believe that the higher figures give a better representation of how long it takes to draw a bitmap. The effect of Refresh is just to raise the Paint event with the above sub. My interpretation is that Graphics.DrawImage does preprocessing for the actual hardward drawing, but the time to return from Refresh includes the actual painting of the image on the screen. Why else would it be significantly longer?

I don't know what implications this could have for your "balance", but it's clear that timing code like this reliably is far from easy.

BB

[bergerkiller]

Well this is what I found:

The onpaint methode returns a graphics object; BUT; this is a referenced object. This means, if the end point changes, the graphics object changes.

Before you can do anything with it, you need to "invalidate" regions. This is nothing more than adding clips to the graphics object, so the graphics object knows what on the form should be painted.

Total invalidate means all graphics will be applies to the screen, but if you only invalidate the region that changes, it reduces lag significantly.

So there is no reliable way of testing the graphics draw image duration. It changes based on what size of area it has to actually draw to the surface.

So, at some point, my old methode will be the faster or equal to the new methode, but the new methode boosts the render speed of small areas, making it overall better.

Try the following to check this:
- Make a new 1 x 1 argb bitmap
- create a graphics object using "fromimage"
- draw the 800 x 600 bitmap again.
The duration should be almost 0 now, since it only has to draw a very, very small surface of the bitmap.

My problem was that it was refreshing the entire 800 x 600 screen all the time, instead of updating the old and new regions.

Conclusion: it seems impossible to store graphics. Graphics are nothing more than "appliers", they apply to the given surface. They don't contain any image data, they only contains some tranformation data. This is why the bitmap changes with fromimage, and you can't restore the old bitmap state.

Hope this is useful for someone

[NickThissen]

I don't think your understanding is completely correct. What do you mean by a 'referenced object'? The Graphics class is a class and hence a reference type. Every Graphics object you have is a reference to some piece of memory, whether you get that object via CreateGraphics or via the paint event handler. The truth is, the Graphics object has to be created somewhere. The only reason you shouldn't do it yourself is because you already have one.

What happens is something like this. It's probably not completely accurate but the details don't really matter:

1. You call Invalidate (possibly with some region or rectangle) on your PictureBox.
2. After a short while, the PictureBox calls its own Update method. You can do this manually too right after you call Invalidate but it doesn't really matter.
3. In the Update method, some API is called that makes Windows want to re-draw the invalidated region of the PictureBox.
4. When that happens, a new Graphics object is created (probably using its FromHandle shared function instead of the CreateGraphics method, but they do the same thing). That Graphics object is encapsulated in a PaintEventArgs instance.
5. The OnPaint method of the PictureBox is called, with the PaintEventArgs instance in step 4 passed along (and thus also the Graphics object).
6. The OnPaint method raises the Paint event, again passing the same PaintEventArgs along.
7. You do your custom painting in the Paint event.
8. Windows finally redraws the PictureBox, but only the invalidated region. If your drawing is outside of that region you cannot see it.



Storing graphics doesn't make any sense. You store an image and its properties such as position and size. When you want to view that image, you draw it. You can't magically make an image appear without drawing it; there is always drawing involved. The Graphics object is merely a class that allows you to easily use GDI+ drawing methods without having to resort to difficult API calls.

[NickThissen]

Here, I tried to convey that in an animated image (was a little bored ). I hope it makes sense



As you can see, the Image is much larger than what you can see after the hardware has drawn it. If you would call Invalidate again without parameters, or simply Refresh, (or wait for windows to perform a refresh), then the rest of the image would become visible too.

Note that Refresh simply calls Invalidate with no arguments followed by Update, so it simply forces a repaint of the entire control.

[bergerkiller]

But you missed one point, which I meant "referenced" about:

When you call g.drawimage(params); what does it actually do?
You might think, add the image to the graphics object.

Nope, it does nothing more than draw the image onto the invalidated (clip) regions of the object it was created on.

So, if you paint a very large image, but the clip size is 20 x 20; drawing will be fast because it doesn't have to draw the entire image.

In your example you say it draws everything and at last only updates the clip region, but this is false. It draws as soon you send a drawimage call, and it only draws the invalidated regions.

So actually; in your animated image it should do the "hardware drawing" during the drawimage, not after all drawing was done.

Referenced object; because it references the bitmap/handle it was created on, it doesn't hold any image data.

[NickThissen]

That is exactly what I tried to indicate with my image, lol. When "You" are using DrawImage, I am showing a transparent image, indicating that that is the 'someImage' image you are trying to draw. It is not drawn yet. Only after the 'e.Graphics.DrawImage' line as finished is the image drawn (no longer transparent). It does the 'hardware drawing' after the DrawImage line. It can't start drawing before it has executed that line because it doesn't know what to draw yet at that point.

[boops boops]

There is another point I would like to see clarified. MSDN stresses that Invalidate does not cause the synchronous repainting of a control. In other words, the effect of invalidating a rectangle of the screen in .Net is to queue the screen update for processing on the next screen scan.

The Refresh and Update methods, according to MSDN, force immediate repainting of the control. This leads me to think that you need to use Refresh or Invalidate(rectangle) + Update when you are measuring timing. But this does not mean that these should be used in normal processing (or that Update is part of the normal drawing cycle as Nick suggested).

After writing this, it occurred to me that a better way to test the time to draw an image on the screen would be to time a Refresh with Graphics.DrawImage in the event sub, and then to time Refresh again with nothing in the event sub. I just tried it and got 10 ms. for Refresh+Image and 6 ms. for Refresh+No Image. This indicates that the actual painting time is about 4 ms. So Bergerkiller, maybe your 2 ms. was closer to the truth.

BB

Edit: I should add that "immediate repaint" can't be taken too literally since the typical screen refresh rate is only 60 Hz. I take it that it means the invalidated area is definitevely scheduled for repainting.

[NickThissen]

As far as I understand it (I'm not absolutely sure about this), when you call Invalidate the region is indeed placed in some kind of queue, and the actual painting happens only when the Update method is called. This allows you to invalidate multiple regions or rectangles in a row, and only call Update when you invalidated all the regions that need to be painted.

At first I thought that therefore one must call Update in order to get a repaint at all, but that doesn't seem to be the case. With all the painting I have ever done, calling Update or not does not seem to make a difference. This suggests to me that the control calls its own Update method when at least one region of it is invalidated. This can't happen instantly (otherwise calling Invalidate multiple times and only then calling Update would be nonsense) so I think that the 'control' simply waits a short time, in which you can potentially call Invalidate a few more times with different regions, until calling Update.

If you are for example moving some shape across the PictureBox, as I am doing in my example project, then you can now Invalidate both the old and the new locations of the shape before calling Update. If Invalidate would immediately repaint the control then the paint code would be run twice (or many many more times in more complicated drawing scenarios), while it is much more efficient to just run it once and then redraw the invalidated regions in one go.

I think the best option is to always call Update after you have invalidated all regions, but I have never seen anything behave wrong or even strange when not calling Update.

[bergerkiller]

It's a tiny bit off-topic; but how can I invalidate a rotated rectangle?

In my program, the graphics object is tranformed multiple times:
- first translation to the main location
- then translation to make it rotate from the middle
- rotating tranform
- offset back (negative of the second translation)
- the above 3 again for a dynamic translation for animation purposes.

Now figuring out a way of calculating the new rectangle location and rotation, guess that wont be too much of a problem.

Even if I have the rectangle location, I can't invalidate it because it is rotated.
At the moment I am converting the 4 edges of the rectangle into points, and then rotating those based on a base point. (rotatepoint function).
With some math it then generates a non-rotated rectangle specifying the bounds of the rotated rectangle.

Problem is it is invalidating regions that don't need it. Any way of invalidating rotated rectangles (with a matrix or something)?

EDIT

For those interested; here is my rotatepoint function:

    Public Function RotatePoint(ByVal Point As Point, ByVal BasePoint As Point, ByVal Rotation As Single) As Point
        Dim width As Integer = Math.Abs(BasePoint.X - Point.X)
        Dim height As Integer = Math.Abs(BasePoint.Y - Point.Y)
        Dim r As Double = Math.Sqrt(width ^ 2 + height ^ 2)
        If r = 0 Then
            Return Point
        Else
            Rotation *= Math.PI / 180
            If Point.X >= BasePoint.X And Point.Y >= BasePoint.Y Then
                Rotation += Math.Atan(height / width)
            ElseIf Point.X <= BasePoint.X And Point.Y >= BasePoint.Y Then
                Rotation += Math.Atan(width / height) + 0.5 * Math.PI
            ElseIf Point.X <= BasePoint.X And Point.Y <= BasePoint.Y Then
                Rotation += Math.Atan(height / width) + Math.PI
            ElseIf Point.X >= BasePoint.X And Point.Y <= BasePoint.Y Then
                Rotation += Math.Atan(width / height) + 1.5 * Math.PI
            End If
            Dim x As Single = Math.Cos(Rotation) * r + BasePoint.X
            Dim y As Single = Math.Sin(Rotation) * r + BasePoint.Y
            Return New Point(x, y)
        End If
    End Function

And to get the bounds of the rotated rectangle:

    Public ReadOnly Property Bounds() As Rectangle
        Get
            Dim br As New Point(Me.X + Me.Width, Me.Y + Me.Height)
            Dim tl As New Point(Me.X, Me.Y)
            Dim points(3) As Point
            points(0) = RotatePoint(tl, Me.MiddlePoint, Me.rot)
            points(1) = RotatePoint(br, Me.MiddlePoint, Me.rot)
            points(2) = RotatePoint(New Point(Me.X + Me.Width, Me.Y), Me.MiddlePoint, Me.rot)
            points(3) = RotatePoint(New Point(Me.X, Me.Y + Me.Height), Me.MiddlePoint, Me.rot)
            For Each p As Point In points
                If p.X < tl.X Then tl.X = p.X
                If p.Y < tl.Y Then tl.Y = p.Y
                If p.X > br.X Then br.X = p.X
                If p.Y > br.Y Then br.Y = p.Y
            Next
            Return Rectangle.FromLTRB(tl.X, tl.Y, br.X, br.Y)
        End Get
    End Property

[NickThissen]

You can also invalidate Regions, although I've never tried it. I think a Region can have any shape.

[bergerkiller]

Would this work?
Can't really test it because my entire program is now build around the rectangle invalidating..

    Public ReadOnly Property Bounds2() As System.Drawing.Region
        Get
            Dim br As New Point(Me.X + Me.Width, Me.Y + Me.Height)
            Dim tl As New Point(Me.X, Me.Y)
            Dim points(3) As Point
            points(0) = RotatePoint(tl, Me.MiddlePoint, Me.rot)
            points(1) = RotatePoint(br, Me.MiddlePoint, Me.rot)
            points(2) = RotatePoint(New Point(Me.X + Me.Width, Me.Y), Me.MiddlePoint, Me.rot)
            points(3) = RotatePoint(New Point(Me.X, Me.Y + Me.Height), Me.MiddlePoint, Me.rot)
            Dim m As New System.Drawing.Drawing2D.Matrix
            m.TransformPoints(points)
            Dim r As New System.Drawing.Region()
            r.Transform(m)
            Return r
        End Get
    End Property

EDIT

never mind, this wasn't working.

Found a nice alternative; made a new matrix object, did the translations and then associated the matrix with both the graphics object as a new region. Screen updates the region and all done.

Now the only bug here is that, when you move an image around real quick, part of the image disappears. Very weird, guess the drawed image location is not the same as the invalidated region.