RotateTransform seems to have no effect.

[jo0ls]

I'd have done it like this

Dim myGraphics As Graphics
        Dim bm As New Bitmap(PictureBox1.Width, PictureBox1.Height)
        myGraphics = Graphics.FromImage(bm)
        myGraphics.TranslateTransform(100, 100)
        myGraphics.RotateTransform(45)
        myGraphics.DrawRectangle(Pens.Red, -50, -50, 75, 75)
        PictureBox1.Image = bm

Don't start drawing on pictureboxes. They are not there to provide a graphics surface. Just create a bitmap of the correct size and draw in that. The advantage of doing it this way is that the image will stick around.
Otherwise, if a paint event is called on the picturebox, then the default drawing will overdraw your drawing (it will clear the background so you lose everything). If you draw in a bitmap and assign the bitmap to the pictureboxes' image property then it will be drawn during that paint event.
Next up, you might aswell just say pens.red in the drawrectangle call. Defining your own pen will just use resources. It's not a big deal...

Finaly rotation.
what your original code did was to:
draw a rectangle
move the origin to 100,100 (the origin is the location (0,0))
rotate both axes about the new origin by 45%

The important thing is that moving the origin, and rotating the axes does not effect the drawing that you have already done. Imaginge you are an animator and use the transparent plastic sheets to draw your image. You draw the rectangle on one sheet, then you grab the next shhet and put it on top. Then you draw the new axes you want to use - with the different origin and the rotation. You do not effect the old rectangle, any subsequent drawing you do uses the new co-ordinate system.

So what I did was to move the 0,0 first, then rotate. Finally I have to try to imagine the next axes. I fail to imagine the new axes and grab a bit of paper... Then I work out where I will want to draw the rectangle.
As the origin is in the middle of the drawing area, I start the rectangle at (-x,-y). The width is easy enough.
I used numbers that allowed the rectangle to fit into my picturebox.

this one shows the axes before and after a translation & rotation

Dim bm As New Bitmap(PictureBox1.Width, PictureBox1.Height)
        Dim g As Graphics = Graphics.FromImage(bm)
        g.DrawLine(Pens.LightGreen, -100, 0, 100, 0)
        g.DrawString("+x", Me.Font, Brushes.Black, 30, 10)
        g.DrawLine(Pens.LightGreen, 0, -100, 0, 100)
        g.DrawString("+y", Me.Font, Brushes.Black, 10, 30)
        'move origin to the centre of the bitmap
        g.TranslateTransform(bm.Width / 2, bm.Height / 2)
        'rotate 45 degrees.
        g.RotateTransform(45)
        'draw the same lines
        g.DrawLine(Pens.Orange, -100, 0, 100, 0)
        g.DrawString("+x", Me.Font, Brushes.Black, 30, 10)
        g.DrawLine(Pens.Orange, 0, -100, 0, 100)
        g.DrawString("+y", Me.Font, Brushes.Black, 10, 30)
        'show the boundary of the picbox:
        Me.PictureBox1.BorderStyle = BorderStyle.FixedSingle
        Me.PictureBox1.Image = bm