When you supply the picbox hWnd to the RegionFromBitmap, you are shaping the picturebox; therefore, the background disappears because it is cut away from the picturebox. The picturebox is no longer a rectangle; this is the key for understanding why you don't get the background color. The region once applied to a window to shape it is no longer yours to play with. Windows owns the region and per MSDN, you are never to reference that region again and are not to delete that region. If you are not setting the region to shape a window, always destroy that region at some point.

So to draw a border also, you'd call the routine 1 more time and frame the region, a different handle will be returned each time you call that routine. Whether you use AutoRedraw or not is up to you. AutoRedraw will keep the border should you minimize then restore the form or drag another window over your "button"; otherwise, it will be erased and you'll have to draw it again.

Last but not least, since this is your first time playing with creating GDI objects that need to be destroyed, I'd strongly suggest you spend some time reading my tutorial on Memory Leaks (linked in my signature below). I'd say it is an extremely high probability that you will write leaky code until you become fully aware of how leaks are created and how to prevent them. That tutorial will help.

As for the order of what you want to do
1) Shape the window first
2) Frame the region
3) Refresh the picturebox as needed
If you want to get a little more efficient, here's what you can do to call the routine only once
Code:
...
' AutoRedraw should be True for this method
RegionHandle = cRgn.RegionFromBitmap(Picture1.Picture.Handle)
FrameRgn Picture1.hDC, RegionHandle, hBrush, 1, 1
SetWindowRgn Picture1.hWnd, RegionHandle, True
DeleteObject hBrush
' don't delete the RegionHandle because you used it to shape the window with SetWindowRgn above