Silverlight had a control called DeepZoom designed EXACTLY for this. It came with tools that would help you define each zoom layer and the rules about when and how to switch between them. I don't know what happened to DeepZoom. I haven't heard of it in years. But it seemed really useful and perfect for this scenario, as when I look back over my explanation above, I feel like I'm describing "concerns for implementing DeepZoom".

Some really shady demonstration of (5) might be like this:

Fudge your zoom code to only have 2 levels, "zoomed in" and "zoomed out". Let "zoomed in" do whatever you currently do normally. Here's how "zoomed out" will work:

  • When loading the image, create a new Bitmap that's big enough to pan around in with the same aspect ratio of your image. I'm thinking make it twice as big as the form.
  • Rescale your huge image into that one. (Graphics.FromImage() on the smaller one, then use that Graphics to draw the bigger one with the bounds of the smaller: that'll rescale it.)
  • When "zoomed out", stop drawing the "real" image, and instead display the smaller one. Don't worry about getting pan just right yet, let it be janky.

See if that doesn't cause some significant speedup. If it does, you need to consider making several zoom levels and figuring out how to swap seamlessly between them. I don't envy you, it seems tough to try and keep all the coordinate systems in sync!