If the application is designed to run on a pendrive, why is everything on the hard drive? Surely at a minimum, the data should be on the pendrive?

If the application is installed on multiple computers that the pendrive could be moved to, simply store the path to the pendrive in the HKCU section of the Registry (or a file in a data folder). To get the initial value, you can either get the user to specify the location, or auto-detect the drive types and search them, or maybe even get the installer to store it for you.

If you want to store template files etc, you can put them (but not the application) in a data directory.
Vista makes a mess when you try to save the data in the Programs folder.
Yes.. and so do Windows 2000 and XP (in a bigger way than Vista does) for users with lesser rights.

If in doubt, check the Windows 2000 developer guidelines (via the FAQ link above), which I'm sure I have pointed you to before - and you should always follow.
That's why I feel "obliged" to target the Public folder for my installation.
No matter how you justify it to yourself, that is just an excuse for being lazy and unprofessional about the issue.