There are lots of new factors in the post-XP world we live in. One place to begin would be to read New UAC Technologies for Windows Vista, though it is very late in the day to start learning about those changes now. They were first described for developers beginning back in 2006.
In this case you might want to read the part on Virtualization first.
Note that there were actually very few "new rules" starting in Vista. Many of these rules have been around since the Windows 95 Desktop Update in 1996 while others came about when transitioning from Win9x OSs to NT OSes such as Win2K, XP, Vista, Win7, Win8, and whatever comes next. The difference is that new mechanisms are now in place that enforce those rules.
There are also some issues for 32-bit processes running under SysWOW64, something a lot of people never saw until they moved to Windows 7. With Windows 7 a lot of vendors saw 64-bit Windows as an opportunity to make consumers buy more hardware, which helps keep the cost of a PC (and thus profits) high. This results in your program triggering another feature called WOW64 file redirection.
What it comes down to is that if your programs try to write to protected system areas of the hard drive strange things may seem to occur. This is Windows trying depserately to let your program run, the alternative being to kill it when you perform such illegal operations.
If you can say what "won't work" really means (exactly what is happening) there might be a resolution. Without the program source this is a little bit doubtful though.