Is it an MS DOS program? If it is then I think you can right click on the file and go to properties and there are some memory options in there.
If not then the only other option I know of would be to just use a virtual machine (Virtual PC and VMWare Workstation are both free downloads) as you can limit the memory available to a virtual machine with ease.