You have stated you are working internally, I can sympathize with your needs. I actually have the same needs sometimes with some of our older applications that don't have an API and will not be updated ever. It makes more sense for us to scrape 3000 internal web pages than to have someone manually do it.

Once you have the source code you need, the first thing you should do is create a watch on the variable and figure out what qualifies the need to perform a redirect. Once you establish the pattern that occurs, its just a matter of pointing to the URL that you are expecting based on the previous variable and initiate another download.