Unfortunately i haven't used that control.
Anyhow, if possible i would get the pictures currently loaded to prettyphoto and modify the pagesize. The problem here is that you are using a client control and a server control together so i am not sure how you can notify the server control about the pictures number.
You could p.e. count all the rows that are loaded at instantiation to the repeater and do a division so you can come to the pics per page you need. I think that could be done at pre-render but not 100% sure.
Another options is to set a web service but it a little vague in my mind how you can set data from the prettyphoto and then get the data from the repeater.
Another option yet would be to use a client side control instead of the repeated to display data. I think slimgrid, slickgrid or something is used with Jquery.