Recordset Size

I know this is kind of a question and depends on how many columns you are returning but is there any generic rule when you should look into hitting the database every time they say filter some values instead of filtering the recordset. I don't know when you really get into trouble because of your recordsets size in memory(100,1000,10000...ect) Can anyone offer some pointers?

A lot of this comes down to the capacity of the machine/network you are working on. If its a network you really need to consider network traffic.