Well, he probably wouldn't get re-elected if he did that. How difficult is it to change parties? In many states the only way you are affiliated with one party over another is if you declare it to be so. It's not like a contract, or anything. What keeps most politicians in the party that sent them to power is the recognition that if they switched parties they'd lose the next election. Specter calculated that he'd lose the next election if he DIDN'T switch parties, so the calculus of staying became backwards for him. There aren't any rules beyond that.