If instead of voting for a person, people voted for issues - You then match people's issue votes to the candidate that best supports them. We'd have completely different outcomes. I think most people would be surprised by where their votes would land....
I suspect you're right though it might get weird in a two party system. Your two party setup creates an inevitable (and often entirely artificial) axis.