Soccer has a giant net with 1 guy who is standing in front of it blocking you from kicking a ball about 1/500th the size of the actual goal net...

Golf you have to hit a ball only slightly bigger than the hole you are trying to get it in, with a club...

If golf is not a physical sport, then what is it? a mental exercise?

When a golfer has a physical injury, that can keep them from playing. Things like poker are not in this category, they are no physical requirements, just mental.