In golf, no one is trying to stop you from doing what you're trying to do...just yourself. A different kind of skill/test/competition, but not a "Sport". A game, recreation, good-walk-spoiled, but not a "Sport" unless you're Happy Gilmore.
So, you are defining what qualifies as a "sport" by whether or not someone is trying to stop you from what you're trying to do?