Hehe love all these comments, my opinion is yes it is a sport, but then again I've had horses and been riding 18 years! And @alicexxx it is hard work, but thats only if you ride correctly, properly, use the correct muscles, don't let the horse/pony carry you. The thing about riding is that you 'work with' the horse and not let the horse carry you around, do the work for you, or force the horse into submission et cetera. I could talk about horses for hours and I could argue for days and days about whether or not riding is easy and you 'just sit there' it is sooooo much more to it that sitting on a horse. That my dear is called plodding about, but actual 'riding' means that both you and the horse work in harmony together, create a bond, a partnership. Okay I'm going to stop there before I rant even more.
But then again, I have never seen you 'ride' so I can't judge your ability, and you can't judge mine or others :)