I’m not sure why we have to exclude things. There’s a lot of sports that I think are dumb that I won’t watch, but that doesn’t mean they’re not sports.
I think outside of Eriks, most of the folks are just trolling. It's a bit hyperbole, as are all the baseball enthusiasts who compare it to basketball as "just standing putting a ball in the hoop".
All sports are different, but baseball probably doesn't lend itself a bit to that thought of not necessarily requiring the same visually fit requirement that sports like basketball and soccer do. For example, Nascar fans are happy to recount just how difficult racing is on the body and the great shape drivers should be in to participate at the highest level, but people are always still going to see it as, well, sure, but you're driving a car.
I think a great way to understand this is to look at youth athletics. Go listen to travel ball coaches. They are looking for kids with "haunches", big thighs and asses", or for pitchers, someone that is all "elbows and assholes" - the latter meaning long legs and long arms, so that when they're pitching all the batter sees coming at them is limbs and a great/hidden release.
Do the same thing in basketball and coaches are looking for height and mature bodies obviously, but also pure athleticism. How they move, dip their hips, fluid, etc. There's technical aspects galore in both sports, but from a pure physical perspective, yes, baseball is not one that is hyper focused on athleticism. It's obviously a sport, but is one of the low hanging fruits that trolls are going to pick when trying to delineate between sports and games.