I totally understand where OP is coming from. Not every game is for every person. Some games are way more social, some are more serious and focused on the game. It only takes one person to throw off the enjoyment of the game for everyone. I play with a lot of folks that I wouldn't invite to a more serious game that I'm planning for next week. They are fun to play with in the not so serious weekly game, but I don't want them to attend this planned game, they frankly don't care about poker as much as the rest of us do, they look at it as an entertainment expense, like gambling and expect to lose, poker is more just the price paid for entertainment and not a game that they are trying to get better at.
I'm glad that the pool of players I know can support both and that we can occasionally have a more serious game at a bit higher stakes, where the hands flow quicker, people know when there is time to step away from the table, when it's their turn and want to play instead of throwing chips in and hoping to get lucky. Folks that are challenging to play against and gets the mind game going. Sometimes it's nice to play against other poker players instead of some folks that get together to drink and toss some cards around.