I tend to be more trusting (probably to my detriment) so I’ve never worried about that. But it did freak me out in a smaller room that I went to, just because a smart dealer who has spent a lot of time dealing to half the players at his table should probably have a significant edge.
This happens occasionally, but it's pretty rare. As
@Anthony Martino points out, most dealers are in fact degenerate gamblers and very poor poker players. But there are also some good ones, most of whom eventually end up just playing full-time instead. This is what happened with me, and with another local dealer who used to play in my home game back when I lived in WA (Tyler Patterson - who has gone on to do pretty well in poker. He was a dealer at Kenmore Lanes back before he was playing/winning big events on TV. And I actually just watched him win a circuit event 2 days ago in Sacremento).
It's a bit of a catch 22 though because the smaller card rooms usually want the dealers to sit in the games to help keep them going during their breaks, in between pushes (a "push" is the dealer's turn dealing at the table). If there are only 2 games going, they'll usually have 3 or 4 dealers in rotation, and the two that are on break will have to sit in the games if there are seats open, in order to prevent the games from getting short-handed and breaking. But a dealer who is a strong player can also end up doing more harm than good. Back when I was a dealer, in my early 20s, I was also playing poker for a living. I was making more money playing than I was dealing, so I ended up just dealing part-time, mostly because I enjoyed it, and because the tips were good and I liked the community and had a lot of friends in the WA poker scene. But it got to the point where I started breaking games because I was winning too much. So then the owners started asking me to soft-play the players, and would handicap me with rules like, "no check-raising allowed". But it still didn't work. I was still breaking the games because they would all just dump their money to me (the player pool was pretty bad at poker, though most were great people). I started checking down big hands heads up, trying not to take all their money, but it just delayed the inevitable. They'd still go broke, I'd still win money, and the only difference was that the house got a little bit more of a rake. Eventually, I just stopped dealing altogether because it was an awkward dynamic when you're just sitting there taking everyone's money and then 30 minutes later, you sit down in the box and they're handing you tips. I made more playing cards than I did dealing anyhow, so it was just sort of a natural progression. Most dealers who are strong players end up taking a similar path, so they don't remain dealers for too long.