Absolutely not a pain at all. In fact, I prefer having it there because it never gets in the way and never gets bumped by elbows/forearms (mine or my neighbors).
Then again, I also like to shuffle behind so....
I have tables setup each way, but I prefer my offset one honestly. FWIW, that is the only way to shuffle, so we are good there.