You don't need to take the felt off.
But you're going to need to cut thru it, the foam, and plywood of the table. Exacto blade to cut thru the felt and foam.
I wouldn't use a reciprocating saw to make the hole thru the plywood. I'd suggest the use of a jigsaw or router. Make sure you cover your table with a plastic dropsheet to account for the sawdust mess.
You also need to be mindful of the placement of other items. Table legs underneath and if you have a drop box or anything else that may interfere.