You Expect Me To Punt?

C-Span today reported Senator Lindsey Graham saying:

“You don’t agree with what I’m doing. You think I’m in Trump’s pocket. I get all that. It’s not lost on me of what you think. And I’m sad because I like you all. But to expect me to punt? You can forget it.”

What does ‘to punt’ mean? It must mean something like ‘to give up’ or ‘to change sides’ – something like that.

The sources suggest it means to give up an attempt or a course of action, so as not to fail.