Here's an excellent message by Mark Devers of Capital Hill Baptist on the subject of "Jesus Paid Taxes", from Mark 12:13-17:
And they sent to him some of the Pharisees and some of the Herodians, to trap him in his talk. And they came and said to him, “Teacher, we know that you are true and do not care about anyone's opinion. For you are not swayed by appearances, but truly teach the way of God. Is it lawful to pay taxes to Caesar, or not? Should we pay them, or should we not?” But, knowing their hypocrisy, he said to them, “Why put me to the test? Bring me a denarius and let me look at it.” And they brought one. And he said to them, “Whose likeness and inscription is this?” They said to him, “Caesar's.” Jesus said to them, “Render to Caesar the things that are Caesar's, and to God the things that are God's.” And they marveled at him.
The rule was that it was one denarius for each man, and the Jew's hated this, because each coin was basically proclaiming Caesar as lord. The goal here, then, was a trap for Jesus: an either-or scenario of popularity (favoring the tax) versus potential death (opposing the tax as a revolutionary.) Jesus here gives a brilliant answer in which He gives a directive to both be obedient to government, as well as giving to God what is Gods. At the same time, Christ is also unhitching his followers from any particular nation. There is no "Christian country", for Christ's kingdom is not of this earth. By unhitching His followers from any nation, the ethnic phase is now done, meaning that there is no more need for circumcision (and likewise infant baptism, since there is no longer a national connection.) Being a follower of Christ is part of a national identity that we share with brothers and sisters globally.