In the days of Jesus tax collectors deserved the scorn of the people because often they abused their position. Here is how William Barclay describes the business of the tax collector in the Roman world:

'The problem of the Roman government was to devise a system whereby the taxes could be collected as efficiently and as cheaply as possible. They had done so by auctioning the right to collect taxes in a certain area. A man bought the right to collect the taxes within a certain district; he was responsible to the Roman government for an agreed sum; anything he could raise over and above that he was allowed to keep as commission. Obviously this system lent itself to grave abuses. People did not really know how much they ought to pay in the days before newspapers and wireless announcements and the widespread diffusion of news; nor had they any right of appeal against the tax-collector. The consequence was that many a tax-collector became a wealthy man through his illegal extortion...

These tax-gatherers were universally hated. They had entered the service of their country's conquerors, and they amassed their fortunes at the expense of their country's misfortunes...They were notoriously dishonest. Not only did they fleece their own countrymen, but they also did their best to swindle the government, and they made a flourishing income by taking bribes from rich people who wished to avoid taxes which they should have paid. Every country hates its tax-gatherers, but the hatred of the Jews for them was doubly violent. The Jews were fanatical nationalists. But what roused the Jews more than anything else was their religious conviction that God alone was king, and that to pay any taxes to any mortal ruler was an infringement of God's rights and an insult to His majesty. By Jewish law a tax-gatherer was debarred from the synagogue; he was included with things and beasts unclean, and Leviticus 20:5 was applied to him; he was forbidden to be a witness in any case; "robbers, murderers and tax-gatherers" were classed together. When Jesus called Matthew, He called a man whom all men hated.'