Why did Haman feel that being invited to the banquets was worth nothing to him?

Being invited to the banquets was worth nothing to him as long as he saw Mordecai the Jew sitting at the king's gate.