They said to him

Matthew does not make clear who answered Jesus. If you need to specify an audience you can translate as "The people said to Jesus."

He will destroy those wretches and make them wretched

"He will bring those bad people to a miserable end" or "He will make those evil people die in misery"