But they took him

"But the vine growers took the servant"

with nothing

This means that they did not give him any of the fruit. AT: "without any grapes" (See: rc://en/ta/man/translate/figs-explicit)