"dirt because he had been taken from dirt." This does not refer to the particular place on the land that the God took man from.
This means to what is needed so that plants grow well.