It was common to use the phrase "go down" when speaking of traveling from Canaan to Egypt. Alternate translation: "My son, Benjamin, will not go with you to Egypt"
The full meaning can be made explicit. Alternate translation: "For my wife, Rachel, only had two children. Joseph is dead and Benjamin is the only one left" (See: [[rc://en/ta/man/translate/figs-explicit]])
To "bring down ... to Sheol" is way of saying they will cause him to die and go to Sheol. He uses the word "down" because it was commonly believed sheol is somewhere underground. Alternate translation: "then you will cause me, an old man, to die of sorrow" (See: [[rc://en/ta/man/translate/figs-idiom]])