Pharaoh answered

It is implied that the members of the court spoke to Pharaoh, and now Pharaoh is replying to Joseph. (See: rc://en/ta/man/jit/figs-explicit)

as he made you swear

"as you swore to him"