It is implied that the members of the court spoke to Pharaoh, and now Pharaoh is replying to Joseph.
"as you swore to him"