go up to my father

It was common to use the phrase "go up" when speaking about going from Egypt to Canaan. AT: "go back to my father"

Come down to me

It was common to use the phrase "come down" when speaking about going from Canaan to Egypt. AT: "Come here to me"