You may not pass ... to attack you

"Your people may not pass ... to attack them". Here "you" is singular and refers to Moses, who represents the people of Israel

I will come with the sword

"I will send my army"