"But you must not let anyone call you"
All occurrences of "you" are plural and refer to all of Jesus's followers.
Here "brothers" means "fellow believers."