How did the brothers respond when they were brought to Joseph's house, and why?

The brothers were afraid they would be arrested and taken as slaves because of the money left in their bags on the first journey.