# Then they said

This refers to the Jewish leaders.

# We will return what we took from them

The Jewish leaders are saying they will return the money which the poorer Jews paid in interest charges.

# made them swear

The word "them" refers to the Jewish leaders.

# Then I called

"I" refers to Nehemiah.