The nobles are saying they will return the money which the poorer Jews paid in interest charges.
## We will do as you say ##
"We" refers to the nobles who are speaking to Nehemiah.
## Then I called ##
"I" refers to Nehemiah.
## and made them swear ##
"and made the Jewish nobles take an oath"
## I shook out the fold of my robe ##
Many times in the Old Testament, oaths were physically demonstrated as a witness to what was promised. Nehemiah is demonstrating to the nobles what will happen if they break the promise they had made.