A covenant is a binding agreement. In this case, it refers to divorcing pagan wives whom the people of Israel had married inappropriately. By doing this, they would renew their commitment to the Sinai covenant.
Women usually received custody of their children when a marriage ended (compare Hagar and Ishmael, Genesis 21:14).