Jeremiah 29:23

Because they have committed villany in Israel, and have committed adultery with their neighbours' wives, .....

Again, the definition of adultery is to have relations with another man's (your neighbor's in this case) wife. Both the man having that forbidden relation and the other man's wife with whom he is having the forbidden relationship are guilty of adultery. See also: Leviticus 20:10, Proverbs 6:32-34, Ezekiel 16:32