When it comes to things like that, affairs in general can spice up a marriage or ruin them. It is all in the perception of the cause. If you had an affair, because you wanted out of the relationship; you're just being cruel. If it was lust/attraction/neglect, be honest. Nothing is wrong with 'swinging', but it is all about how you go about it.