Two channel home theater speaker integration

I have separate two channel and home theater equipment but want to use the same set of speakers. I originally had two sets of speaker wires (one from each amp) with bananas and just swapped cables when switching from 2 channel or home theater. I found a nice switch box that allowed me to input both sets of speaker cables with output to the one set of speakers. After testing for a while, I decided that I compromised my two channel sound. So I decided to put both sets of speaker wires on the speakers. I figured I was only using one amp at a time so what's the big deal. However I have been noticing that sometimes when I switch from two channel back to home theater, the home theater amp goes into a protection mode and shuts down. I have to turn in it off, unplug and replug, and then it works fine. Am I messing up the amps up here with this set up?