2 amps 1 pair o' speakers.

My friend has "made" me spend lots of money and now I own a Cary SLI-80 integrated amp. I have a Sony DA5-ES for HT and the Cary for audio. The source is the Sony NS999ES with 2 outputs, a separate pair of ICs into each the Cary and the DA5-ES I'm wondering if I can run speaker cables from both amps into the main speakers allowing 2-channel with the Cary and the same speakers for mains with the Sony for HT. What will happen with one amp off but hooked up to the speakers while the other amp is on? Will there be any problems with both amps on? Are there any other solutions you can suggest?
If you don't mind using the Cary for music and HT you could pass your front L+R outputs through it for HT. You will need to use your pre-outs from the Sony to the Cary. If you do this, you may want to use the Cary at 80 watts for HT. It's easy to do but will require tubes to be on for both.

The only other thing I can think of is for you to run two sets of speaker cables and swap them at the speaker for each use.