Main L R speaker connection from seperate amps


A short background, I'm running 2 seperate systems, one analog stereo only and one HT setup using seperate L and R front speakers. I'd like to remove one set of speakers and use one set of speakers for both the pure stereo and HT driven by 2 seperate amps. 
Is this possible? I was thinking of doubling up the L and R speaker connection from the stereo and and the HT amp at the speaker terminals. What would be expected or happen if I did this. What kind of harm could it do to speakers or the amps? My thinking is that as long as one amp is turned off there should be no cause for concern but I'm not really sure about that and am not ready to blow either system. 
I'd appreciate any input one might have. Has anyone tried something like that?
Current configurations are, Stereo: Rogue Pharaoh driving B&W 803d and Lexicon MC12 Emotiva 5 ch amp driving Polk TL3.1. I was thinking of removing the Polks to make my listening space more pleasing visually.
I'm happy as is with both systems and they both do their functions superbly. It's just very cluttered up front.
gillatgh
Yes. Use a speaker switch in reverse. :)

They're cheap, if you don't like the effect, you can always take them out.
Appreciate the suggestion erik, not so sure about the reverse capability though. I do have an old Adcom 4 speaker selection box I used for auditioning. It’s an idea for sure :) Never thought of that.