Main L R speaker connection from seperate amps

A short background, I'm running 2 seperate systems, one analog stereo only and one HT setup using seperate L and R front speakers. I'd like to remove one set of speakers and use one set of speakers for both the pure stereo and HT driven by 2 seperate amps. 
Is this possible? I was thinking of doubling up the L and R speaker connection from the stereo and and the HT amp at the speaker terminals. What would be expected or happen if I did this. What kind of harm could it do to speakers or the amps? My thinking is that as long as one amp is turned off there should be no cause for concern but I'm not really sure about that and am not ready to blow either system. 
I'd appreciate any input one might have. Has anyone tried something like that?
Current configurations are, Stereo: Rogue Pharaoh driving B&W 803d and Lexicon MC12 Emotiva 5 ch amp driving Polk TL3.1. I was thinking of removing the Polks to make my listening space more pleasing visually.
I'm happy as is with both systems and they both do their functions superbly. It's just very cluttered up front.
Yes. Use a speaker switch in reverse. :)

They're cheap, if you don't like the effect, you can always take them out.
Appreciate the suggestion erik, not so sure about the reverse capability though. I do have an old Adcom 4 speaker selection box I used for auditioning. It’s an idea for sure :) Never thought of that.
After much research and education I resolved my issue for the enormous cost of $7.69 and some old fashioned ingenuity. Thanks to some enlightening information on the web and here I combined this to make my own. Since I already had an old Adcom speaker selector box I used it and modified it for my purpose by disconnecting the existing wiring from the binding posts, removing the common ground and adding a 4ptd switch and wiring it up to 3 of the existing binding  post sets. Naturally the original function of speaker selection is non existent but easily restored by reconnecting the left in place (stored) original wiring and common ground and disconnect of the modification. Best of both world with minimal effort. Works like a charm. Looks good in my rack too. :)
Follow up, although it works as expected it also introduced an unacceptable amount of distortion. Especially at the high frequency ranges. I surmise that this may be due to the added connections between amp and speaker in addition to mismatch of materials within the switch itself or added resistance. In any case, even though it worked as intended it had to go.