Its not even just damage to worry about, the sound will be backing up thru a passive circuit in the other amp causing major impeadance changes, this is thru the Dead amp sitting on the end of the line that is, cause your signal will still pass thru the circuit from the binding posts backwards thru the other amp that is not being used..
The damage can occur especially with tubes if you forget to turn one amp off before turning the other on.
Myself and a friend have experimented with this method, totally screws up the sound and adds un-needed resistance, especially when running the Tube amp, it sounded like one channel was scratchy, and the other channel had no bass, once we disconnected the HT receiver "on the fly" even though the HT was off at the power supplys, the Running tube amps completley straightened out and were perfectly balanced again immediatley..
So yes no matter what it will screw with the Tube circuit it seems when having the Dead circuit at the other end of a solid state amp in our experience connected to another set of cables sharing the binding posts on your speakers.. And even if in your system its not audible for some reason that you can tell, I would bet it would not sound as good or be as safe as it should be...
Only thing is to run a switch box of some type to disconnect that amp physically or pull the wires off the amp not in use each time.. Real pain in the A$$.