You need switcher box.

Here is an example of remote control switch box.

https://www.youtube.com/watch?v=_yGEmO7GfBE

https://www.parts-express.com/Fosi-Audio-LC30-Y-Speaker-Selector-with-Orange-VU-Meters-235-288?quantity=1

Following @jea48 advice here is a good alternative if you’re system is up to it. 
https://alleliteaudio.com/products/luxman-as-55-passive-speaker-level-switch?variant=44189892313141&country=US&currency=USD&utm_medium=product_sync&utm_source=google&utm_content=sag_organic&utm_campaign=sag_organic&utm_source=google&utm_medium=cpc&utm_campaign=23352083182&utm_content=198399029708&utm_term=&gad_source=1&gad_campaignid=23352083182&gclid=Cj0KCQjwj47OBhCmARIsAF5wUEEJJmzDTrXFbYNgl85tgMcujPGjgrWINSF75yDYUgXXtvlzki84Q-EaAjPrEALw_wcB

Used option. 
https://www.ebay.com/itm/358269581669?chn=ps&_trkparms=ispr%3D1&amdata=enc%3A1dSwuc_Q6RoO-f9vMZQe_lQ23&norover=1&mkevt=1&mkrid=711-173151-913341-5&mkcid=2&mkscid=101&itemid=358269581669&targetid=&device=m&mktype=pla&googleloc=9033780&poi=&campaignid=23517807481&mkgroupid=&rlsatarget=&abcId=10414390&merchantid=5560932007&gad_source=1&gad_campaignid=23508089004&gbraid=0AAAAAD_QDh9WSDsGuV9wo-Tf3TEzwyqaO&gclid=Cj0KCQjwj47OBhCmARIsAF5wUEHS-Z1IQ3w_5tQywpQcG689C22xUpK2NRqlOLa1VC_3a_b3K1VdQYoaAiI6EALw_wcB

here is a thread for a new guys product. 
https://forum.audiogon.com/discussions/nereneas-arca-new-audiophile-rca-switcher

+1 for the Luxman speaker selector switch. I got a used one from Japan a couple of weeks ago. You can connect 3 amplifiers to 1 set of speakers, OR, 1 amplifier to 3 sets of speakers. Connectors are (3-4 way) speaker binding posts (incl. banana). You can select 1 connection path (A, or B, or C) at a time. Simple, but works well. Heavy construction, good quality. Obviously there are many other and much cheaper speaker selector switches on the market but none as high quality as the Luxman.

Luxman AS-55 Passive Speaker Level Switch

What actually goes wrong, if anything?  Speakers generate BackEMF which amplifiers withstand.  Why should signal affect another connected (but unpowered) amplifier?

And what would happen if both amplifiers were playing different music?

Please only answer if you actually know