It’s generally not a good idea to put two voltage sources connected to one input. They will fight each other. With preamps, usually all this means is severe distortion, with amps it can cause a fire. Either use can cause damage to one or the other preamp, though because of the low voltages and currents available this woiuld be unexpected.
The issue is that the inputs on amps with balanced and unbalanced connectors are not usually isolated. That is, there’s a direct connection between the RCA center pin and one of the XLR pins.
When you have two preamps (i.e. voltage sources) connected, and one is playing music, one says "this should be 2V" and the other says "no, this should be 0V" and they will duke it out via current. The other issue is the possibility even when a preamp is off of the preamp creating a second "input impedance". For instance, your amp may have 47kOhms input impedance, an easy load for a preamp, but the second preamp, when off, may appear to be 200 Ohms. The impedance of the output stage of a preamp when off is not really well defined.
So, generally, the only times it’s safe to do this is if you know for sure that the inputs are isolated. That is, have a very high impedance between them. Mixers are an example of a device which DOES do exactly this.
Why aren’t the inputs isolated? It’s extra cost which 99.9999997% of users will never need. they use either the XLR or the RCA connectors.