... the sound is much fuller than when going from RCA to RCA. Could this be because the XLR output has a higher gain than the RCA?
Yes, that is normal. The sound quality has nothing to do with RCA or XLR Design, when you have both it depends on matching components and what technical quality the Designer has implemented.
RCA/XLR adaptors degrade the sound normally because they are internally super lousy wired. Open them and you will find even in very expensive adaptors super thin, ultra cheap wire. When you want a good one, contact SignalCable, they do custom order with Silver wire, not expensive and very, very good. You can also order better connectors (for signal transfer).