Both of your sources will likely yield a 6dB higher output level from their XLR outputs, versus RCA. How any why they do this is another topic. The +6dB is more likely to be useful for analog sources (e.g. to make up for a low-output cartridge) than digital (typically a "hot" output compared to traditional analog sources).
Does the "common mode rejection" of balanced signal transfer (including AES standard, which your sources are probably not, blah blah) really matter for our typical applications (a few feet / meters of typically shielded cable in a reasonable environment)? Not really. Does the actual RCA vs. XLR input & output circuitry implementation of your components matter? YES, this can impact the sound. You’re right to let your ears guide you here.
On another practical matter, there are pros and cons to XLR vs. RCA connectors. If you have stiff cables, twisting the XLR pins to line up for connection can be a huge PITA. Some of Audioquest's models will start to separate connector from cable body if you swap cable/components a lot (eventual internal connection fail). RCA plugs can be inserted at any rotation angle, so they're a lot easier to manage. That said, once you line up the XLR pins, you get a nice satisfying lock & click.