Besides the drop in gain, the main thing you are losing is the common mode rejection that balanced cables (and connections) provide. In other words, any noise picked up by the cable is usually picked up by both the positive and negative phase conductor in the cable and cancelled out by the differential receiver at the destination end.
Common mode noise in a home environment isn't generally a big issue unless you live next door to a radio broadcasting station or a factory with lots of big motors. And it probably has less impact on the bass than the upper frequencies.
If the cable is constructed such that the conductor that is connected to the negative pin on the XLR connector is connected to the shield (outer part) of the RCA connector at the other end, then you are going to get at least some of the common mode rejection benefits anyway.
Common mode noise in a home environment isn't generally a big issue unless you live next door to a radio broadcasting station or a factory with lots of big motors. And it probably has less impact on the bass than the upper frequencies.
If the cable is constructed such that the conductor that is connected to the negative pin on the XLR connector is connected to the shield (outer part) of the RCA connector at the other end, then you are going to get at least some of the common mode rejection benefits anyway.