Considering switching to XLR cables,

panzrwagn

In every imaginable parameter the XLR is superior to the RCA connector.

It's all about the implementation. I've seen some really horrible XLR connectors that I wouldn't want anywhere near my system. On the other hand, I've seen some really exquisite RCA connectors - some of which even lock - from manufacturers such as WBT. So there is good and bad quality of each.

But that's not the question. Is it audibly superior at lengths less than 5 meters? The answer is no.

I think it unlikely you'd be able to reliably hear differences between XLR and RCA terminations provided each are well built. If the XLRs terminate into balanced connections, you may very well be able to hear and measure differences. For example, many balanced components (such as Audio Research) will have 6 dB greater gain on the balanced connections. If you don't need that gain, you might want to consider that as 6 dB better S/N. Either way, that's a substantial difference and cable length has nothing to do with it.

I highly recommend looking into Cerious Technologies cables. I’ve used very expensive cables until I found Cerious. Bob’s XLRs are more resolving than other cables three to four times the price. Just take a look at their website. 
 

https://cerioustech.com
 

There is really very little difference between XLR and RCA. Most of us aren't going to need a cable longer than 2m.

I've had some great RCA cables and poor sounding XLR cables and vice versa.

Many high end valve amp manufacturers only provide RCA inputs.

I would say the OP's dilema is in finding synchronicity in his system - we've all been there!

Over the years I've used Cardas as my reference, they get along with most electronics.

I am sure it's gear dependent but in my experience, with two highly regarded sources, a Modwright Oppo and a Lumin A1 music server, the XLR outputs were far superior to the RCA's. 

In a little second system I ran a variety of amps and speakers with a Lumin D2 as source....because the pre amp and most amps only accepted RCA I was forced to go all single ended.  Amps were SS and tubes, speakers were Omega (2 different models) Sonus Faber, Harbeth, Sterling Broadcast and Klipsch.  I always had the same issue....the midrange was awful no matter what combo I tried.....kind of hollowed out and tinny.  Finally realized the one common denominator was the Lumin, thought about selling it then realized in all my Lumin experience I had never run single ended output cables in my primary system, always XLR, so I purchased Cardas XLR to RCA adapters on the preamp side with XLR cables....mid range magic returned.  Pisses me off I didn't realize it earlier as I fired some really nice speaker amp combos.

Unless the equipment is actually true balanced differential there is no reason to use XLR’s . As an example I have a Mark Levinson no.585.5 amp that is balanced. But my DAC is a Chord Dave. Even though my Dave has XLR outputs it’s not balanced (per Rob Watts) so using XLR’s is pointless. BTW I use Audience FrontRow Reserve interconnects