Simultaneous XLR & RCA outputs = bad?

OK, from trial and error, I know the answer to this one (at least on my gear). Answer's yes. Looking for some help in understanding why.

Short version, have an Ayre QB-9, and the most convenient place to stick a headphone amp into the system is by running XLR from the DAC into my preamp, and then RCA from the same DAC into the headphone amp. Effectively, it's feeding two preamps from the same source, one from the balanced, and one from the single ended output.

Trouble is, with both outputs connected, the XLR version (feeding the speaker end of the equation) developes a nasty distortion on the (for lack of appropriate terminology, what I'll call) the leading edge transients of mid-bass. Call it hash, distortion, fuzz, but clearly, no-BS significant audible artifacts screwing up the sound. Replicable, and kindly disappear once the simultaneous RCA output is disconnected.

So, lesson one: stop. Loud and clear (and not just figuratively). But, don't get me wrong, I'd love to be able to do this. Anyone got a read on why it's apparently not such a great plan? Many thanks.
I had the same problem when I was demoing rca vs xlr from my cd player to my integrated. For me only the xlr sounded horrible when both cables were connected. My guess is that the circuitry from the xlr was delaying the signal (compared to the rca) and somehow getting mixed. That is only a guess and I would have thought the rca should have sounded horrible too (when both were connected).

Good question
Something I read once, don't ask me where, mentioned that screws with the CMMR.
Does the same thing happen regardless of whether the headphone amp is turned on or turned off? It seems conceivable to me that you MIGHT get better results when the headphone amp is on, because its input impedance could conceivably fluctuate unpredictably as a function of signal level when it is off.

In any event, the QB-9's rca outputs are probably driven by the same output circuit that is driving the positive polarity xlr output signal (xlr pin 2). So the two signal polarities that are received by the preamp (xlr pins 2 and 3) would be unbalanced, at least to a slight degree, by the unequal load impedances on the two xlr signal polarities, and also at high frequencies by the unequal total cable capacitances. Ground loop effects might also be involved.

Unequal load impedances on the two xlr signal polarities result, at least to a small degree, in degradation of CMRR, as NGjockey indicated. CMRR = Common Mode Rejection Ratio, which relates to noise rejection (including ultrasonic noise or rfi that might cause unpredictable effects in the preamp at audible frequencies).

What model headphone amp is it, and if you can readily determine it, what is its input impedance? I see that your Rowland Concerto has an input impedance of 40K.

If the headphone amp's input impedance is on the low side, connecting the QB-9's rca outputs through short cables to the inputs of an external buffer stage having very high input impedance, with the buffer's outputs connected to the headphone amp, would probably help.

Or if the headphone amp happens to provide xlr inputs as well as rca's, using a pair of xlr y-adapters to connect the QB-9's xlr outputs to both destinations would probably help as well, as it would keep the two signal polarities in balance.

-- Al
Ooh, that's a whole lot of useful info -- our resident engineer to the rescue, again. Thanks. Haven't experimented with them both on -- habitually turned off the headphone amp when listening to the speakers. Worth a shot.

The headphone amp in question is a Headroom Home Amp (circa 1999). Yea, took it out of the closet after quite a while.... Not sure what it's input impedance is, and some looking on the web provided no additional info. It's RCA only.

Truth be told, really don't resort to the cans all that often (that would be the same ones that have been in the closet for years, so there you go...), so simply leaving the Headroom disconnected unless I'm using it may well be the simplest and easiest fix. But if I were to reach for a higher-impedance buffer stage to put between the DAC and the Headroom, what would such a beast look like? (Forgive my ignorance, just not familiar with such animals). Little disappointed that the beast in question isn't a Canadian mutant mouse repository, but don't suppose that's where I might start looking for one...;). Thanks again.
If I were to reach for a higher-impedance buffer stage to put between the DAC and the Headroom, what would such a beast look like?
The Burson Buffer is one example, and seems to be well regarded. 500K input impedance, 15 ohm output impedance. It costs around $550 though. Undoubtedly there are lower priced devices that can perform the same function, but I have no particular knowledge of them, or their sonic quality.

Another possible approach, for about half that amount, would be to send the xlr outputs of the QB-9 into a pair of xlr y-adapters, and connect the xlr signal pair to the headphone amp via a Jensen Transformer, which would convert the xlr signal pair to rca. If you want to pursue that approach, I'm sure that a phone call to Jensen would be very helpful.

Best regards,
-- Al