Simultaneous XLR & RCA outputs = bad?

OK, from trial and error, I know the answer to this one (at least on my gear). Answer's yes. Looking for some help in understanding why.

Short version, have an Ayre QB-9, and the most convenient place to stick a headphone amp into the system is by running XLR from the DAC into my preamp, and then RCA from the same DAC into the headphone amp. Effectively, it's feeding two preamps from the same source, one from the balanced, and one from the single ended output.

Trouble is, with both outputs connected, the XLR version (feeding the speaker end of the equation) developes a nasty distortion on the (for lack of appropriate terminology, what I'll call) the leading edge transients of mid-bass. Call it hash, distortion, fuzz, but clearly, no-BS significant audible artifacts screwing up the sound. Replicable, and kindly disappear once the simultaneous RCA output is disconnected.

So, lesson one: stop. Loud and clear (and not just figuratively). But, don't get me wrong, I'd love to be able to do this. Anyone got a read on why it's apparently not such a great plan? Many thanks.