Does the same thing happen regardless of whether the headphone amp is turned on or turned off? It seems conceivable to me that you MIGHT get better results when the headphone amp is on, because its input impedance could conceivably fluctuate unpredictably as a function of signal level when it is off.
In any event, the QB-9's rca outputs are probably driven by the same output circuit that is driving the positive polarity xlr output signal (xlr pin 2). So the two signal polarities that are received by the preamp (xlr pins 2 and 3) would be unbalanced, at least to a slight degree, by the unequal load impedances on the two xlr signal polarities, and also at high frequencies by the unequal total cable capacitances. Ground loop effects might also be involved.
Unequal load impedances on the two xlr signal polarities result, at least to a small degree, in degradation of CMRR, as NGjockey indicated. CMRR = Common Mode Rejection Ratio, which relates to noise rejection (including ultrasonic noise or rfi that might cause unpredictable effects in the preamp at audible frequencies).
What model headphone amp is it, and if you can readily determine it, what is its input impedance? I see that your Rowland Concerto has an input impedance of 40K.
If the headphone amp's input impedance is on the low side, connecting the QB-9's rca outputs through short cables to the inputs of an external buffer stage having very high input impedance, with the buffer's outputs connected to the headphone amp, would probably help.
Or if the headphone amp happens to provide xlr inputs as well as rca's, using a pair of xlr y-adapters to connect the QB-9's xlr outputs to both destinations would probably help as well, as it would keep the two signal polarities in balance.