Inexpensive headphone amp paired with an audiophile integrated amp

When one's integrated amp doesn't have a headphone jack and one routes out to a headphone amp, is the quality of the signal compromised by an inexpensive headphone amp?
Thanks. I actually have a Schiit Magna sending to Shure 1840 headphones from a nice musical fidelity integrated—as of yesterday. I just have no frame of reference to know if this makes good sense.
I have a Magni. The Vali with the right tube sounds much better.  
My xDuoo with two right tubes is even better.
Balanced in and out.