I agree with the recommendations for an integrated amp.
It's what I have, and I'm very happy with it.
Also cuts back on the power cords and interconnects required, rather than with a pre/power amp.
Most of my system I bought used, which enabled much better equipment with my money.
If you sold your Pass Labs X1 and Bryston 4B, you then could have $10,000-$12,000 possibly.
Buying used you could easily end up with a $25,000-$30,000 system.
I cannot suggest much, as I have no access to most of the US brands available on EBay, here, or US Audio Mart.
I would suggest at some stage you trial a PPT Omega+ Emat on your breaker panel. Sceptics will soon disagree. But you get to trial, and hopefully notice an increase in base (as I did), an increase in depth (as I did), an increase in detail (as I did), and better instrument separation.