I've had the opposite journey. I have built three nice headphone rigs and have about a dozen headphones including some high-end cans like Audeze LCD-4, Senn HD800, and Sony MDR-Z1R.
As my two channel systems improved I find myself so involved with the detail, soundstage, imaging, and dynamics of the system that I don't want to limit myself to what headphones are able to present. And it's nice not to have anything sitting on your head while you're listening.