How Do You Set Up a Headphone System?

I’ve finally got a headphone setup that I really enjoy listening to. What’s curious is that it’s about as simple as it can get: a portable CD player and headphones with cable and battery upgrades. I’ve tried using many of the same components from my 2-channel rig but I’ve always gotten mixed results. Why is that? Why do the more expensive components not help in this situation? I must be screwing up the implementation, right? So how do you setup a great headphone system for amazing sound?
You need a good source, a good set of headphones, and a good headphone amp. I'm assuming you're talking about full size headphones, not portable or in ear monitor?

Innerfidelity might be a good place to start. 

It would be helpful if you elaborate on what you've tried in your main system. 
I’d opine it’s more what you do with what you have than it is coveting the best headphones, best amp, etc. Tweaks are all the rage for battery headphone set-ups since you don’t have to worry about room anomalies, power cords, speaker cables, interconnects, fuses, in other words - all the things we’ve come to know produce noise and distortion.
The beauty of a good headphone rig is that it takes the room out of the equation.  I have three headphones I'm currenty using, a portable battery powered amp, a Rupert Neve amp and the Quicksilver Headphone Amp.  

Depending on the music and my mood,  I'll use different amp / phone combos.... you could never do that easily with amps and speakers.   It's like having a room full of speakers at your disposal.  But more practical