Which Component Has the Greatest Affect on Low-Level Listening

I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
I would look for a good class A amp that will not change over to A/B until 5 watts. Low level class A sounds really good.
I’ve had very good results at low volume, with 97db horns and about 4-5 watts (300B).
mapman Interesting answer about sub, I think trying a REL 9Ti with my stereo gear.  Thank you
Start with a very low noise room.  A rule of thumb is that your ambient noise level should be at least half of your average music level.  It's hard to get below 35dB ambient.
If I were to name a single component, I’d have to agree with yyzsantabarbara that a very well executed line preamp with attention paid to stable linear gain attenuation will have the most dramatic impact of fullness at ALL playback levels.