Which Component Has the Greatest Affect on Low-Level Listening


I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
bgm1911
Your amp should have a "stiff" power supply. A good power supply retains it's character at low-levels...
A good power conditioner and power cords.  Drops the noise floor.  Lower noise floor increases signal to noise most dramatically at low listening levels.

Look into the Core Power stuff
My Benchmark HPA4 preamp had the biggest positive on low level listening. It has a volume control with 256 settings and it is quieter than any other source component you will use with it. Once of the reasons I got it was for late night low volume listening. It has exceeded expectations on all fronts.
Speakers. You probably need a powered sub or two practically with most setups to sound best at low volume because our ear sensitivity drops off and we do not hear nearly as much as we do at louder volumes.

Equal-loudness contour - Wikipedia