Which Component Has the Greatest Affect on Low-Level Listening


I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
bgm1911
In my experience high sensitivity loudspeakers do a good job on low volume. At least my 98dB Ocellia’s do a very good job at low level listening. 
Start with clean power feeding your Amplifier Perhaps a conditioner regenerator or as stable as you can get it into your system. Then as much power as you can supply with your amplifier.
I work from home and 75 % of my listening is low level
High level electronics will make good speakers sound great
Speakers with high sensitivity and paper cones usually sound good at low volume. I own a pair of Tekton Enzo 2.7 which are rated 98 db sensitivity and have 2 8" woofers with paper cones per speaker. Due to an upcoming move, I had to move the speakers into a small room, where I sit about 6 ft away and play at a low level most of the time. They sound good loud, soft or medium. The bass is surprisingly good at very quiet levels even without any eq boost and as long as you’re sitting relatively close. I also used to own Zu speakers which were similar in this regard - again high sens. and paper cones. In my experience, speakers with poly/plastic cones need to be cranked before coming alive. 
mapman

Modern Equipment without 'Lousness' filter:

Perhaps a sub with remote control would be nice, to boost bass to your ears for any volume/listening location/content. Or for full range speakers, remote tone, boost just the bass like you would boost the sub individually.

Remote, from my listening position is what I find so beneficial.

I listen to a lot of Jazz, and maintaining the Bass Player at low volume is what keeps the music involving at those low levels, maintains my awareness, while 
All the audio components through which electricity passes will have an impact on listening at any level.