Which Component Has the Greatest Affect on Low-Level Listening


I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
bgm1911
mapman linked

https://en.wikipedia.org/wiki/Equal-loudness_contour

You want a system with automatic ’loudness’ circuit. It is completely misnamed, it is a ’low level boost’ to boost bass progressively as volume lowers, (boosts highs also, less importantly). Maintaining bass at low volume will maintain involvement with the music, otherwise it becomes background noise.

To compensate ONLY when listening at low volume. Raise the volume, eliminate the boost, otherwise the bass is too prominent as well as the highs will get screechy.

As you raise the volume, you want the circuit to automatically/progressively reduce/remove the ’loudness’ boost, back to normal for normal or louder.

This unit has a built in automatic fletcher munson circuit, and remote volume, remote balance, I wouldn't live without it.

https://www.ebay.com/itm/Chase-Remote-Line-Controller-RLC-1/254708363331?hash=item3b4dcd3043:g:huIAA...

Vintage receivers have ’loudness’ filters.


Loudspeakers. Good lightweight cone loudspeakers that are designed to sound good at low levels. Triangles come to mind. 
I lean towards speakers also, especially high efficiency with 10" or better woofers. My Cornwall 3's sound great at the low listening levels where they spend most of their time playing. 
Speakers make the biggest difference, followed by amplifiers.  Horn systems are particularly dynamic and engaging at low volume.  If you look at Japanese audio magazines, it is common to find serious audiophiles putting gigantic horn systems in their small homes and apartments.  Given the close living conditions and cultural emphasis on being polite and considerate, systems cannot be played very loudly, hence the desirability of horn systems.

I find amplifiers also play a big role.  Tube amplifiers, particularly low-powered tube amps, deliver lively and engaging sound at lower volume levels.  I find output transformerless tube amps to be particularly dynamic sounding.
Bass boost or loudness control is the right idea but powered sub is better. It provides extra muscle to provide the power hungry bass without taking it away from the rest. An amp driven less hard typically sounds better with lower distortion. Power demand increases exponentially as frequency lowers. It takes a lot of power for flat response to 20hz even at lower volume.  Especially in a larger room.