Which Component Has the Greatest Affect on Low-Level Listening

All the audio components through which electricity passes will have an impact on listening at any level.  

As an audio salesman for over 45 years, I will give you my answer...given countless times to clients...a "properly calibrated" loudness contour, and speakers that do not lack for extended bass response.  Start with bass shy speakers and they will shut down all the faster at low levels.  This is why the "high efficiency" response is counterfactual...high efficiency is achieved at the expense of bass extension in any given cabinet size.  True, with bass reflex a speaker designer can engineer a "bump" to mask the lack of extension, and that may make it more palatable at low levels, but then when played a "normal" audiophile levels, it will boom.
The "properly calibrated" loudness is rarely achieved.  If you have separates, and the preamp has a loudness function, then the amp MUST have input level controls to allow the system to be calibrated to the speakers' in-room response.  A "Variable Loudness" control, a staple of Yamaha products, is a fair compromise.  My 1962 McIntosh C-20 preamp has an excellent variable control, but hardly anyone is doing that nowadays.

@andrewkelley

My 5.1 was Marantz preamp and amp with Paradigm speakers all the way around, nothing high end. I want to get out of HT and go 2 ch. I visit a couple local shops and realize my equipment is poor for 2 ch.

Sell all that and upgrade to some quality gear, but it's important that when I just want to play music with the family around, it still sounds reasonable - my Marantz/Paradigm was terrible ITR.

As someone who grew up in the late 60's through the 70's and had always been used to having hi-fi amps and receivers with tone and loudness controls, I find it ridiculous that many hi-end amplification systems have nothing but an on/off and volume control. If you want tone control, you need to purchase yet another piece of equipment to add in the chain. Still others say tone can be achieved through use of certain cables. IMO- which is clearly "old school", having an amp or receiver with tone and loudness controls is a no-brainer. Why wouldn't you want to have more options to tailor the sound to your ears as opposed to fewer, or none at all? Don't want that much bass, mid or treble, cut them back, or run them flat. Many receivers have a switch to bypass the tone controls altogether. Nearly all recordings will differ in some way or another, through microphones, analog vs digital, studios, producers, mixing, mastering, you name it. The tone controls allow the listener to very easily make changes tailored to their own enjoyment.

Higher efficiency speakers are far better at low volume levels than power hungry higher spl speakers. You can overcome some of it but there are physics behind it as well that you can not overcome as well.