Not convinced that low distortion equates to a very low black noise floor or higher fidelity.
You’re certainly correct in that it doesn’t necessarily equate to a lower noise floor. Only low noise gear and quiet rooms can do that. But low distortion is in essence the definition of high fidelity. The latter should be immediately apparent when one finally experiences high-end, ultra-low-distortion speakers at spirited levels in a well treated room. I’m talking speakers with SOTA drivers, not just expensive models ascribed some BS tech claims (brands I’ll refrain from naming).
Loudness really becomes a factor when you want to hear a kick drum that sounds anything reminiscent of an actual kick drum, the same for low octave piano notes. The majority of speakers, even some rather expensive ones, fall apart at those levels, or often sacrifice the two lowest octaves of bass in order to achieve decent headroom. Sometimes it comes down to a lack of amplifier power/headroom.