Quality inversely related to power output?

Not true. As Gregm says - it is simply more difficult and more expensive to build more powerful amplfiers with similar performance to lower powered amps.

If your only measure of quality is distortion then stick to headphones.

As far as I am concerned, dynamics and ultimate SPL output is also quality. If an unamplified piano or drum set can produce north of 110 db SPL on transients then the low distortion qualities of an underpowered system is rather a mute point - as it can never realistically reproduce the range of natural sound of musical instruments in the first place...

All this points to the efficiency of the loudspeaker as being quite important overall

However, this now requires "all other things being equal" in speaker design. It is generally not as easy to find a full range speaker of 100+ db SPL with acoustical performance that is competitive with the many speakers designed between 85 to 91 db sensitivity. (Generally designers equalize drivers downwards to achieve a desirable acoustical performance (crossovers etc.) because the raw driver output is rarely flat and smooth on and off axis acros the frequency range.