Gain and Loose b/t SET amp and Maggies route?

Generally speaking, without referring to any particular speaker: the higher the sensitivity the bigger the speaker, the lower the overall accuracy of the frequency response and the lower accuracy of the low frequency extension.

To add to Twl's comments:
Most solid state amplifiers sound better driving lower efficiency speakers since those amplifiers are more linear at higher power, while their distortions (solid state 'nasties') are most easily noticed while working at under 1 watt (or at clipping, many watts later). Basically most solid state designs have no listening advantages at 1-2 watts output, besides possibly more quiet operation.
With the Maggies you'll get impressive sound with a good high power solid state amp, and the ability to listen to symphony orchestras a high levels. But count on at least 250 watts of really good amplification. That can get expensive. For lower listening levels, the Pass Labs amplifiers are good because they have fewer gain stages and actually use a single ended design. Many say the result is tube like sound that is a good compromise between tubes and solid state. Again, expensive.
Some of the best sound I've ever had has been in a small room driving a pair of Magnepan MMGs with a 65 watt Counterpoint (hybrid) design which was modified by the designer. The sound both had body and transparency, and sounded fine at high levels - a great combination!
With speakers at over 95 or so dB sensitivity, the vast majority of the time you'll be listening to the amplifier's first couple of watts. A single ended tube design will do less to the signal, making it go through fewer parts and actually will be more linear than a transistor design under those conditions. On a good recording where all the phase information isn't already destroyed by the recording process and your CD player, you will get more magic and more body to the sound (try analog for this, it's addicting, I notice you do have a turntable) - more realism. And then when the transformer saturates under high power demand, you'll get a kind of compression or lessening of dynamics.
For the 5 watt SET amplifiers, you probably need over 97 dB sensitivity to cover most listening situations, and a higher impedance speakers with few dips in the impedance curve. The Coincident line comes to mind. The Galante line is also very promising, I have heard both sound very good at shows.
--Tom

Well, even though there seems to be some kind of prevailing consensus around here lately that tube amps are too distorted to listen to, I will forge ahead with some more "folly".
Generally, a shorter signal path with less gain stages and circuitry will pass more signal integrity than longer signal paths and more gain stages. In SET amps, especially the low power ones, the signal path is very short and there are very few gain stages.
A high efficiency speaker is needed with low power amps, whether they are SET or not.
A low efficiency speaker is an option with higher power amps, because the high power amps have enough power to drive them to the desired listening levels.
If the high power amp loses any information or signal integrity during its many gain stages, that part of the musical signal will not be reproduced by any speaker, efficient or not.
Ideally, some lower efficiency speaker designs could be better at some aspects of sound reproduction than high efficiency ones. However, as previously mentioned, if part of the signal is lost in the amp, then the ability of the speaker to produce it is lost also.
So which way do you go?
Perhaps there is a high power amp that passes the signal better than one with a much shorter signal path and fewer gain stages. If you can find one, you may use whatever speaker you wish. If you can't, and you default to the lower power amp with very short path, then you need to get a speaker system that will operate well with it, and properly reproduce the good signal that the amp passes through to it.
So, there are trade-offs. If you like a high power setup, you may have to settle for slightly less signal integrity, in favor of higher volume levels, or greater choices in speaker selection. If you like the low power setup, you may pass a better signal, but the speakers you like may not work with that amp.
IMO, if you can find a very good low power amp, and a very good pair of efficient speakers that will work well with that amp to properly reproduce the music, you CAN get a better result. Not in every case, mind you, but it is possible.
Again, I submit that, with good design, the shorter signal path will provide better/less degraded signal to the next component down the chain, as long as there is no mismatch, as sometimes can happen with passive preamps and the input/output characteristics of the components used.
Naturally, I expect to be soundly excoriated for this outlandish point-of-view.