SET v. Pushpull

 One thing I would point out is that some of the distortion from push pull comes from the crossover region which can be more objectionable to your ear. SET distortion is mostly even ordered and usually is perceived as pleasant. But you do want to run SET very lightly in which case it’s one of the lowest distorting topologies around.  In my experience, running very efficient easy load speakers, I preferred SET over push poull and 0TL over SET by no small margin. 
 And in my experience SET running a proper load is not warm and fuzzy. It’s quick light and dynamic. I just found O TL to offer an even better picture into the musical event. Hard to describe. 

So much depends on topology!
*Some* pushpull amps have crossover distortion problems but most do not. What push pull amps often do have trouble with is the so-called 'first watt'. Many do have higher distortions in that area, but its usually not crossover distortion, although that is often what gets trotted out as the explanation.
SETs have a distortion character where as the power level is decreased, the distortion linearly decreased to unmeasurable. Some push-pull amps are like that too- ours are, and in general I would suspect any fully differential push-pull amplifier to have a similar quality if designed correctly.
SETs often do not employ loop negative feedback! This accounts for both their high distortion at full power and also their higher output impedance. The tradeoff is that first watt- which tends to have lower distortion on this account. That can be done with push pull amps too, but it is trickier.
Generally speaking, a sensitivity or efficiency of a loudspeaker that is less than 85 db is bordering on criminal- it gets nearly impossible to find an amplifier that simultaneously makes enough power and actually sounds like music in most average rooms. Exceptions might be if the speaker is only heard nearfield (within 3-5 feet) or the room is quite small. You just wind up having to push the amplifier too hard, and in the case of an SET this really won't bring out the strengths of the amplifier!
Someone already mentioned my comments about SET amplifier power above so I won't repeat them. But here's a thing in a nutshell:Tube amplifier power whether SET or not is expensive so its to your amplifier investment dollar's advantage to use a speaker that is more efficient and easy to drive. In particular its to your advantage to avoid four ohm loads as all amplifiers (tube, solid state or class D) make more audible (and unpleasant) distortion into lower impedances! If high quality reproduction is your goal, again your amplifier investment dollar is best served by a higher impedance (8 ohms or more) loudspeaker. If sound pressure is your goal you have a 3db argument for going with 4 ohms, only you have solid state.

Low powered amps need efficient speakers, but beyond that I found that my little 12 watt per side SE tube amp (Dennis Had Firebottle HO) simply sounds astonishingly better than any other amp I've heard, tube or SS. But then I don't get out much...

@atmasphere  +1 If people want to make 85 dB @ 4 Ohm or lower speakers they need to cough up and make active versions or just make them more efficient. People listen to speakers at show rooms with amps that are not practical at home. They get the speaker and then the sticker shock of the amp required hits them. 

In the lower to mid end of consumer/audiophile (a term!) gear speakers should be 92 dB @ 8 Ohm stable or higher. Else they need to come with a warning to help ordinary folks like me. New people like me are totally stunned at shows / show rooms when we see amps that cost twice the amount of speakers or more sometimes. The speakers sound great at the dealer but once they are home it is a different matter.

I understand that we need things at every price point however.