Gs: The frequency that the power supply switches at is typically well above the audible range in an audio component. Most will be at 80 KHz or above from what i've seen. Some are quite a bit higher than that and / or are quite "dirty", producing strong harmonic output. This can result in not only a non-linear load being placed on the power line, but RFI being generated and fed back into the AC system.
Bigtee: What you described i.e. switching between two rails of different voltage potential was designed by Hitachi in the late 1970's and is called "Class H". Soundcraftsman used something similar and called it Class G. Don't ask me about the differences as i don't know what they are.
Kkm: Think of a "sine wave" aka "the hill and valley" waveform. Rather than having one continual line that swept up and then down and then back up again, draw the same thing using dashed lines rather than one steady flowing line. You now see the difference between a "Class A" sine wave and a switched sine wave. The switched supply actually pulses off and on at a pre-determined rate and adjusts the amount of signal / power as it needs to. As such, it is more efficient since it only supplies as much power as it needs to during the dashed lines and is not turned on during the gaps between those dashed lines. As such, efficiency is increased since less power is used and less power is wasted. Since less power is wasted, thermal losses are reduced and heatsinking can be minimized. Production costs are reduced, so the manufacturer thinks "this is good", etc...
The drawback to this approach is that it can't respond instantaneously to a signal change as it may be turned off ( the gaps between the dashed lines ) at that moment that the signal chance occurs. As such, the device would have to respond on the next cycle that it was pulsed on. In effect, this is kind of like trying to "counter steer" after you've already gone into a skid rather than staying in control of the situation to begin with by monitoring the situation 100% of the time. The end result is a lack of detail ( you've lost all the information that was included where the "gaps" in the dashed sine wave are ), lack of control ( the amp isn't "locked" onto the driver since it is being pulsed off and on all the time ), dynamics suffer ( can't respond in-between pulses, so there is a slight lag ), blackness of background suffers ( there is less of a difference between the amp being off or on ), etc...
On top of that, pulsed or switched supplies tend to produce their own types of distortion due to all of the switching off and on. This can result in increased RFI and other types of distortion. The RFI or pulsing can seep into other components tied into the same supply line, resulting in further degradation of their performance also.
Even with all of the technical disadvantages that they have compared to the "accuracy" and "control" that a Class A or a "richly biased" Class AB amp has, these devices are getting better and better and can sound amazingly good right now. As Gs mentioned, what you hear is more about the way that designs are implimented rather than the technology being used. Sean