but nobody talks about phase shifts.
I do and all the time. The ear can't hear phase shift in single notes, but over a spectrum it can hear phase shift in a variety of ways- as a tonality or as a change in the soundstage. The engineering rule of thumb to prevent phase shift is to have bandwidth exceeding the highest frequency to be amplified by about 10 times.
Now that is generally interpreted to be a bandwidth of 200KHz (but might be higher- for example nearly all LPs have bandwidth well past 30KHz in record and in playback; usually this is limited by the source tape or file or the microphones themselves...). However you need the bandwidth in the playback electronics even if the source lacks the bandwidth if you expect to reproduce all the phases correctly! We get that out of our amplifiers but to do that with class D is still a bit of a trick. We are barely getting switching speeds much past 250KHz which means true 200KHz bandwidth isn't there yet.
Now the AES says that if you can get about 2 octaves above the limit of human hearing (80KHz IOW) that you are doing OK for a monitor amplifier. Audiophiles routinely operate gear that has performance in excess of that. And one of the things I don't like about many of the class D amps I've heard has been the lack of speed and spaciousness on the top end. Like mapman, there's no way my hearing is as good now as when I was in my 20s but I still notice this stuff nevertheless.
So if you're going to get that 200KHz response you need a switching speed of around 1 MHz and even then its going to be a bit spotty (2Mhz would be much better). There's no problem getting the converters to run that high. A prototype we're building here in the shop can do conversion at 3MHz easily no worries (the chips we're using have a bandwidth product of about 50MHz). The real problem is the switching speed of the output devices (and whatever junk occurs between the output of the converter and the inputs of the output devices...). If one were to spend the extra cash to get the really high speed stuff then one of the primary advantages of class D goes away real fast: low cost/high profit margin. Really fast, powerful switching transistors aren't cheap and you don't see them in 99% of all class D amps! So to compete against traditional solid state and tube amplifiers in a nutshell Class D has to improve bandwidth, and you don't have to be a bat to appreciate the difference.
This is one reason why class D is still a rising star. Its also why some class D amps are quite diss-worthy and others are not too bad.
