First, almost all high-powered amplifiers use a lot of negative global feedback to stabilize the circuit, and negative feedback, in most amps, makes them sound unnatural and uninvolving compared to an amp that does not use global feedback. The Ayre monoblocks, which put out about 300 watts per channel, and the darTZeel, which puts out about 150 watts per channel, are both examples of "no negative feedback" amps that sound superb. Rowland's Model 625 amp uses no feedback and even Audio Research now makes a no global feedback amp. With a really good speaker like the Temptations, you can plainly hear the difference.
Second, high-powered amps use a lot of output transistors or output tubes, which also generally makes an amp sound worse. When a lot of output devices are paralleled, the sound becomes unclear. Each output device amplifies only a part of the music and the signal might not arrive at the exact same time from all of the transistors, even if they used matched transistors. Two or three pairs can give good sound, but with more than that, it is quite difficult to do. The best sound is with one pair, and the best example is the darTZeel, which uses only one pair of transistors per channel.
In conclusion, as long as your speakers are efficient, a lower powered amp, especially the elite amps that use no global feedback, will generally sound better unless you like to listen really loud or unless you have a huge listening room that is hard to fill with sound. With a great speaker like the Temptations, quality is better than quantity.