There is some logic to first starting with finding the speaker that you really like, and then finding an amp that works well with that speaker. The demands of certain types of speakers might rule out certain amps (e.g., low efficiency speakers ruling out SETs; speakers that dip very low in impedance might rule out high output impedance amps).
However, things are not necessarily that simple. Some of the most obviously different and great sounding gear that I've heard were amps. Some amps are so much better than run of the mill types that it makes it worthwhile finding speakers that work with such amps rather than the other way around. One such amp I heard is a custom-built OTL, that is quite astonishing sounding with many speakers, but, I thought not quite so great sounding with a very low efficiency, low-impedance speaker. The other great amps I've heard include a pushpull 252 amp and a pushpull 300b amp; both of which sort of demand high-efficiency speakers.
The amps I own, are not slouches either: an Audio Note Kageki (parallel single-ended) and a Western Electric 133 (pushpull 349). Again, both require high-efficiency speakers. Both amps are of high enough quality to justify finding speakers that will play nice with them.