I totally agree, pick your speakers first. Listen to everything you can and find speakers you love. Then choose an amp that will work well with them. An amp that does not have enough power will sound somewhat thin and closed in. I would not go for the bare minimum of power.
Take your time. Make a long term plan as to what components you want and work toward that goal. Synergy is the most important thing. It does not matter how good or expensive the different pieces are if they don't work well together. Browse the forums and see what people are using together. And listen to everything you possibly can.