Most Important?

you are right on track building off your speakers as the foundation...amp matching is next, then front end and last is wire....but before amp I'd get the electricity nice and clean, room acoustic anomalies sorted out then off you go.
Larry

Source. Not players or other electronics - source material. I'd already gotten to the point of getting almost all gear used (or doing my own for room stuff), so the only significant expenditures for really new stuff is source.

I also measure this more in terms of overall investment (money + time + whatever) than just funds. Yes, I freely admit to having put most money into speakers (the only things that actually make sound...) and there is just a little money in amps, etc. If there is any truth to "you get what you pay for" then if you want good sound you have to have something that makes good sound - ergo I am in the "speakers first" camp (and if you're into analog sources then there is also some consideration there, too). You can find good gear and good deals if you spend some darned time looking, so the actual dollar outlay may not be nearly as big as some others have spent. Other than that the investment in finding decent electronics and getting the room(s) set up is kinda ongoing and is now measured more by time than $$$. But the investment in source material is never-ending...

My opinion is that if you skrimp on the preamp your system will suffer. I would rather have a system with 500 dollar speakers and a 5000 dollar preamp than the opposite.

Wow - what an insane thread. Some of the polarizeed views suggest that Bose and Noel Lee has been extremely succesful at marketing their cheap speakers and expensive cables.

Anyone who thinks that any old speaker will do clearly does not understand the relative difference in distortion levels between electronics and speakers. I know the math can be daunting for some but fwiw we are talking factors of over a thousand times worse for good speakers and hundreds of thousands of times worse for an el cheapo Bose or its equivalent speaker.

Headphones anyone? (for $500 you can get a decent pair)

I have 4 systems, and in all of my mixing and matching etc, I have found that source and electronics are more important.

One of my 1st heavily front ended systems was a Linn Lp12/Ittok/black koetsu/Mitch Cotter step up into a NAD 3020 into a pair of ADS 320 bookshelf speakers.
Total cost of that system was around 3700, with around 3k of it in the front end.
The sound was amazingly good and fun to listen to.