The most important piece in a stereo system???

Fmpnd is right on (in my opinion).
Great speakers with poor electronics sound terrible. I'd add that cheap CD players (<$200) are pretty listenable through all but the highest end spkr/amp setups. Therefore if I were to assemble a system bit by bit over many months (assuming I already have some sort of cheap CD and speakers ala circuit city) my order would be (I would buy in this order to ensure that at all stages I had a system I could bare to listen to):
1. Amp (this will set final budget)
2. Speakers (spend approx same as amp)
3. cables (spend about 1/10 of total system cost)
4. CD/DVD player. (spend approx 1/2 of cost of amp)

Matching is important, and I think the CD source has the least overall impact on the soundless even than the cables (again .. just my experience, no flames please).
Note if you want an analogue front end then I think the turntable/arm/cartridge is equally important to amp and speakers, and should be purchased first. In this respect I think that digital front ends are much more forgiving.

In theory, as Linn states the source is the most important. In reality it is a complicated interaction between the source material, all components and the room. However what I think is a better question is. How can I get the best sound for my budget? I actually assemble my system in my own strange method. I methodically try to, over time get the best of each type of component for my budget. For example When a pair of Aerial 10Ts were available at the right price my a/d/s speakers went, Linn LP-12with Lingo Bye Denon ,when I found a sonic frontiers SFCD-1 at the right price my Tandberg 3015a went. CJ Premier 14 bye bye CL35 MK III, 10B , goodbye Luxman T110 , and so on. The same goes for cables etc. Everything matters , but I think cables can be a bad investment if you go too far in a moderately priced system I stop at Monster 1500i m 1000i and kimber kable. To acheive the above I sell vintage audio as a hobby online to help trade up whenever I can. Lastly being that cost is a key issue for most people (not the reviewers of Stereophile), In certain systems there is a sensible proportion you should spend. Is the $ 20,000 Linn CD-12 great ? Yes but in a $ 25,000.00 system its a bad balance. You would be better off with a Rega or other sub $ 1,000.00 or less player etc. etc. I almost forgot dont forget about using good source material like good clean vinyl, 180 heavy remastered vinyl. MFSL cds etc. Enjoy!

That's a false dichotmomy Fmpnd, because no one is going to pair no-fi speakers with high-end source or no-fi source with high-end speakers. So, construing the question to assume decent hi-fi components all through the system, whether budget or expensive, would you rather listen to a NAD cd player through Eidolons or an Accuphase cd player through PSB mini-speakers?

I've made the weakest link argument before, since speakers are the least accurate component in a hi-fi system, and all speakers are colored in one way or another. But, that said, there are a lot of inexpensive to moderately priced speakers that are quite decent.

The biggest changes in sound for me, from mid-fi to hi-fi came with changes in amplifiers and cd players.

As much as I love my Harbeths, I have to disagree with my friend Charlie and say source, but amp is a close second.

1. own the best preamp that you can reasonably afford
2. buy the speakers that you intend to keep for life & then build the rest of rig around them

1. Speakers. Speakers are the most important component when you are first starting to build your system.

When first setting up your system, you should consider:

a) Budget.

b) Room size. Small rooms, small speakers, vice versa or somewhere in between.

c) Speaker placement. If you cannot place the speakers properly, do not expect to get good sound. Save yourself some time and money and buy a rack system from Circuit City.

d) How loud do you listen you listen to your music.

e) What type of music you listen to.

f) Aesthetics. Some speakers are big and ugly and you would not want them in your home no matter how good they sounded.

The above will dictate what Speakers you should get. You may not get the "right" speakers if your budget is too small. The speakers will then determine what amp you should buy. CD players offer the most bang for your buck so you can save here. Avoid expensive cabling.

Speakers 50%, Amp 25%, CD player 20%, Cabling 5%