2 channel system?

in a 2 channel system (cd player or integrated amps) which component affects the sound the more? im looking to upgrade gradually and im wondering which component will have a more direct impact on the sound? i understand that matching of components is very important and all parts play an important role in the overall sound? but in your opinion which one affects it more? thanks in advance.
You should go with the best source (cd or turntable) your budget can afford. No matter how good is your preamp or integrated, they only will amplify what you feed them. In all of my years in this hobby, over 25, an upgrade in the source always have gave me a noticeable improvement in sound quality. Upgrades in the preamp and the amps will give you improvements, but more subtle ones. Go to www.linn.co.uk there is a good explanation on how even the best preamps and amp can't improve a low quality signal.
Hepl offers a good rule of thumb. However, it depends on what components and speakers we're talking about. If your source is pretty good, but your amp is very weak, you could realize a more noticeable improvement by upgrading the amp. Every system is different.
rule of thumb is you can't hear what doesn't exist... buy the best source you are able to afford so that if the software has the info the hardware can read it and send on down the line.
Linn was very influential in developing the theory of priority that went beyond the foolish 50% on speakers syndrome that used to be so popular. The source is far and away the most important! If you don't get the music off the source, LP or CD, your componants will not add it back later. Buy the best source you can afford and work your way back to the speakers. Don't forget to have fun though!
All very good suggestions. But don't forget about speaker cables and interconnects, either. As someone who first started out with lower end Monster, MIT, and XLO cables and is now using mid-line Transparent and Audioquest cables; the differences can be quite amazing. Your music takes on a new dimension; and details you never heard before are clear and prcise. And as Nrchy mentions, do have some fun and enjoy the great music!
I third or fourth the source recommendation, what's lost(or not read) can not be recovered down stream. Buy the best source you can this way there is no guessing about what the recording is suppose to sound like. For a while(recently) I was running a $9k cd player into a pair of monoblock amps that can be picked up used at agon for under a grand(they are no longer made) and it sounded much better then when I was using my $5k amp with a cheap CD player- and guess what deductive logic will tell us that combining the two(good cd player and good amp) yielded the best results. Avideo brings up a good point about not forgetting cables, once you've tried good cables you simply can not go back, it is often what makes or breaks a system. One piece at a time and it'll be done in no time :)
I'd suggest going with amp/preamp first. I only say this because it is good to see what the source you currently have can sound like. It's hard to judge sources with bad preamplification/amplification. Get amplification/preamplification that matches your speakers.

You said that you wanted to upgrade the integrated amplifier and did not wish to go with preamp/amp.I would suggest you get one of the following: Rowland Concentra I or II,Gryphon Tabu Century,Gryphon 2100 or 2200,YBA Passion Integre.Do not stay in the middle,no source will help you achieve a good sound if you do.Then try different players and judge them accordingly.In the beginning this may sound like a waist of money but in the long term,I think,you will be pleased.CD-players come and go,good amplification stays.That's what I would do.
I agree with amplification first, and if separates, the preamp before the power amp. What good is a great source sent through inferior amplification.

I also agree with the cable comments. Most people put a cheap cable on a lower cost source. You'd be surprised how good a lower cost source will sound connected with better cables.
I've no quarrel with starting with the source. I would caution against starting with amplifiers, at least until you have selected a style of speaker. Will you choose cones, planars, electrostatics ... monitors, full-range .... a subwoofer ...? These choices will have a huge impact on what amplifiers will be necessary and suitable, let alone preferences among them. At least the source components are pretty much free of this critical interdependency.