Which is the most important part of a stereo system?


My system consists of a pair of B&W 630's, an old Denon 50 watt reciever (DRA-550) from the mid 80's, a Marantz CD5004 cd player, and now a Pro-ject Debut Carbon DC Turntable.  I'm pleased with the speakers and the cd player and while the Denon sounds good it has some issues and I want to upgrade.

I'm planning on returning the Pro-ject TT and getting a higher end TT.  I'm also looking into getting a new amp pre amp with a internal DAC.

Is the source the most important? The speakers? 

Please Help!
klimt
As a manufacturer of audio components, I was a firm believer of the source because I felt that all tone and sound stage came from there first.  You know garbage in and garbage out.  My partner and I recently built a hybrid 6SN7 power amp - think Counterpoint but way more advanced and two our items to a buyer.  He loved the DAC the preamp and power amp.  He had them there for a few weeks and he concluded that the amp was where he would start first.  So away we go!

Happy Listening.
@klimt for integrated amps with dacs

Modwright KWI 200 hybrid with dac is $6200 new, but used is under $5000

Hegel h390 class a/b with dac and streamer functionality is $6000 new 

NAD m33 class d with dac and streaming and bluetooth functionality is $5500 new

All admittedly over your budget but only marginally and imho worth a stretch 
I'd go speakers first followed closely by the room itself.  Find your taste in the sound of the speakers you like--everyone is not the same where that is concerned.  All the other stuff is important, just not as important as those 2 things.  Know what particular cables do especially well helps to get it right near the end of assembling a system.  

Bob
@Terry9
 I definitely built a few sets of speakers, back in the day. Going to try single ended boutique amp kit next maybe with or from a Nelson pass design. 
I find that everything matters.

For me, the most important thing is the amp/speaker interface. Get it right and you are on your way to a system that can sing. Get this wrong and well, you may never get it right.