Is it the table, arm, cartridge or phono section that is most noticeable to you?
The answer to your question is yes. They all make a difference. Sorry, no easy solution so you need to spread your budget and chose each component wisely. I think this holds true no matter if your budget is $2K or $20K.
Thinking of the cartridge/phonostage match first is a good idea because that synergy will probably have the most impact on the sound potential. Your choice of a cartridge will help narrow down what tonearms would work best. Then you'll have a good idea of what tables you can use. This is just a general suggestion of a possible selection process, but I think it will give you an example of one way to proceed. Still, it can be an iterative process.
IMO, if you want to budget a little less in any one area to start it should be the cartridge.