Upgrade amp. or speakers first?

I have a Krell KAV 300i, and a pair of B&W N804's, all about 12 years old. Want to upgrade and want to know which should be first to go. The speakers or the amp? I listen to mostly jazz, some small scale classical, and R&B. I am leaning towards the 803D's or 804D's. Will budget up to 8k for amplifier(s). I have listened to some McIntosh, the new ARC integrated and Rotel amps. I recently got an Ayre QB-9 for digital source ( ITunes-Macbook ). Will upgrade, but not all at once. Thanks for any suggestions.
Always pick speakers first than find the right amp to drive them. Speakers are the voice of a system. They pretty much dictate how your system will sound.
Speakers first.
Speakers without a doubt.
When I was in need of a new amp I had a 18 yr old Denon PMA 700. I got the Bryston B100. Great improvement with my 17 yr old Mission/Cyrus 782's. Couple of years later since no replacement parts seemed to be available I decided to shop speakers. That's when I found the Dynaudio line up. I had to have the C1's. As good is Bryston is the B100 wasn't what the C1's needed. So I just bought another new amp. Thought about moving up in the Bryston world but wanted to hear something else for comparison sake. I got the Octave V70SE and all I can say is the Dyn's woke up at all listening levels.

Anyhow that's my story and had I bought the Dyn's first I would have looked for a more ideal amp.
Thanks everyone. I assumed that the weak link in my system was the amp so that's what I thought I might change first. But it seems you folks, and others that I talked to, think the speakers first. Since that is the case, perhaps I'll go for MUCH better speakers and change the amp later. Thanks for your info.
Always is not a good answer. Amplifiers and preamplifiers are the heart of the music. It seems to me that to often people purchase speakers and then must search for an amplifier that will drive that speaker, and not just drive the speakers but create a sound stage and believability, texture and nuance. Some speakers are hard to drive and require expensive amplification.
This would be a good time to consider amplifiers and preamplifiers. They just might transform your 804's.

I used to come at this problem from the opposite direction but experience has changed my mind.

Good luck
Post removed 
I'm listening to everyone's suggestions and appreciate your input. I thought I would upgrade the weakest link in my system and from what I can tell, the weakest componet is the Krell KAV 300i. I may keep the N804's for awhile longer. My system ( with Ayre QB-9 ) sounds at least as good as some other ( 804D's without QB-9) systems using McIntosh or Rotel amps. So I thought if it sounds good now, if I upgrade the amp., to Pass, ARC, the new MD-309, integrated or possibly some of the pre/power comb's., I would get even better results. Even though I like the B&W sound, perhaps I should listen to some other speakers
( $7K- $10K).
I attended RMAF this year and made a point to stop into a room that I had really enjoyed last year. The only thing that changed significantly was the speaker. A different make and model was chosen this year. The amplification remained the same. This room year was a disappointment.

What's the point? Well, in my opinion, the speaker selection changed the sound significantly, which affirms why I suggest selecting speakers first, and then selecting amplification that will drive them well.

Obviously, if you own or are considering buying tubed amplification...and any of the sub sets of tube amplification, you must be careful to select speakers that can be driven by the amplification that you own or that's on your radar.

That said, there's no right answer, and there are several roads that will get you to your destination.

Good luck.
10-27-10: Tvad
This room year was a disappointment.
Very sorry for the lazy editing.

Should read..."The room this year was a disappointment".
For sure you can change speakers and the sound will always change dramatically but is it change you can live with? Been there done that. Sure for a different sound get a different pair of speakers.

The single ended guys go at it the other way, They start with the amps and then of course choose speakers to go with the usually limited power they have.

From my experience a superb sounding amplifier will sound superb or at least very good on a set of capable speakers.
pick amp and speakers as a synergistic pair.

I mean, say you just bought some speakers you really liked.....for some reason, you now want to go amp shopping and risk not liking the result. Nope, I'd buy 'em together.......
But that's just me.

I'll toss my hat into the front to back method for who ever is counting.

My short answer though is I’d upgrade both…. Asap.

RE which one first?
I've heard very expensive speakers sound not so good on modest front ends, and I've heard modestly priced speakers sound outstanding off a great front end.

In other words, what the speaker see's is the most important thing. The signal quality, and at times, how much there is of it. The speakers needs for power (juice) is always a consideration, as well as is it’s voice.

What sort of voice & how much juice you want to feed them regularly is going to be your main concern?

Gonna stay SS forever? Wanna try tubes? How about SET?

As signal purity is IMHO 'PARAMOUNT' I prefer the source to speaker plan.

Consequently, less money needs be spent on speakers that way right off. For a good long while too. Ultimately though you'll want to upgrade them most likely.

By that time however, you will have established a far better degree of signal quality, and adding fine speakers will let you have a better feel for what it is you want in a loud speaker by then... Two ways, two and one half... full range... panels.. horns... etc.

I prefer that path versus jumping on expensive speakers and then trying to catch up to them by adding the proper match for them in terms of power amp.

Great front ends will make modest speakers sound great!

Great speakers with modest front ends won't sound like they should or could.

so it comes down to this for me, Do I want a great sounding system now? Or a great sounding system in a year or two or three?

If you're in a hurry... and on a budget too... go front to back. If there's no budget constraints well, have at it all, naturally. I assumed there is some money concerns given the initial question though.

About the only thing one can take from threads like these is the speaker & amp combinations are nearly THE key to a good system. That blend is indeed very very, integral.
We have two schools of thought talking here. But both are in agreement that it is the amp/speakers match that counts.
I would audition many different combinations, find the most satisfactory and then begin the upgrade in any order depending on price, mood, dealer's attitude etc. Or would wait and get both at the same time.
Personally I tilt a little towards "speaker school" and tend to think of speakers (and the room)as musical instruments. After all I have Michael Green free resonance speakers - not the last word in resolution and many other aspects of sound reproduction but they play in a very natural way with any amplification I tried.
Speakers first as they will dictate your amplifier selections.
I would suggest speakers first because you want them to be suitable for your room.
Thanks for everyone's input and advice. One problem that I have run into is determining how any speaker candidate will sound in my system, fed by the Ayre QB-9 in particular. It made my system much more lifelike. I'm thinking that new speakers may sound great at the dealers but at home, they may be too aggressive or forward. Should I take my DAC and computer with me as I audition speakers, or limit myself to the one dealer that already has the DAC. I mentioned that I heard the 804D's in systems without the QB-9 and the speakers lacked that live feeling that I'm getting at home now.
Take your DAC, computer and amplifier to the dealership. There's no other way to hear the combo unless the dealer happens to have the same electronics.

Otherwise, slow down a little and buy a plane ticket to Las Vegas to attend CES and T.H.E Show in January to listen to a wide variety of speakers (and matching amplification). You'll get a good idea pretty quickly what appeals to you and what doesn't.
if at all possible I would try for a audition in your own home.