Hard to say which of the two will have more impact in your system, but I generally like to start with upgrading the speaker first, because I find it's generally easier to match an amp / preamp to your speaker, than the other way around.
And as others have mentioned, you typically get the most bang-for-buck with the speaker, so might as well pick your ideal speaker, and build the rest of the system around that.
And as others have mentioned, you typically get the most bang-for-buck with the speaker, so might as well pick your ideal speaker, and build the rest of the system around that.