As I mentioned above, I have not heard your amps, but the description suggests that it is very well done--the construction is good, it uses an interstage transformer to help drive the 300B properly, etc. Nonetheless, if you find the sound is not quite what you want, it is worth exploring alternatives. I would not automatically assume it is the speaker that should be replaced, particularly since it is not that easy to find speakers that have all of the positive attributes of the AN-E. I would at least try different speaker placement, room treatments, perhaps different tubes in the amp, even different amps if you can find loaners. In looking at different amps, don't automatically assume that SET amps are "magical" and cannot be touched by other topologies. I've heard great pushpull amps that also sound great with high efficiency speakers and OTL amps that also deliver killer sound.
At Deja Vu, I've heard the AN-E's with Synthesis Audio amps (pushpull KT66) and prefer them over Audio Note amps because they have a punchier sound. Perhaps you can find something like that to try.

