Speaker swap or Amp,which impacts more????

What you like/dislike about your current setup?
For all you know, it could be some other things you can improve. 4 BST is pretty solid amp. Quick pace, great for recording industry. A reasonable match with B&Ws.
As far as SF line 2 goes, there are plenty of room to move up from there if you can live without the remote.

The preamps has a lot to do with sonic character.
That's why some purist prefer not to use preamp. However, if your source doesn't have enough gain then it would sound lack of dynamic. I'm not a big fan of passive preamp for that reason.

You are missing out on the great preamps. You'll find that most of tube preamps doesn't go as low as SS counterparts. However, most of tube preamps sounds more open than SS.

If you plan to go with balanced setup,
the Audio Research LS-5 MK III is still one of the good preamps.
For non-balanced RCA output,
Audio Research SP-8 MK II and SP11 MK II do get great bass.
The SP-8 is not as clean and transparent sounding but it is warmer sounding.

If it is warmer tone you're looking for, you can also consider many other good preamps out there.