Speaker swap or Amp,which impacts more????

hi,system is:SF-CD1,SF-Line2 Pre,Bryston 4B-ST driving B&W 802S3.Was thinkin'two ways to upgrade-- bryston 14b-sst or some similar amp,or going to 802N speakers.budget only lets me do one.new to hi-fi and am on the fence over this one.will do both in time but callin all for a little advice in the matter.do not like talkin to salesman.thks for any help.