There are some parallels to where you’re starting from and where I started. I used an Arcam AV40 and PA240, both of which I still have. It was good for what it is and both still remain although the AV40 is purely for AVR duties.
I now have a fully separate 2 channel system and there is no comparison with the Arcam. Lumin U1 -> Denafrips Terminator Plus dac -> Pass Labs XP22 -> PA240 -> Focal Kanta 3s. It's my first foray into 'hi-fi' and I'm glad I've upgraded piece by piece because you learn a lot about how every item in the chain matters.
Just adding the XP22 alone was night and day over the Arcam as a pre-amplifier, not to mention the other components that replaced the AV40 on the streamer/dac side. But it all comes at significantly higher cost so it’s all relative. The amp is the last item in the chain I will be upgrading.