upgrade the amp or the head unit?

speakers are a 2.1 system, B&W 685s and a HSU VTF-1 sub. current Yami changer does not have a digital output. On the back of my Yami receiver under the spring clip speaker terminals, there is a bit of text stating the amp is stable to 8 ohms with only the A speakers driven, with both channels it is only stable to 12 ohms. The sub is hooked into channel B, but the amp shouldn't see it because the sub's impedance is so high.

I have a V-DAC and highly recommend them. I would get that first and use your current changer as a transport , assuming it has a digital out. VDHs cables are good, I sell them but have never used the optical ones. You haven't mentioned your speakers, hard to tell about the Yamaha without knowing what it is driving.

IMO, you should start with your budget, then your room, then your speakers, work back from there, do cables next to last and finally check the room again.