USB seems to be the best choice since it does not produce jitter directly. It can still induce jitter indirectly by injecting electrical noise into DAC. You could try USB cable with data line only (no +5V, GND) or/and install optical isolation. Other than that choice between optical and coax is system dependent. Coax should be better, having faster transitions, but can inject electrical noise, create ground loops and is prone to standing waves (transmission line effect). Toslink has 10x slower edges inducing jitter when system is noisy - noisy level threshold translates to jittery timing.
I use Toslink, but my DAC is very quiet and has strong jitter rejection. When you have problem you can always add external reclocker. I such case I would run Toslink from streamer to reclocker and then very short (less than a foot) coax from reclocker to DAC.
I use Toslink, but my DAC is very quiet and has strong jitter rejection. When you have problem you can always add external reclocker. I such case I would run Toslink from streamer to reclocker and then very short (less than a foot) coax from reclocker to DAC.