I love Toslink myself. But that shouldn't matter. I'm more likely to believe that the dacmagic is your weak spot considering the rest of your gear. IMHO. Have you considered trying out some other higher-end dacs in your rig? You may be surprised.
Regards
Regards