Toslink is only a benefit where the galvanic isolation is needed. It relies on light bouncing off the inside walls of the fiber as it travels so yes glass is most definitely better. Whatever transmits light better.
Different sounding Toslink is just different levels of data loss, jitter, degradation.
Always use USB or I2S whenever possible.