In my experience, coax is better than optical, and USB can be better than coax. But this is very system dependent. Some DAC's just sound better with one type of input compared to the others. So there is no hard rule here.
And to muddy the waters even more, the quality of the cable can effect the quality of the sound and change the order of things. Some USB cables may sound the same or not as good as a high quality coax, while others will sound better.