Itunes should not make any difference, since you're transferring data and not the music. As long as it is bit-perfect you should be OK. One possibility is that there is something in the MACs software (or USB) settings (or Bryston settings) that is not right. Perhaps USB port is faulty - try different one. It is always possible that you're injecting noise into the DAC (try 2 wire USB cable), but it is not very likely. It would not make it sound that awful.
You can always borrow another Mac and repeat it. Async USB should give you the best sound available.
You can always borrow another Mac and repeat it. Async USB should give you the best sound available.