FWIW, are you sure you want to eliminate the preamp? Below 80% of full volume, most digital volume controls will shed bits in order to attenuate the signal level, which might explain some of the problems you heard. I know nothing about Audirvana, so perhaps it deals with this issue, but otherwise, you should have at a least a passive, non-digital volume attenuator for volume control. Also, do you have any power-line filtering in use? Dirty a/c (like a/c polluted by computers on the same circuit) can contribute to some of the treble nastiness you described.
