The problem is that the changing impedance levels of a passive preamp can cause coloration. That's why they usually seem to lack dynamics at lower listening levels. The Ayre sounds excellent at any volume level.
True, however, with such a low output impedance of your source, and using a high impedance amplifier, the coloration should be minimal. It always depends on the implementation, of course.
FWIW The mechanical switch on the Ayre only comes online to change the volume, then it is silent again. In fact, it is adjustable on a per input basis so theoretically you never need to change the volume again after you've set it to the right levels for each component that you normally listen to.
Maybe I wasn't clear enough. When I said "I hear them" it meant that I could hear the added coloration coming from a mechanical switch. It does not matter how well and expensive the actual contacts of the switch are. This said, I use switching devices that do not have mechanical contacts.
Best wishes,
Alex