Elgordo is correct about many recordings being recorded in reverse phase. On some very phase-coherent speakers, this will make a difference and having a polarity inversion switch on the preamp or DAC is a convenient thing to have. That said, many systems are not phase coherent, which makes switching polarity a moot point. Also, in my personal experience, the effects of polarity inversion are harder to hear in an all solid-state system than a system that uses tubes. But it appears that you are using tubes, so in your case, you may well hear the difference.
This seems to be one of those issues that can make you crazy if you let it. For example, I have two DACs and prefer the performance of one to the other. Unfortunately the one I like best lacks the polarity invert feature. Yet, my preamp has an invert feature but I don't use it because I feel that going through that extra switch degrades its performance to a small degree. As you can see, I just can't win!
This seems to be one of those issues that can make you crazy if you let it. For example, I have two DACs and prefer the performance of one to the other. Unfortunately the one I like best lacks the polarity invert feature. Yet, my preamp has an invert feature but I don't use it because I feel that going through that extra switch degrades its performance to a small degree. As you can see, I just can't win!