There are two very different ways in which a stereo system can be wired out of phase. In the first case, one channel is 180° out of phase with the other. This produces some dramatic negative effects, imaging is damaged severely, and it’s just quite noticeable. In the second case, both channels are wired in the same phase with respect to each other, but the phase of the system is 180° different from the phase with which the material was recorded. Frankly, I have never been able to hear that second difference in my system, perhaps because I use dipolar electrostatic speakers. However, there are golden ears who claim that system phase makes a great deal of difference to them. Nevertheless, this second sort of phase issue is unlikely to have anything to do with your observation because it’s quite subtle, at most..
So please tell us exactly what you mean when you say you “switched cartridge leads”. Thanks.