Yes, only switch one. If you run multiple sources, leave the cartrige leads alone and switch the + and - speaker leads. That way all the sources that the pre inverts phase gets corrected. Some will tell you that phase is over rated due to the lack of recording standards.
VPI cartridge install f/phase inverting pre
Classic manual says that, "if phono section inverts phase, hot color becomes the ground color". My pre (w/phono) inverts phase, but I've already switched the speaker cable leads for this. So, do I still need to swap cartridge wires or is this now unnecessary?