Caps are relatively cheap, replace them all. But instead of looking at your phono preamp as an assemblege of caps, resistors and diodes, try to conceptualize it as a power supply and the regulation of that power supply, an input stage, RIAA EQ and an output stage.
I tend to attack the power supply first, as I think that this has the most positive result and it is the foundation for all other upgrades. Therefore, if you can source a higher amperage power supply, that would help. Improving the electrolytic storage caps and changing the rectifiers to soft recovery HexFet or HexFred, or at the least UF4007, will be of benefit and it will be very cheap. From there, your coupling cap upgrade will have much more value. Then again, you said "poly cap", but if you have the real estate to use paper in oil caps, they are a viable alternative with a different flavor.
While you are in there you may want to look at any ICs as well and don't forget any wiring. Good luck!
I tend to attack the power supply first, as I think that this has the most positive result and it is the foundation for all other upgrades. Therefore, if you can source a higher amperage power supply, that would help. Improving the electrolytic storage caps and changing the rectifiers to soft recovery HexFet or HexFred, or at the least UF4007, will be of benefit and it will be very cheap. From there, your coupling cap upgrade will have much more value. Then again, you said "poly cap", but if you have the real estate to use paper in oil caps, they are a viable alternative with a different flavor.
While you are in there you may want to look at any ICs as well and don't forget any wiring. Good luck!