Hum in phono that comes and goes

I have a strange hum that I hear when I touch my tone arm, but not all the time. When I move it off of a record that I just played, it does not hum. If it sits for a while like before I get another record to play, it will hum when I touch the tone arm to place it in the lead in groove. If I repeatedly touch it the hum reduces and after several touches will go away, so this seems like some sort of capacitance issue.
Is this the cartridge, phono amp, or pre amp, and how do I track it down.
And just to add to the strangeness, if I touch the arm and the pre amp chassis (not phono amp), the hum will disappear.