Symptoms of damaged drivers / crossovers?

What type of symptoms show that a driver or a crossover is damaged and in need of repair or replacement? - I.e. what type of bad sound indicates that the culprit is the loudspeaker, as opposed to the amplifier or the source?