Please discribe what you hear in more detail. How loud. How big and what brand are your amps and speakers. if the source is changed, does the sound change. Please be as specific as possible.
The implication is that your speaker drivers are continuously in motion. This could cause premature aging of the speakers' mechanical components. If your speakers are dynamic cones, parts that often fail are (1) the "spider" that keeps the voice coil in proper location, (2) the thin wire that runs from the voice coil to the signal terminal on the basket could break or detach, and (3) the soft surround that attaches the outer edge of the cone to the basket or housing. Normally, these components are designed for years of use and flexing, but they do eventually fail. Some, like the surround, fail more commonly due to age than usage, but continuous usage such as you've described could shave a little time off your speakers' usable lives.
Very efficient speakers and even good amps can have this occur. I had a pair of Klipsch La Scalas (104dB). Pretty much always had a some white noise. I was using a 40 watt class A/B amplifier of good, but not great quality. Will it damage the speakers over time? Very hard to say, and probably very equipment dependent as Richard already pointed out.
Sounds like a power grid upgrade is needed. Try some different power cords or conditioners. Does it quite the sound? That is the sound of power noise, this is what you must listen through to get the music. This is why all the threads on power. You will be amazed to hear your system for the first time when this noise is eliminated.