Is it taboo to mod a speaker?

I often read and hear about modifications done to source equipment, amplifiers, DACS, and ocassionally I read about someone doing stuff to the crossover portion of a loudspeaker. I myself plan to have someone modify my Pioneer DVD in a few months. However, I rarely read about anyone modding a loudspeaker by exchanging drivers. I have never heard of anyone exchange or upgrading to driver ABC from drive XYZ. Is it done? Does anyone do this professionally?