I have often wondered about this myself. I certainly understand the concern that a sub might sound slow, bloated, or one note. But it seems to me that there is ample evidence available to audiophiles that manufacturers like REL, Velodyne, and JL make subs that have few or none of these problems. So the question remains, why do so few audiophiles seem to use them? Maybe it is a prejudice, left over from an era when musical subs were few and far between. Maybe it's that many audiophiles value system simplicity, and they view subs as adding unnecessary complexity.
I use a sub in my 2 channel system. I initially decided to use it as a solution to space limitations, but now that I have, I'm not sure I would give it up if my space limitations were to change. Among the biggest advantages, as I see it, is the fact that the sub has a powerful, optimized amp to drive the woofer, which left me free to choose a low power, sweet sounding amp for my satellites, without having to worry about the amp's ability to balance both muscularity and finesse.
Of course, some manufacturers, like YG Acoustics and Evolution Acoustics, use powered sub modules in their tower speakers, which is essentially like having subs that act as stands for your satellites, integrated and optimized by the manufacturer. I think that is a very intelligent approach, and I often wonder why more manufacturers don't adopt it.
Among the downsides of using a sub, as others have mentioned, is that it is very difficult and time consuming to integrate a sub properly in a system. It took me several full days of experimentation, microphone measurements, and computer assisted equalization. Proper integration also requires a high quality crossover, which are expensive and somewhat difficult to find. As a result of these difficulties, it is rare to hear a system in which a sub is truly well integrated. Perhaps this explains some of the reluctance on the part of audiophiles to embrace subs.
Ultimately, the standard against which success should measured is your INABILITY to hear the contribution of the sub until it is turned off. If you can reach that level of integration, the results can be very musical, IME.