Downside of sensitivity?

I'm new to world of audio, and still puzzling out a number of things. One question in particular that I have concerns speaker senstivity. It seems as though higher sensitivity is a positive, i.e. it allows one to use lower-power amplifiers, sound is more relaxed, generally more flexible with equipment. Yet there seem to be a large number of quality speakers with lowish sensitivity. Given that speaker design seems to be an exercise in compromise (short of the cost-is-no-object level), what qualities does one give up with higher-sensitivity (say, > 91db)? Conversely, what qualities does the speaker designer gain with a less sensitive design(86-87 db)? I've quickly learned that there are no absolutes when it comes to audio equipment, but any thoughts as to the trade-offs in general? Tks.
There are many factors that influence the final sound character of speakers, not just sensitivity. To choose a speaker based solely on this one parameter would be folly. That said, picking inefficient speakers limits your choice of amplifiers to higher-powered models.

There are a couple of approaches that work well. You can find speakers that will work well in your room and then find a suitable amp for those speakers. Or you can do the reverse, i.e., decide on an amplifier that you really like and then pick your speakers based on what will work with your amplifier.

Ideally, I'd want the amp/speaker combo to play comfortably without distortion or compression in my particular listening room environment.
One of the drawbacks of having an extremely sensitive speaker system, eg 103db @ 1metre, is that everything, and I mean, EVERYTHING, can be heard- funky ground- big time sound through those horns, tube rush or other noise associated with electronics- welcome to a proctological exam of your system.

Wasn't the original rationale of the sealed box to give big bass in a small enclosure? And, that coincided with the intro of solid state electronics, which had the power to drive them?

I agree with Plato that the interaction between components ultimately means that you could own two absolutely great pieces of gear that really don't mate well.
High sensitivity speakers sometimes bring out hum and hiss that would not be noticed with low sensitivity speakers.
The downside of high sensitivity speakers had been that there were so few models available which were of high quality and low coloration.Over the last 5 years or so that has changed greatly.As you may be aware,there is something of a renaissance underway in older audio reproduction technologies,vinyl e.g..High sensitivity speakers have benefitted and there is a growing pool of high quality offerings out there.In fact many would argue that in terms of the assault on ultimate audio reproduction,high sensitivity designs lead the way.Dynamics are the key to lifelike reproduction and dynamics is what high semsitivity/efficiency designs are all about.
In general,high sensitivity designs are going to sound more lively,engaging and alive than lowish ones with complex crossovers.
The transformation to lower sensitivity speakers was a result of high demand for more linear speakers that measured well. In general, (down boy!) sensitive speakers can be more dynamic. This is especially true if you are of the single ended, triode crowd. I agree with the comment that approaching a system with either an amp or a speaker in mind can dictate what the other needs to be to perform at its best.
High sensitivity typically comes at the expense of wide bandwidth, power handling, convenient size, etc... So long as the other criteria can be met within reason i.e. speakers that will go low enough, don't go into dynamic compression rapidly and / or are of convenient size for your specific location of placement, the more sensitive the speaker, the better off you are. How the higher than average level of sensitivity is achieved may bring about other side effects that make things less desirable. Like anything else, do your homework BEFORE spending your money and / or audition as much gear as you can to form your own opinions. Just remember to compare apples to apples if you want to arrive at logical conclusions. Sean
I think Sean covered the issue quite well.

In my opinion the single biggest downside to high efficiency is that you will need a bigger box to get the same bass extension. If you want the advantages of high efficiency (improved dynamic contrast, for instance) you either have to live with a bigger box, reduced bass extension, a powered bass section (built-in or as a separte subwoofer), or corner speaker placement.

Thanks all for the great responses. In poking around on the internet, I came across the following which I thought was interesting: