Check out the North Acoustics small monitors, Spirit and Kitty Kat Revelator, made specially to go near the wall.
If you mount in a bookshelf or entertainment system - do yourself a favor and install with a picture frame around the baffle and integrate into the walls in a flush mount fashion. If you do not then you will surely face strong edge diffraction issues. (This also means you need a forward facing reflex port in your choice of speaker and to fill any void space around the speaker with acoustic batting)
Alternativley use a wall bracket with a long extendable arm so that you can bring the speaker well forward from the cabinet shelves by a speaker length (even if you only do this for serious listening).
Pricing -- in the $3-500 range. Keep the recommendations coming as I'm not familiar with bookshelf speakers as have never peaked my interest -- these would be for my sisters' system & I don't have a clue as to what amp they are using -- probably along the big box store line; i.e., Yamaha, Pioneer, Sansui, etc.
Energy Loudspeakers manufactures some good bookshelf monitors at reasonable prices. I had a pair of the Take 5's for a while. No bass but good for a 2nd pair in a small office or for surrounds. They have a C-100, C-50, and RC-Mini that will fit your space requirements. I was going to suggest the Proac Tablette 8 Ref speakers which will fit in your space but they are a very high priced speaker line out of the UK. Even their small units are way up there. B&W produces some mini-monitors such as the LM-1 and M-1. I have 3 of the M-1's as my surrounds and center channel. Super sound but again, no bass. The M-1's I believe sell for $200.00/each.
I bought a pair of Usher S-520's for a system for my daughter and have been very impressed with them. They're small enough for the entertainment center, are front ported and cost $400 new. Here's a Stereophile review.
Ontjesr, I second the Amphion recommendation.
Another possibility is a variation on the Loki kit from Madisound. The Loki uses a 6.5" coaxial driver from Seas. I was going to manufacture a speaker based on this driver, and even showed a pair at an audio show in Dallas earlier this year. But the Loki kit undercuts my price by so much that I abandoned my plans, and my prototypes are now sprinkled around the living room in my home theater system.
Now the only problem is the Loki's enclosure is 15 inches tall. But Madisound sells a 12 inch tall enclosure that will also work - you'll have to cut the holes yourself though. This box is 7.5 liters, and just leave it sealed for response down to 70 Hz (and probably down to the mid-50's with boundary reinforcement). Get the shielded version of the driver, and use Madisound's Loki crossover.
The advantage of the coaxial driver in a less-than-optimum acoustic environment is that the woofer cone acts as a waveguide to narrow the tweeter's radiation pattern in the octave or so above the crossover frequency. Imho this is very valuable, as otherwise you can easily end up with a speaker that measures well on-axis but sounds harsh because there's an extra 6-12 dB or so of reflective off-axis energy in the lower treble region, right where the ear is most sensitive. The trade-off is in dynamic capability; the coaxial's woofer has a smaller linear excursion capability than a comparable stand-alone woofer does.
The Amphion uses a separate dedicated waveguide instead of the woofer cone to control the tweeter's radiation pattern.
Other companies that make compact coaxials include KEF, Tannoy, and Gradient. In the interest of full disclosure I'm a Gradient dealer, but even the smallest ones are above your price range unless you find a used pair. And as you can see this post is a Madisound plug, not a Gradient plug.
I had owned the ELS-3 speaker and upgraded to the Epos M5. The speakers are on a bookshelf. One of my concerns about any speaker being placed on a bookshelf is the degree to which the bass gets muddy/boomy. I had not noticed any significant bass problem with the ELS-3, but that's most likely due to its lack of bass extension (maybe 60 hz). The M5 is significantly better than the ELS-3. The ELS-3 gets edgy/ragged in the mids/highs at times; also there is a portion of the midrange where it sounds 'boxy'----this is hard to describe. None of this exists with the M5; its frequency response is fairly flat and its bass response extends lower then the ELS-3's. I have not noticed any bass problem given my shelf placement. They produce classical music in a clear and non-fatiguing manner. Please read Reina's review in Stereophile. A used pair can be found for around $500.
I also second the recommendation of the B&W 600 series speakers, although I haven't heard them. But please be careful with respect to bass response given bookshelf placement. IMO few things are worse than a boomy/muddy bass! Good luck---try to audition in an environment similar to yours.
Check out the Eltax Monitor III,they have downward firing port,they are only sold in the states by AudioWaves for $299 in California they are very impressive with deep tight bass,and they are very extended on the topend.Also look into the Dana 630 from the: www.audioinsider.com for $349 a pair they are very impressive,they dont have quite the bass power but are one of the best mini monitor's you can get for under $800 bucks with a very nice real wood veneer very very accurate with nice midrange bloom and dynamics,good luck!