Actually bookshelf, or monitor speakers would work excellently with good woofer sellection, integration, and set-up! With HT, you're going to configure the mains(and all speakers) as "small" anyway, and the subs will handle the bass. The monitors would be handling from 65hz and above(considering roll of down from 80hz), while the subs do 80hz and bellow. Using dual woofer/mid diappolito (or similar, even THX) designs would work best for sure(especially with low ceiling in that room), and I don't see how you'd loose! Surely you could find some used Dunlavy SCIav's at excellent pricing, and mate some very fast subs with authority(M&K and Earthquake come to mind).
Both Robert Deutsch and Steve Stone from Stereophile Guide to HT magazine used the "Dun's"(even though out of Biz) for HT dubties, as well as the current Dunlavy set up's(plural) at the AV Interiors editors complex! They are fast, dynamic, ultra detailed, coheren't, and just plain sound good for HT/music dubties(with emphesis on HT. They're also Diappolito designs.
Of course, you could use other "centerchannel" design speakers all around the front*(at least), and do dipoles/bipoles all around.
3 Dunlavy SCIAV's up front, with 4 M&K SS150 tripoles(very close match with SCI's!) in back/sides, with a couple of M&K or Earthquake subs for bass, and you have a dynamite sounding HT system if you set it up all correctly!(good acoustics, and proper crossover and coupling of speakers and subs in perfect phase!).
I don't see how you'd loose with this "tried-and-true" HT speaker set up.