Sure, there is a point where you will not get the best out of a speaker but most folks cannot hear electronics as well as they think they can. A pair of $5000 stereo mains would sound great. They'd sound better on better electronics but it's a stretch for many to hear it.
And, it would be helpful to better understand what 'expensive' means in regards to a speaker budget, low end, and high end. Five speakers (including the two mains) and subs or more?
Other considerations: Primary use of system, TV or two-channel audio? Hopefully two (or more) good (not expensive) subwoofers as well.
I love movies but I'm an audio guy first. In your case I'd get better mains, a fair center and smaller surrounds plus two good inexpensive subs. I have the Elac Debut F5.2, center and B5.2 bookshelf surrounds for the smaller TV room (and an old REl sub). Elac is great for the money and there are a lot of other good economical mfrs these days, Golder Ear, PSB, etc.
I would imagine at some point, the limitations of the receiver will outweigh the benefits of purchasing higher-end speakers? Yeah, IMHO you’re already there as the AVR just doesn’t justify higher-end speakers and will be a significant bottleneck to good 2-channel performance. To meet their competitive price points, mass market AVRs use cheap components (power supplies, etc.) so both its preamp and amp sections are severely limited in their respective performance levels. And no, it’s not hard at all to hear the difference between a mass market AVR and decent dedicated stereo electronics with good speakers. If you’ll only be using the 3700 I wouldn’t spend more than $1000/pair on floorstanding speakers. I’d recommend the SVS Prime Towers as they have an excellent Prime Center speaker to match and that’s much better than most center speakers near its $350 price, and a good center speaker is really critical to fully enjoy the HT experience. And SVS offers a completely risk-free, 45-day trial so you can see if they work for you in your home and with your electronics. Read the reviews that are universally positive.
However, there is another option if you think you might get more serious about 2-channel down the road. Your Denon has pre outs so you could buy a good stereo integrated amp with a HT Bypass feature and seamlessly integrate it into your HT setup (with the 3700’s pre outs) such that the integrated alone powers the front L/R speakers for 2-channel and the AVR is completely out of the signal path. If that’s a possibility then you could absolutely justify getting much higher-end speakers— it just depends how serious you wanna get into 2-channel. Just my $0.02 FWIW, and best of luck.
Save you the trouble: you can never get the best out of anything. No matter what you do, something can always be better. Any speaker however good or bad you will hear improvement with a better amp. Any amp, however good or bad, same thing. Including even AVR which you are right to wonder about that.
Diminishing returns are another fallacy. Except in the case of channels. From mono to stereo, huge returns! From stereo to add a center, nada. Nothing. Stereo already has a good center image. From stereo to surrounds, worse than nada, the processor ruins the stereo quality.
One simple decision- to go stereo instead of multi-channel- frees up so much money you can afford much, much higher quality. Vastly higher quality. As in no comparison higher quality.
Run the HDMI direct from your video source to your video monitor. Run your video sound direct from your source to your stereo integrated amp. Thank me later, when you hear how freaking awesome this is.
+1 millercarbon: a good or better two- channel stereo system always beats multi-channel! All those extra sound sources (speakers) just cause confusion! Let the rock throwing by the HT crowd begin!
Buy the best floor-standers you can afford and keep the Denon. Later add a good stereo amp and use the Denon as a preamp. No need to junk it!
soix is correct about being able to hear mass-market receivers vs good audio gear. My bad. I was thinking of all the higher end audio comparisons I've done over the years. That can be work to differentiate above certain levels of good quality. SVS is a good call on speakers as well as subs.
A lot of other interesting and fun suggestions. Since this system appears to need to do double duty, and since the picture implies two people watching, it means two people hearing, so each gets a center image that is either far left or far right depending on seating. Does not sound like fun to me. (Of course, the OP and sig O could sit audio style, one in front of the other in wood chairs...) Losing your HT center image is my gripe with telling them to ditch HT.
"....to go stereo instead of multi-channel- frees up so much money you can afford much, much higher quality. Vastly higher quality. As in no comparison higher quality..."
Quality not quantity.
“I’m currently in the market for floor standing speakers that will serve a dual purpose as both 2ch audio, and the L/R in a 5.1.”
Sadly, this place is run by 2-ch fanatics (majority of them are deep into Vinyl or should I say anti-digital) Do you see the pattern, they have a one track mind….digital is bad, home theater sucks. Never mind if someone like you loves movies and music alike. Instead of suggesting a good pair of speakers, what you get is biased
advice to give up the idea of hybrid system and focus on two center stereo…fabulous!
I suggest read up on some of the speakers listed on hometheaterreview.com and see what peaks your interest within your budget. Once you have narrowed down your choice of speakers, either find a way to audition them at nearest dealer or seek advice here from the actual users. Good luck! https://hometheaterreview.com/equipment-reviews/audio-reviews/floorstanding-and-audiophile-loudspeak...
I love music and videos. I am fortunate enough to have a separate audio and home theater… which is ideal as they have somewhat different requirements and can get in each other’s way. But they can be combined effectively. Some good speakers and a good amp can raise the 2-ch part to a very high level of performance… ultimately limited by the receiver. But people’s lives change… more money, maybe move. Then those expensive speakers and amp can look like a tremendously good decision. Since my interest in music and home theater has persisted over my life, every time I reached for better gear than I necessarily could justify with my other components it has been a great decision in retrospect.
With that Denon it looks like the compromise is already made so might as well run with it, and you can hear whats coming up behind you in the films etc.
I run TV sound via optical through a two channel system set up for music, there’s no problem with centre fill from either end of a three seater but the speakers needed careful placement to achieve a wide listening area with a stable image and tuneful bass. I only have depth into the screen however, not behind me and they’re a bit too dynamic. If I set the volume to hear the dialog the sound effects get to neighbours annoying levels (we’re in a third floor apartment, temporarily) and I end up arguing with my wife about just how loud it needs to be.
"...Sadly, this place is run by 2-ch fanatics (majority of them are deep into Vinyl or should I say anti-digital) Do you see the pattern, they have a one track mind….digital is bad, home theater sucks..."
I think you mean, luckily this place is for two channel enthusiasts. I still have my 60s and 70s LPs but I only listen to digital sources.
Depends on resolution of your system to determin that point and how much you have to spend.
Ah yes, two-channel 'enthusiasts'. I admit to being one totally, but I still thought to suggested a center and surround solution (with better mains for two-ch use). It seemed to be what was needed per the O.P. unless I misunderstood. Sometimes it's not about what we want I guess.
I remember the days having to work it out for stereo and video in one system (adding TV sound in the 80s to a receiver). It was more challenging as the system grew and eventually separated. These days I feel very lucky to have individual rooms/setups for high end two-channel audio, regular TV, and a home theater. It just takes time and if the O.P. builds carefully those items are very usable later. (plus I probably never sold a piece of audio gear in my life...)
Keep the Denon and buy speakers you really like. They will sound great (assuming a common sense match).