Speaker manufacturers or box manufacturers with off the shelf drivers?

Rule #1, never take anything kenjit says as legitimatize advise.

My speakers are Audio Note AN-J/lx's, Audio Note makes their own drivers and is obsessive about matching drivers in a stereo pair. From their website:

Consistent performance is a major issue in loudspeaker design and unfortunately all drive units vary slight from each other, even if they look the same and have the same basic specification. Many loudspeaker manufacturer’s will tell you that they provide “computer matched” crossovers, and whilst this may be true in one sense (each crossover may have been matched to have the exact same capacitance, inductance and resistance) this essentially “passive” method does not adequately take into consideration the mechanical and acoustic variance present in the drive units themselves, where minute differences in acoustic behaviour will result in quite substantial differences in performance and sound. Therefore, to obtain the best possible combination of drivers and crossovers, we have developed a dynamic matching process. This ensures that each loudspeaker in a stereo pair matches a ‘master curve’ and also its partner, to within 0.4dB; to the best of our knowledge, no other loudspeaker manufacturer achieves such close matching and tests 100% of its production.


Also kenjit is an expert on trolling but not speakers. :D

The design is as important as it's execution. And too many do not ever address even the easy details. Let alone the hard one's.

I Think it comes down to being a smart consumer an educated audio enthusiast and training your ear/brain. Speaker design is an art and a science and its up to you to decide if a speaker represents a good value for your budget and tastes.

Also don’t forget that driver designers are experts at what they do and make them often at an economy of scale that would be much more if they were bespoke for each speaker company.

To use a car analogy, a lotus Elise
I once rented had a Toyota engine and transmission. It was a fantastic exhilarating ride and if I didn’t live in a NYC I’d probably own one. Did the Commuter car components (though tweaked) take anything away from the car? No, in fact it made it affordable and reliable.

This is fascinating.  I am a small speaker manufacturer and drivers to me are one of the thing that stressed me less.  There is a lot of work that goes into choosing the right driver in terms of testing with your cabinets and materials but there are lots of bigger manufacturers that use stock drivers that make AMAZING speakers.  
To name a few, Wilson (small mods in some cases) Avalon, Marten, Joseph Audio  all use stock drivers.  These guys make some of the best products in the world and they use stock drivers from Scanspeak, Eton and Accuton, Accuton and SEAS from the companies above. 
I have been told that Revel has since moved production internally but the drivers in The Performa3 line were SB Acoustic when it launched.  Doubt this?  I still have a pair from years ago with SB drivers in them.  The tweeter had a waveguide designed by Revel.  That was what made it their own.  
There are crappy speakers out there and you can spot them and avoid them.  Check out this series of articles from audioholics:

https://www.audioholics.com/loudspeaker-design/high-fidelity-loudspeakers 

See links at the bottom of the page for other relevant topics like this one on the explicit elements of a driver:

https://www.audioholics.com/loudspeaker-design/loudspeaker-drivers

in the series they touch on all aspects of speaker design other than utilization of exotic cabinet materials (aluminum, fiberglass, carbon fiber, stone, etc...) but those materials are clearly superior to MDF based on guiding principles outlined in this series.  
I will admit that i designed my products so that if Audioholics took them apart, they would be pleased with what they found.  
Good luck to you in your shopping.