All things being equal there’s no replacement for displacement...But use the sportbike analogy and the experienced will say smaller and lighter is better on a tight track.
The problem with analogies is finding the ones that apply properly and illuminates a matter. Given proportionately stronger motor force a larger diameter driver will see no lack of nimbleness compared to a smaller unit, within the constraints given re: upper frequency bandwidth dictated with the use of bigger cones. Moreover added surface area AND better efficiency to boot equals less inertia build-up in the moving system via less cone movement, not to mention the benefits of a surplus in headroom with all that entails in regards to lower distortion and minimizing thermal compression.
It’s also worth keeping in mind there are 5" woofers and 12" midranges; a smaller diameter low efficiency woofer in a 2-way system acting as well as a midrange isn’t necessarily the last word in performance prowess in its upper range (nor in the bass for that matter), whereas a suitable 12" or even a 15" high efficiency unit can do very well into the lower to central mids, not least when high-passed below 80-100Hz - something that would turn these drivers into "mitten raketen," as they say in German (i.e.: midrange rockets).
Even though bigger units are limited in regards to upper band frequency response they can be properly met within their safer operating range (typically between 500-1kHz) with a compression fitted to a horn/waveguide, something a dome tweeter would never be able to achieve unless mounted to a waveguide; the power response in the cross-over region between a dome tweeter and a woofer/midrange isn’t ideal, to say the least, something that can be attained quite successfully with the use of a waveguide/horn crossing over from a larger woofer/midrange - all in the name of a more (energy) coherent presentation.
My old fischers with 15’’ woofers would allow me to feel the air pressure in the radio studio move when the door opened and I’ve never experienced that from any other design, but I could also watch that paper driver flex and distort its shape when playing music. To make a 15’’ woofer that’s as rigid as a 6.5? is it even possible.
This should be relative to what I outlined above.