Not necessarily flat, but smooth.
Agreed - exactly! Very few people are aware of this. When I talk about tone control adjustments audiophiles cringe ....but they should not. It is no crime to tweak in this way. In fact a ruler flat on axis response is NOT all that important to our listening pleasure...it is actually the off axis that matters most.....in fact the MOST important thing is that the on axis and off axis behave in at least a uniform way with consistent gentle roll off as you get to wider angles...this is what sounds natural!
What sounds unnatural to our ears is a speaker transducer that has either strong directivity or wobbles and bumps off axis such that the on axis and off axis curves do not all behave together in a smooth way - in many poor designs they often crisscross as the cone breaks up or there is a sharp discontinuity at the crossover region (suckout or sudden abrupt change in dispersion pattern)
See this discussion SM75-150S. You will see the author correctly emphasizes that it is NOT the smooth on axis frequency response that is so important in a driver but the smooth consistent behavior between on and off axis and a wide dispersion.
SMOOTH is the key ingredient. This is why many designers pay so much attention to smooth phase response which is a warning sign of unwanted things going on with the design.