Some speaker companies (like Vandersteen and the old Thiel) stake their design philosophy on the premise that these time domain issues are crucial. Listen to those designs yourself and see what you think. I'm a Vandersteen guy, so you know where I stand.
Polarity of speaker drivers
Many speakers do NOT have all the drivers aligned in the same polarity. This is seen on many of the stereophile speaker measurements. In certain designs, I guess this is done for better summing of driver output. Is the time domain compromise audible?