Good point, Erik.
What has always seemed ignored in these debates is that human perception and brain science are incredibly young fields, relative to what there is still left to understand.
The supposition that acoustic measurement is adequate has to ignore the fact that half the equation -- our brain and even our ears -- is so poorly understood. Acknowledge that truth, and the sufficiency of acoustic measurement (objective, physicalist or subjective, listening testing à la Harman) is shown to be seriously incomplete (though still useful for some purposes).