How much can be measured -- and how much cannot?

There has been a lot of discussion over the years on Audiogon regarding the measurement of components and other audio products. Some people claim everything is either measurable now or will one day become measurable with more sophisticated measuring equipment. But others say there are things in high end audio that will never me measurable and that measurements are really not that important.

Here is a typical example -- a quote taken from the Stereophile forum regarding their review of the Playback Designs MPS-5:

"JA 2/17/10 Review Measurements of Playback Designs MPS-5
Posted: April 13, 2011 - 8:42am

John Atkinson's 2/17/10 review measurements of the Playback Designs MPS-5 revealed less than stellar technical performance even though Michael Fremer really liked the player. I've included JA's closing measurement remarks below followed by the manufacturer's comments.

To my knowledge there was never any followup in Stereophile regarding the manufacturers reply the MPS-5 could not be adequately measured with traditional measurement techniques.

I believe Stereophile should respond to this reply in the interests of its own measurements credibility.


How important do you think measurements are? Are the ears really the only true arbiter?
"How important do you think measurements are?"

Important, but do not tell the whole story. Not likely that they ever will in the foreseeable future.

"Are the ears really the only true arbiter?"

In the end, for the end user, yes. No two ears along with the rest of the human auditory sensing system behind them are exactly the same, just like stereo gear. What else could possibly take their place?

Specs and measurements are most useful to help determine what gear might/should play well together up front during the decision making process, but its all you and your ears from there.
I second Mapman's comments. As I see it the main usefulness of measurements, and also specifications, is that they allow one to identify and RULE OUT candidates for purchase that would be poor matches to either the rest of the system (e.g., impedance incompatibilities, gain and sensitivity mismatches, etc.) or to the user's requirements (e.g., maximum volume capability, deep bass extension, etc.).

Also, they can facilitate diagnosis of issues that may exist with components that have already been purchased, but that may not be optimally matched to the rest of the system.

The number of times that I and many others have referred to John Atkinson's measurements in helping Audiogon members to make purchase decisions or to diagnose problems is practically countless.

A secondary benefit is that measurements and specifications can help to provide a better understanding of the design concepts and approaches that underlie a product, which is something that I always prefer to have.

-- Al
Measurements are a beginning but they are not the end of the journey. It gives an idea of what the piece will do or can do then you have to figure out how that works with what you have and or are willing to acquire to make it work for you.
"The number of times that I and many others have referred to John Atkinson's measurements in helping Audiogon members to make purchase decisions or to diagnose problems is practically countless."
Actually I have never found measurements to help me in any audio decision to by a certain product. Measurements are helpful in the design process but do not tell you anything how a component will sound. Use your ears and they will guide you on the right path
If we spent more time listening than thinking we would be better off.
And I will add that learning to read and understand measurements will provide a great education to an audiophile. When I compare the specs I looked at originally compared to the factors that I try to take into account now, I realize how much I have learned from this hobby.

Off the top of my head, factors that I now take into account:
output voltage
output impedance
input impedance
input sensitivity
harmonics distortion (2nd, 3rd order, etc)
damping factor
speaker distortion (hard to find in general)
crossover point
crossover slope (1st, 2nd, 4th order etc.)

I'd also add to that list the standard plots vs. frequency graphs that one sees in Stereophile's measurements section.

And when I see many of Al and others' posts, I realize that there is much more to learn.

BUT all of that having been said, I too ultimately come down on the side of letting your ears make the final decision. Psychoacoustics are pretty wild, can be tough to get your head around, and aspects of them are difficult/impossible to measure. Heck there are very likely aspects to psychoacoustics that remain to be identified. But trust your ears, they are excellent psychoacoustic instruments connected to all the bits of the brain that matter for a satisfying experience of music.
>But others say there are things in high end audio that will never me measurable and that measurements are really not that important.

Speaker preference has a very strong correlation (if you weren't a scientist you'd say it was causal) with with uniformity of amplitude response curves at angles representative of a direct listening window and where front-wall, side-wall, and floor/ceiling reflections come from in average rooms.

Sean Olive has reportedly taken that to the next level producing a formula that predicts peoples' speaker ratings based on such measurements.

Stored energy is important too, although that shows up in polar response curves (ripples in the on-axis response can come from diffraction effects that aren't that audible, but ripples at many angles tend to be resonances which are) with fine enough frequency resolution.

>How important do you think measurements are?

Very. Good enough to predict that you're not going to like a speaker before you go to the trouble of listening.

>Are the ears really the only true arbiter?

If you can hear it you can measure it and if you can measure it you can fix it or decide it's not worth the price tag which goes with the fix.

You'd do well to read _Sound Reproduction: Loudspeakers and Rooms_ by Floyd Toole.
Emotions and logic are two different contexts. Perhaps both are needed to fully evaluate and appreciate equipment and systems. For music, the final evaluation is how much is emotionally moving to the listener. One can ask, "do I hear detail, staging, extension, depth and evenness of frequency response," but in the end, whether any of these characteristics is emotionally involving is the ultimate question and challenge, and for that, to my knowledge, no quantitative measurement has been, so far, predictive.
The froofaraw around stuff you cannot measure?
The problem YES there are qualities no one has figured out how to measure. But NO the problem is others abuse this to make rediculous claims no one can podssibly verify realistically.
So we get all sorts of crazy advertising drivel of Quantuum tunneling, and various 'new age' sort ofdrivel to 'explain' whey one item (which looks just like the $10 one) should cost $1,500. instead.

So it really thee FACT some stuff cannot be measured which is important. and then the fact other abuse that first fact to obfuscate issues and sell 'tin plated junk' as is it were gold.

And that bunch of folks pushing to line with bs are the ones I blame.
So all in all yeah i find there are real things which measurements cannot quantify.
On the other side are all sorts of bogus balony being
spouted claming improved sound but no one can show that particualr stuff IS crap due to the aformentioned hustlers being able to pull the "can't be measured card"
So I hope all those con artists (even the one's who seem to actually believe thier own BS) will end up in a Hell of thier own creation.
As PT barnum said: a Sucker is born every minute. In the Audiophile world.. same comment.

I go for plenty of tweaks. so i guess i try to walk a line between folly and science
I think the original question and all the responses miss the central point of the modern audiophile pursuit. Despite audiophile claims to the contrary the science of of sound reproduction is extremely well understood, particularly if we're talking analog signals. Audiophiles focus on what sounds good, which has become more a function of marketing and mass psychology. As a group we have been indoctrinated and conditioned to think about sound quality in very specific patterns and a group of talented entrepreneurs are profiting by exploiting these patterns. The products we buy are luxury goods and marketing involved is well understood - product differentiation and market segmentation predominate.

Audiophiles have two fundamental challenges to their world view, musicians and engineers. Each group basically thinks we're delusional. To paraphrase comrade Marx, "who you gonna' believe, me or your lying ears?"
"Audiophiles focus on what sounds good"
Of course we do if it does`nt sound good why bother?
I don`t know what your priorities are but I only want audio components that allow me to 'enjoy' the beauty and emotion of music. If you can`t connect to the music what are you listening for?
This is like Rashomon. It is amazing how many differing points of view there are on this subject. Each one seems to be a valid part of this complex puzzle. IMO. The question remains, which aspect is the predominating factor -- or is this a matter of co-important factors?
Regarding the example in the introduction regarding John Atkinson's measurements of the Playback designs MPS-5, I would have been interested in hearing a discussion between the reviewer and John Atkinson as they listened to the PD together after the measurements were taken and after John Atkinson wrote about his observations.

Another recent example comes to mind -- John Atkinson's recent measurements of the AMR DP-777 opposite the reviewer's observations. John Atkinson expressed disappointment with the measurements of the AMR DP-777 where the reviewer praised the unit very highly.

Is it possible for the twain to meet? Would it not be interesting for John Atkinson to sit down and listen to some music with the reviewer when all is said and done to see if he finds the actually listening experience as irksome as his measurements? I think it would be interesting to hear their cross-talk.
"Despite audiophile claims to the contrary the science of of sound reproduction is extremely well understood"

No doubt. One is always at risk when proven scientific principles are ignored or not applied in solving a problem.

Here are the most useful audio specifications/measurements for me that I tend to always take into consideration:

Input and output impedance so these can be matched between components for better dynamics and lower distortion.

In addition, for amps, I look at power ratings, damping factors and current delivery related specs to get an idea of how well an amp might be able to drive a particular set of speakers.

Sensitivity related information can be useful as well,especially when dealing with various phono section designs and speakers.

I do not usually pay too much attention to distortion specs. I will look at various charted lab measurements if availble from a reliable source in order to gain more insight when possible.

That's mostly it that I can think of off hand.

I would say that a basic understanding how to interpret and apply audio metrics is a very advantageous skill for any true audiophile who wants a systematic means of improving their sound reliably to develop. It's something I am still working on and I try not to lose any sleep over these things though I probably have in practice from time to time.
To me the best speakers give me an emotional connection to the music. The final of the Sibelius 2nd will give me goose bumps on certain speakers. I know of no measurement that will tell me which speaker will do that for me. Remember early solid state and early digital cd players measured great but sounded terrible. The statement above that measurements alone will tell you if you will like a certain speaker is just nonsense
03-19-12: Ahendler
Actually I have never found measurements to help me in any audio decision to by a certain product. Measurements are helpful in the design process but do not tell you anything how a component will sound.
They won't tell you that the component will sound good, but they will often enable you to predict that the component will sound bad in a particular application, as a result of the kinds of mismatches that have been cited.

Basically, they allow you to narrow the field of potential candidates for purchase, to minimize trial and error, and to reduce the likelihood of expensive mistakes.

-- Al
"The final of the Sibelius 2nd will give me goose bumps on certain speakers. I know of no measurement that will tell me which speaker will do that for me."

No, but to Almargs point, when shopping for gear with the best chance of doing it, impedance specs allow matching from output to input that if accurate and applied properly as a factor in the decision making process is more likely to result in better dynamics and lower distortion, two attributes that are generally desirable from a goosebumps perspective regardless of what speaker is used.
In reply to Charles1dad, good sound is a comparative thing. If you don't know bad sound it might be hard to recognize good and nearly impossible to fully understand great sound.

Your attribution of beauty and enjoyment to components is exactly the type of marketing conditioning I was talking about. Over time each of us has developed a belief system about sound and music that largely predetermine our opinions on certain audio subjects. This is not necessarily a bad thing, but once you take it into account questions of measurements vs. subjectivity become largely irrelevant. We can trade opinions, but our belief systems, which I believe are marketing driven, prevent us from being educated, at least to some large extent.
I honestly don`t get your point. When I attend concerts it`s for the pure joy and involvement that music provides. With my home audio I attempt to come as close as I reasonably can and with my current system I`m extremely pleased with it. It`s that simple and straight forward for me.We just have different approaches and apparently different objectives.What ever works for you is the way to proceed.

Marketing has nothing to do with my choices, only the end result which is the sound and my response to it(it moves me or it does`nt).
I appreciate your different perspective.
No, marketing is not driving what I think sounds good. I may be enticed to purchase a product, but after it is in my listening room the marketing is all over and forgotten. If it sounds good to me, I keep it. If not, I sell it or send it back.

Markering influences the buy decision, but never the keep and enjoy decision. Marketing's influence is very short lived in the end.
03-19-12: Ahendler
Actually I have never found measurements to help me in any audio decision to by a certain product. Measurements are helpful in the design process but do not tell you anything how a component will sound.
Let me supplement my previous response with some specific examples, that illustrate what I, Mapman, Roscoe and others have been alluding to:

1)Someone has read good things about the sound quality that SET amplifiers can provide. He or she is considering purchasing a SET amp and using it in conjunction with a speaker having a specified impedance of 8 ohms. JA's measurements reveal that the impedance has wide swings over the frequency range, including dips to low values at highly capacitive phase angles. Purchasing the SET amp would be an expensive and/or time consuming mistake.

2)Someone is considering adding a powered sub having only line-level inputs to his or her system, and driving it from a second set of output jacks that are provided on the preamp. As is commonly the case the two sets of output jacks are not separately buffered. The power amp is solid state and has an input impedance of 20K, the sub has an input impedance of 10K, and the preamp is tube-based and has a specified output impedance of 400 ohms. JA's measurements reveal that the output impedance rises to 3K at 20 Hz, which is not uncommon. Purchasing the sub would be an expensive and/or time consuming mistake.

3)Someone is considering purchasing a tube amp to use in conjunction with a speaker having a specified impedance of 6 ohms. JA's measurements reveal that the impedance is close to 4 ohms at low frequencies, and close to 8 ohms at high frequencies. That may work OK in some cases, but it definitely suggests the possibility that the speaker was designed with the expectation that it would be used with a solid state amp. The result stands a good chance of being an expensive and/or time consuming mistake.

Countless other comparable examples could be cited.

Disclaimer: I have no affiliation with JA, Stereophile, or anyone else associated with the audio industry.

-- Al
I agree with your clear reasoned examples, they will certainly have value in determining likely(or unlikely) compatability. They won`t help in determining the sound quality of various component matching.
I like to refer to a statement from Kevin Voekes (sp?) of Revel speakers fame. He states that measurements are indeed important in designing equipment but the ultimate test is double blind listening sessions. His summation? A speaker that measures bad never sounds good, but conversely a speaker that measures good does not ALWAYS sound good.(but stands a better chance of sounding good) I think this may apply more heavily to transducers than electronics, but I feel most of the improvements over the past 20 or so years are due to either improved materials and/or increased ability to take the measurements that DO matter. Without this, we are only left with the Art of design, which is falling off in favor of the Science of design as capabilities improve to measure the aspects of performance that truly do make a difference.
My brother, who's a professional audio engineer, recently had to develop his own test gear, sensitive to smaller than picoseconds.
This is a very good point that you make:

"A speaker that measures bad never sounds good, but conversely a speaker that measures good does not ALWAYS sound good.(but stands a better chance of sounding good)..."

Knowing a speaker's measurements and knowing how to interpret them may allow us to avoid making a costly buying error.
"I agree with your clear reasoned examples, they will certainly have value in determining likely(or unlikely) compatability. They won`t help in determining the sound quality of various component matching."

Disagree somewhat.

Accurate impedance specs or even estimates are more likely to help assure quality of certain attributes (like dynamics and distortion) of the resulting sound than anything else.

However true that there is no guarantee that specs are accurate and the complete nature of the resulting sound still cannot be known until heard regardless.

Perhaps it would help to look at certain specifications and their application (like impedance matching)as a form of quality control one can practice before buying, though as has been stated repeatedly, the complete final results cannot be known until heard.
"the complete nature of the resulting sound still cannot be known until heard regardindless"
Exactly! when all is said and done you still 'must' rely on your ears to judge the sound quality.I don`t believe we disagree on this conclusion.
And around and around we go. In all cases where JA didn't like much in measuring and Fremer liked the component I have always agreed with Fremer. A classic example was the WAVAC SH-833 monoblock amps. Fremer loved it as did I on risking hearing something I could not afford as CES. Atkinson panned it and it does look pretty awful. My conclusion is that we can measure what is not important to what we hear.
You state: "We can trade opinions, but our belief systems, which I believe are marketing driven, prevent us from being educated, at least to some large extent."

Some of how we hear may sometimes be marketing driven -- for some people. But our brains are wired for music, the same as they are wired for language and the same as they are wired for enjoying beautiful things. Marketing is only a small part of how we perceive music and the extent to which we enjoy music. IMO. This is a highly complex matter where many individual factors contribute, as well the collective unconscious playing an important role. IMO.
When I say marketing one of the concepts I'm referring to is that only through the meticulous selection of and then synergistic mating of equipment can audio satisfaction be attained. I suspect some people will find the last sentence nonsensical. And it could be, but have you ever had a moment of musical bliss when listening via clearly non-audiophile equipment? It happens to me all the time and I think it's because music and the emotions I bring to listening to music trumps the quality of equipment I'm using. Has all the knowledge each of us learned in our individual audiophile journeys actual conditioned us to enjoy music less if not reproduced via high end equipment? If so are you really better off than when you started?
I think you may have hit the nail on the head here: "My conclusion is that we can measure what is not important to what we hear."

I wonder if John Atkinson sits down to listen to a component after it shows bad measurements to see if he can hear anything wrong? If, for instance, he has bad measurements for a given component and Fremer or another reviewer says it's a great component, does John Atkinson sit down and try to reconcile the measurements with what his ears say -- if his ears tell him something different from what his measurements show? Or is this just an intellectual exercise for him? I wonder.
You say "music and the emotions I bring to listening to music trumps the quality of equipment I'm using." This is an important matter that I started a thread on a while back called "What is Musicality?"

Nevertheless, getting closer to the essence of the music often helps get closer to the emotions, IMO. And for many of us getting closer to the essence of the music involves equipment mating to optimize sound quality. I used to get tremendous satisfaction from my transistor radio. But I doubt that I would get that same satisfaction from it today having been spoiled by the high end sounds of my system. There are degrees of satisfaction. The song is the same from a transistor radio and a high end system. But the extent to which you can appreciate and enjoy it can increase greatly if you hear the music in a refined system. IMO.
I'll admit the obvious: we can't know all that we don't know. In as much as we might not be able to know in entirety what a system will sound like from specs alone, we can still get an understanding of many of the sound attributes a system will have from specs. Not only can specs be a valuable tool in shortlisting components for consideration, we can also use them as a baseline to share an understanding of how a system comes to sound as it does. Despite what so much of the reviewing community does to create a mysticism of how audio gear comes to sound a certain way, let's not forget that audio gear is still an exercise in engineering. The experience derived from audio gear might be predominately subjective, but getting to that point might be predominantly objective. I doubt we could have a satisfaction with either without the other. IMO, being able to correlate one with the other is an ongoing quest that needs to vigorously pursued.
I think one of the ineffable aspects of this is system synergy. You just never know what a cable or component or tweak change or addition will do to your system until you plug it in and turn the system on. No amount of understanding specs will give you this information. IMO.
I feel the same as sabai, you really made a point. If a component sounds good to M, Fremer and he raves about it then the 'bad' measurements by JA tell us exactly what in regard to the subsequent listening experience?
Tbg, I respectfully disagree. Things like frequency response and dBW just a couple of examples of measurements that can be important in regards to what we might expect to hear.
Unsound, you are right, speaker frequency response and sensitivity matter. All the cases I was alluding to were electronic components.
Tbg, how about an amplifier's dBW into impedance? Or the predictable roll off at frequency extremes when there's issues of impedance mismatch or capacitance issues? Antenna and tuner specs can often tell some how well they can receive certain stations at a given locations. Those are just a few examples.
Sabai, what do you mean by the phrase "the essence of music" and is it related to high end audio?
You say "Things like frequency response and dBW just a couple of examples of measurements that can be important ...". I agree. There are specific measurements that can give us useful information and can help eliminate contenders when there are clear compatibility issues.

But my point is that once this process has taken place and there are a number of possible contenders left in the ring, all with compatible measurements, you can never tell which one will be the "best" for your system until you let your ears take the reigns. IMO. Measurements can never tell us everything we need to know. Once obvious mismatches are eliminated, measurements may tell us nothing further about the sound. The next step is system synergy which is totally unpredictable. IMO. This is where the ears are the final arbiter.

You hit the nail on the head. What do John Atkinson's measurements really tell us about the listening experience if the reviewer is in complete contradiction with his measurements and accompanying observations? Not much. IMO.

When I talk about the "essence of music" I mean the listening experience itself -- outside of intellectual considerations that consist of various observations that we make about the listening experience, sometimes kept to ourselves, sometimes shared with others.
Unsound, I have certainly heard the impact of some of these, but I don't know of much concern with any of these save what is audible. A tuner or an amp that fails to deal with such issues probably will not sell given how it performs or sounds.
Tbg, What that tells you is that M Fremer is either not sensitive to the results of the 'bad measurement', that his equipment isn't sensitive enough to reveal the result of the bad measurement, or as often, that the 'bad measurement' is insignificant in real world conditions.

Unsound's comments about FM tuner spec's are correct, but as always with measurements those specs alone tell you nothing about how the unit will sound beyond its ability to pick up a clean signal. Yet a lot of folks can't tell the difference between the various tuners actual sound and buy them based on these spec's. It's curious that in a famous FM tuner site which has published a lot of reviews and has ranked tuners this is more true than anywhere. Very few of the reviewers are actually audio freaks, they a radio freaks, yet folks talk about the review rankings as if it were gospel.

FWIW to make measurements meaningful you must know in real life what they mean and how you can use the information productively. Just seeing and comparing spec's is a meaningless activity - often the most meaningful spec's, especially from manufacturers, is missing. Impedance curves for speakers and amps for example, which are very important, are hardly ever seen except in reviews by folks like Atkinson. Although I do recall one speaker manufacturer who did, but if you saw the curve you would know why - dammed near flat gentle curve between 5 and 8 ohms with nominal 6ohm rating. Just as with spec's, reviews are for the most part useless unless you have a good handle on the reviewers preferences and competence.

Bottom line, knowledge is great but hard to come by and even when possessed it isn't worth a crap if you don't know how to use it. :-) Sooner or later, if you last long enough, you will identify the spec's that will be meaningful to you and those you can safely ignore (most of the time).
Newbee, I couldn't agree more.
You make a heck of a lot of good points. Don't you think it would be interesting to hear the cross-talk between Fremer and other reviewers and John Atkinson after both have finished their written observations if they sat down together and listened to the equipment being reviewed? It seems to me to be a very logical next step for Stereophile to take with their reviews. A no-brainer, really. I wonder why this is not being done by them?
Newbee and Unsound, I certainly would take no exception to your centering on specs that you find closely associated to your tastes, but I will continue to trust my ears. I have never found any measures other than whether the unit is on or not that are associated with quality music reproduction. Michael Fremer and I certainly are in total agreement about the WAVAC SH-833 monoblock amps. He is the only reviewer in Stereophile or TAS, that I would trust in recommending a component.

At one RMAF, John Atkinson presented a seminar where he had a Boulder amp and an unnamed amp hooked to a THD meter. Under various loads the unnamed amp was horrible and the Boulder exemplary. He went on and on. He had no capability to listen to the two amps. Having heard the Boulder and not liking it, I asked whether he thought the designer of the unnamed amp thought THD was a major concern in design. He was flabbergasted as were most in the audience. I left.

The problem with that is when its all said and done you are still just getting opinions which will lead you to believe that one of three things have occurred; 1) the measurement is inaudible to the most sophisticated ear using the most sophisticated equipment which as to that particular measurement it is meaningless except there it is on the test equipment; 2) that these same folks with the same equipment can hear the the effect of the measurement but thinks it doesn't materially affect the sound they deem important; or 3) that their listening skills or equipment are not up to standard for evaluating equipment, or that they just can't hear it because of their actual hearing limitations. What would a magazine have to gain by pursuing the testing/review you suggest. The finding has to affect the magazine and/or its reviewers negatively.

Interestingly I can think of one internet mag that uses two reviewers on many/all of its review who review the product separately. They publish each review but make no attempt to reconcile any differences which there often are.

Hobby magazines in general rarely publish negative reviews of anything, its just bad for business. As close as they will come is when someone like Adkinson measures spec's and points out deviations from manufacturers spec's or things HE thinks are meaningful for users and lets you draw your own conclusions whether you feel they are relevant, or on a rare occasion a reviewer will parse words in a way that MIGHT alert a potential user that it ain't up to snuff. But the reader has to put on his thinking cap to sort it all out. That is why personal knowledge is so important.
For me, the ears must be the first guide, and are much more important than any spec or measurement. One must listen to many different types of equipment/systems and decide what one's priorities are sonically. Only then can one begin to use measurements and specs to help in a purchasing decision, by learning how and why the equipment types you like sound like they do. At least, this was my approach, since I was/am not very mechanically inclined at all, but I am blessed with very good ears, and as a professional musician I live the "absolute sound" literally almost every day. Very, very often in the world of audio reproduction, the equipment that measures "best" does not actually sound the best. So the actual measurements may not be very important, depending on one's priorities. What is more important, after one has determined one's priorities, is figuring out which measurements/specs are more important to you, and why, based on what you hear/want to hear. Hope this makes sense, I'm tired and probably shouldn't be posting right now. :)
You're absolutely right. You will rarely read a negative review. And if anything negative is said it is usually couched in terms that make it sound not so bad at all, really. Or a matter of personal preference. You really do need to read reviews with your thinking cap on.

Regarding Stereophile reviews, my point is not that we have 2 reviewers commenting on the same aspects of the same products on Stereophile. If this were the case, two differing opinions could easily stand side by side without the need for further explanations.

What we have at Stereophile is 2 reviewers commenting on different aspects of the same products. When you have the latter happening and John Atkinson says "I don't know how he (the reviewer) could have liked the product given my measurements", he is implying that his measurements supersede the ears of his esteemed colleague.

This is a whole other ball game. Under these circumstances, I think it behooves John Atkinson to sit down and have a listen to the component under review with the reviewer whose ears are being called into question. Otherwise readers are left to consider what it all means without any attempt by the magazine to clarify matters.

The choices are:

1. The reviewer must be right since he is the only one of the 2 who actually listened to music on the component.
2. John Atkinson must be right because measurements are more important that the ears of a reviewer.

Leaving readers in limbo to sort out a matter that could have been clarified or reconciled by the 2 people in question getting together (but not doing so) is not the best way to present audio reviews in a widely-read audio magazine for high end consumers. IMO.
LOL, while your choices are correct I do not think they are the only ones available. You might choose to think that not only Atkinson's valuation of the results of the component's deficiencies might be wrong, but so could the reviewers conclusions from his listening sessions.

One of the things all audiophiles experience in evaluating components is recognizing initially everything that is happening at one time. Usually subtle changes brought about by components deficiencies only creeps in with time, sometimes a long time. I'll spare you examples. But in this case I think it sez a lot about JA's integrity (if not his sonic preferences) that after discovering the measurements discrepancy he didn't simply call Fremer and tell him what he measured which would have allowed Fremer to incorporate it in some way in his review. A very pratical solution from a PR point of view - nobody loses and the audience never knows.

I have no idea what your LOL means. So I will return it in good humor. LOL.

I am assuming the following:

1. That John Atkinson's equipment is working when he takes measurements.
2. That Mr. Fremer and other reviewers are reporting accurately what they hear.

If we cannot assume these 2 basic things then there is no point in this whole exercise because everything becomes smoke and mirrors and nothing can be believed. If you start to question "valuations" of observations of what others report you can turn anything in any direction you wish. IMO.

I believe emphasizing JA's integrity is misplaced here. This is simply the way they do things at Stereophile. Integrity is part of their work, not something that would be extraordinary to expect from them. I am not questioning their integrity at all. I am questioning if there is not something missing in their evaluation process. Something very simple. They finish their work. They read each other's reports. They sit down and listen together and Fremer or another reviewer listens for the measurement side of things and Atkinson listens to the music as well as to any measurement factors he may be able to discern. Simple.

In this way, if one side or the other was missing something they can write a codicil to their report. This means instead of Atkinson saying "I don't understand how the reviewer could like that component after what my measurements show" he might well say after listening to some music that he can actually hear and report some good things -- in spite of what his measurements showed and the valuation he gave to those measurements. And the same for Fremer or another reviewer.

This does not mean Atkinson or a reviewer are changing the valuations placed on measurements or audition of components. Those observations stand. What they are doing is giving a second valuation based on listening in a different way. I mean, this is audio we are talking about, is it not? Or are we talking about the preeminence of the oscilloscope over the ears.