How much can be measured -- and how much cannot?


There has been a lot of discussion over the years on Audiogon regarding the measurement of components and other audio products. Some people claim everything is either measurable now or will one day become measurable with more sophisticated measuring equipment. But others say there are things in high end audio that will never me measurable and that measurements are really not that important.

Here is a typical example -- a quote taken from the Stereophile forum regarding their review of the Playback Designs MPS-5:

"JA 2/17/10 Review Measurements of Playback Designs MPS-5
Posted: April 13, 2011 - 8:42am

John Atkinson's 2/17/10 review measurements of the Playback Designs MPS-5 revealed less than stellar technical performance even though Michael Fremer really liked the player. I've included JA's closing measurement remarks below followed by the manufacturer's comments.

To my knowledge there was never any followup in Stereophile regarding the manufacturers reply the MPS-5 could not be adequately measured with traditional measurement techniques.

I believe Stereophile should respond to this reply in the interests of its own measurements credibility.

Len"

How important do you think measurements are? Are the ears really the only true arbiter?
sabai

Showing 19 responses by sabai

This is like Rashomon. It is amazing how many differing points of view there are on this subject. Each one seems to be a valid part of this complex puzzle. IMO. The question remains, which aspect is the predominating factor -- or is this a matter of co-important factors?
Regarding the example in the introduction regarding John Atkinson's measurements of the Playback designs MPS-5, I would have been interested in hearing a discussion between the reviewer and John Atkinson as they listened to the PD together after the measurements were taken and after John Atkinson wrote about his observations.

Another recent example comes to mind -- John Atkinson's recent measurements of the AMR DP-777 opposite the reviewer's observations. John Atkinson expressed disappointment with the measurements of the AMR DP-777 where the reviewer praised the unit very highly.

Is it possible for the twain to meet? Would it not be interesting for John Atkinson to sit down and listen to some music with the reviewer when all is said and done to see if he finds the actually listening experience as irksome as his measurements? I think it would be interesting to hear their cross-talk.
Spinaker01,
This is a very good point that you make:

"A speaker that measures bad never sounds good, but conversely a speaker that measures good does not ALWAYS sound good.(but stands a better chance of sounding good)..."

Knowing a speaker's measurements and knowing how to interpret them may allow us to avoid making a costly buying error.
Onhwy61,
You state: "We can trade opinions, but our belief systems, which I believe are marketing driven, prevent us from being educated, at least to some large extent."

Some of how we hear may sometimes be marketing driven -- for some people. But our brains are wired for music, the same as they are wired for language and the same as they are wired for enjoying beautiful things. Marketing is only a small part of how we perceive music and the extent to which we enjoy music. IMO. This is a highly complex matter where many individual factors contribute, as well the collective unconscious playing an important role. IMO.
More from Kevin Hayes of VAC:

"During this time, [voicing for the 70/70 amplifier] small changes in the physical arrangement of the parts were tested, different types of capacitors and wires were auditioned, and even various chassis materials tried. The differences often eluded the linear test bench instrumentation but were plainly audible to the human ear. This often frustrating work and attention to detail is required to bring a design to its peak of performance, and is the deciding factor between mere hi-fi and music."
Tbg,
I think you may have hit the nail on the head here: "My conclusion is that we can measure what is not important to what we hear."

I wonder if John Atkinson sits down to listen to a component after it shows bad measurements to see if he can hear anything wrong? If, for instance, he has bad measurements for a given component and Fremer or another reviewer says it's a great component, does John Atkinson sit down and try to reconcile the measurements with what his ears say -- if his ears tell him something different from what his measurements show? Or is this just an intellectual exercise for him? I wonder.
Onhwy61,
You say "music and the emotions I bring to listening to music trumps the quality of equipment I'm using." This is an important matter that I started a thread on a while back called "What is Musicality?"

Nevertheless, getting closer to the essence of the music often helps get closer to the emotions, IMO. And for many of us getting closer to the essence of the music involves equipment mating to optimize sound quality. I used to get tremendous satisfaction from my transistor radio. But I doubt that I would get that same satisfaction from it today having been spoiled by the high end sounds of my system. There are degrees of satisfaction. The song is the same from a transistor radio and a high end system. But the extent to which you can appreciate and enjoy it can increase greatly if you hear the music in a refined system. IMO.
I think one of the ineffable aspects of this is system synergy. You just never know what a cable or component or tweak change or addition will do to your system until you plug it in and turn the system on. No amount of understanding specs will give you this information. IMO.
And here is John Dunlavy:
"Oh, no. Listening comes later. Because if you stop to think about it, no loudspeaker can sound more accurate than it measures."
More from JohnDunlavy:
"So we try to get the greatest accuracy we can achieve from measurements. Then we begin doing what some people might call "voicing," because the best set of measurements are still open to interpretation."
Finally, from John Dunlavy:
"So, to those who like to call us "technocrats" or whatever, we would suggest that those who design by "voicing" loudspeakers are working with an enormous number of perturbations."
Unsound,
You say "Things like frequency response and dBW just a couple of examples of measurements that can be important ...". I agree. There are specific measurements that can give us useful information and can help eliminate contenders when there are clear compatibility issues.

But my point is that once this process has taken place and there are a number of possible contenders left in the ring, all with compatible measurements, you can never tell which one will be the "best" for your system until you let your ears take the reigns. IMO. Measurements can never tell us everything we need to know. Once obvious mismatches are eliminated, measurements may tell us nothing further about the sound. The next step is system synergy which is totally unpredictable. IMO. This is where the ears are the final arbiter.

Charles1dad,
You hit the nail on the head. What do John Atkinson's measurements really tell us about the listening experience if the reviewer is in complete contradiction with his measurements and accompanying observations? Not much. IMO.

Onhwy61,
When I talk about the "essence of music" I mean the listening experience itself -- outside of intellectual considerations that consist of various observations that we make about the listening experience, sometimes kept to ourselves, sometimes shared with others.
Newbee,
You make a heck of a lot of good points. Don't you think it would be interesting to hear the cross-talk between Fremer and other reviewers and John Atkinson after both have finished their written observations if they sat down together and listened to the equipment being reviewed? It seems to me to be a very logical next step for Stereophile to take with their reviews. A no-brainer, really. I wonder why this is not being done by them?
Newbee,
You're absolutely right. You will rarely read a negative review. And if anything negative is said it is usually couched in terms that make it sound not so bad at all, really. Or a matter of personal preference. You really do need to read reviews with your thinking cap on.

Regarding Stereophile reviews, my point is not that we have 2 reviewers commenting on the same aspects of the same products on Stereophile. If this were the case, two differing opinions could easily stand side by side without the need for further explanations.

What we have at Stereophile is 2 reviewers commenting on different aspects of the same products. When you have the latter happening and John Atkinson says "I don't know how he (the reviewer) could have liked the product given my measurements", he is implying that his measurements supersede the ears of his esteemed colleague.

This is a whole other ball game. Under these circumstances, I think it behooves John Atkinson to sit down and have a listen to the component under review with the reviewer whose ears are being called into question. Otherwise readers are left to consider what it all means without any attempt by the magazine to clarify matters.

The choices are:

1. The reviewer must be right since he is the only one of the 2 who actually listened to music on the component.
2. John Atkinson must be right because measurements are more important that the ears of a reviewer.

Leaving readers in limbo to sort out a matter that could have been clarified or reconciled by the 2 people in question getting together (but not doing so) is not the best way to present audio reviews in a widely-read audio magazine for high end consumers. IMO.
Newbee,
I have no idea what your LOL means. So I will return it in good humor. LOL.

I am assuming the following:

1. That John Atkinson's equipment is working when he takes measurements.
2. That Mr. Fremer and other reviewers are reporting accurately what they hear.

If we cannot assume these 2 basic things then there is no point in this whole exercise because everything becomes smoke and mirrors and nothing can be believed. If you start to question "valuations" of observations of what others report you can turn anything in any direction you wish. IMO.

I believe emphasizing JA's integrity is misplaced here. This is simply the way they do things at Stereophile. Integrity is part of their work, not something that would be extraordinary to expect from them. I am not questioning their integrity at all. I am questioning if there is not something missing in their evaluation process. Something very simple. They finish their work. They read each other's reports. They sit down and listen together and Fremer or another reviewer listens for the measurement side of things and Atkinson listens to the music as well as to any measurement factors he may be able to discern. Simple.

In this way, if one side or the other was missing something they can write a codicil to their report. This means instead of Atkinson saying "I don't understand how the reviewer could like that component after what my measurements show" he might well say after listening to some music that he can actually hear and report some good things -- in spite of what his measurements showed and the valuation he gave to those measurements. And the same for Fremer or another reviewer.

This does not mean Atkinson or a reviewer are changing the valuations placed on measurements or audition of components. Those observations stand. What they are doing is giving a second valuation based on listening in a different way. I mean, this is audio we are talking about, is it not? Or are we talking about the preeminence of the oscilloscope over the ears.
Newbee,
Thanks for explaining. No problem.

I agree with you about not having enough experience. Years ago I was in the very same position you found yourself in -- not enough experience and lots of information trying to influence my purchases coming from various vested interests. This is a very steep learning curve that really never ends.

This has been a very rewarding if expensive process that is nearing an end because of serious financial constraints. But not only because of the latter. My system has got to such a satisfying level that I am at the point of diminishing returns. It would take a lot of money to get to a much higher level and I don't know if I did get there that all the effort and expense would translate into much more listener enjoyment because of the high level my system has reached.

It's time to sit back and enjoy what I have without chasing the latest and omnipresent breakthrough products with the attendant manufacturer claims. Over the years most of the improvements to my system have been what I would term incremental. Certainly, there have been some increments that have been more impressive than others. Some I would even call game-changers in their class. Nevertheless the effect on the sound has been incremental and improvement has not always come with the addition of an expensive component or cable. Sometimes even a very inexpensive tweak has given a wonderful improvement in the sound.

In spite of the impressive claims of even the most respected manufacturers in favor of a single component or cable being able to transform one's system, there has not been a single purchase that comes to mind that has measured up to the inflated claims of the maker.
Unsound,
The technical points you make are excellent and clearly explained. Regarding "audio rags", I would have thought it would be in their best interest to be as clear as possible although reading some of their reviews leads me to believe they seem to think otherwise. I will not name names but I think you will be able to easily find fine examples of unintelligible use of the English language by perusing some of the major audio review magazines. They often feature the most convoluted and confusing language that I can imagine a writer composing about audio matters. Florid language and audio cliches abound.

For the life of me, I cannot see how contradictory or confusing reports help the interests of audio magazines. They only lead to skepticism on the part of savvy readers, IMO. Each side of the coin in Stereophile reviews is valid. But if the two sides contradict each other and the review leaves it all hanging in mid-air it begs the question: don't these fellows realize their business is unfinished?

An example of this was the Stereophile review of Playback Designs MPS-5. Let me preface my remarks by saying I have no affiliation with Playback, I have never had any contact with them, I don't own their player, I have never heard it and I am completely neutral regarding it. I am not in the audio business and I have no connection with any audio company.

John Atkinson states at the end of his measurements: "So while I was impressed by the player's standard of construction, I can't say the same about its technical performance. The relatively high level of background noise limits the MPS-5's resolution with SACD and external 24-bit data to not much better than 16-bit CD. I am puzzled, therefore, why Michael Fremer liked the sound of this player so much."

Could John Atkinson not have picked up the phone and given Michael Fremer a call to arrange a chat and an audition together of the component in question? I mean, this seems to me to be a no-brainer. They work for the same publication. Are they not on speaking terms? You have one voice pointing north and the other voice pointing south and a little voice is heard in the background saying to their readers "up to you". How have they served their readers other than going to the trouble to confuse them? Frankly, I find this a very curious way for John Atkinson to end a review of one of the major contenders in the high end CD player market.
I find this quote from Kevin Hayes of VAC very interesting:

"We require that all VAC components sound superb and measure at least reasonably well. Careful attention is paid to sound engineering principles, but we recognize that theory is just that."