Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

@jayctoy

If I remember Robert Harley was warned by either Mike Moffat or Jason from Schiit audio that yagdrassil dac don’t have good measurements ? It turned out the yagdrassil got a stellar review from Robert Harley.

An outcome which he hands to any product that costs money. Why would that in any form or fashion mean anything in the context of measurements being reliable or not? I only recommend 1/3 of all products I review. What is that percentage for RH? 99%? 100%

BTW, Schiit publishes measurements of everything they build now. As noted, this came from work that we did at ASR, highlighting the value of such things. So I would not pick this as an example of anything.

@amir_asr ,

I think the test you gave yourself is too easy 😀  Your point is well taken in regards to trained listeners and MP3. I think a better test is to serve up 10 different tracks which may be MP3 or may be wave, and then test how well listeners do at accurately assessing if the track is compressed or not. I wonder if even the trained listeners will be challenged in that case without a reference.

@nyev 

All this to say, that sometimes, maybe a lesser performing component may sound better given lesser surrounding components in one’s system. Which again, judging a piece by measurements would never provide any useful context in these situations.

Thank you for another measured response.  On this point, we never foreclose such possibility.  All we ask that evidence of sonic superiority come in the form of only auditory senses.  And have statistical rigor to be reliable.  Should this evidence come about, we certainly will throw out the current measurements and investigate further what is going on.

What we face unfortunately is so and so says it sounds good.  Well, I can get you people who say the opposite.  And neither can be shown to be reliable.  Someone mentioned Robert Harley.  He raved about my Mark Levinson amplifiers.  Stereophile editors hated it.  To their credit, they showed some measurement issues although that was not normative with respect to comments from subjective reviewer (which no doubt was biased against class D amps).

So what we ask is simple: please conduct a test where all factors have been removed other than sonic fidelity of the two devices.  Match levels.  Play the same content on the same system.  And repat at least 10 times and see if you can get 8 out of 10 right.  It is not much to ask for as I perform many blind tests to add back up to my arguments.  If that is too much, and it can be, then we better run with a) measurements of the device and b) science and engineering behind how the device works and why something would or would not be audible.

I should note that when it comes to transducers, are measurements are less predictive and I have in a number of occasions liked something that didn't measure well.  Wilson Tunetot speaker review was one such speaker.  It was an expensive bookshelf speaker ($12K), so would have been easy to go with the flow of bad measurements and expensive so let's damn it.  But I could not in my listening tests and reported that.  Got heat for it from my own crowd but so be it.

@thespeakerdude 

I think the test you gave yourself is too easy 😀  Your point is well taken in regards to trained listeners and MP3. I think a better test is to serve up 10 different tracks which may be MP3 or may be wave, and then test how well listeners do at accurately assessing if the track is compressed or not. I wonder if even the trained listeners will be challenged in that case without a reference.

I didn't give that test to myself.  I was challenged on a major forum by an objectivist to be able to tell MP3 from original with him claiming that no one could.  At the same time, there had been a challenged on that forum to tell 16 bit content from 24 bit.  Content for that was produced by AIX records which is well known for quality of its productions.  So to remove any appearance of bias in selection of material, I grabbed the clips from that test and compressed them to MP3.  And post those results.  The clip was not at all "a codec killer" where such differences are easier to hear.

On the type of test you mention, I am not a fan of them for the reason you mention.  It is harder to identify the original vs compressed that way because you have to now know what the algorithm does to create or hide sounds.  In other words, is an artifact part of the original content or was it removed.  

Our goal with listening tests should always be to try and find differences, not make it hard for people to find what is there.  Because once we know an artifact exists, we can fix it.  Making the test harder to pass goes counter to that.

That said, I and many others were challenged to such a test on the same major site above.  We were given a handful of clips and asked to find which is which.  Results were privately shared with the test conductor.  When I shared my outcome, he told me I did not do all that well!  I was surprised as I was sure two of the clips were identical and thought that was put in there as a control.

Fast forward to when the results are published and wouldn't you know it, I was "wrong."  We had a regular member with huge reputation for mixing soundtracks for major films and he got it "right."  Puzzled, I performed a binary comparison and showed that the two files were identical!  Test conductor was shocked.  He went and checked and found out that he had uploaded the same file twice!  He declared the test faulty and that was that.

Despite that, as you saw, I will repeat again, I don't want to make blind tests too hard on purpose.  We need to be interested as much in positive outcomes as negative.

@amir_asr

Problem is, when most of you are put to blind tests, you flunk being able to tell the source from the compressed one. And no, resolving system has nothing to do with it. The fact that you say that tells me you don’t know what it takes to hear such differences. As a trained listener in this domain, I can tell differences with just about any headphone on any system.

Yes, this is, I find, one of the most common myths among audiophiles. Whenever someone raises skeptical doubts about claims made about, say the audible character of cables, the response if the skeptic doesn’t agree is the "ears or gear" - either you don’t have the hearing acuity the Golden Eared Audiophile does, or your system just isn’t "resolving enough" like you need a "super resolving system" to hear these differences.

There are various problems with this idea:

1. Subtle sonic differences, if real, can be heard across a large range of transducers. Sure we can get to something like the worst laptop audio or whatever, but it really doesn’t take THAT much to produce speakers on which you can hear very subtle differences. I’ve worked in tons of different studio conditions, different monitors, headphones of varying quality and ALL have allowed me to hear and make the subtle changes I need to for my job. Give me an old pair of radio shack minimus 7 speakers and if I do a subtle EQ tweak to bring out the upper mids, you will hear it!

2. The type of sonic differences often ascribed to (for instance) cables is often fairly dramatic - the "highs opening up" the bass becoming more punchy or extended, more forward in tonal balance, more laid back etc. These are all qualities, if that obvious, should be audible on most speakers. It’s why different mastering is obvious on most speakers, from cheap to expensive.

3. We have audiophiles reporting these "obvious sonic differences" across a wide range of systems and speakers. It’s not just the well-heeled audiophiles with the Super Resolving Systems. Just go to the typical amazon page for some set of audiophile cables (even not expensive ones) and you’ll see audiophiles with very modest systems reporting "obvious differences" with the cables in question.

From this you have either two implications for those making the Resolving System demand:

A. It’s a red herring to demand that someone must own a Very Resolving System in order to evaluate whether a cable is making a difference.

Or:

B. If these differences aren’t obvious on less resolving systems, plenty of those audiophiles with modestly ’resolving’ systems are imagining they are hearing differences between cables. Which would only emphasize the problem variable of listener bias in the first place.