Why blind listening tests are flawed


This may sound like pure flame war bait - but here it is anyway. Since rebuilding my system from scratch, and auditioning everything from preamps to amps to dacs to interconnects to speaker cable etc, it seems clearer than ever.

I notice that I get easily fooled between bad and great sounding gear during blind auditions. Most would say "That should tell you that the quality of the gear is closer than you thought. Trust it".

But it's the process of blind listening tests that's causing the confusion, not a case of what I prefer to believe or justify to myself. And I think I know why it happens.

Understanding the sound of audio gear is process of accumulated memories. You can listen to say new speakers for weeks and love them until you start hearing something that bothers you until you can't stand them anymore.

Subconsciously you're building a library of impressions that continues to fill in the blanks of the overall sound. When all the holes are filled - you finally have a very clear grasp of the sonic signature. But we know that doesn't happen overnight.

This explains why many times you'll love how something sounds until you don't anymore? Anyone experience that? I have - with all 3 B&W speakers upgrades I've made in my life just to name a few.

Swapping out gear short term for blind listening tests is therefore counter productive for accurately understanding the characteristics of any particular piece or system because it causes discontinuity with impression accumulation and becomes subtractive rather than additive. Confusion becomes the guaranteed outcome instead of clarity. In fact it's a systematic unlearning of the sound characteristics as the impression accumulation is randomized. Wish I could think of a simpler way of saying that..

Ok this is getting even further out there but: Also I believe that when you're listening while looking at equipment there are certain anchors that also accumulate. You may hear a high hat that sounds shimmering and subconsciously that impression is associated with some metallic color or other visual aspect of the equipment you happen to be watching or remember.

By looking at (or even mentally picturing) your equipment over time you have an immediate association with its' sound. Sounds strange, but I've noticed this happening myself - and I have no doubt it speeds up the process of getting a peg on the overall sound character.

Obviously blind tests would void that aspect too resulting in less information rather than more for comparison.

Anyone agree with this, because I don't remember hearing this POV before. But I'm sure many others that have stated this because, of course, it happens to be true. ;
larrybou
'When doing A/B comparisons, its not necessary to do it blinded. What's important is to make the switches rapidly, because audible memory fades quickly.’

That’s really the heart of the matter. Listeners would be surprised at how short our sensory memory actually is... It can be surprising how two things might sound the same, but when put right next to each other, obvious differences emerge. I do this everyday at work, you really do learn to listen better, and trust your hearing more and more...
Sciencecop, what particular audiophile nonsense are you referring to? There's so much to choose from. Are you saying that long tests are valid and short ones are not? I think they accomplish different things.

I would differ with your opinion on personal taste. My dad and I recently did a comparison between a CD and a high rez file of the exact same performance. I thought the high rez file was superior, but my Dad preferred the CD. He has no experience with high rez, and to him it sounded too detailed and less musical.
One standard I have is if I could tell if a change had been made walking in blind. In a short a/b comparison I can tell a difference but I would be guessing if I came in and had to correctly identify the configuration. It would be a guess.
In respect to the OP, my experience parallels Larrybou's, but I find that critical listening for differences in gear (or whole system evaluation) requires at least two efforts on the part of the listener that tend to oppose to each other. On an ongoing, momentary basis, you must be ready and able to take inventory of a rather exhaustive length and breadth of various sonic categories and traits in the attempt to track and analyze them - to remain vigilant and on point enough to follow the sonic aspects you're concerned with. That is to say we must concentrate. And yet not lose sight of the role of those aspects in the overall picture, or the Gestalt. That is to say we must also be relaxed. This is not only a hard state, I feel, to both attain and to maintain, for whatever we might want to consider a useful purpose, but I find it's sometimes further complicated by an additional circumstance.

When I first heard the 25th anniv. edition of McCartney's Band On The Run CD, on it Paul spoke of an instance in which he recalled he had spent so much time and studio effort in an attempt to narrow down his choice between 2 different versions of one song (the title escapes me) to be chosen for the initial single release ahead of the album, that he found he no longer could rely on his own instincts for knowing which version he should choose for that single's release...IOW, he had lost his perspective on how to gauge the public's reaction on hearing it for the first time. After his producer learned of the choice between the two Paul had made, he called him up and told him that he thought Paul had made the wrong one. Not long after, it began to click for Paul that his producer had been right.

That actually happens often with artists and it's referred to simply as "getting too close to your work". It happens to audiophiles, too, I believe. And we're not known as audio"philes" for nothing. Listening for evaluation's sake is likewise a creative process that also requires a certain amount of passion, aesthetically, to see through to completion (however you may objectively define that completion). We must be sufficiently on point and yet also remain relaxed as possible, but there may come a point in it where you can simply get too close to the work and you are forced to back off for a while to keep from just spinning your wheels. Sometimes the best cure for it is a self-imposed absence...whether that's days, weeks, or whatever is always up to the individual. But, over the years I've found that when I've returned from such a prolonged absence as that, it is Far easier for me to get a handle on things than when I last left it - a very good 'reset button'. But, the give and take between concentration and being in a relaxed state does seem to force evaluation to be more of a long-term thing, to me. In all this, the truly relaxed state for us may end up being the most elusive.
Sciencecop, what particular audiophile nonsense are you referring to?

Well, you just pointed to one good example. More “detail” if indeed that is what you think you are hearing, will inevitably have to be “more musical”, don’t you think? You can only detract from the recording that was captured for you to listen to, not add anything to it that is worth listening to (like “musicality”). If you think you can improve the original capture, by hearing less of it, I would say you got an issue, and it’s not necessarily your stereo (-;

Having said all that, you most likely don’t know how these recording you are talking about were taken/transferred. So many so called “high-res” recording are not really high res, and even if they are, they are not always better due to the format they are converted to. All things equal, with the same quality DAC there is no real option for a 44.1 version of a higher sampling rate recording (or analog) to sound better. Not possible.