2. misc-audio
  3. 147670-20253

Blind Testing is Dead - Long live My Wallet testing.

Hi Everyone,

I was seeing some discussions around cables, and reading other discussions about A'gon members asking for opinions on different alternatives for hooking up a DAC, or TV sound, or whatever, and it made me think of this.


I want to tie a few things together:

  • Most technical measurements consumers read were defined by the 1970s. It is fair to describe them as stagnant.
  • The cost to benefit ratio of a lot of products can vary a great deal.
  • I hear things I can't yet measure in cables and crossover components.
  • I like measurements. 
  • Someday measurements commonly discussed among consumers will improve and better tie our values to technology.

A lot has been made about double blind testing, and a lot of readers rely on taste masters (web sites, magazines and social media) and whether in fact these taste masters can hear anything at all. Reminds me a lot of blind testing of wines, or an article I read recently about how much super rare whiskey is fake.


When deciding on a bit of kit, I could not care less about double blind testing. I care about :

  • What audible value can I perceive?
  • Is the price proportional to that value?
  • Is my money better spent on a vacation or liquor?

We should also note that I'm a bit of an iconoclast. Most consumers also care about:

  • Brand recognition
  • Style
  • Perception of modernity (is it cutting edge no one else has)
  • Perception of construction (how much does it weigh, how is it packaged)
  • Ability to create envy.
  • Price ( if it's too inexpensive, it can't be good! )


What is my message then? My message is that this is all cute, like reading about movies or books or music shows, but in the end, it's my wallet, no one else's. John Atkinson is not buying my speakers for me. I am. My hard work creates value which I use some of (sometimes too much) to buy audio related products. The more you detach yourself from brands, costs and worries about measurements the more frugal, and happier  you will be.


Best,

Erik


erik_squires
mikexxyz
66 posts
 
Get your headphones and give it a try.  The test ranges from 6 dB to 0.1 dB.  My old ears can't discern anything below 1.0 dB.

https://www.audiocheck.net/blindtests_level.php?lvl=0.5
prof
3,438 posts
 

I don’t see many, or any, agon members blindly following reviewers.

That said, Art Dudley is probably my favorite audio writer just in terms of talent and style, and Michael Fremer is probably my favorite reviewer (of speakers) because Fremer seems to capture in print the character of a speaker so well. (When I’ve heard speakers Fremer has reviewed, they usually sound just as he’d described).

As for blind testing, as I’ve said many times before, no gastopo squad is breaking in to anyone’s home to force blind testing, and no one should bother with it if it doesn’t interest them. I don’t bother blind testing the majority of what I buy. At the same time, I can’t pretend our hobby is magically exempt from all the same variables in perceptual bias that humans suffer from in every other endeavour. I have no problem whatsoever admitting that my perceptions may be in error, for any number of reasons. But it’s damned fun to trade subjective impressions of gear.


geoffkait's avatar
geoffkait
23,365 posts
 
prof
As for blind testing, as I’ve said many times before, no gastopo squad is breaking in to anyone’s home to force blind testing, and no one should bother with it if it doesn’t interest them. I don’t bother blind testing the majority of what I buy. At the same time, I can’t pretend our hobby is magically exempt from all the same variables in perceptual bias that humans suffer from in every other endeavour. I have no problem whatsoever admitting that my perceptions may be in error, for any number of reasons. But it’s damned fun to trade subjective impressions of gear.

>>>>>Why pick on perceptual variables? There are too many physical and electrical variables and potential errors in the test system or test procedure to be able to control. No one seems to even know what all the variables are. For that reason alone test results should be suspect. Besides not everyone is impressed by the cost or appearance or advance buzz of audio devices or components. Maybe newbies are, maybe not. Who knows? Did I already mention the ulterior motives some people might have in pushing for blind tests?

In addition, there is no comparison between audio and “other endeavors” - audio is infinitely more complex than say, wine tasting or medical trials. Thus, many more things can go wrong in the test. Audio test results have little or no significance.

Note to self: Blind test proponents apparently can’t help pushing blind tests even while pretending to be nonchalant about it. Must be a new strategy. 😄
cd318
2,636 posts
 
Blind testing is not without flaws, but it's easily the best test available to us when comparing products. Easily.

When testing, the weakest link in the entire chain from the recording to our enjoyment of it, the one most subject to variability, is inevitably ourselves.

In an ideal sound test we would not be involved in any of the actual assessment because our impressions are simply too subject to change. Mainly because they all too often depend heavily upon our moods. Heck, on a bad day we might even start believing that the Beatles 2009 remasters were any good. Or one day we prefer Presley's version of Blue Suede Shoes, the next day Perkins etc.

When listening blind some may find that not knowing the make, the price, the technology behind the sound can act as a wonderfully liberating experience. Wonderful because there's only the sound to think about. Ultimately even a blind test will still depend upon the current mood and frame of mind.

Therefore a better way to evaluate tech is by comparing side by side, preferably without sight of it. This at least ensures that our mood preferences will apply equally to the equipment we listen to. [Unless of course we are afflicted with some kind of disorder whereby our moods (preferences/biases) are changing from moment to moment.]

I have rarely had the chance to do an unsighted comparison test but they do tend to be eye opening - sorry. One favourite is comparing different masterings of favourite albums - level matched of course. It's amazing how your mind can desperately scramble around for any data it can find in order to form an impression. Even displayed album artwork can contribute towards the impression formed.

A lesser kind of unsighted test is encountered when you walk into a place where there is music playing but you have no idea what it is being played on. So far I have never walked into a store and found the sound there to be superior to what I enjoy at home. Often it's almost unbearable, but just one of these days, who knows?



erik_squires OP
15,667 posts
 

I don’t see many, or any, agon members blindly following reviewers.


I don’t either, but I think the repeated controversies over this is notable.

As I’ve said among many issues, from amplifier design to cables, it’s a real shame to me that testing is stuck in the 1970s.

Best,
E

More to discover