Amir and Blind Testing

@chayro 

Great topic. You have accurately described the problems which arise when trying to assess a new component. IMO the next step is to find ways to overcome the biases and desires of our mindset. Lets face it, when we spend $1, 2, 5, 8K on an upgrade we want a return on our investment. And if one isn't there, we can easily fool ourselves into believing it IS there. I've done it. And I suspect many others have done so as well.

But you and others have given the answer to the problem. @spenav  has described my (and others) method saying,

It takes me months to assess a new piece of hardware and I usually won’t know until I remove it and see if I missed it.

This seems to be the method of you and most every poster so far. I  don't take months to assess an item as others have said, but weeks for me. And like spenav,

I usually won’t know until I remove it and see if I missed it.

That, for me, is the key. I believe it is easier to see or hear what we've missed than to hear what was gained, though that is in a mental database for comparison

Following this logic, which I agree with, wouldn’t blind testing, or any A/B testing be compromised because our brains are always in analytical mode and therefore feeding us inaccurate data?

No, unless it's not a proper conducted test.

Seems to me you need to relax for a few hours at least and listen to a variety of music before your brain can accurately assess whether something is an actual improvement.

Which is why in a properly controlled ABX test you can take as long as you like. 

Following this logic, which I agree with, wouldn’t blind testing, or any A/B testing be compromised because our brains are always in analytical mode and therefore feeding us inaccurate data?

The data is the signal. The brain is what does the analysing of this signal, not the feeding.

Some form of blindfolding is necessary by definition, and a near instantaneous switching mechanism needs to be used. In a perfect test environment you wouldn’t be aware of when it has been switched back and forth.

Together with dB level matching - that might sometimes be the tricky bit (that’s an understatement), and the nuances that have been noted may well be because this step isn’t followed correctly.

That’s a good start.

Part of the problem with the blind-test brigade is that I have been in a position to be vulnerable to expectation/confirmation bias many times ((over ~40 years in the game), yet there have been more than a few that failed to produce the "expected" result.

I don’t doubt that such biases are frequently responsible for the positive reactions of audiophiles who are assessing newly acquired gear, and I am certainly not immune. But if it were so simple, and the biases so predictable and strong, then why have I, and countless others, no doubt, been disappointed with components when the opposite would be expected?

On a tangential note, regarding blind testing, I do not believe that listening to short passages is necessarily fair, or best-practice. I have been able to distinguish between components when allowed to listen to full songs with which I am very familiar, but not when confined to snippets. And I know that others have had similar experiences.

But if it were so simple, and the bias so strong, then why have I, and countless others, no doubt, been disappointed with components when the opposite would be expected?

If what has been promoted was marketing and snake oil and the earth didn’t move for you? It is common, yes, if people are honest.

Alternatively, you may have preferences, and the reviewer/influencer or whoever it was that lead you to have this preconceived expectation, had other preferences. And again, that is common.

I’d like to think these observations are reasonably self-evident, but perhaps not.