I never understand the resistance to true A/B blind testing. Sure, it is not the be all and end all, but it is an important data point in comparing competing equipment. I remember doing an imperfect AB test (not blind) on a digital and optical cable and it was really only because we could switch between them that we were able to detect subtle but existing differences. OP makes an interesting point as well about people's varying ability to detect differences. My wife has the nose of a GMO bloodhound; she can smell whether our cat took a dump in the Litter Robot upstairs and must run up and clean it because it smells so badly. On the other hand, I can't smell it even standing next to the litter robot. Does that mean that out doesn't smell, or can't be detected. No. It just means that I can't detect the minute differences in the air, but she can. Perhaps the same sort of analogy applies to audio equipment. Some people have very sharp and discriminating hearing, or even the same sharp hearing but respond to one set of frequencies more than another. For example, I don't like sharp treble -- it sounds "bright." Other people might hear the same sound that I dislike as "bright" and believe that it has amazing "realism." So of course, that leaves us pretty much where we started. Measurements are an objective data point, AB testing is a datapoint, but ultimately, bad measurements or not, the question is how something sounds to you. Just because it sounds different or better does not mean that it is the result of a placebo effect or confirmation bias. To my wife, a piece of cat poop stinks up our entire two story house. To me, I can't smell it at all. Doesn't mean that its not there!
Take it on faith: A cease-and-desist letter to those who only believe in measurements
Faith is a firm belief in something for which there is no proof (https://www.merriam-webster.com/dictionary/faith). Faith is often considered to be distinct from and even contrary to science. I argue science is based on faith. Specifically, it is faith in the belief that measurements are always correct, and they alone can reveal the world around us. However, there is no evidence that this approach will always provide a correct and complete depiction of our environment.
I am not anti-science. In fact, I am all about science. I was a science major in college. I taught high school biology and chemistry. I employ science every day in my current career. I also use it to make decisions when it comes to audio, and I can point to a scientific basis behind my equipment decisions, speaker/listener locations and room treatment. I believe John Locke’s scientific method is a wonderful boon to mankind. But although data may rule my life, I know that science has its limitations.
The scientific method is an empirical approach and relies on our eight senses or extensions thereof to measure phenomena, enabling us to better understand and control our environment. People who embrace this approach believe if something cannot be measured, it cannot exist. They have total faith in this approach and deny the credibility of others whose senses do not or cannot yield something in units. In essence, these disciples take it on faith that measurements are the only true way to make sense of the world. However, we just may not have developed the instrument that enables us to measure the event. Early digital is a good example of our senses superseding the limitations of our understanding of the technology and hence, our measurements. Other examples of this include our past beliefs that we could destroy mass, that the earth is flat, and the universe is not expanding. And cables and amplifiers all sound the same.
Others find their senses can reveal events that are not apparent to some and may not even be measurable. Some people can smell faint odors or feel a slight breeze that others cannot. My wife can find a Petoskey stone on a beach out of thousands of rocks; I cannot see it even when I am standing over it. Different cables, fuses, amplifier topology, or cartridge design may or may not result in the same or even any data points and may or may not sound alike. But just because you cannot hear a difference nor measure a difference does not mean there is no difference. Beauty is in the eye of the beholder, just as good sound may be in their ear.
Some of us have at least as much faith in our ears as we do in our REW software and associated hardware. I start room setup with acoustic theory and then confirm with measurements, but the final placement is always a result of what sounds most pleasing. I would not know how to determine speaker toe-in using a microphone.
While I will always have to trust my senses, I am not handicapped by relying solely on those that are associated with a number.
“…not everything that can be counted counts, and not everything that counts can be counted.” William Bruce Cameron, 1963
- ...
- 195 posts total
Anybody who tune his system room takes hundred of simple informal blind tests(Not double blind test )... No one with a brain can be against double blind test in acoustics... But in acoustics unlike ASR they takes subjective hearing seriously because they study it... In ASR they use the formal complicated test to eliminate subjective hearing as the main player and to promote a short set of established measures as the gold standard for sound quality... It measures well : it must sound good... It is marketing ploy or pure ignorance ... It was well debunked by Milind Kunchur...
|
@tcutter interesting initial post, thanks for starting the thread. @newton_john is this the post you wanted to reply to? ”Bringing this back to audio reproduction, some of the biggest unsettled arguments remain how different cable materials and configurations could change the sound we hear in our rooms, or how two amplifiers or DACs that measure nearly identically could sound different in our listening rooms, or how an amplifier or DAC that measure worse on the bench could actually sound better when experienced in our listening rooms. And don’t even get me started on soundstage… As you noted, the reductionist method works very well when the systems under consideration are uncomplicated and conducive to solving with simple experimental design. When we add complexity, controlling variables becomes much more difficult. In the case of evaluating hifi equipment, those variables include but are not limited to the quality of the supplied power and any power conditioning, the assemblage of equipment in the test, the configuration of the listening room, its materials and the placement of the speakers, and the seating position and listening skills of the person evaluating it. As commonly discussed here and elsewhere, the results of an evaluation of stereo equipment A versus B are only really valid for that particular “system”. To get any statistically valid results requires careful ABX blind testing procedures and multiple listening “subjects” for both statistical power and to cover the potential variance in listening skills and hearing acuity. I have yet to see a really well designed “experiment” to test this because it is time consuming, and requires a large number of subjects tested one at a time in the sweet spot. This is still a reductionist approach, it is just a very complicated, and logistically and cost prohibitive in service of a point that has little transferable utility. Based on all of this, the appeal among potential hifi shoppers to lean on bench test results derived with proven measurement equipment and protocols testing gear in complete isolation and calling that “Audio Science Review” is apparently immense. The alternatives are to trust the claims of manufacturers, the collective “wisdom” and experience on audio forums like this one, and/or trial and error testing at home in your listening room. Given the time and expense of the latter, there is ample motivation to disregard whole segments of products out of hand as voodoo, and take measurements as an insurance policy that an investment is well advised. High end audio is for people with time and money. For many with limited time and a basic understanding of classical physics and electrical engineering and who just want to listen to music, a spec sheet is apparently plenty good.” kn |
- 195 posts total