Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera

Another reason why reviews are almost always favorable is because the magazine and or reviewer doesn’t really want to kill a small company, especially in view of the fact that sometimes things happen, you know, such as damage during shipment to the reviewer, operator error, failure to follow instructions, that sort of thing. Also, sometimes there are trade-offs for the device under review which would be spelled out in the review. For example, cost, size, complexity, peculiarities, etc. Finally, reviewers and magazines are not all the same. Reviewers and magazines can certainly have unique or different approaches to conducting reviews. It certainly wouldn’t be unusual to see products reviewed that are trending or that the reviewer and or magazine would suspect of having interest for their readers.

I am not trying to insight a so-called flame war here, but I have to agree with willemj.

If audiophile magazines are "hobbyist" journals, then I would stop the product reviewers from including what appear to be plots of measurements of impedance versus phase, spectral decay, and amplitude versus frequency. I enjoy looking at these graphs but in almost all cases, the reviewers will only hint at issues (e.g., speaker impedance spikes or drops), but the review will end with statements such as "best speakers I have heard, so I bought the review pair" and so forth. I can only believe that there is some economic incentive that is not disclosed.

Comparisons to a hobby and a profession:

1) I used to be involved in Porsche club racing, so I would read porschephile and generic car magazines. This would certainly be considered a wealthy person's hobby, but the car magazines would provide head-to-head comparisons of performance between brands and would make disparaging statements about cars that were poor performers. I have never read head-to-head comparisons in a given category of hardware in any high-end audiophile magazine (e.g., different speaker brands in the same room at the same time, driven by the same audio hardware);

2) In biomedical research, there are many problems and challenges, but to suggest that our collective intent is to do anything but try to arrive at so-called "ground truth" is not accurate. That does not mean there is no publication bias (see [1]), but at least there is recognition that this issue exists, and many attempts to deal with the problem.

- Gerry

[1]  A. César-Razquin et al. A call for systematic research on solute carriers. Cell. 162 (3), (2015), 478-487. https://www.snijderlab.org/pdf/26232220.pdf 
 

You missed the elephant in the room. Ordinary bits of wire. Not long ago, bits of wire were correctly viewed as bits of wire. Today gargantuan sums are paid for ordinary wires in the misguided belief that ordinary bits of wire do more than anything else in the audiophile’s setup. The fact is they don’t. What they do is lead to massive profits and mark ups for something cheap with no substantive benefit to the user. Audio reviewers and press have jumped on the band wagon (incentivized by advertising) and stores have jumped on the opportunity to sell additional cable trinkets and jewelry at extremely high mark ups to buyers.

The advertising lie is that “everything affects the sound” and that “exceptional ears and exceptional systems will require these trinkets to sound their best”. A story just like the emperor’s new clothes which audiofools eat up readily. Smugly the audiofools look down on anyone with less than $1000 bits of wire and snear at those that don’t hear a difference as being deaf or not having a high end system. The whole domain is like high end skin care - packed with excruciatingly expensive products that do nothing more than appease the ego of the vain. The bits of wire are made of fairy dust meme also undermines real audio innovation.

Are you at Johns Hopkins? My daughter is studying there currently.

Reviewers, and audio magazines in general, are built around the idea of only reviewing gear they know to sound good from their own listening at shows and at various other locations and locales.

It would be economic suicide for any published magazine to review stinkers, and trash them.

It’s a very simple bit of economics as tied to the human condition.

Stereophile has said this flat out, and said it fairly often.

No negative reviews as negative reviews are never allowed to have a chance to form in the first place. Everything reviewed is filtered on multiple levels, before the given review is published.

The best you’ll get (toward the idea of a negative review) is to have the given item ’damned with faint praise’, if the selected item they thought would sound great and review well, does not quite meet that high quality minimum.

It does not mean that smaller publications or non paying (no advertisers and no paying to read the reviews) scenarios where equipment trashing is done...are somehow more honest, it’s just that there is ZERO, I repeat ZERO economic position available for anyone trying to run a company (of a publishing nature) that produces only (or high levels of) negative reviews of commercial products that are in this area of the economy.

Importantly, Stereophile additionally states -far more often than they speak on only publishing good reviews-..that if they don’t review something...that does not necessarily mean it is bad or not good. There’s lots of gear out there and they can only review a very very small percentage of it. They state this openly, regularly in the magazine, about 2-3 times per year, one in every instance of ’recommended components’, and in other places and ways.

There is a film called the ’American ruling class’, where the ’fictional’ main character in the ’pseudo-documentary’ interviews the actual and real ownership and publisher of the NYT (at the time). (everyone interviewed in the film is for real) He asks a real question and gets a real answer. The editor says they only publish editorial that is favorable and in line with their advertisers and that’s the way it is done. Period. Any other way and the newspaper would simply not exist. The end.

Now, let’s talk about the US medical research INDUSTRY, that + trillion dollar medical/pharmaceutical juggernaut’s ’factualization’ arm.

Additionally, if the audio world was about 2-3-4 times bigger than it is, then publications could get into the idea of a more ’biased toward negative’ review standard. Then a world with audio magazines with some negative reviews could stand and survive.

Also audio is specifically NOT a ’first past the physical post’ type endeavor. Where everything is in full black and white factual norms. We still don’t have a full and correct handle on how people hear and understand audio quality, nor how distortions and micro-distortions interfere and integrate with our not yet understood hearing functions.

We can send out the exact same audio cable to four different people and get back four very different assessments of how the cable sounds.

Eg, the post right above this one illustrates the complexity of the audio world and marketplace, quite well. It’s a mess that has no zeroing point or norms that can be discerned beyond a basic direction in desires. Sometimes.

We’re dealing with mental wiring and associated system which are all INDIVIDUAL in end points. There is NO capacity to perfect anything on paper or in the so called real world of audio.

If you want to normalize all individualism (hearing and associated wiring, intelligence, etc) down into a black and white repeating standard and give away all the things that make you--you ..and kill all of humanity down to a repeating widget that comes off an assembly line of repeating identical items..and do the same for everything you know in the human world ---basically kill off humanity and the world into a dead, non-living nothingness..THEN.. you can have your black and white perfection in high end audio.