Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

@nyev

elementary school audio science = ASR
The measurements done for audio electronics in particular do not paint an accurate picture of what we are supposed to hear. Much less rated output power (also false in my experience) Therefore, I have to use the Sabaj A10h as a basic tool to test if amplifier inputs are working (units I’m selling) rather than listening to / enjoying it.

I would always suggest listening first; and measurements later. There are obviously reasons why such a site exists...and I don’t think the founder is out to save audiophiles from themselves and provide 100% honest reviews to save people money; all for nothing....

People make donations on ASR. Long ago, there was proof of collusion between said site admin and now-popular Chi-Fi audio brands (execs and designers) on various forums. Of course, before starting ASR, those tracks were paved over....so anyone who thinks ASR is an audio science charity is fooling themselves.

it is a cult - where noobs and audio novices gather to talk about how their latest $10 headphone dongle or $99 DAC is far superior to "audiophile jewelry" all while never having stepped foot in an audio store to listen...because those who are most influencial on that site tell them that higher THD is the only thing that can impact audio quality....lol

@hilde45

Long ago, there was proof of collusion between said site admin and now-popular Chi-Fi audio brands (execs and designers) on various forums.

I’m glad you think so. And it’s true. If you do enough digging online, you would be very lucky to find even 1 forum post that confirmed this as fact....like I said..it’s been paved over. Think about it - asking for donations + an overabundance of audio gear from particular audio manufacturers. No 3rd party testing or validation of his measurements. But why would he need to?! He’s a genius afterall...send it to audio precision...get them to do even more measurements and we’ll see who’s right.

That forum is where you go for digital conspiracy theories ....

Like my father used to say: "nobody gives you anything for free in this world. There is always a cost."

I actually enjoy posting here on audiogon because it’s a great community. I ask a question, people answer with useful info. I try my best to reciprocate. It’s fun being on here...

@decooney ✅️

@nyev

Sensory experience is all we’ve got.

Go to a costly steakhouse - order a steak. Does it taste just good or much better than a $30 steak? we’re talking 3 - 6 times the price. If it is dry, not properly cooked (medium or medium rare etc. to your liking, you will notice). Even if you were blindfolded and asked to try, you would notice.

Ask a friend to audition 2 different bluetooth headphones with the same song on their smartphone. Then once they’re done, ask them which headphone they liked more and why. You could aruge they were not perfectly volume matched - but evenso, the differences in sound quality will be obvious.

Sensory experience is certainly imperfect; like all human senses. We are only human - not gods. Because we experience music from our audio systems with our senses, that sensory experience is all-important. All of it - sight, hearing, etc.

Blind listening for audio is a flawed practice. I don’t know anyone in pro audio that uses it. For example, at AIR studios in London, a power amplifier for their main control room (I believe) was chosen based on listening sessions. They bought a Class A/B power amp from a UK-based company called ATC.

😂Those who are fond of conducting blind tests for audio believe that a certain number of successful trials is sufficient for proving whether we can hear a difference or not... how did we arrive at this number of successful trials?!

😅 8/10 or even 10/10 successful trials could be riddled with guesses and inaccurate auditory memory recall. The test subject may not admit they were unsure, because they wanted to be correct and prove their ability to be golden-eared to their peers.

Wearing a blindfold also creates problems that make an objective listening test more difficult. Blindfolds may hamper with the frequency response characteristics of speakers and headphones.

Because blindfolds are made of soft fabric with padding or a sheet of fabric, placing them over the eyes creates a sound-absorbing pocket, whereby the sound waves from speakers would not disperse as evenly with it on. And since our eardrums are levelled to our eyes, they can be a direct source of sound wave absorption/reflection from the blindfold strap or covering.

A blindfold may interfere with achieving a proper seal with over-the-ear headphones and on-ear headphones. If conducting such a test, the test subject may already be attuned to how said headphone fits and feels on or over their ears... thereby giving them a sense of pseudo-preferential vision, much like figuratively having another set of eyes to visualize the set of headphones being auditioned in their mind during the test.

Lastly (for now anyway), you must acknowledge at some point in your subconcious that a "blind test" which you believe is wholly unfallible is being conducted. And as the subject of that test, you start to believe in the testing methodology. If you haven’t been the subject of such a test before, that is blind faith being invested...no better than losing teeth as a kid and asking the tooth fairy for 20 bucks.

@amir_asr 

As far as I am aware after having done research long ago, Topping and SMSL are owned by the same company: AOSHIDA 

Maybe I will...

@steve_wisc

Right on. ✔

Also testing just one speaker at a time; not both. Driving them to Nabraska. Then saying they don’t produce a linear response. Also, like @kota1 said, and he’s right; Amir’s room is untreated.

I mean, if you have non-ported speakers (small) that are right infront of you - in a nearfield setting; room treatment is not nearly as important.

For floor standing speakers; which he also measures, room treatments are signficantly more important. Especially since those speakers are meant to fill a room with sound and have way more potential/reflection points. One could suggest the machine alone negates all of this, but that is unlikely. Since of course, those machines for measurements may have also been made with human error and engineering choices that the owner is unware of, which adversely impact the results.

I could go further, but this wack-a-mole is so bruised I almost feel bad.

Who cares. 

If someone who bought it was able to derive some kind of enjoyment from it, why bother with testing it? 

Please stop supporting the man. Let him speak for himself.

I have found some major disappointments that I might post in a photo on my profile today.. talk about huge discrepancies!!

@p05129 

testing for all variables would be extremely expensive for manufacturers. You are talking about impossibilities...and also not practical.

@bobitto

We can go opposite way: if audio waves are unmeasurable and so sophisticated for current audio measuring equipment, how companies then produce the equipment then? Do they measure the results at all? Or maybe they just use alchemy, philosophic stone and any other magic when engineering the equipment to reproduce audio for our fragile ears :). Sure no need to measure, just price it $1,000 000 and it will instantly become the best “audiophile” product or the year, lol.

That’s not how it works. Of course they measure the audio products they build - during and after the design process.The most important things are a waveform of the output stage (null test), measuring with an oscilloscope, and a select few measurements that go beyond just the standard 5 or so. SINAD is an outdated way to measure audio equipment. Yet it is used as a gold-standard on ASR.

Then you must also consider the cost of parts, paying their employees, advertising costs, office space etc. There is obviously a lot that goes in to running a business.

Common sense tells us that for a hundred bucks, we shouldn’t be able to get a DAC with superlative performance, but ASR (Audio Science Review) tells us of course we can!

The word "Science" in the website should hint at a hypthosesis for why audio gear meant for the same purpose sounds different; and should welcome 3rd party testing - like other real scientists. However, that is not allowed over there...just try to challenge the results - suggest further measurements. Open the device up. Take a picture of internals and indentify the parts used. Do a reliability test. None of those things are done....not to mention countless errors in testing.

We actually do have fragile ears - since once hearing loss is an avanced stage (bad genetics or excessive exposure to loud environments) we can’t get it back. Using a hearing aid is not fun. I have some family members who unfortunately lost their hearing in one or more ears...

Human hearing (in particular the sensitivity; to detect changes in tonality) is incredibly advanced. As such, we can hear small differences easily; providing our hearing is not compromised.

I have a few products that ASR have trashed and a few they most likely would and they all sound great.

Smart design choices. They built a machine that could provide great sound quality; and left out meaningless specs like 0.0000000000002 % THD and SINAD as a yardstick measurement...

Buying based on ASR recommendations is kind of like having a vegetarian tell you that a steak house is bad.

Golden statement...

Totally different impressions and MEASUREMENTS on Head-Fi for the same product. @amir_asr likes to suggest that his "instrumentation" is so much more accurate than what others are using. Well with that logic, upgrade every 3 months or whenever AP releases a new flagship audio analyzer. This means that every former product was substandard or less accurate in some way. Right? Check out my profile to see a photo as proof of this IEM review...

And the way he EQs headphones is painful to see. It makes me furious. He simply drags up/down a line on a log EQ so it inherently influences the frequencies around that octave as well; rather than fine-tune with proper notches in place and compensate with a preamp option in the software so the levels are not compromised. You’re welcome @amir_asr

Then he has the NERVE to message me on here and tell me that I can’t hear a difference. Nobody is golden-eared - sure, however if we have good hearing and are trained listeners with knowledge, we can certainly discern all kinds of differences in audio equipment.

he says:

"And on topic of the above research showing how poor people like you in discerning differences in speakers let alone electronics:"

Also bad grammar. I am not poor, but he is suggesting that "people like me" do poorly at discerning differences in speakers and other electronics...people like me...lol

@oddiofyl

@1extreme 

Right on.

This is common sense folks... what @milpai wrote.

Is there a moderator in the house?

Gentle reminder that this is not ASR where you cry "mommy" and the moderator comes out and bans the poster.

Exactly! One of my discussions has some back-and-forth, posters started chatting about food! Am I going to get upset and reach out to a moderator...asking them to stay on topic? No. Because I'm not the boss of anyone on this website. I can accept that some posters will add value to what I've written, while others may not. And that's perfectly fine with me. I don't mind...so long as we're having a good time.

I’m fairly sure that @kenjit and @thespeakerdude are quite popular and active on ASR.

Alright guys, I’m sorry if I hurt his feelings (Amir).

This is an online forum where freedom of speech is allowed. If you disagree with what someone else posts, you certainly have a right to defend your own viewpoints or critique what others have posted. Picking apart little tidbits of what others have posted is not really a strong way to win an argument...I barely try to win, yet strong retorts from any so-called objective audiophiles are very, very, rare.

Reading what @milpai wrote above tells me he is a mature adult. Do you have some instrumentation to test that, or will you try and read what he wrote again to formulate a new response; using your good judgement and intellectual honesty?

lol so my post was removed... lol

ASR does use a 32 tone test - or something like that...which is better than just 1 KHz.

@amir_asr

My post about your graphs (the bar graph) with different types of listners in a selective order was removed by a moderator! I’ve reached out to her a few times regarding site issues that have since been fixed! 

Was this done in error? Please consider that the content of my post tried to get the crux of an issue regarding ASR measurements. Anyway, I can post some of it again. Wish I would have saved it...wasted my time writing all that out.

@amir_asr

I’m not here to argue with you, but since you are here and alive/well, I figured that sharing some of what I believe is right is worth a shot.

Audio _ Science _ Review

Like I said before,

Would you consider having your test results/measurements verified by other industry experts? I don’t see a problem with that. If I were you, I would welcome 3rd party validation of my work. Why? because that is a scientific process. I have a few friends who are actually scientists, work in labs etc. And we’ve spoken about this process many times before. If I’m actively posting reviews (like you) I would certainly want others to credit my work as correct, with repeatble results.

Perhaps an online petition could be made public; then have audiophiles from this forum, your forum, and others shoot their shot. Either you get audited, or you don’t. It’s that simple. Would you be open to such a challenge?

If no one it auditing your work (much like at a workplace within a finance department); we can’t just assume that junior accountant is doing flawless work. A CPA would be a watchful eye, so to speak. I think ASR needs one as well. Nothing against you or the folks on your site.or even @kenjit

Let’s have a balanced back and forth about this. + 6 dB or whatever. Cheers.

@thespeakerdude

You say that publishing is normally the audit process in the scientific world.

Scientific papers are subject to peer review. Publishing the results instantly is not a great idea. There is always someone more qualified to do a job. And that person ends up auditing and making sure the work is suitable for release.

Audibly transparent is a catch-phrase I hear all too often. More descriptive language is required...such as using: Sounds Like? An Audio Glossary J. Gordon Holt |

Furthermore, large discrepancies between mass customer reviews and measurements still exist with plenty of gear reviewed on ASR. You can find an audio product that has plenty of positive reviews, mentions particular sonic traits, etc. from numerous online stores (not just amazon). and even impressions from audiophiles in person whose experience lined up with yours almost exactly.

Then you go to ASR, only to find that product does not score well there. And yet, we find a mountain of information from others that suggests otherwise...actually the total opposite.

I think it would be great to hear from @amir_asr about 3rd party testing his measurements/results...let the man speak. 

@clearthink I agree 100%. The experience is all-important vs measurements alone.

@amir_asr

mp3s are not great. Sure, you could fool someone in to thinking that 2 files are the same on a smartphone over bluetooth, but upon further inspection; in a more resolving system, you could tell the original .wav file and .mp3 file apart easily, no matter what the kbps was, even 320 kbps.

For example there is a feedback loop from the brain to the hearing system to seek out information in a noisy environment. This is the so called "cocktail party effect"

This effect does not exist in a quiet listening environment. You go to visit an audio shop. Walk up the stairs and they’ve got a listening room. No one else there but you and the sales guy.

You’re at home - in your office listening to headphones and/or speakers. The room is at probably 30 dB, perhaps even less. My office for example can be a bit lower than that. The music playing on speakers and headphones will overpower the environment in this case for 2 reasons -

First, the level is louder and...

Midrange frequencies that are louder or within the same octaves effectively cancel out other midrange frequencies. Bass and treble would also soften the resolve of background noise.

I’ve found a bunch of posts on other forums where people are refuting your measurements with their own. I will share them later today....just been busy.

@andy2 

Nope. I’m asking because I had posed a response to him above. Also, other members on different forums claiming his measurements were done incorrectly. Just saying...will post later when I’m home.

@speakerdude @prof lots of points to respond to. Let's hear from @amir_asr 

@thespeakerdude 

A lot of what you're posting sounds like market research. (To be honest).

@alexatpos 

I guess I predicted the future?

Wanting to be seen as smart and being intelligent are not the same thing folks.

@alexatpos 

One of them probably has an answer to this too...

lol

@amir_asr

They can spend months and it wouldn’t make their reviews reliable. If you know what you are doing, including science and engineering of the gear and what the measurements show, you can zoom in and find issues. You don’t sit there listen to random track after random music for weeks. That tells you nothing.

More time spent on a task can net better results. This is not always true. However, when it comes to listening (headphones, speakers, DACs, amps etc.) spending more time evaluating a product before releasing a review can help in a few key ways:

1) Testing for Reliability

2) Features & Functionality

3) Overall sound quality analysis

4) Small important details

I’m sure that most of these people doing reviews have a standard set of reference tracks; or at least a background/strong interest in audio; enough so to make their impressions reliable. They were hired to do a task and might be very good at it. We have no way of knowing how much audio knowledge they have...

Tyll Hertsens - of innerfidelity (now defunct) was probably the best reviewer of headphones on the net. Like I said in one of my discussions, he took time to describe what each headphone sounded like with a particular track. He also did measurements and highlighted key areas in the frequency response or octaves where performance could have been better...while still quoting the measurements he took. And on top of all this, comparing it headphones in the same price-range with the same factor - open/closed back!

Since he went above and beyond, this gives a potential customer huge insights! You can go ahead and test with the same track. You can wait a week, be really busy, visit an audio shop, listen to that headphone (or headphones) and jot down your impressions on the notepad app your phone has installed.

Then you can compare and contrast your review with his - figure out if this headphone is worth the money for you. I’m sure that anyone who has bought audio products based on solid reviews like his will agree...

Now back to reviews...I bought a SABAJ A10h based on your review. I also bought a DROP THX789 based on your review. In both cases, not only did I find that output power was severely lacking; each of them also had their own sound signature. Based on your measurements and overall write-up of both units, a potential buyer would actually believe that each of them were a wire with gain!

Please see my profile for a photo that illustrates this. Looking inside one of these devices tells you it is cheap to build. Uses an OP amp and tons of subtractive distortion limiting - like negative feedback in a circuit. Tons of this, much like dynamic range compression in mastering will limit perceived dynamic range in a track. You’ve got to wonder how they put something together at that price-point and sold it. You can easily look up parts by just looking inside a unit and doing a parts inventory check...they are both not state-of-the-art ! lol

Alright..you can have your cake and eat it too! All I’m saying is....live and let live. Your tone and how you almost bully people into listening to you is rather rude. Hence why virtually every audio forum on the web has labelled you all kinds of silly names.

You need to stop listening to your lay intuition and embrace science of how to do such evaluations correctly. Formal testing shows long term listening to be much less revealing than instantaneous ones. See this published research on that:

I implore to you start paying attention to decades of research on what it takes to properly evaluate audio gear. The lay understanding and intuition stuff needs to go out the window.