Blind Testing

You are participating in a community which through decades of experience has placed a far lesser value on objective blind testing to that of sighted comparisons, technical measurements, and observation based comparisons. Despite the fact that you admit you are neither qualified or experienced enough ...

... to perform, evaluate, or publish these tests yourself, you still cling stubbornly to your point of view. I call that willful ignorance. Not once did you ask someone to explain their point of view. You just kept arguing the same point over and over again. And then, in one last flaming douchebag act of defeat, you then go and call us a cult. Classy. If you want to participate meaningfully here, try asking more questions and make far fewer untenable statements.

It's my belief that you should go submit yourself for testing, as I suspect you suffer from a neurobiological disorder on the higher-functioning end of the autism spectrum. (Perhaps more on the lower-functioning end. Only your doctor can say for sure.)

 

But you can’t trust medical diagnosis unless it’s double blind. Potential bias renders all non-ABX judgments meaningless.

 

As a science teacher, I need to chime in. Double blind is the gold standard when testing things such as new medicines. I’ve mentioned before that brain science regarding sound preferences, and human hearing isn’t as conclusive. Most double blind tests with Audio have involved training subjects before the test. This introduces investigator bias. So as @atomicbob stated, double blind tests in Audio is just one tool, but doesn’t give us the ultimate answer to what sounds better. In the end really all that matters is if you enjoy the music.*


* Except for the S19. That thing sounds like shit.

 

It is not a cult if we all come to similar conclusions on our own. Sometimes we don't, but those differences are more often down to personal preference.

Anyways, this thread is getting nowhere.

 

Ding! Ding! Ding! We have a winner.

This is one thing our "cult" determined pretty much unanimously. Listening, measurements, etc. all in agreement. S19 is the picture you see next to the visual definition for auditory excrement.

 

In some cases I have to say "blind" (or, at least, "blissfully uninformed") testing is necessary.

One example is in getting me to spend any more f'ing time with anything based on the goddamn CS4398 chipset (as featured in the Topping D30). @atomicbob threatened me earlier in the day with the "opportunity" to listen to the one he bought. (And I thought we were friends ... ).

You're only going to get me to go down that path again if I do not know there's a CS4398 even POTENTIALLY involved.

Life's too bleedin' short.

And the shortest known production run of any DAC I'm aware of!

I'm amazed I managed to find other units in the hands of actual customers.

 

I'm not sure why the OP is getting so many dislikes. Double blind is the gold standard. I don't see any reasoned argument around that. So are randomized experiments in the social sciences. In both cases the gold standard is often not feasible, but that does not mean the shouldn't acknowledged as the gold standard and something to aspire to.

Bias is real in hobbies like ours where large pleasure comes from parsing tiny and diminishing returns at large costs. For example: https://www.theguardian.com/lifeandstyle/2013/jun/23/wine-tasting-junk-science-analysis

 

When RBCDs first appeared, any DB testing to LPs would have produced this result: "OMG SO DYNAMIC! AND NO SURFACE NOISE! OUR QUEST FOR MUSICAL NIRVANA HAS ENDED AT LAST!!" Only later, after extended listening to CDs, would this question be raised: 'Holy shit why are my ears bleeding?" DB testing in audio may or may not have its place, but I'll go with thoughtful, long term decision making every time.

 

Double blind testing is the gold standard for aesthetic appreciation? Nope.

 

Kinda late to the game here, but this comment, to me, is important. If the market were to make ABX testing of comparable devices (and repeatable results showing not only a difference, but also a preference for a specific device) a product differentiation, we'd see much more of it. In fact, if those circumstances were the norm, the market would dictate that a website like SBAF would be redundant and useless. Why bother reading subjective drivel when there's published, peer-reviewed science that states that box-A is better than box-B?

In other words, if (just kinda picking on Benchmark for being overly "objective tech-ey"...) Benchmark thinks they've got a better DAC than errrbody else cause it measures better at -140db, they'd blind test it against 7 other DACs and publish the results. But they don't. Neither does JDS labs. Or Lampizator, or MSB, or Schiit, or PSAudio, or Topping, or Fiio.

Re: dipshit @Zbells, without going wall'o'text here, if you're really going to get interested in "science," you might want to start reading up on the last 15-20 years of neuroscience and behavioral psychology. We are bias machines. We've managed to survive for about 300,000 years thanks to our adaptations to filter stimulatory input, and unfortunately our little lizard brains have made it pretty difficult to be the homo-economicus that redditors seem to think is required to make rational purchase decisions about electric boxes that make pretty music.

 

Haha, thanks Purr1n for the thread. Damn a lot more hate since I bid adieu. Maybe cult was a strong term (but you're kind of a dick Elnrik), but I do think there's a lot of "groupthink" going on here (no not universally or across the board). I didn't say to blind test for every listening session or a blind test should be required to post an opinion or an impression. That's not what I believe at all.

But when writing a review/comparison that the company then provides on their website when promoting a a piece of equipment, I do think there should be some kind of standard. Whoever used the wine analogy, that was a great one because (https://www.realclearscience.com/blog/2014/08/the_most_infamous_study_on_wine_tasting.html). Hilariously enough, I'm a level 1 Somm (lowest level - years ago lol) and I've worked in the restaurant industry and I know a high level Somm who can absolutely blind test very accurately (and others who are very good). But those are the exception, not the rule, even among people who know their wine. I don't know for a fact, but I imagine it's similar in the audiophile community.

 

Along those same lines, I'm really into craft beer and home brewing and have my BJCP judging certification as well as a level 3 Cicerone Certification.

Not everyone is capable of getting these certs, theres a large amount of people that are completely anosmic to certain off-flavors (or even non off-flavors) and are unable to pass these tests. But there isn't a large group of people that can't taste these flavors claiming it's all bullshit like there is with audio.

 

And to twat @yotacowboy, (only being a dick because you were) that's why for medicine and other things that matter, they do their best to make sure human bias doesn't play a big role. No, audio doesn't matter like medicine and I'm not saying people should rely on blind tests for everything. Because all ABX testing tells is what that particular tester perceives. People hear differently. Still is a great tool though!

 

Seems apropos.



https://kottke.org/18/05/more-trippy-audio-illusions

 

And when the reviewers that get sent products are friends with the company that sends them products (which I assume you are, could be wrong), then that adds significantly more bias. I know you're just trying to have fun, but if a review is going on a product site, personally I believe there's an obligation to make sure it's as accurate as possible. Any other time, do whatever the hell you want.

 

@Zbells

I don't think blind testing in audio hobby (in general) is very controversial.

1. If you like it, and you want to DB for yourself. That's fine. Go for it. Enjoy yourself.
2. If you like it, and you want others to DB. Do the right work and pay the experimented people enough to attract/motivate. To be specific,

• Buy a decent ABX tester.
• Develop DB test/experiment plans - This is much more difficult than typically overlooked.
• Contact the quality reviewers and propose an attractive - at least agreeable - amount (Note: in this case, their time value may be very high) - say, $100-150 per hour or even much higher. Of course traveling cost must be considered separately. De-identification or disclosing results may increase the fee further.
• Schedule and arrange
• Pay fee to the participants
• Analyze the results. You may hire an external stat analyst.
• Enjoy the results for yourself, or disclose it to a blog or forum post (should be discussed at early stages).
• You can consider single blinded test instead.. but process will be largely the same.

Of course participants can be drawn from average people or volunteers if you're lucky. This may cost less (still I bet total experiment cost will be a lot though).

Now, do you really think blind-testing is worth every penny? If yes, fine. Use your money to do the above, rather than cowardly bitches (=generates noise) here.