Blind Test Part Deux: Schiit Heretic vs Magni+ (with statistical analysis)

I'm pretty sure a simple CATV splitter will do what you want. It will keep the 2 outputs at 75 ohm impedance. They use F connectors, but F to RCA adaptors are cheap.
I have been looking at this myself, as I'm considering bypassing my AVR for my 2 channel rig.
Or F to BNC adaptor

 

Pi2AES should be able to output to two dacs over coax and bnc (with an adapter)

 

For the validity of the test, the source connector should be the same for both.

 

not really. Assuming the DAC’s process the signal the same. Mercury and Pi2AES drive the BNC and RCA from the same buffer chip.
The only difference is the voltage level. Which for cables less than 1 meter (3ft) is totally irrelevant.

 

I can do it going from my PC via USB to Singxer SU-6 - and then taking coax or optical, etc. to the DACs.

 

Oh I absolutely believe you... but you just know someone is going to cry foul over the test results if different cables/connectors are used.

 

Excellent. As you said, Amir will keep making excuses, but having credible witnesses is probably the most valuable tool you have to make their excuses appear more and more asinine.

 

YESSSSS!! As I told Tyll when I met him way back, I would love to see blind test competitions for fun, or $$$. I want to see people put their money where their ears are. I would fail and finish last or maybe second last, but would love to see others put in their place!

 

I almost laughed. This A-blahblah guy obviously doesn't know what the garbage he is talking about. A few interesting points:

• "The trick is to get > 95% confidence that there was no guess" --> This sentence hints he has zero background in stat analyses. Those who understand the nature of TWO types of errors (type-1 and type-2 errors) won't say like this.
• "For 95% confidence, they need to get 9 answers right" --> The most obviously wrong part in his claim. Critical number of success with 0.05 alpha is, assuming one tail test (because we are not interested with the other end), 4 out of 5, 8 out of 10. Maybe he intended two-tail test but why (no rationale provided)??? Let me elaborate the process a bit:
  • Because he mentioned confidence, I'm guessing he meant type-1 error. This means everything is conditioned on random guesses. Type-1 error is, roughly speaking, the probability of getting more extreme outcomes conditioned on the null hypothesis.
  • Binomial distribution parameters with p=0.5 (random guess), n=10 (total trials), and x (critical number of successes)
  • We want to find the minimal x such that the cumulative sum of Binom(n,p,z) between z=0 and z=x is equal to or greater than 0.95 (=1-alpha=1-0.05) --> x=8
• "They must run the test at least 10 times" --> In general, determining the required test size needs to consider the TWO errors together. Often more consideration toward the type-2 error (related to so-called "Statistical power"), requiring proper formulations of alternative hypotheses. This also introduces a number of relevant questions including "to what resolution we want to distinguish measures". Any co-statisticians or myself never talk solely about type-1 error when it comes to necessary sample sizes.
• "No blind test is valid without statistical analysis" --> This sounds a bit funny. Of course statistical analysis is an umbrella term. So, the argument per se isn't too wrong. But I doubt if he really meant just two numbers lol.
  • For the reference,knowing the sample size and first moment (mean)--"A successes out of B trials" -- is enough from informational perspectives in the context of binomial distributions. The way Marv presented his results this time (full result sequence) is even dominantly better because of greater informational quantity (e.g., order of outcomes).
  • Amir seems to think statistical analysis is a big deal, but it's not. It's mostly about reducing dimensions and complexities. For simple scenario like binomial trials (a sequence of respectively independent Bernoulli trials), his wording sounds pretty awkward.
  • The only good point he made was Marv didn't report the results specifically in the Magni 3-heresy thread. But 10/10 is reported this time. lol

 

@Vtory , I would recommend two-tailed in this instance which is why I went with a 95% confidence interval in my earlier post.

If someone gets 0-1 correct, it is statistically significant and there's a reasonable chance that they were probably NOT guessing. They are just confused about the properties of the amps...they have associated the wrong amp name with a particular sound.

Or, if the test is flawed, there is something else that is tipping them off. Either way, 0-1 correct guesses is just as significant as 9-10 correct guesses if all we are trying to determine is whether they are guessing randomly.

 

I see. That makes sense in the test you assumed, where inverse prediction is possible. My basic blind testing method is 3-point abx. Each Bernoulli trial is defined as hearing A and B then matching randomly selected X to either A or B. And in this test, I'd avoid two tail because of power loss other things being equal.

PS. I think it's typo.. but isn't it that 1-10 correct guesses (1 correct out of 10 trials) is equivalent to 9-10 correct guesses (9 corrects out of 10 trials)? 0-1 is totally different story (i.e., different distribution at all). Type-1 error is drastically different.

Pr (one incorrect out of 1 trial | random guess) = Pr (one correct out of 1 trial | random guess) = 0.5
Pr (nine corrects out of 10 trials | random guess) = 10 * 0.5^10 = 0.0098

Disregard above. I might misunderstand. You're just saying the range instead of outcome-total combination.

PS2. Just for fun.. Heretic test's type-1 error..
Pr (10 corrects out of 10 trials | random guess) = 0.5^10 = 0.00098

 

If you're worried about power, just raise the sample size. As I tell my students, whenever you don't know the answer just say that you should raise n and you'll be right 90% of the time.

 

I was schooled that way decades ago, too. But the realty is trickier because we have to think about the costs. That's why I emphasized "Ceteris paribus". Increasing sample size is not a free lunch, particularly in human experiments. Even for my hobby experiments, considering each trial needs its respective wash-out/rest time, reducing trial size would be way more favorable -- even bigger benefits when we involve our friends or families. In other words, I'd prefer to maximize power given allocatable resources. I also prefer to design experiments to systematically avoid inverse-predictability when I'm allowed to design structure.

Aside: I've been conducting some ABXs for myself (for curiosity). At the end of the day, I really always wanted to reduce as many # trials as possible...

Anyway, back to the topic, I really don't think Amir meant two-tail. Look at his statement. "they need to get 9 answers right". He should've added "... or they must get 1 or less answers right".

My original point is that two result numbers will be enough to report. Choice between one vs two tails in testing might be slightly subjective and/or context-based, I think.

 

I don't think Amir had any clue what he was talking about, so I don't take his intent into consideration

Sorry for the nerdy diversion, everyone. Back to your regularly scheduled (super cool and non-nerdy) programming.

 

I very much appreciate the stat nerd discussion!!! I think that's part of the allure of SBAF. Simple comments like, "that's not exactly right," that come from folks that don't necessarily tout their academic/experiential accomplishments but often hint at a deeper understanding of the nuances involved in applied mathematics and the practice of science show the general restraint I've come to recognize as hidden gold on this forum.

Hints, nuance, selective word choice, etc., are indicative of deeper understanding gained through something other than a textbook, imho.

 

This is also why I love SBAF, when I saw what some of you do via the Jobs IRL thread, I wasn't surprised one bit—that we have scientists, PhDs, grad students, and other occupations of the intelligentsia.

So true, the extreme restraint in selecting just the right words for something, an idea, concept, even impression, is usu indicative of careful, very subtly nuanced, but ultimately rigorous thought.

 

Has anyone used any of the VAA products? Their power amp looks beefy and priced well considering.

 

Stats people, correct me if any of this is wrong, but:

I believe the intention of using a 95% confidence interval assumes the distribution of values is gaussian. In the event that the distribution isn't gaussian, (ie. it's bimodal or Mandelbrot-ian) then the stats may be misleading.

An example of this might be that you have outliers that either are deaf or have golden ears. The average Joes would form a Gaussian distribution but the gifted or deaf listeners could create a skew. If you ignored the non-gaussian distribution and simply took the forced mean you might come to a wrong conclusion. The last time someone did this at scale it crashed the world economy, re: Black-Scholes Model

Additionally, 95% confidence doesn't mean if the value isn't within 2 standard deviations it's a random result. My understanding is it means there's a 95% chance that the observed values fall within 2 standard deviations of a predicted gaussian distribution.

As for getting 9 answers right, this is wrong. The hypothesis "People can't tell amps apart" doesn't require everyone is 90% accurate. If we have two comparable groups, where the control listens to two of the same amp (but is told they are different) and the test population listens to two different amps, and the test group identifies the different amp correctly and we use statistical analysis to be confident the test group result does not fall within the control group distribution (95% CI or 99% CI if you have a large sample) then the hypothesis is not supported.

As for running 10 times, my understanding from college is a sample size of 30 is table stakes for statistical power at a 95% CI.