Blind Test Part Deux: Schiit Heretic vs Magni+ (with statistical analysis)

Discussion in 'Blind Testing and Psychoacoustics' started by purr1n, Feb 16, 2023.

  1. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Background and part un of the test is here: https://www.superbestaudiofriends.org/index.php?threads/magni-3-heresy-and-3-blind-listening.8663/

    I think Amir brings up a good point, from a scientific point of view, that we need statistical analysis.
    upload_2023-2-16_18-5-14.png
    I routinely run blind tests where I get 5 ot 6 right out of 10 and then get lost after that showing these were lucky guesses -ASR

    So here we go. Similar methodology as last time:
    1. 1kHz test tone, measured via DMM, to match levels within 1/100 Vrms
    2. All units left on for at least one hour.
    3. Double blind test: I got my kids to change up the wires into the amps headphone output randomly while I wasn't in the room and then I asked them to leave the room in another direction of which I entered the room.
    4. Wires occluded so I could not see them during the ABX test.
    Additional notes:
    1. Used Modi Multibit 2 in OS (normal mode).
    2. Instead of last time with high gain where the differences are more obvious, especially with respect to soundstage, I used low or x1 gain (which I subjectively, at least initially, felt it was more difficult to tell the amps apart) with both amps.
    3. I was allowed practice to learn the sound of the amps (I never listen at low / unity gain - just not my subjective preference - so this was bit of different experience). I took the test 5-6 times to practice, until I felt comfortable I knew the sound of the amps, before the official test.
    DSCF0311.JPG

    DSCF0312.JPG
     
    Last edited: Feb 16, 2023
  2. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    From my practice sessions, which did include many wrong guesses initially, I determined that these were the following in differences sound which were most discernible. Basically once I knew to key in on these differences, it became a cakewalk.
    • Magni+ was smoother and less grainy. Heretic had grainier jumpier highs. This was the biggest giveaway.
    • Heretic was flatter and boring sounding, but not by much. This was the second giveaway.
    After "learning" the amps' sound, the test was easy peasy with respect to guessing which one was which. I mean, it was ridiculously easy for me to discern the difference between these two 103+ AmirNAD amps.

    Test results:
    Attempt 1: 2 is Heretic CORRECT
    Attempt 2: 2 is Heretic CORRECT
    Attempt 3: 2 is Heretic CORRECT
    Attempt 4: 1 is Heretic CORRECT
    Attempt 5: 1 is Heretic CORRECT
    Attempt 6: 2 is Heretic CORRECT
    Attempt 7: 2 is Heretic CORRECT
    Attempt 8: 2 is Heretic CORRECT
    Attempt 9: 1 is Heretic CORRECT
    Attempt 10: 2 is Heretic CORRECT

    What's the confidence level for getting 10 in a row correct?

    P.S. Amir saying that he routing gets 5 of 6 right before losing it, that seems rather anecdotal and not scientific at all - especially because he's implying people other than himself would not be more capable (this does fit his ego however). We all know that different people have varying abilities to discern stuff. Most of the time, it comes down the practice. The thing is, Amir is hostile to the fact that people can actually have the ability to hear differences with amps that have well over 102+ SINAD (2V into 300-ohms 20kHz bandwidth), so perhaps this affects his hearing ability?

    BTW, here are some examples of how different people hear differently:

    https://www.superbestaudiofriends.org/index.php?threads/absolute-polarity-test.10019/
    https://www.superbestaudiofriends.org/index.php?threads/how-much-distortion-can-you-hear.8694/
     
    • Like Like x 25
    • Epic Epic x 19
    • List
    Last edited: Feb 17, 2023
  3. atomicbob

    atomicbob dScope Yoda

    Pyrate BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    18,652
    Trophy Points:
    113
    Location:
    On planet
  4. Skyline

    Skyline Double-blindly done with this hobby

    Pyrate
    Joined:
    Sep 28, 2015
    Likes Received:
    3,426
    Trophy Points:
    113
    Nonsensical question from a statistical perspective. Amir's rant about confidence was also...uhhhh...misinformed?

    You set the confidence level before constructing a confidence interval or set the significance level before conducting a significance test. Neither of those things are really happening here. But, anyway, getting off my stat teacher soap box that nobody cares about.

    A 95% confidence interval for this scenario is kind of meaningless with such a small sample size, but it would be:

    (.1871,.8129), which is to say that we are 95% confident that the true proportion of times guessed correctly by someone purely guessing in this scenario would be between .1871 and .8129. Since you named all 10, and 1.0 is outside of this interval, it would seem that we have statistically significant results to indicate that you weren't guessing.

    The question then turns to whether your experiment had any flaws (beyond the small sample size). Assuming the test was theoretically perfect and there were zero lurking variables other than the differences in the amps themselves, then someone purely guessing would have a .00098 chance of guessing all 10 correct.

    For the record, I don't doubt that people can hear differences when properly trained. But mentions of statistical analysis are triggering for me. :p
     
    • Like Like x 20
    • Epic Epic x 3
    • Agreed, ditto, +1 Agreed, ditto, +1 x 1
    • List
  5. futbutts

    futbutts Friend

    Pyrate
    Joined:
    May 15, 2018
    Likes Received:
    469
    Trophy Points:
    63
    Location:
    Shermie Oaks, CA
    Funny how, for them, any idiot can do a half-assed ABX and conclude 'Wellp couldn't tell the difference, subjectivity is dead' and they'll eat it up -- meanwhile someone qualified like Marv can sit and conduct a studied ABX test and score well beyond what could be considered margin of error (or perfect, like in this case), and they'll still move goalposts to incorporate some other "oversight" that they'll say nullifies the test.

    Guarantee they'll have some other parameter they'll say this test needs to cut mustard... and failing that they'll just say the results can't be trusted.
     
    • Like Like x 7
    • Agreed, ditto, +1 Agreed, ditto, +1 x 3
    • List
  6. rfernand

    rfernand Almost "Made"

    BWC Contributor
    Joined:
    Oct 10, 2022
    Likes Received:
    728
    Trophy Points:
    93
    Location:
    Kirkland, WA
    Me as well. What are the odds?
     
    • Like Like x 8
    • Epic Epic x 2
    • Agreed, ditto, +1 Agreed, ditto, +1 x 1
    • List
  7. CEE TEE

    CEE TEE MOT: NITSCH

    Pyrate IEMW
    Joined:
    Sep 25, 2015
    Likes Received:
    3,721
    Trophy Points:
    93
    Location:
    California Bay Area
    Home Page:
    @Hands went to Montana and spent a weekend blind testing amps with Tyll for "Big Sound". He also got 10/10.
     
  8. Cspirou

    Cspirou They call me Sparky

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    8,200
    Trophy Points:
    113
    Location:
    Northwest France
    • Like Like x 6
    • Epic Epic x 1
    • List
  9. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Amps did not have good enough AmirNAD of 100db or higher.

    Need this to happen at least 9/10. That's only 1/1.

    :D
     
    • Like Like x 3
    • Agreed, ditto, +1 Agreed, ditto, +1 x 1
    • List
  10. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    I believe ASR dogma says 110db AmirNAD is necessary to achieve transparency, at least I remember reading that somewhere. Of course this wasn't arrived through science or experiments. It was just something Amir just pulled out of his ass. Of course it then makes no sense to plot DAC SINAD past 110db, unless one wanted to shill, like here: https://www.superbestaudiofriends.o...g-and-smsl-with-asr-punking-us-on-dacs.12808/

    Moral of the story: Science only applies to others, but not to Amir.

    Anyway Magni+ is only 103db AmirNAD. I may have to borrow a Magnius to compare again the Heretic. Subjectively the Magnius, unlike the +, has a much different sound to the Heretic, so in theory would be easier to pick out in a blind test (with statistical analysis).

    ---

    With respect to statistics, we can actually take this approach through inquiry based on people's subjective experience. This is how painkiller effectiveness is measured. There is no way to objectively measure pain. Just as there is no way to objectively measure pleasure, auditory pleasure in this case (or preference).

    It would be interesting to ship a sealed box of say two DACs with only the input outputs exposed (with extender cables to hide the backplate), and see which one was preferred.
     
    Last edited: Feb 17, 2023
  11. rfernand

    rfernand Almost "Made"

    BWC Contributor
    Joined:
    Oct 10, 2022
    Likes Received:
    728
    Trophy Points:
    93
    Location:
    Kirkland, WA
    That could be cool - we’d learn the percentage of participants who can tell them apart, and for bonus points, which one is preferred.
     
  12. JK47

    JK47 Friend

    Pyrate
    Joined:
    Oct 31, 2022
    Likes Received:
    2,015
    Trophy Points:
    93
    Location:
    Surf City USA
    Sealed with tamper proof foil...
     
  13. Vansen

    Vansen Gear Master (retiring)

    Staff Member Pyrate
    Joined:
    Nov 19, 2016
    Likes Received:
    3,893
    Trophy Points:
    93
    Location:
    Seattle
    Send me two DACs. I'll make cases for them and kick off a loaner.
     
    • Epic Epic x 6
    • Like Like x 2
    • List
  14. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
  15. Armaegis

    Armaegis Friend

    Pyrate BWC
    Joined:
    Sep 27, 2015
    Likes Received:
    7,461
    Trophy Points:
    113
    Location:
    Winnipeg
    Put extra weights in one of the boxes just to mess with people.
     
  16. Cspirou

    Cspirou They call me Sparky

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    8,200
    Trophy Points:
    113
    Location:
    Northwest France
    How do you feed two DACs from a single source? I am not sure a cable splitter applies the same way with digital signals as with audio.

    Two USB inputs is probably fine with the appropriate software tricks however in this particular case the DACs would be exposed right away in the OS
     
  17. gixxerwimp

    gixxerwimp Professional tricycle rider

    Pyrate
    Joined:
    Sep 30, 2015
    Likes Received:
    5,773
    Trophy Points:
    113
    Location:
    small island claimed by China
    I run this splitter from the AES output of my DEQ2496 to two DACs.
     
  18. Biodegraded

    Biodegraded Friend

    Pyrate Contributor
    Joined:
    May 28, 2017
    Likes Received:
    7,975
    Trophy Points:
    113
    Location:
    Vancouver BC
    For a while I was accomplishing the reverse (two sources to a single DAC, over coaxial S/PDIF) using the video ports of the same cheap switch box as in purr1n's photos above. However, the connection would occasionally drop so I eliminated the box and I'm pretty sure it led to a sound quality improvement from both sources (pi2AES and CD transport).

    The box specs claim 75 Ohm impedance for the video path, but apart from the video jacks being yellow there was no discernible internal or external difference between it and the audio ones. The internal wiring is very short so I guess that just assumes the interconnect cables are the right spec.
     
    • Like Like x 1
    • Epic Epic x 1
    • List
  19. Armaegis

    Armaegis Friend

    Pyrate BWC
    Joined:
    Sep 27, 2015
    Likes Received:
    7,461
    Trophy Points:
    113
    Location:
    Winnipeg
    The various usb decrapifiers or digital-to-digital converters usually have multiple outputs that you can use to feed multiple dacs.

    The transformer balanced splitter that @gixxerwimp wimp linked above is another way.

    Depending on the devices on each end, sometimes even a simple cable splitter might work for aes/coax. Maybe not "optimally", but it could still work.
     
  20. Roderick

    Roderick Facebook Friend

    Joined:
    Oct 22, 2018
    Likes Received:
    208
    Trophy Points:
    33
    Location:
    Finland
    I use voicemeeter banana . https://vb-audio.com/Voicemeeter/banana.htm

    It creates a virtual source that runs all the selected sources simultaniously. If you try to cheat and click sound settings icon it only shows the name of the virtual source not the dac. And you don't need to touch the OS anyway. I've been using voicemeeter with Schiit Sys and SPL Phonitor 2 (has two XLR inputs).
     

Share This Page