DAC Blind Test: Modi2 vs ODAC

Discussion in 'Blind Testing and Psychoacoustics' started by purr1n, Nov 17, 2015.

  1. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    TEST PROTOCOL

    So I am about to embark on a blind test for DACs, just for fun. I'd like to get some input on the test protocol.
    1. I will first listen to each DAC to get a sense of its sound (already done). The DACs being compared are the ODAC and a tweaked Modi 2. The amp is something I cannot say at this point other than it is a good amp, but isn't near TOTL or anything like that. Basically the real reason is that I'm too lazy to setup the ZD next to my desk.
    2. Those are stepped attenuators into the Ratshack switch. If more granularity is needed, I can easily parallel a resistor to level match DACs precisely. On second thought, maybe I should move the steppers to where the volume control on the amp is at at approximate location and start off each test with the volume turned completely down. A non-descript volume knob without any indents will be used. I know level matched testing is one method, but how would people feel if the levels were simply reset and I was allowed to set the volume at any level I wanted to. This method is closer to real usage compared to static set volume levels; and it would seem that this method wouldn't favor either one DAC over the other. I am thinking, if the volume was turned down to start every time, and there were statistically significant results, then obviously the volume level had nothing to do with it.
    3. I will write a script to hide the JRiver MC Window to change the DAC output back and forth randomly. The AHK script will capture the F11 key which will toggle the DAC output in JRiver. (I need to refine the script to move the window a second monitor not viewable by me, make the changes, and then move the window back.) It's a bit clunky since I then I have to press random buttons on the RatShack switch to find the correct output.
    4. I'll get Julie to randomly change the inputs and shove the back half of the switcher into box so I don't see where the connectors go. There are four switches for two devices. She can plug them into random spots.
    5. During the tests, I get to take my time and I get to go back and forth between the DACs before I say whether the DAC is the tweaked ModiU or ODAC. After I guess, I will open up the JRiver settings to see if I was right or wrong. I will document the results along the way.
    6. I will get walk away and take breaks and go about daily life. During these times, Julie will change the inputs.
    [​IMG]
    [​IMG]

    ROLLING TEST RESULTS - DAY 1
    1. 11/17/2015 7:54pm: Hit F12 to randomize. Then F11. Turned down volume. Pressed play in JRMC: Daft Punk Lose Yourself to Dance. Turned up volume slightly. Found output on switch as "VCR". Adjusted volume. I'm thinking, this is fricking easy. This is the ODAC. Sounds a bit flat and bass is not well defined. Pressed stop in JRMC. Hit F11 to toggle. Turned down volume. Pressed play in JRMC: Daft Pank Lose Yourself to Dance. Turned up volume slightly. Found output on switch as "AUX". Adjusted volume. Yep, first one was definitely the ODAC. The second one is the Modi 2U. More dynamic sounding, better clarity and less mush on bass. Flipped over cover. Yep I was right. Going to take a piss now. Asking Julie to change wires. CORRECT. 1 / 1
    2. 11/17/2015 8:07pm: Hit F12 to randomize. Turned down volume. Then F11. Pressed play in JRMC: Daft Punk Lose Yourself to Dance. Turned up volume slightly. When through switch and found output on switch as "VCR" again. Adjusted volume. Ah. I think this the Modi. Pressed stop in JRMC. Turned down volume. Hit F11 to toggle. Turned down volume. Pressed play in JRMC: Daft Pank Lose Yourself to Dance. Turned up volume slightly. Found output on switch as "GAME". Adjusted volume. This must be ODAC. Flipped over cover. Wrong! Haha, too cocky. WRONG. 1 / 2
    3. 11/17/2015 8:16pm: Hit F12 to randomize. Then F11 three times. Turned down volume. Pressed play in JRMC: Daft Punk Lose Yourself to Dance. Turned up volume slightly. When through switch and found output on switch as “DVD”. Pressed stop in JRMC. Turned down volume. Hit F11 to toggle. Turned down volume. Pressed play in JRMC: Daft Pank Lose Yourself to Dance. Turned up volume slightly. Found output on switch as "VCR". Adjusted volume. Went back all over to do again with same track three more times. Can’t tell the difference. Not going to guess. Taking a break. ABORTED. 1 / 2
    4. 11/17/2015 8:34pm: k.d. lang Constant Craving. DACS still seem to be on “DVD” and “VCR” switch positions. LOL Julie, didn’t change anything! I made this call on ODAC stage being closer in between ears and didn’t throw out things in front as well as Modi 2U. CORRECT. 2 / 3.
    5. 11/17/2015 8:55pm: k.d. lang Constant Craving. DACs on AUX and DVD. Staging gives it away. More separation and depth to instruments on Modi 2. ODAC stage is flatter. lang's mixed in background vocals at near 1:00 give it away. CORRECT. 3 / 4.
    6. 11/17/2015 9:05pm: k.d. lang Constant Craving. DACs on GAME and AUX. Staging gives it away. Guessing GAME is Modi 2. CORRECT. 4 / 5.
    7. 11/17/2015 9:29pm: k.d. lang Hallelujah. DACs on DVD and AUX. Switched back and forth three times. Tired. Can't tell the difference. ABORTED. 4 / 5.
    8. 11/17/2015 9:48pm: Daft Punk Get Lucky. DACs on DVD and AUX. Decided to relax and listened through entire song with one DAC. Then the next DAC. Guessing AUX is ODAC. Second DAC has flatter, less clear, and less engaging. CORRECT. 5 / 6.
    9. 11/17/2015 10:03pm: Daft Punk Get Lucky and k.d. lang Constant Craving. DACs on DVD and AUX. Listening through most of songs. Guessing Modi 2U is ODAC. Seems more engaging. WRONG. 5 / 7.
    10. 11/17/2015 10:12pm: DAC output script failed. Adding extra code to click "Audio" on left side before changing output to make script more robust. ABORTED. 5 / 7
    11. 11/17/2015 10:33pm: Duran Duran Save A Prayer. I can't tell the difference. Switched three times. Going to take a break. ABORTED. 5 / 7
    12. 11/17/2015 10:51pm: Lionel Richie Stuck on You. VCR and GAME. Guessing ODAC is VCR. Thinner tonality. CORRECT. 6 / 8.
    13. 11/17/2015 10:59pm: Talking Heads This Must Be The Place. AUX and GAME. Guessing AUX is Modi 2U. Heftier bass. Not as thin up top. CORRECT. 7 / 9.
    14. 11/17/2015 11:06pm: Talking Heads This Must Be The Place. DVD and GAME. Guessing DVD is ODAC. Digital sounding snares. CORRECT. 8 / 10.
    15. 11/18/2015 8:31 AM: Daft Punk Lose Yourself to Dance. AUX and GAME. Guessing AUX is Modi 2U. More foundation to bass. Smoother treble. CORRECT. 9 / 11
    16. 11/18/2015 8:51 AM: Daft Punk Lose Yourself to Dance. GAME and VCR. Guessing VCR is Modi 2U. More foundation to bass. Smoother treble. CORRECT. 10 / 12
    COMMENTARY: http://superbestaudiofriends.org/in...lind-test-modi2-vs-odac.582/page-2#post-14366

    ROLLING TEST RESULTS - LEVEL MATCHED 0.01db - DAY 2

    1. 11/18/2015 9:49PM: Daft Punk Lose Yourself to Dance. More dynamic range and nuance with volume levels with what I guess as Modi 2. CORRECT 1 / 1
    2. 11/18/2015 9:56PM: Daft Punk Lose Yourself to Dance. ODAC is just plain louder all the time with less microdynamics. CORRECT 2 / 2
    3. 11/18/2015 9:56PM: Daft Punk Lose Yourself to Dance. ODAC treble is sharper. Modi 2 more nuanced in volume levels. CORRECT 3 / 3
    4. 11/18/2015 10:06PM: Daft Punk Lose Yourself to Dance. ODAC has sharper snares, more grainy treble. Modi 2 is warmer. CORRECT 4 / 4
    5. 11/18/2015 10:11PM: Michael Jackson. Man in the Mirror. MJ's vocals grainer on ODAC. More variances in volume and decay with intro synth on Modi 2. CORRECT 5 / 5
    6. 11/18/2015 10:19PM: Michael Jackson. Man in the Mirror. Same observations as above. Also noted MJ vocals pushed out more and higher in terms of height on Modi 2. CORRECT 6 / 6
    7. 11/18/2015 10:33PM: Peter Gabriel In Your Eyes. Modi 2 staging is deeper less flat. With ODAC, Peter Gabriel's vocals are inside head. CORRECT 7 / 7
    8. 11/18/2015 10:33PM: Peter Gabriel In Your Eyes. Exact same as above. CORRECT 8 / 8
    COMMENTARY: http://superbestaudiofriends.org/in...lind-test-modi2-vs-odac.582/page-5#post-14752
     
    Last edited: Dec 4, 2015
  2. TMoney

    TMoney Shits on SBAF over at Head-Case to be cool

    Joined:
    Sep 27, 2015
    Likes Received:
    760
    Trophy Points:
    93
    Location:
    Sausalito, CA
    Godspeed with this, Marv. I'll be very excited to hear your impressions.

    I've found that even sighted A/B testing shows many DACs to be a lot closer in sound to what the hype might suggest.
     
  3. velvetx

    velvetx Gear Master West/Vendor Spotlight Moderator

    Staff Member Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    1,067
    Trophy Points:
    113
    Location:
    Bay Area, CA
    I am definitely following this thread but was kind of interested in potentially setting up a Bay Area A/B Test Lab :) as I think it takes more than one person to do A/B testing. The big issue after reading your foray into this was volume matching which to me would be the biggest pain.
     
    Last edited: Nov 17, 2015
  4. mkozlows

    mkozlows Friend

    Pyrate
    Joined:
    Oct 6, 2015
    Likes Received:
    512
    Trophy Points:
    93
    Tweaked how? Because I mean, one obvious objection is that it's been tweaked into audible distortion.

    The volume protocol seems sensible to me, with the caveat that it'd be easy to get sloppy about resetting when switching.
     
  5. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    LOL, no tweaks to change distortion characteristics or anything like that. (I've been checking measurements. Honestly, spurs via USB cable differences are making a bigger difference in measurements.) Think parts changes a la what Bill did with Wyrd. I'll say exactly what it is after I'm done in a month. I didn't want to get Yggdrasil because the differences would be too big.

    On volume control resetting, that's on my checklist, to turn volume all the way down (and also with every switch), if Julie doesn't do or forgets, I will check as well. This is one was of the things we uncovered via our dry runs last night.
     
    Last edited: Nov 17, 2015
  6. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Here's the AHK code. F11 for toggle. F12 for randomizing to start off each cable change.
    Code:
    DAC = 1
    
    $f11::
      WinActivate,JRiver Media Center 19
      sleep 50
    
      ;####### Bring up Options Window and Move it Offscreen
    
      sendinput ^o
      sleep 500
      WinMove,Options,,-1200,500
    
      ;####### Open Audio Device Listbox
    
      CoordMode Mouse, Relative
      mousemove 242,101
      click
      sleep 500
    
      ;####### Toggle DAC
    
      If (DAC = 1) {
          mousemove 65,258
          click
      }
    
      If (DAC = 0) {
          mousemove 65,298
          click
      }
    
      DAC := 1- DAC
    
      ;####### click OK
    
      sleep 100
      mousemove 592,559
      click
      sleep 100
    
      CoordMode Mouse, Screen
    return
    
    $f12::
      Random DAC,0,1
    return
    
    
     
    Last edited: Nov 17, 2015
  7. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Results of six tests posted in first post. I need to take a poop now.
     
  8. The Alchemist

    The Alchemist MOT: Schiit - Here to help!

    Pyrate
    Joined:
    Sep 24, 2015
    Likes Received:
    1,189
    Trophy Points:
    113
    Location:
    The SchiitR
    Home Page:
    :Violin:
     
  9. Hands

    Hands Overzealous Auto Flusher - Measurbator

    Staff Member Pyrate MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    12,285
    Trophy Points:
    113
    Location:
    Colorado
    Home Page:
    I will accept no less than one million attempts. But even then, I'll still just assume you had a lucky guessing streak or flawed methodology.
     
  10. lm4der

    lm4der A very good sport - Friend

    Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    1,461
    Trophy Points:
    113
    Location:
    Seattle, WA
    Impressive, young Skywalker.
     
  11. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Going to bed now. Wife doesn't want to get up to change wires.

    I have a few takeaways:
    1. The dry runs I did yesterday and knowledge I've gathered from blind tests I've conducted on other people in the past did help. Like any test, it can be practiced. One starts to develop techniques to take the test.
    2. These DACs are more similar than different. I should have done Yggdrasil vs. ODAC instead.
    3. This is fricking stupid and a colossal waste of time. Moffat is right. It is much better to get laid.
     
    Last edited: Nov 17, 2015
  12. The Alchemist

    The Alchemist MOT: Schiit - Here to help!

    Pyrate
    Joined:
    Sep 24, 2015
    Likes Received:
    1,189
    Trophy Points:
    113
    Location:
    The SchiitR
    Home Page:
    Goodnight purrin
     
  13. thune

    thune Friend

    Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    50
    Trophy Points:
    18
    Having the abort/concede/tie option is quite novel. I find it thought provoking: what should be the significance attributed to such a trial? Does having this option reduce the stress and confusion associated with having to make a selection, when sometimes "don't know" is the more honest, mentally consistent, less frustrating answer? Anyway, cool stuff.
     
    Last edited: Nov 18, 2015
  14. T.Rainman

    T.Rainman Acquaintance

    Joined:
    Oct 12, 2015
    Likes Received:
    57
    Trophy Points:
    18
    Moffat is right.
     
  15. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,129
    Trophy Points:
    113
    Location:
    India
    Brilliant project.
     
  16. drfindley

    drfindley Secretly lives in the Analog Room - Friend

    Pyrate
    Joined:
    Sep 25, 2015
    Likes Received:
    1,533
    Trophy Points:
    93
    Location:
    Austin
    I've found the biggest problem is humans aren't computers. We're not programmed to do comparisons to detect small differences. It's counter-intuitive because we're fantastic at detecting small things. It's the difference engine that's broken.
     
  17. JewBear

    JewBear Almost "Made"

    Joined:
    Sep 28, 2015
    Likes Received:
    270
    Trophy Points:
    63
    I too think you have something novel by allowing aborts. Actually think it's quite a good idea as it takes the stress out of ABX as you never feel pressured to make a decision when you don't really want to.
     
  18. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,129
    Trophy Points:
    113
    Location:
    India
    Seems fine to me, but I am no expert in the kind of logic that the gods of ABX use to define test conditions and procedures.

    In my book, any blind test, whether it is postable on hydrogenaud.io or not, is a good experiment
     
  19. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,129
    Trophy Points:
    113
    Location:
    India
    Perhaps the problem is that are computers, with complex loops, feedback mechanisms and filters that come between what people call "the ears" and hearing as it happens in our conciousness.
     
  20. SKiring

    SKiring Friend

    Pyrate
    Joined:
    Oct 3, 2015
    Likes Received:
    203
    Trophy Points:
    43
    Location:
    Rotterdam
    Hurrah! Haha same conclusion on my end, at least with Bifrost Multibit vs. ODAC I feel there's a certain sense of space and placement and an analog undertone that distinguishes between the two. Haha, you were better of posting this than my tests on Reddit.

    I might do one follow-up with a statistically significant amount of runs or not, we'll see. It's a pain and fun at the same time when you partner up with a friend to do it, alone is easier to get through but also much less fun.

    Thanks for doing this @purrin, however expect to hear the same amount of "this amount of testing is statistically insignificant".
     

Share This Page