Tutorial - Lies, Damn Lies and USB DAC technical measurements

Discussion in 'Measurement Techniques Discussion' started by atomicbob, Feb 11, 2018.

  1. atomicbob

    atomicbob dScope Yoda

    Pyrate BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    18,652
    Trophy Points:
    113
    Location:
    On planet
    Or how to manipulate measurements for fun and profit.

    This visual tutorial is Microsoft Windows oriented.

    Let’s examine two hypothetical reviews from two separate hypothetical websites professing to participate in audio equipment reviews and discussion.

    Reviewer A:
    It’s a giant killer, the best thing since the invention of the transistor! The inexpensive objective odac outperforms a costly studio grade Violectric V800. Just look how much better the odac performs than the V800 on our expensive analyzer:
    graph1.png
    As you can see we provide the truth other sites can’t or won’t because we have an expensive analyzer. Did I mention we have an expensive analyzer which makes us better than everyone else?


    Reviewer B:
    Huzzah, a DAC of quality! The Violectric V800 is definitely a cut above the ordinary. Just look how much better it performs than the common objective odac on our expensive analyzer:
    graph2.png
    As you can see we provide technical insights other sites can’t or won’t because we have an expensive analyzer. Did I mention we have an expensive analyzer which makes us better than everyone else?


    Ok, so which of those graphs above are real?
    Give up?


    They both are real measurements.
    Whaaat??!! Ok AB, now I know you’re messing with me. How can that be?
    They are not fair comparisons. Let me explain.


    Notice the graphs and associated text are devoid of measurement conditions other than both appear to have been made with a 48KHz sample rate by the analyzer analog input given the 24KHz x-axis. Oh, but you have seen text overlays on graphs with a few conditions stated you say? If those reviewers have a history of meticulous attention to detail, then they might deserve the benefit of the doubt. However text entered by hand has a potential for typos, mistakes or worse. I agonize over my measurement posts for this very issue. Let’s look at the following in detail:

    1. Odac measurement s with pertinent setup parameters included
    2. V800 measurements with pertinent setup parameters included
    3. RME ADI-2 pro as DDC measurements with pertinent setup parameters included
    4. Shiit Eitr DDC measurements with pertinent setup parameters included

    Hopefully it will be appreciated that this is a rather complex tutorial and has taken significant time to create. While every effort has been made to assure accuracy, there still may linger a mistake or two. If a legitimate mistake(s) is found, please call to my attention and it will be corrected.


    There are many analyzer parameters to adjust in making a USB DAC audio measurement, not all of them inside the analyzer control program.

    Orientation, a few specific parameters to watch as we progress through these examples:

    graph3.png
    A. Soundcard audio driver selection – USB ODAC in this example
    B. Residual noise floor and spikes
    C. DAC analog output measured in dBu (reference 0.775 Vrms)
    D. FFT size (number of points used)
    E. FFT averages – number of FFTs averaged, lowers residual bin noise
    F. Not used
    G. Generator output level



    graph4.png
    First up, odac measurement. let’s adjust the audio analyzer with the following interesting parameters:

    1. Analyzer generator set for sine, 12 KHz, 0 dBFS
    2. Generator audio goes through Windows Audio Architecture (WAA) WDM driver
    3. Odac analog output measured at 8.119 dBu
    4. FFT size set for 8K
    5. FFT averages set for 8

    Note all the spikes on the residual noise floor.


    graph5.png
    Let’s make just one adjustment, the route by which the audio is sent to the odac. Using an ASIO driver we observe two changes:

    1. The number of residual spikes have diminished substantially, implying lower jitter
    2. Odac analog output has increased slightly, from 8.119 to 8.250 dBu.


    graph6.png
    Now returning to the WDM audio path, let’s make a different adjustment, change the generator from 0.0 dBFS to -0.2 dBFS. Not a large change in amplitude but watch the results

    1. Similar reduction in residual spikes, though not as large as ASIO driver change
    2. Odac output has again changed slightly from 8.119 to 8.050 dBu, but not as much as expected.

    Those amplitude changes may not seem like much but they are indicative of a problem. Changing -0.2 dBFS should have resulted in 8.119 dropping to 7.934 dBu in the WDM path. Also note that 8.050 dBu is -0.213 dBFS from 8.250 dBu from the ASIO path indicating that we should be measuring 8.250 dBu in the WDM path. Note that 8.119 is -0.139 dBFS lower than 8.250 dBu. These results will be important in the DDC section below. Combine the unexpected levels with the rise of spikes when the generator is set for 0 dBFS through the WDM path and we can be reasonably certain there is a limiter in the WDM path. So when we see spikes at 0 dBFS that do not occur at a slightly lower level or with a different audio path through the WAA, we should question the validity of our 0 dBFS WDM path measurement. More on that in the DDC section below.


    graph7.png
    V800 measurement: let’s adjust the audio analyzer with the following interesting parameters:

    1. Analyzer generator set for sine, 12 KHz, 0 dBFS
    2. Generator audio goes through Windows Audio Architecture (WAA) WDM driver
    3. V800 analog output measured at 5.664 dBu
    4. FFT size set for 8K
    5. FFT averages set for 8

    Similar profusion of spikes on the residual noise floor as the odac through WDM at 0 dBFS.


    graph8.png
    Again, making just one adjustment, the route by which the audio is sent to the V800. Using an ASIO driver we again observe two changes:

    1. The number of residual spikes have diminished substantially, implying lower jitter
    2. V800 analog output has increased slightly, from 5.664 to 5.792 dBu.



    graph9.png
    Now returning again to the WDM audio path the generator amplitude is changed from 0.0 dBFS to -0.2 dBFS.

    1. Similar reduction in residual spikes, though not as good as ASIO driver change
    2. V800 output has again changed slightly from 5.664 to 5.590 dBu but not as large a drop as expected.



    In our hypothetical example:

    1. Reviewer A chose to measure odac using ASIO at 0 dBFS
    2. Reviewer A chose to measure V800 using WDM at 0 dBFS
    3. Reviewer B chose to measure odac using WDM at 0 dBFS
    4. Reviewer B chose to measure V800 using ASIO at 0 dBFS

    Additionally:

    1. Reviewer A chose to measure odac using FFT size 32K, FFT averages 32
    2. Reviewer A chose to measure V800 using FFT size 8K, FFT averages 8
    3. Reviewer B chose to measure odac using FFT size 8K, FFT averages 8
    4. Reviewer B chose to measure V800 using FFT size 32K, FFT averages 32

    Observe how ragged the poor performing DAC traces are in the first two reviewer comparison graphs and how much smoother the better performing DAC traces appear. Also note the use of color with poor performing DAC traces in Red and the better performer in Yellow with nice contrast to the Black background. Color choice is intentional.

    The comparisons weren’t fair.


    A fair comparison:
    graph10.png
    Both odac and V800 have been measured as follows:

    1. Analyzer generator set for sine, 12 KHz, 0 dBFS
    2. Generator audio goes through ASIO drivers
    3. Analog output measured for 0dBFS: Odac 8.250 dBu, V800 5.792 dBu
    4. FFT size set for 32K
    5. FFT averages set for 32

    In this fair comparison odac is not the giant killer claimed by Reviewer A, but doesn’t perform as badly as indicated by Reviewer B. V800 measurement by Reviewer A was also clearly deficient and the validity should have been questioned before publishing for thousands, perhaps millions of readers to see. The same applies for odac measurement of Reviewer B. Both reviewers have demonstrated their measurement skills are either at what Real Engineers™ call IAO level or are deliberately misleading. I leave it to the reader to interpret the acronym with the hint that O stands for Operator.

    To those using my graphs in an attempt to establish their measurement legitimacy, please note that my screen name is atomicbob, not AtomicBomb or other variant. It does not instill confidence in ability to correctly use a complex measurement system when one can’t get the simple details correct, though there is possibly an attempt at a subtle depreciative implication.
     
    • Like Like x 76
    • Epic Epic x 1
    • List
    Last edited: Feb 11, 2018
  2. atomicbob

    atomicbob dScope Yoda

    Pyrate BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    18,652
    Trophy Points:
    113
    Location:
    On planet
    DDC section


    Microsoft Windows Audio Architecture
    WindowAudioArchitecture_3_with_ASIO.png
    Audio that traverses the MME shared mode path is subject to processing by the KMixer Audio Engine. DirectSound Core Audio API may or may not be mangled by the KMixer Audio Engine depending on whether shared mode or exclusive mode is engaged. WASAPI exclusive mode will avoid the Audio Engine. ASIO avoids audio data bit manipulation by communicating directly with the vendor USB audio hardware layer. Some vendors have provided ability to test for audio data path manipulation. RME ADI-2 Pro and ADI-2 DAC, for example, have short wavefiles which when played will display whether bit accurate data has been transmitted. A message appears on the display indicate Bit Test Passed if successful. JRiver Media Center allows the user to set which of the many audio paths to the Hardware Abstraction Layer is employed. This is a big deal for Windows users.


    Let’s compare routing audio data through WDM and ASIO. Two devices will be examined, RME ADI-2 Pro and Schiit Eitr. Below is an example of the test setup to visualize the data paths. Either ASIO or MME (labeled WDM in dScope) will be selected.

    DDC testing with dScope v3.png


    graph13.png
    RME ADI-2 Pro spdif output to dScope spdif input via WDM driver path, 0 dBFS.

    Observe all the spikes in the noise floor. What is causing those? Also note signal output is 0 dBFS but return is -0.131 dBFS. We’ve seen that before in the odac WDM vs. ASIO measurements above.


    graph14.png
    RME ADI-2 Pro spdif output to dScope spdif input via ASIO driver path, 0 dBFS.

    No spikes, single perfect line at 12 KHz as expected. Also note signal input now agrees with signal output, is 0 dBFS for both.


    graph15.png
    RME ADI-2 Pro spdif output to dScope spdif input via WDM driver path, -0.2 dBFS.

    Look at that. -0.2 dBFS sent, -0.2 dBFS received. No spikes. Apparently below the threshold of whatever is causing a problem in the WDM path, which is likely a limiter process.


    graph16.png
    Schiit Eitr spdif output to dScope spdif input via WDM driver path, 0 dBFS.

    Observe all the spikes in the noise floor, same as RME ADI-2 Pro above.. Also note signal output is 0 dBFS but return again is -0.131 dBFS.


    graph17.png
    Schiit Eitr spdif output to dScope spdif input via WDM driver path, -0.2 dBFS.

    Look at that. -0.2 dBFS sent, -0.2 dBFS received, again! No spikes.


    Ok, so is there anything happening below -160 dBFS?
    graph18.png
    RME ADI-2 Pro spdif output to dScope spdif input via ASIO driver path, 0 dBFS.

    Y-axis expanded to -250 dBFS. Nope. No spikes. Nothing. Just a perfect data stream as expected.


    graph19.png
    Schiit Eitr spdif output to dScope spdif input via WDM driver path, -0.2 dBFS.

    Y-axis expanded to -250 dBFS. Again, no spikes. Nothing. Just a perfect data stream as expected.


    When making USB DAC or ADC audio measurements, driver paths matter. Levels are especially important when using non-ASIO paths.

    Closing thoughts. I have displayed pertinent analyzer parameters embedded in the graphs in this discussion. Measurements should have a statement of test conditions so others may repeat and verify. In the past, I have assumed that others performing audio measurements would be equally meticulous and careful. Bad assumption. Look at any measurement graph with a critical eye and skepticism. Even mine. Ask yourself if there may be any biasing motivator by the publisher of the data, even mine. Don’t assume!
     
    • Like Like x 60
    • Epic Epic x 1
    • List
    Last edited: May 31, 2018
  3. atomicbob

    atomicbob dScope Yoda

    Pyrate BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    18,652
    Trophy Points:
    113
    Location:
    On planet
    reserved 3/3.
     
  4. Kattefjaes

    Kattefjaes Mostly Harmless

    Pyrate
    Joined:
    Sep 5, 2016
    Likes Received:
    4,521
    Trophy Points:
    113
    Location:
    London, UK
    Good stuff, AutomaticBarb!

    I particularly like the point about bit-perfect output, oft overlooked.

    It might be worth mentioning WASPI, as increasing amounts of stuff likes using that in event mode with exclusive device locks for reliable playback- it looks like it could supplant ASIO at some point*. It's also worth mentioning it to distract people using consumer cards without ASIO drivers from thinking they can get magic benefits by layering an ASIO wrapper on top of their WDM drivers (ASIO4ALL and similar).


    * Especially on "prosumer" kit. As much as good ASIO drivers save a lot of headaches and control specific hardware properly, manufacturers love an excuse to cheap out on the software they ship. Staunchly professional stuff will probably continue to ship them though.
     
    Last edited: Feb 11, 2018
  5. Taverius

    Taverius Smells like sausages

    Pyrate
    Joined:
    Dec 27, 2017
    Likes Received:
    3,026
    Trophy Points:
    113
    Location:
    Rapallo, Italy
    It is fascinating that just -.2dB is enough to hide whatever mangling the limiter in the mixer is doing.

    Also, WASAPI runs like ASIO/OpenAL, right?
     
  6. Kattefjaes

    Kattefjaes Mostly Harmless

    Pyrate
    Joined:
    Sep 5, 2016
    Likes Received:
    4,521
    Trophy Points:
    113
    Location:
    London, UK
    WASAPI (Windows Audio Session API) is a modern sound API for Windows- particularly good if you don't have native ASIO drivers for your sound device.. It's a supported "official" Windows API, and it allows bit-perfect playback with optional exclusive device access. It's fairly efficient and light, and lets you avoid all that mixer crap.

    It's practically Windows 7 and later only- but that's no hardship. No-one should be running XP and Vista.. well, is Vista (so it is in theory supported but eh, bets are off).

    Gory details:

    https://msdn.microsoft.com/en-us/library/windows/desktop/dd371455(v=vs.85).aspx

    (Incidentally, the Foobar 2000 guy suggested that people stop using his Kernel Streaming output mode, and switch to WASAPI, if anyone out there is still using that.)
     
    Last edited: Feb 11, 2018
  7. Lyander

    Lyander Official SBAF Equitable Empathizer

    Pyrate Contributor
    Joined:
    Feb 25, 2017
    Likes Received:
    10,961
    Trophy Points:
    113
    Location:
    Philippines, The
    I made the switch from ASIO to WASAPI Event, if only because the visualisations on my foobar screen play smoother on WASAPI. I was rather proud of my being able to discern ASIO from KS and WMD a couple years ago, but I guess that's about the minimum required from critical listening round here? :p

    (Also it was a sighted test, so yeah.)

    Anyway, many thanks @atomicbob! Again, your content proves immensely educational for us laypersons. Hope you don't mind I shared a link to this on reddit, if only to spark some necessary conversations over there.
     
  8. Elnrik

    Elnrik Super Friendly

    Pyrate
    Joined:
    Jan 10, 2017
    Likes Received:
    8,973
    Trophy Points:
    113
    Location:
    Denver CO
    Home Page:
    I have to say it... I about fell out of my chair laughing.

    Anyway, this is why we love @atomicbob. Kill'n fools with facts. No mercy. Sweep the legs.

    I hope this gets over to that Amir dude on HF. I'd love to see egg on that face.
     
  9. obsiCO

    obsiCO Thai Fish Experiment Gone Wrong

    Pyrate
    Joined:
    Sep 18, 2017
    Likes Received:
    634
    Trophy Points:
    93
    Location:
    Bangkok, Thailand
    Thank you for the knowledge bomb, AromaticBomb! Educational is an understatement. Just the kind of content I lurk SBAF for. Nice to know about DirectSound as well, didn't know there's quite a lot in the way -- not so direct now, is it?

    I hope Mr.A (or N?) is reading this, should give him a good jolt.

    Me too. Hard not to steal. :D Sorry @Kattefjaes
     
  10. MWSVette

    MWSVette Almost "Made"

    Joined:
    Jan 3, 2016
    Likes Received:
    22
    Trophy Points:
    53
    Location:
    Orlando, FL
  11. atomicbob

    atomicbob dScope Yoda

    Pyrate BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    18,652
    Trophy Points:
    113
    Location:
    On planet
    Now go listen to Tom Waits - Step Right Up - for perspective.
     
  12. insidious meme

    insidious meme Ambivalent Kumquat

    Pyrate Contributor
    Joined:
    Sep 26, 2015
    Likes Received:
    4,999
    Trophy Points:
    113
    Location:
    Sector 8023 of the Third Quadrant
    [​IMG]
     
    Last edited: Feb 14, 2018
  13. Kattefjaes

    Kattefjaes Mostly Harmless

    Pyrate
    Joined:
    Sep 5, 2016
    Likes Received:
    4,521
    Trophy Points:
    113
    Location:
    London, UK
    Yep, event mode in particular is super-efficient. It lets the actual driver "pull" data from client buffers as it needs it. It's lighter on CPU and often more robust too, as well as generally lower latency. It's good shit.

    If event mode works properly on your drivers (early ones didn't always) then it's a great choice. If you're listening to music, consider exclusive mode too, and you're all set.
     
  14. Vtory

    Vtory Audiophile™

    Pyrate MZR
    Joined:
    Sep 12, 2016
    Likes Received:
    10,831
    Trophy Points:
    113
    Location:
    East Coast
    Thanks for your great job @atomicbob !

    I've always assumed that quantitative reviewers at least measure and compare products under the same settings and configurations (aka fair comparison). What a naive assumption it can be!
     
  15. Lyander

    Lyander Official SBAF Equitable Empathizer

    Pyrate Contributor
    Joined:
    Feb 25, 2017
    Likes Received:
    10,961
    Trophy Points:
    113
    Location:
    Philippines, The
    @Kattefjaes exclusive mode as in hog mode/renders all other system audio silent during playback? Yep, been running that since I got back into the audio hobby proper a couple years back, haha. It does make it easy to miss critically important IMs, but that's a tradeoff I'm willing to make :p

    ... Or I could get a stronger PC that doesn't choke on ASIO. Eh, I'd rather get audio gear. Or pens and ink.

    P.S.
    Mshenay also seems to be an AnhedonicMob fan. Props to him standing up to the influx of populists on reddit.
     
  16. Kattefjaes

    Kattefjaes Mostly Harmless

    Pyrate
    Joined:
    Sep 5, 2016
    Likes Received:
    4,521
    Trophy Points:
    113
    Location:
    London, UK
    It's not exclusive for efficiency, but quality. It stops anything buggering about and interrupting/changing the DAC sample rate etc.. The last thing you want when you're deep in the music is Facebook going "bingle bongle" too :)

    Anyway, you don't really need to use ASIO in this situation. WASAPI has your back. ASIO is an older low latency driver protocol cooked up by Steinberg, which was invaluable in talking to pro audio gear without Windows getting in the way- as there was no sensible Windows sound API to achieve this. Lots of things support it, and it's a safe choice in the pro audio world, but there's really no reason to stress if you're just listening to music and your WASAPI is working well.
     
    Last edited: Feb 11, 2018
  17. JohnnyCanuck

    JohnnyCanuck Acquaintance

    Joined:
    Oct 24, 2015
    Likes Received:
    29
    Trophy Points:
    18
    Location:
    Ottawa, Canada
  18. Taverius

    Taverius Smells like sausages

    Pyrate
    Joined:
    Dec 27, 2017
    Likes Received:
    3,026
    Trophy Points:
    113
    Location:
    Rapallo, Italy
    That is annoying, but if you ever need to hear other stuff you can always stick a good resampler at the end of the audio chain, like SoX.

    During casual listening, there's no noticeable difference, and then the dac only has to deal with one rate.

    I'd love to know what it is that's making all that noise though.

    I don't agree with why they did kmixer the way they did - there really isn't a need to worry about the cpu hit from resampling multiple streams, even at the base win7+ specs, which seems to be their overriding concern - but it mostly appears competently coded.

    I wonder - and will likely never know - what design brief ended up making those graphs.
     
  19. Elnrik

    Elnrik Super Friendly

    Pyrate
    Joined:
    Jan 10, 2017
    Likes Received:
    8,973
    Trophy Points:
    113
    Location:
    Denver CO
    Home Page:
    Looks like the head-fi censors are in high alert on @baldr blog. I've seen several references to bob's tutorial be removed in the last hour, including my own.

    It's as if head fi were part of Amir's attempt to besmirch Mike and Schiit's good name.
     
  20. JohnnyCanuck

    JohnnyCanuck Acquaintance

    Joined:
    Oct 24, 2015
    Likes Received:
    29
    Trophy Points:
    18
    Location:
    Ottawa, Canada
    Mine, too. It used to be ok to say "the site that cannot be named" but that appears to be unacceptable now. :(
     

Share This Page