Audio Science Review Review

Discussion in 'Audio Science' started by purr1n, Aug 30, 2020.

Thread Status:
Not open for further replies.
  1. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    A while back, a few members suggested that we do a review of Audio Science Review. I didn't think it was a good idea back then because I wanted to avoid drama. Now I think it's about time.

    bs.jpg

    One thing I wanted to get out of the way is that I don't have a problem with people who want to use measurements as a yardstick in determining what audio gear to buy. It's extremely confusing out there trying to separate the wheat from the chaff. Heck, I've experienced gear that costs thousands of dollars, recommended by someone at HF, that sounded like poop. I've bought gear based on recommendations from HF randos (or respected HF'ers who I've learned over time not to trust) and had my heart sink after realized the thousands of dollars I just spent had been wasted. And seriously, I wouldn't blame people at all for not understanding what the heck we mean by "microdynamics" or "incisiveness". Utilizing the measurement method for buying gear is a legitimate method. (However, this doesn't mean other methods are not legit - thinking so would be faulty logic).

    The problem with the measurements based approach is when it gets taken to far. As I've always liked to say, not too loose and not too tight. HF is too loose. Audio Science Review is too tight, especially when they take things to an extreme where a few measurements (at best) and a single number SINAD (at worse) is used as purity test of sorts. If it were just the measurements left as they are, I could care less. The problem arises when there is the following:
    1. Someone who insists on providing color commentary "not recommended, sort of recommended, and highly recommended, etc." This is effect creating a purity test. Also, any time there is a human involved, objectivity is lost.
    2. Changing the goal posts depending on whether the manufacturer is favored or not favored, or using voltage levels not in proper context of the gear being used.
    3. The measurements become something purely for the sake of measurements, to the point where better doesn't matter. And what would otherwise be good measurements under normal circumstances just a few years ago suddenly become bad.
    4. When a limited set of steady-state measurements are implied to be the end-all be-all of sound, without any correlation of what we may or not hear. Focus on easily accessible numbers, rather than looking at behaviors or the whole.
    5. Not taking into consideration the intent of the design. (For example, tube amps are often used by audiophiles for intentional coloration - these are automatically dismissed).
    In the next few days, I will show cases of why ASR is "full of crap" at best, and disingenuous at worst. In other words, I'm calling bullshit on stuff.

     
    • Like Like x 80
    • Epic Epic x 2
    • List
    Last edited: May 8, 2021
  2. dark_energy

    dark_energy Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    1,040
    Trophy Points:
    93
    Location:
    Forest
    I guess people could learn a lot from this. Sometimes the why and how are quite complex so to get into it you need to see the details and the bigger picture as well. Seeing the patterns and focusing on the right things is key.

    I am not concerned about AVR, at all ,but the problem is.... Many times in life some important things are hidden behind some "magic" wall so things could be quite a lot easier if someone told the truth. It seems to me that transparency makes the modern world work better. (P.S not saying everything should be open and free)
     
    Last edited: Aug 30, 2020
  3. gixxerwimp

    gixxerwimp Professional tricycle rider

    Pyrate
    Joined:
    Sep 30, 2015
    Likes Received:
    5,773
    Trophy Points:
    113
    Location:
    small island claimed by China
    Would be happy to do a proofread* of the final version before public release, as I'm sure it'll be highly scrutinized and it would be better to keep the focus on the content and away from typos.

    *It's part of my day job.


    edit: It's public now, so post irrelevant.
     
    Last edited: Aug 30, 2020
  4. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    CASE 1: Unheard distortion from an amp with "high" distortion

    First off, I am going to start off with two amps reviewed by ASR, the Behringer A500 and NAD T758, which scored poorly. I'm not here to say whether these amps sound good or not, but I do want to establish are both of these amps were panned at ASR. Their measurements (0.05% THDN at around 5W into 4-ohms) and snippets of the color commentary are provided below. Just hold your horses for now because what I am veering at will come in the next few posts.

    Behringer A500
    https://www.audiosciencereview.com/...easurements-of-behringer-a500-amplifier.5070/
    upload_2020-8-30_12-6-18.png

    "Given the high distortion levels and somewhat uneven frequency response, I can't recommend the Behringer 500. If you just want loud amplification, then you do have a bargain here."


    NAD T758
    https://www.audiosciencereview.com/...-t758-v3-avr.8912/#lg=attachment32844&slide=0
    [​IMG]

    "Alas, objective performance is quite shameful. It appears that no testing was performed for the digital processing or even simple distortion measurements. As much as I hate to say it given the good looks and feeling of the NAD T758 V3, I cannot recommend it."
     
    Last edited: Aug 30, 2020
  5. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    So let's take a look at this power amp's measurement. I'm using the same 5W into 4-ohm load that ASR uses. Take a look at the distortion:

    Mystery Product
    1kHz 5W into 4-ohms
    upload_2020-8-30_12-37-16.png

    Here is another look with the signal normalized to 0dbFS
    upload_2020-8-30_12-37-49.png

    Mystery Product
    dBu vs THDN
    (The x-axis is the output, subtract 3.7dbU to obtain dBU at the output)
    upload_2020-8-30_12-44-48.png

    Note: even in its sweet spot, SINAD is no better than 55db! That puts it right here:
    upload_2020-8-30_13-42-45.png

    What would you think this measurement piece (or a very similar variant of it) was used in a product with the following comments on ASR (some items redacted to remove identity of the product):

    Subjective Listening
    XXX It gives a smooth, very realistic sound impression, beating many "hi-fi" XXX ... And it runs out of power in a XXX

    Review Conclusions
    It is hard to believe such great performance can be had for so little money in the form of XXX. We get it because of research and measurements like I have been showing that resulted in the XXX. So I can definitely recommend the XXX on what good sound is like, and correlation between that and the measurements I am showing.

    What "correlation" is this which ASR speaks of? The power amplifier is absolutely shit by Audio Science Review standards? How can such a product be recommended?

    I'll let you guys guess.
     
    Last edited: Aug 30, 2020
  6. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Here's a hint:

    IMG_20200830_111140.jpg
     
  7. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    OK, I'll admit that this is a slightly different model, the 306 whereas Amir reviewed the 305P. However, the boards are the same (JBL pushed up the power 6W to 50W from 44W on the 306), maybe with a higher spec STA amp chip.

    Here are ASR comments in full:

    Subjective Listening
    The JBL 305P Mark ii is a delightful little speaker. It gives a smooth, very realistic sound impression, beating many "hi-fi" speakers. Problems are two fold: there is a tweeter hiss that you can hear in near-field applications. And it runs out of power in a large room and high volumes. Not an issue if used as studio monitors.

    Review Conclusions
    It is hard to believe such great performance can be had for so little money in the form of JBL 305P Mark ii. We get it because of research and measurements like I have been showing that resulted in the advanced waveguide that it has for the tweeter. So I can definitely recommend the 305P Mark ii. Get a pair so that you can get calibrated on what good sound is like, and correlation between that and the measurements I am showing.

    So evidently, ASR feels that the 305P (which I had in the house and loved, but happily gave them away to an video editor in need) possess the following subjective qualities:
    • delightful
    • smooth
    • very realistic
    • beating many "hi-fi" speakers
    Now I am wondering how ASR can even subjectively love this JBL self-powered monitor given how BADLY the amp measures using the same criteria that ASR uses.

    Other concerns:
    • Also, given how much emphasis that ASR puts upon SINAD or THD+N, why is the "tweeter hiss" not even a consideration? Imagine if PS Audio or Schiit had produced a product with this much audible hiss!
    • Does noise all of a sudden not matter when it comes out of the damn speaker? And only matter when indicated on an amplifier SNR measurement in isolation to itself?
    • Given how easily ASR condemns the smallest bit of distortion in amps, how it is possible that the distortion from this amp was not heard? Perhaps his trained ears aren't as trained as he thinks. Or perhaps there is something out of context.
    • To the above point, where is the correlation ("and correlation between that and the measurements" of bad amp measurements to good sound quality?

    P.S. I happen to love the cheap JBL monitors, their waveguide tech, and happen think they punch well above their price level. One may ask me, how you can possibly love them given their high distortion? The answer is: well, I like measurements and all; but i was never a religious measurement extremist!
     
    Last edited: Aug 30, 2020
  8. dmckean44

    dmckean44 In a Sherwood S6040CP relationship

    Pyrate
    Joined:
    Nov 17, 2016
    Likes Received:
    1,425
    Trophy Points:
    93
    Location:
    Peoria, IL
    This is what happens when you focus on measurements of individual components rather than measuring the system as a whole. The drivers in the loudspeakers will have THD a couple orders of magnitude higher than even the worst measuring amplifier. How everything sounds together ends up being way more important.
     
  9. Cspirou

    Cspirou They call me Sparky

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    8,200
    Trophy Points:
    113
    Location:
    Northwest France
    At some point ASR could have been ignored, but they've become too influential in the audiophile world. Search results list ASR links at the top and many audio groups I am a part of regularly link to ASR as an authority. Even stores like Audiophonics list some ASR award as a stamp of approval.

    They are a worthy target
     
  10. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    THD+N isn't additive in a strict sense. The math behind it can be discussed at another time.

    The distortion of each component in the chain does adds up, but not as much as we think. It's always going to be the most distorted component, the weakest link in the chain, which contributes the most distortion, and by nature also masking the components with a lesser amount of distortion.

    The reason we don't hear the crappiness, or most of the crappiness from the STA amplifier chip in the LSR306, is because the speaker drivers are probably responsible for most of the distortion at higher levels. And at whisper levels, there is the hiss from the amp!

    We'll move on to case study #2 maybe later tonight :)
     
    Last edited: Aug 30, 2020
  11. skem

    skem Friend

    Pyrate
    Joined:
    Nov 1, 2017
    Likes Received:
    1,911
    Trophy Points:
    93
    Location:
    Charles River
    [​IMG]

    I was always amused that, despite Amir’s SINAD proselytizing, in private he used a Mark Levinson 360S DAC for 21 years until January of this year. What happened? He finally measured it and, according to Amir’s own measurements, the 360S is worse than the MEIZU dongle:

    [​IMG]

    I ditched my Levinson 360 back in 2012 because it was clearly getting long in the tooth a decade ago, but I sure as shit didn’t replace it with a dingleberry DAClet.

    It begs the question: why after testing nearly one hundred other DACs that all measured better than the 360S did he not, time and again, feel compelled to rip the guts out of his main system?

    There can be only one explanation: he cannot hear the difference.

    In January, when he finally did measure his nice Levinson—a DAC lovingly engineered using Burr Brown chips by Harmon’s elite team—and discovered it “sucked”, he decided to replace it with a ChFi, MQA-enabled Sabre DAC that has the analog design sense of a Motorola flip phone. I hope he enjoys it. And he probably does... just as much as he would a dingle dongle.
     
    Last edited: Aug 30, 2020
  12. Luckbad

    Luckbad Traded in a unicorn for a Corolla

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    3,408
    Trophy Points:
    113
    Location:
    Holly Springs, NC
    If this goes public, we should get you the matching pair of JBLs to measure or people will say that the 306 is clearly worse than the 305.
     
  13. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,130
    Trophy Points:
    113
    Location:
    India
    An aside...
    Just... Whatever you do, please don't let mr A back in. Natural justice? Giving the man a chance to respond? Bugger all that. There is no end to a conversation with A: let's not have one.

    If he wants in, the message should be like the kids used to say at school: we're talking about you, not to you.
     
  14. insidious meme

    insidious meme Ambivalent Kumquat

    Pyrate Contributor
    Joined:
    Sep 26, 2015
    Likes Received:
    4,999
    Trophy Points:
    113
    Location:
    Sector 8023 of the Third Quadrant
    It's not like he doesn't have a forum to fashion his own reply.

    EDIT: And it's already started https://www.audiosciencereview.com/...t-ps-audio-now-sbaf-more-smack-talking.15686/
     
  15. HotRatSalad

    HotRatSalad Friend

    Pyrate
    Joined:
    May 4, 2018
    Likes Received:
    1,684
    Trophy Points:
    93
    Location:
    NH
    Yeah I don't know why anyone acknowledges them.
    I do this to some of my bosses at work that are pieces of shit. I don't even acknowledge them lol.
     
  16. yotacowboy

    yotacowboy McRibs Kind of Guy

    Pyrate Contributor
    Joined:
    Feb 23, 2016
    Likes Received:
    10,682
    Trophy Points:
    113
    Location:
    NOVA
    Home Page:
    Here's what drives me bonkers. ASR measures a speaker with his fancy spinomatic royale with cheese. Predicted in room response looks pretty good with maybe a dB or two deviation from a nice falling response. No impedance wiggles, and no weird stuff going on in the CSDs. Sure, there's a bunch of distortion in the bass when playing 96dB test tones through a miniature bookshelf speaker, but how is that shocking to anyone?

    Final verdict?

    "I can't recommend a speaker that doesn't sound good to me and that is where I stand with SVS Ultra. I suspect measurement score will be good, making me look bad. So be it!"

    So what is it? Measures bad, sounds bad = VINDICATION! Measures good, sounds bad = So be it!

    [​IMG]

    (N.B. - I could give two shits about SVS, so I'm not defending the speaker per se. Just pointing out a major failure of the ASR measurement to "good sound" correlation.)
     
  17. YMO

    YMO Chief Fun Officer

    Pyrate Contributor
    Joined:
    Apr 1, 2018
    Likes Received:
    10,518
    Trophy Points:
    113
    Location:
    Palms Of The Coasts, FL
    I never seen these many users in one thread at once.

    [​IMG]
     
  18. wadec22

    wadec22 Almost "Made"

    Contributor
    Joined:
    Sep 28, 2018
    Likes Received:
    418
    Trophy Points:
    63
    Location:
    USA
    this is sorely needed. evaluations that are put out there leading readers to believe are scientific in nature require; consistency, discipline and context. all of these are lacking there.

    I find gear-review to be a great resource and many folks do it different ways but the way it is presented there shows extreme bias and has really opened the door for some Chinese-based companies to exploit the intentions there and create a demand for a sub-par product. very obvious generalization there, but you get what i mean.

    We end up with products that test off the charts in areas that don't improve the listener experience and then the user spends more on it but loses out on functionality and aesthetics.... sucks...

    people are led to believe this is a science based rating system when it's closer to a guy drinking wine in his man cave testing what products he wants, changing parameters all the time and only testing specific features he is interested in (often not what the product is for). very frustrating....
     
  19. perogie

    perogie Facebook Friend

    Joined:
    Aug 16, 2020
    Likes Received:
    153
    Trophy Points:
    43
    Location:
    Ontario
    This is similar to the cult like following that can occur regarding the end all be all importance of the Harmon curve.

    It's unfortunate that they have become the influence they have. Handled correctly it could be a great and resourceful site.
     
  20. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Here are the 305 and 306 boards. Maybe JBL worked on magic on the newer 306's boards to make measure worse? I should be easy enough to find out.
    lsr305-1.jpg
    IMG_20200830_123558.jpg
     
Thread Status:
Not open for further replies.

Share This Page