Merv's Heptagons and Presentation Techniques

Discussion in 'Blind Testing and Psychoacoustics' started by Boops, Dec 1, 2015.

  1. spoony

    spoony Spooky

    Friend
    Joined:
    Sep 28, 2015
    Likes Received:
    241
    Dislikes Received:
    2
    Trophy Points:
    43
    I've found the graphs really helpful in the context of a comparison, and in that sense they should be not be offered separately. Being able to highlight the areas like in the d3.js examples is just an improvement over the static presentation IMO. Otherwise the charts are serving their purpose.
     
  2. Jeb

    Jeb Friend

    Friend
    Joined:
    Sep 28, 2015
    Likes Received:
    419
    Dislikes Received:
    0
    Trophy Points:
    63
    Location:
    Bristol, UK
    I think the charts are a great visual representation of every headphone ultimately being a mixture of weaknesses, successes and compromises.

    Obviously there's no substitute for a listen but I think the charts are a really nice antidote to the finding the "Perfect Headphone" mentality (of which I have been guilty). In that respect, I find the charts to be very grounding.

    I don't think anyone half-sensible believes they tell the whole story or every nuance. They're a very useful visual accompaniment to a more complex story.
     
    MoatsArt likes this.
  3. keanex

    keanex Martian Bounty Hunter - Friend

    Friend
    Joined:
    Oct 1, 2015
    Likes Received:
    315
    Dislikes Received:
    0
    Trophy Points:
    63
    I love spider charts and I might consider using them in my reviews. I've always been a sucker for them but my concern is figuring out what to put on them and possibly having conflicting ratings over time. Decisions decisions...
     
  4. spoony

    spoony Spooky

    Friend
    Joined:
    Sep 28, 2015
    Likes Received:
    241
    Dislikes Received:
    2
    Trophy Points:
    43
    That's why you don't want to make a cummulative global rating system ala |joker|, he had to adjust his scores several times to accomodate his experience listening to more gear over time. Comparative subjective charts should remain isolated in the context of their creation so they don't expire.
     
  5. keanex

    keanex Martian Bounty Hunter - Friend

    Friend
    Joined:
    Oct 1, 2015
    Likes Received:
    315
    Dislikes Received:
    0
    Trophy Points:
    63
    To understand you better, are you to mean that each graph would be taken as it's own without comparative context to another?

    I just wrote out some ideas for doing it and came with 12 areas to rate, which assuredly is too much. I have a hard time narrowing things down, so I'll likely just scrap the idea and keep to words.
     
  6. Claritas

    Claritas Friend

    Friend
    Joined:
    Sep 26, 2015
    Likes Received:
    521
    Dislikes Received:
    16
    Trophy Points:
    93
    Location:
    R'lyeh, Carolina
    The radar charts are fun to look at, but comparative number rankings are fairly uninformative to me. (I suspect they're also unintentionally misleading.) Maybe using a scale of 1 to 5 would be better--or worse; I really don't know. Either way, I learn a lot more from descriptions and helpful similies.
     
    MoatsArt likes this.
  7. spoony

    spoony Spooky

    Friend
    Joined:
    Sep 28, 2015
    Likes Received:
    241
    Dislikes Received:
    2
    Trophy Points:
    43
    Nope, the opposite; if you performed a comparison session with headphones A, B, C, D, the chart should be produced, displayed and most importantly understood in the context of the session, with no absolute ratings. The chart of a single headphone should not be taken in isolation or compared to the charts of headphones belonging to other sessions unless you already know some of the gear and the language of the reviewer. I'd even go as far as to normalize the charts and get rid of the numbers.
     
  8. keanex

    keanex Martian Bounty Hunter - Friend

    Friend
    Joined:
    Oct 1, 2015
    Likes Received:
    315
    Dislikes Received:
    0
    Trophy Points:
    63
    Oh okay, so assuming I understand you, I would potentially do a comparison between say the HD600 vs the R70x vs the HE400s and rate them in regards to each other? But the chart would hold no relevance to one with three different headphones?

    Basically if I had a Blackwood, cherry and walnut omni a chart might be useful in that context?

    Sorry again if I'm misunderstanding you. Sometimes I'm slow to understand concepts.
     
  9. purr1n

    purr1n Finding his inner redneck

    Staff Member Friend BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    45,254
    Dislikes Received:
    70
    Trophy Points:
    113
    Location:
    Antarctica
    Yup, each graph set is self contained and not comparable to other sets in the past or future.

    Set #1: HD650, Ether, HE-560
    Set #2: HD650, Omni, X00, Nighthawk

    The scores within a set can be compared to each other, but they cannot be compared across different sets.

    The key is not to be hasty and sit on it for a while. The sonic aspects that chose to use were influenced by various people here (Sorrodje, Hands, Negura, yourself, etc.) over time. I wanted to communicate in established lingo. At some point, it all came together in head and I decided to go for it; and even then I can see the sonic aspects undergoing revision. You may have noticed that the sonic aspects were slightly changed between the DAC/amp and headphone reviews. You should give it a shot, if at least only for yourself or a few others to see if they work well. My only advice is to keep it simple and resist the urge to increase comparison size.
     
    Last edited: Apr 1, 2016
    keanex likes this.
  10. spoony

    spoony Spooky

    Friend
    Joined:
    Sep 28, 2015
    Likes Received:
    241
    Dislikes Received:
    2
    Trophy Points:
    43
    Yes, it would hold the most relevance in the context of the comparison session, which encompasses the source gear, amplifier, headphones, music selection, and personal biases like sound signature preference. I would add the comfort category to any chart.

    It would be most useful in that context, yes. But it doesn't mean that the reader cannot extrapolate based on his readings of the reviewer and what he interprets from the results of multiple comparison sessions.

    No need to apologize, I may have not been clear in the first place. This is all, of course, IMO.
     
    keanex and Claritas like this.
  11. Boops

    Boops Friend

    Friend
    Joined:
    Nov 8, 2015
    Likes Received:
    1,385
    Dislikes Received:
    2
    Trophy Points:
    93
    Location:
    New York
    It could work to change the numbered scale to words:
    • Bad
    • OK
    • Good
    • Best
    You would lose some of the nuance and the web shapes would be less varied, but might be worth experimenting with. Although at some point, the spider chart stops being the right format as the rating scale gets coarser.
     
  12. keanex

    keanex Martian Bounty Hunter - Friend

    Friend
    Joined:
    Oct 1, 2015
    Likes Received:
    315
    Dislikes Received:
    0
    Trophy Points:
    63
    Thanks purrin, I will take it into advisement. I may shoot you a PM and work out my own parameters and thoughts; as long as it is not a bother. I do love spider charts though so the temptation is huge for me.

    Thanks a lot for your input, it's super valuable.

    Good points, and now I understand why it's a fine line to walk with caution in regards to warning users for future use.

    No no, I think you were fine. I misunderstood you likely due to working and trying to understand on my phone. Thanks!
     
    Last edited by a moderator: Apr 1, 2016
  13. spoony

    spoony Spooky

    Friend
    Joined:
    Sep 28, 2015
    Likes Received:
    241
    Dislikes Received:
    2
    Trophy Points:
    43
    BTW: I enjoy your reviews and appreciate all the effort you put into making this a better hobby. Same goes for all reviewers who are not shills.
     
    FlySweep and keanex like this.
  14. keanex

    keanex Martian Bounty Hunter - Friend

    Friend
    Joined:
    Oct 1, 2015
    Likes Received:
    315
    Dislikes Received:
    0
    Trophy Points:
    63
    Thanks man, I really appreciate that.
     
  15. Armaegis

    Armaegis Friend

    Friend BWC
    Joined:
    Sep 27, 2015
    Likes Received:
    3,395
    Dislikes Received:
    2
    Trophy Points:
    113
    Location:
    Winnipeg
    The issue with spider-charts for comparisons is that the shape encourages judgement/comparison by area, but those areas are so arbitrary and easily skewed by simply rearranging the axes.
     
    RakiRaki, ultrabike, Thujone and 4 others like this.
  16. Boops

    Boops Friend

    Friend
    Joined:
    Nov 8, 2015
    Likes Received:
    1,385
    Dislikes Received:
    2
    Trophy Points:
    93
    Location:
    New York
    This is interesting. I hadn't thought to try rearranging the axes to see if the general shape of the areas holds. I'll give it a try when I have some time and see if you're right.

    @MoatsArt My feeling is the taste/knowledge/bias of the reviewer is always relevant. You can measure THD with a machine, but the spider charts here, despite using numbers, are just a visual representation of the critic's judgements. Trust in the critic is included in the price of admission.
     
    Last edited: Dec 3, 2015
  17. Tyll Hertsens

    Tyll Hertsens Grandpappy of the hobby - Special Friend

    Friend
    Joined:
    Sep 26, 2015
    Likes Received:
    2,809
    Dislikes Received:
    0
    Trophy Points:
    93
    Location:
    Bozeman Montana
    Home Page:
    Personally, I think Marv is a pioneer exploring the future of the hobby; prying open various doors on what's possible.

    Is it dangerous? Yes, sure. Who knows whether his thoughts will trigger productive or destructive ideas...probably both really.

    For example, he asked we not total the scores, but Boops simplification of his charts essentially does just that by making the area covered by the chart apparent.
    But I don't necessarily see that as bad, it's just playing with the features of possible futures.

    I will caution this: As soon as we put numbers on subjective impressions we are producing an illusion. Human experience is VERY difficult to quantify. Using numbers to describe subjective experience produces an illusion of understanding the ignorant people will latch on to and miss use.

    OTOH, the activity of leadership is bound to produce mistakes and unintended consequences, but that's not a good reason to give up the attempt.

    Try.

    It's really a much bigger word than we give credit for.

    So, I applaud the movement here which attempts to explore new ways to understand headphones. We have to be careful and humble in the knowledge that we do so from a place of amateur involvement. We should always question what we're doing; we're not Sennheiser engineers. Hell, knowing that even they produce crap with some degree of regularity should cause us to take pause. My caution would be to not "buy your act" (all of us) and harden your current opinions.

    Certainty is the death of learning. Remain unsure; embrace "undetermined". Strong beliefs are for ignorant fundamentalists.

    But try.

    And, maybe most importantly, have fun. This is a hobby after all.
     
    RakiRaki, n3rdling, TMoney and 18 others like this.
  18. purr1n

    purr1n Finding his inner redneck

    Staff Member Friend BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    45,254
    Dislikes Received:
    70
    Trophy Points:
    113
    Location:
    Antarctica
    Couple of thoughts:
    1. Thanks for the feedback. The results of this discussion will result in a better disclaimer or notice of proper use of these plots in the future.
    2. Instead of simply adding the numbers up, the sonic aspects can be weighted. For example, people like me who place a high premium on reproduction of low level information, granularity with tiny volume level changes, slam, and frequency response can assign the micro, macro, and tonality aspects a higher weight. In order words, personalize the numbers to suite your preferences.
    3. Assign minimum requirements or identify showstopper issues. For example, say that you are sick of the "slow" or "unclear" sounding HD650, and looking for a new headphone that has much faster temporal separation or clarity. There are two spots on the plot that can help you there.
    4. As far as numbers one to give or words to represent the numbers, that's a double edged sword. The use of numbers denotes relative performance without having to say "best" or "crap". The range of one to ten was chosen intentionally to show that most of the headphones or DAC/amps weren't so wide apart in terms of performance. With the exception of the Nano, none of the gear scored less than 6 in their worst attributes. This indicates that the gear being discussed is good or at worst mediocre in only one way or two. I could remove the the numbers and force people to use their eyeballs.
    5. I intend to keep making each heptagon set not comparable to others. This will prevent the creation of a master list a la the 2D DAC chart. The 2D chart was sort of an exception because there were only two-axes and things were still quite nebulous. As as other have said, it can be a pain in the ass to go back and keep revising, especially if the gear is no longer on hand and we have to rely on our memories.
    6. Per Tyll, better to lead from the front than to lead from behind. Someone will do something like this sooner or later.
     
    Last edited: Dec 3, 2015
    drfindley, spoony and keanex like this.
  19. spoony

    spoony Spooky

    Friend
    Joined:
    Sep 28, 2015
    Likes Received:
    241
    Dislikes Received:
    2
    Trophy Points:
    43
    This is great, that's why an interactive chart would be most helpful, I believe. You could allow the viewer to set their own weights and personalize the graphs this way.
     
  20. Thujone

    Thujone Friend

    Friend
    Joined:
    Sep 27, 2015
    Likes Received:
    180
    Dislikes Received:
    0
    Trophy Points:
    43
    Location:
    Kansas
    It would be beneficial to have explanations of the 7 traits used on the chart. Is this information somewhere else?

    In general, I believe I understand what they represent. Couldn't hurt to have an in depth description to show what they represent to you. I may have just missed this information somewhere. If so, apologies.
     

Share This Page