How should I A/B test HP amps?

I’m wanting to A/B test two headphone amps from a single DAC.

The two ways I thought about doing this are:
1) Use a switcher between the dac and the amps and have a 2 male to 1 female 3.5mm splitter plugged into the output of both amps. This way my headphone is plugged into both amps, but only one amp is getting a signal passed to it via the switcher.

2) Use RCA splitters from the DAC to feed both amps and have a 3.5mm switcher on the output of both amps. This way the amps are both amplifying the signal to their output, but the outputs to the headphone are switched between them.

Would there not be a difference between these two setups? Could a 3.5mm splitter cable damage the outputs of both amps, even if only one is being passed an audio signal?

 

Listen to one, with music you have heard a billion times, then unhook and listen to the other one. Level match as best you can.

 

Got the 3.5mm switcher today, had a few of my audiophile buddies come over, and proceeded to question our reality for the rest of the day.

I'll start off by describing my A/B testing setup. I played flac to my Modi Multibit via toslink, which fed two headphone amps at the same time with RCA splitters. Both amps' outputs fed a MC102 3.5mm switcher that switched between each output to feed my HD58X headphones. I used a Fluke 117 multimeter to level match each amp to 0.08v while playing a 400Hz test tone with the headphones connected.

A variety of genres were tested such a pop, electronic, country, and classical. I sat with my eyes closed while my friend clicked the button to switch the inputs to eliminate bias, so I had no idea which amp was which and I was not picking up visual queues from his body language.

Amplifiers tested were the Schiit Jotunheim, JDS Atom, JDS O2, Massdrop O2, and Schiit Vali 2 (with 6BZ7 tube and Amperex ECC88/6DJ8 Orange Globe Logo O-Getter tube warmed up).

Comparison Results Prior to the A/B Setup
My initial impressions with each amp when I was comparing them by only unplugging my headphones from one amp and plugging them into the other as fast as possible were:
Atom sounded more bright/trebly than the rest
JDS O2 sounded more full/slightly more textured than the Jotunheim
Vali 2 sounded more textured than the Jotunheim
Orange Globe tube sounded better than the 6BZ7

Results After the A/B Setup
All the amplifiers sounded the exact same. Tube amp and all. Same quality. Same soundstage. I could not tell when my friend was pressing the button besides when there was a very faint pop sometimes when the signal was switched. I had to open my eyes to make sure he was pressing the button to switch them because I didn't believe it. My friends had the same results, they couldn't hear any differences. I even unplugged one cable from each amp to ensure the switcher was actually working. The tube amp is what shocked me the most. There was zero perceived difference between every amp tested.

 

Hey, at least you won't have to worry about tubes now, or getting better gear ┐(￣ヘ￣)┌

Messing about aside (just found out my phone keyboard had built-in shruggies, had to test it out), the proper way of going about testing gear is... living with it. There's a lot to be said for blind A/B testing, both good and bad (placebo, nocebo, confirmation bias, fatigue), and while I'm not inclined to think you botched the test process or that you're being disingenuous it is genuinely interesting to hear that there was no audible differences between the amps you had lined up. Could just be my crap listening skills but it sometimes takes me a while to discern qualities inherent to gear.

But again, I wouldn't worry about it. Go with what your gut says and don't get caught up in gear nervosa.

 

Amazing, sell all your shit and buy an iPod

 

I totally agree. Many a times subtle differences in an amp or a DAC is not noticeable upfront and when we spend some quality time with it that we start noticing the subtle variance, musicality that it brings to the table

 

I stopped A/B ing a while ago. @purr1n is right. It never really does work well. Especially if it’s towards the end of a long meet, or it’s 3 A.M. . Or it’s 3 A.M. and different scotches have been involved. For me it’s a bit like cars. When I drive one that is new to me, it mostly is great. But if I drive it for a week, I get a much better sense of its weaknesses and strengths. There was a meet a couple of years ago where I brought my BW 1 and Craig had an EC Af there two tables over. Listened to the BW with my HD600’s. Quickly ran over to the Af. The two sounded the same. I didn’t know what to say to Craig other than “sounds good”. Loaners are like that in a way. The first night I’m in love. Really great. After a couple of days the magic is gone...sometimes. Well, a lot of times.

 

Quick A to B switching is just going to confuse your brain like purr1n said. And when the brain is confused, the logical conclusion is that every switch muddles together and you can't tell anything apart, and so everything sounds the same.

Let things warm up, and spend a half hour with each at a time. Don't focus on analyzing things, just enjoy the music and you'll notice the differences more easily. For me the less I focus on trying to hear new things or different things, and just sit back and relax, the more the differences appear obvious. I realize this post sounds like audiophool crock a shit but it's legit. I've "downgraded" my whole setup before, both headphones and speakers, and after a week or two been left wanting more. I think for a lot of people that's a required step in the audio journey to realize that folks in the HF sound science forums are full of shit.

Take your time, trust your ears, try shit out. If your 58X out of an iPod still rocks your world after a week, let us know. I'd bet good money that it wont.

 

He's not talking about "an iPod," though, he's talking about a handful of low/mid-priced amps. I bet if he just uses the Vali or just the Jotunheim, it will in fact be totally fine.

As for dismissing the whole idea of blind testing... man, you must experience audio stuff a lot differently than I do. Whenever I try to compare two pieces of gear in a sighted, just-unplug-it-and-plug-it-into-the-other-one casual way, it always goes like:

1. Plug the headphones into the new one: "Holy shit, this is completely different, I can hear [x] in this recording that I've never heard before, it's so much richer/brighter/whatever."

2. Plug it back into the old one: "Okay, no, wait, all that stuff is here, too. Maybe it doesn't actually sound that different, after all?"

3. Plug it back into the new one: "Yeah, okay, I can't tell any difference at all."

Comparing headphones, yeah, I can tell differences there, always and easily; but comparing amps or DACs, my perceptions always waver between being certain I hear a difference and then being totally unable to hear it. Which means that whether there is a difference or there isn't, my judgment is completely wrong at some point in the process. So the idea of actually having a systematic, eliminate-all-the-variables approach to the comparison seems not at all ridiculous.


(And the "live with it long term" thing never really works for me, because long-term, whatever I'm listening to (if it doesn't have egregious flaws) just sounds... normal. Like, long-term, I even lose some of the difference between headphones -- when I had my Aeon Flow Closed at work and LCD-X at home, I had managed to convince myself that they basically sounded almost the same; bringing the AFC home and listening to them next to each other, it was immediately obvious that they weren't. Short term comparisons are always much more informative to me than long-term live-with-its.)

 

My point is that if you compared low/mid priced amps and found no significant difference then you likely won't find an iPod to sound any different either. The reason for that is because your comparisons done in a hurry aren't worth shit. Go take a look at the HF sound science forums, you'll find plenty of convincing arguments to ditch your DAC/AMP combo and get an iPod because they "sound the same".

What matters, in my opinion, when comparing is how different things tend to sound after you've had time with a piece of gear. Short term comparisons can be useful if you're experienced and know what to listen for, but ultimately unless you have a ton of experience with gear, a week down the road your opinion on things is likely to change radically and trusting initial reactions isn't worth squat. The vast majority of listeners are not experienced enough for that.

Just like your example above, comparing amps and DACs, you're expecting to hear differences in what's there vs what's not, which is really besides the point. Unless you're comparing really crappy gear, you're not going to hear things in the same recording you didn't hear before, you're going to hear differences in presentation, in detail, you won't suddenly hear an instrument on one DAC you can't hear on the other, but you might hear things presented as if you're sitting closer or further away for example, or variations in note decay, in slam/dynamics, nuances in voice vibrato, etc. Everything you hear on one DAC is still there on the other, it's just rendered differently, with more or less difference depending on what gear you're comparing.

Short term comparisons are good for one thing - deciding if you like how a piece of gear sounds better than another, and for the vast majority of people this is only useful with headphones/speakers where differences are obvious. Comparing amps and especially DACs, unless you have plenty of time with them, and can throw a wide range of music at them, you won't be able to tell anything apart most of the time. After an hour or two though, you'll start to notice that for example, a bright DS DAC will be more fatiguing than a darker sounding NOS DAC. If that matters little to you, then great! Stop wasting your money, get an iPod and headphones you like, and start listening to music more.

Point is, if you want to do a proper comparison of several amps, spending a couple of hours ABing them blindly with 50 different songs isn't likely to be anything more than a waste of your time.

 

@elmoe I wish I could express myself as well as you sir.
His reply is not an insult. Just his opinion.

 

I will say that catching new details listening on my main desktop rig makes it more easily audible listening to the same off my phone (same headphones, yay for efficient cans). While that isn't always the case especially with low-level detail, knowing something's there makes it easier to discern just because I'm now aware of its existence, not necessarily looking for it.

Also, I listen at much lower volumes on my phone. Desktop rig has a bit too much gain and just for nervosa purposes I don't always use DVC— I'm listening maybe +5dB more than on phone because I run into channel imbalances any lower on the pot (yes this could merely illustrate how crap and unresolving my upstream is relative to a smartphone)

 

After so many years in this hobby, I seriously miss being "ignorant". I miss the days when an ibasso amp was enough to drive every headphone I had (and was unnecessary for half of them). I miss the days when I had to stretch to even hear the difference between compressed and uncompressed. I miss having the proverbial wool in my ears so I wouldn't obsess over having the best or newest toys. I miss not being able to tell the difference between dacs, which despite my ears telling me otherwise my brain tells me it's still lunacy. I miss being able to just put on music and listen, and not get angry at the speakers, or the amp, or whatever other component in between. I miss being the days when I could just listen when friends put on music in the background and not feel snooty that I have a better recording than they do.

If you can't hear the difference, spend your money on new transducers for fun, or a home improvement project. Go spend it on a date, go see live music, maybe at the same time. We're all nutjobs here.

 

^ this

The iPods and the apple dongles are pretty well engineered devices. It’s no insult to “just get an iPod”. The lightning and usb c dongles measure very well and clean. It helps that apple’s R&D budget for their accessories is more than the GDP of many countries.

 

Agree with @bixby and @purr1n .
I am about as far as an expert as you can get but after having lived with a Phonitor XE, Cavalli AUX and Heed Canalot PSU for a couple of years now, I am pretty certain I could easily pick them out in a blind test and define to an extant the varying qualities of each.