 
|
Audio Asylum Thread Printer Get a view of an entire thread on one page |
For Sale Ads |
|
|
Audio Asylum Thread Printer Get a view of an entire thread on one page |
For Sale Ads |
124.189.36.153
In Reply to: RE: Let me get this out of my system... posted by Ryelands on March 01, 2010 at 02:33:26
*** You're right to argue that Gordon's wav/lossless "blind test" performed in an unknown manner on unknown subjects at a trade show (!) without statistical analysis of the "results" is not robust and that his listening tests are probably more valid. ***
You have hit the nail on the head - what Gordon described is not a statistically valid test. It wasn't "double blind" (which would imply neither the observer nor the experimenter knew what is being tested), and it wasn't ABX, simply AB, hence subject to all sorts of biases.
What I found amusing was the implication that *if* Gordon was right (that is, if differences between formats were audible and that CPU processing capacity was a factor) *then* it indicates that his products are not working effectively (because async USB is *supposed* to isolate such differences away from the DAC). Not sure he intended to convey that implication to his readers.
Follow Ups:
Async does not isolate electrical noise from the computer that is passed down the USB cable. All it does is reduce the jitter between the computer and the DAC. ASYNC is not a magical interface that makes everything in the computer no longer matter. Perhaps you should try some ASYNC DACs on your system and get some first hand experience.Please don't respond to me in Japanese :)
Edits: 03/01/10
Mercman, I do have an async USB DAC in my system and I am well aware that it is not perfect.
I am not sure what you mean by "isolate electrical noise from the computer that is passed down the USB cable".
I have done electrical engineering and have designed my own circuits but I'm still scratching my head on this one.
If you are completely reconstructing the signal timing, which is what an async USB DAC is supposed to do, then any noise from the input should be isolated from the DAC. Unless of course you were such a poor designer that the input signal is causing power rail modulations.
Since you mentioned distortion measurements I'm curious what tools you use...If you don't mind my asking and going OT for a moment. I'd like to do measurements and can't really justify an analog distortion analyzer but would certainly look into low cost PC software/soundcard solutions.
I guess I better leave this to the engineers to explain. But there is high frequency noise generated by the switching power supply that is not related to jitter. But you know all of this.Thanks
Edits: 03/01/10 03/01/10
You mean the switching frequency (generally 50kHz-1MHz)?
That shouldn't affect even a synchronous signal - gets filtered out.
Or are you talking about EMI/RFI? Yes, that could affect the DAC, but Gordon was advocating a higher powered CPU, which generates *more* noise than a low powered one.
A higher CPU doesn't necessarily generate more EMI/RFI. Famk has previously pointed out "The effect depends on many factors including chip design, interaction, board design, power supply decoupling, audio section design etc etc."
Christine, my expereince with my 3 macs support Gordon's findings. If I can't trust my hearing, I better find another hobby.
Of course it depends a lot of factors. That's stating the obvious.
However, higher thermal power plus higher clock frequency = higher RFI. Can't walk away from that one.
But it is also true that modern CPUs automatically underclock themselves when idling - definitely true for Apples (controlled by the AppleIntelCPUPowerManagement.kext kernel extension).
In the end, we don't know unless we measure it. And even then, the impact depends on a whole host of other things.
I don't think Gordon's point was based on RFI - previously he has posted that he believed the differences were due to CPU "spikes" which are more pronounced for slower processors which he claims affects the THD.
Now, that is *exactly* the sort of effect that a good async USB design is supposed to isolate against. I have measured my DAC on slow and fast systems, and can confirm there is no measurable change in THD. Then again, I don't notice any CPU "spikes" on my systems ...
That's why I pointed it's ironic that if what he claims is true, then it does imply that the async USB design is ineffective in preventing exactly the sort of issue it was designed to protect against.
Fmak has previously pointed out "The effect depends on many factors . . .
Fair comment - but not exactly right IIRC. The argument at that time was about whether under-clocking reduced RFI, not about changing to more powerful processors. Besides, some of us felt it was never quite resolved.
Dave,
What if I slip you a $50 ? Would you then agree with me? :)
Steve
What if I slip you a $50 ?
Come, come, I have my principles.
$75 or no deal.
D
PS Or, as Groucho Marx put it:
"You don't like my principles? Well, I have others."
C'mon guys, a fair exchange would be a brand new MacBook, surely? :-)
Slip me the latest greatest model, and I'll happily agree yes, the faster and more powerful, the better the music sounds to me :-)
You're easy to please!!
Actually, I'm much cheaper than that.
A good dinner, plus wine, and I'll probably agree to anything!!!
Oops - probably shouldn't have revealed that tidbit. Okay okay, I need at a minimum a diamond-studded iPhone!!!
When you combine async transmission with optical isolation it's pretty hard to see any mechanism for computer noise to affect the DAC except through the power supplies.
I am aware of the QB-9 design. I just discussed one example of something that influences the sound. I support Gordon's views when dealing with OSX.
If noise is really the issue, then I have trouble understanding the logic behind some of Gordon's recommendations. Let's take the "headroom" argument for example. If more computing power = more headroom = better sound, why stop at a Mac Mini? The Mini is also a curious choice when you consider RFI/EMI issues because everything is packed closely together and the GPU is on the same board. I'd think there's far more opportunity for RFI to pollute the USB connection with such packaging. Ditto for a laptop. If minimizing noise over the USB interface is important, wouldn't it be better to take USB off a daughter card installed in a regular desktop case and run the computer headless controlled via Ethernet?
Even the choice of OS X (or Windows in Cics case) is curious. If having other non-audio processing going on impairs audio playback, wouldn't it be better to use a stripped down embedded Linux that only plays audio?
Being skeptical, I often wonder whether Gordon's choice of Mac Mini + OS X is primarily aesthetic.
It wasn't "double blind" (which would imply neither the observer nor the experimenter knew what is being tested), and it wasn't ABX, simply AB, hence subject to all sorts of biases.
What is the procedural merit of an experimenter not knowing what he/she is doing?
I have to confess I'm not even sure (heck, probably haven't a clue) what an ABX test is and that reading up on the technique generally leaves me more confused than I was before I started. "AB vs ABX" strikes me as a phoney distinction between two equally suspect procedures.
I'd like to know the rationale for the apparent assumption that perceptible differences can be detected reliably using only one sampling procedure. What little I've read on the subject seems also to assume that subject populations and the like do not matter.
And so on - but I'm happy to be proven wrong.
AB=you compare A en B and you know that you are listening to A or B
ABX= you listen to A, you listen to B and now a third party (might be Foobar ABX comparator) randomly plays A or B. You score without knowing if it is A or B playing.
After 10 or so trials you compare your score with the real score and check if you are above change level.
Compared with DBT this is probably SBT
What is the procedural merit of an experimenter not knowing what he/she is doing?
None of course, in all experiments the experimenter makes the design, the experimental setup and is supposed to know what he is doing.
In DBT, the experimenter knows, the participants and the raters (often patients and doctors) don’t.
A proper conducted ABX is in essence a unsighted test. Of course the results are not conclusive, it claims just one but not unimportant thing, removing the bias from the experiment.
It says nothing about the conditions (program material, gear, room, etc)
The Well Tempered Computer
In all experiments the experimenter makes the design, the experimental setup and is supposed to know what he is doing.
Quite - the notion that an experimenter has to be ignorant of what he/she is doing to "prevent bias" is so fatuous it is not even worth discussing. (I can see why there is a case for doing it in market research but that's a different matter.)
In the audio sphere, many of those who advocate "DBT" methodologies seem each to have their own idea of what the optimum procedure is but are reluctant to tell you what it is, apparently rely on the naive assumption that essentially the same test design is adequate for every parameter you care to investigate and definitely have a thin skin if you question their ideas.
Almost invariably, it turns out that they have not conducted tests of the type on which they claim expertise and cannot even point you to competent reports of examples conducted on the lines they recommend. (If anyone can, I'd greatly appreciate it.)
It's a mess - I don't blame manufacturers and designers for trusting their ears rather than the aggressive meanderings of experimental wannabees.
You're confusing experimenter with moderator. Further evidence that you are out of your depth.
P
Further evidence that you are out of your depth.
Can you explain what, in the context of a psychological experiment (or come to that any scientific experiment), a "moderator" is. I do not know what you mean by the term.
Trust me, I'm not out of my depth here (nor am I clear how you would know if I was) but it may be that there are cultural/semantic differences at play.
and just to add, the main reason for double blind is so that the experimenter does not (knowingly or unknowingly) bias the results.
A classic example of bias is the following: I am at an audio trade show and I tell an audience of guinea pigs (sorry, audiophiles) that I have two systems that I want to see if they can spot the difference.
Well, simply by me making that statement, I have preconditioned the audience to expect differences. This will encourage an audience member to say there was a difference even when there isn't, and to guess which is which.
I'm not saying this is what Gordon did, simply illustrating that it is so easy to get the result that you were aiming for. The percentage figures he gave are meaningless without confidence levels attached to them, and frankly given the testing conditions and the sample size are not inconsistent with pure speculation on the part of the audience.
Simply by me making that statement, I have preconditioned the audience to expect differences.This is true and you are right about the need for stats - but the scenario you describe is not by any stretch of the imagination an experiment, it's a trade-show participation game. Harmless, good fun and possibly informative in a loose, back-of-the-envelope sort of way that no-one with a modicum of scientific training would take too seriously.
Inevitably, as soon as you ask someone to participate in an experiment, you confess that you are looking for something. If you weren't, you wouldn't be doing the damn thing in the first place and you certainly wouldn't be funded for it.
So, you address the problem with measures that have stood the test of time in a myriad of fields - a prior and clearly-defined null hypothesis, appropriate controls and defined control groups. You do NOT do it by losing control of your own experiment. If your own antics prejudice the results, the experiment is either badly designed or poorly conducted - and probably both.
Must dash - I hear there's a real wheeze on Stand 413 - if you do their cable-shootout test and give the right answer, you get a free drink.
Edits: 03/02/10
*** You do NOT do it by losing control of your own experiment. If your own antics prejudice the results, ***
It's not really about losing control, it's about ensuring that your personal biases and knowledge do not influence the results.
A classic example of this is "psychic" experiments where they found out that subjects were getting the correct answer simply by reading the body language of the experimenter.
But it is also possible that you can do everything right (use all the right buzzwords and even apply them correctly) and still get it wrong. An example of this is that infamous study which "proved" by exhaustive double blind testing that people can't differentiate between SA-CD and DVD-Audio.
Problem was, the researchers didn't take into account that it takes time for someone to get used to the differences. And hearing acuity can be improved with training. An average person would find it difficult to distinguish between MP3 and CD, and yet people can train themselves to hear the differences. The study ignored the fact that there were at least two subjects they tested that were able to spot the differences consistently - they just brushed those subjects aside rather than investigate the reasons why these two can tell the difference and others couldn't.
This shows how powerful experimenter bias is in clouding thinking. The experimenters wanted to believe there was *no* difference, so they deliberately interpreted the results to fit their prejudice (*Most* people can't hear differences, therefore there *are* no differences).
See you at stand 413 - except my ability to distinguish differences is so poor I may need to steal a few sips from you!
It's not really about losing control, it's about ensuring that your personal biases and knowledge do not influence the results.An experimenter who cannot control "personal bias" in tests of this sort is out of his/her depth - it's undergraduate stuff. The fact that some who perform ever-so-scientific tests in the audio field are out of their depth is sad but true.
A classic example of this is "psychic" experiments where they found out that subjects were getting the correct answer simply by reading the body language of the experimenter.
The story is often told the other way round - it's the "psychic" who reads the body language of the subject :> ) Whatever, even if serious "psychic" research (so help me, there's a Chair of Parapsychology at Edinburgh University) is, for most of us, an oxymoron, competent researchers first addressed the issue 100-odd years ago when there was a craze for things "spiritual" in Britain and loads of sensible folk keen to debunk it.
But it is also possible that you can do everything right (use all the right buzzwords and even apply them correctly) and still get it wrong. An example of this is that infamous study which "proved" by exhaustive double blind testing that people can't differentiate between SA-CD and DVD-Audio.
The Meyer and Moran paper (M&M - the one that "proved" we can't differentiate between RBDC and SACD) did almost nothing right: it was deeply flawed from the off. See e.g. these posts:
see: http://db.audioasylum.com/cgi/m.mpl?forum=pcaudio&n=57852
and:
http://db.audioasylum.com/cgi/m.mpl?forum=pcaudio&n=57904
from July last year where I try to explain why. (Sorry about all those ' thingies - they turned up after a recent AA archive revamp.)
(Ironically, I was also debating with PP who resorted as readily then as now to ad hominem abuse and a mix of sarcasm and Google.)
Problem was, the researchers didn't take into account that it takes time for someone to get used to the differences. And hearing acuity can be
improved with training.M&M was flawed on several fronts, not just on "training" issues (though I'm sure you're right to raise them). The authors' competence in other fields didn't protect them from schoolboy howlers in this one.
An average person would find it difficult to distinguish between MP3 and CD, and yet people can train themselves to hear the differences. The study ignored the fact that there were at least two subjects they tested that were able to spot the differences consistently - they just brushed those subjects aside rather than investigate the reasons why these two can tell the difference and others couldn't.
Assuming we're talking of the same paper, M&M didn't get five per cent significance for any subjects so I'd hesitate to accuse them of brushing aside awkward results.
On a personal note, I find that mp3 quality varies from execrable to acceptable. I have some mp3-sourced recordings that sound really quite good but others, made from what I know to be a good source (as I have the CD), I simply cannot listen to. Whether that's inherent in the process, down to bad engineering, a bit of both or something else altogether, I can't say.
I suspect that the mp3 v CD thing would actually prove tricky to test properly because you have two variables at play - quality differences within and between formats, not to mention vested interests.
This shows how powerful experimenter bias is in clouding thinking. The experimenters wanted to believe there was *no* difference, so they deliberately interpreted the results to fit their prejudice.
Experimenter bias is powerful in clouding the thinking of incompetent experimenters only. Any half-way decent design addresses it early doors. I read M&M several times before writing the above critique and have just re-read it. I find no grounds for suggesting that the authors "brushed aside" any results and I don't think they fairly can be accused of "deliberately interpreting" them to suit preconceptions. They genuinely got them - but they were always going to get them because of flaws in the experiment's design.
Inevitably, those who wanted to believe the results did so without reading the paper with much care. Now those folk were biased . . .
Edits: 03/03/10
*** The story is often told the other way round - it's the "psychic" who reads the body language of the subject :> ) ***Well, it's the subject's "psychic" ability that was being tested. The experimenter knew what the answers were, and was subconsciously conveying the answers through body language to "successful" subjects.
The M&M paper is interesting, but wasn't the paper I was thinking of.
My memory is terrible these days, and it may well have been the paper, I'll have to look around to see if I can still find it.
I agree that the M&M paper is kind of ... "interesting" reading (sarcasm hat on).
MP3 quality varies by encoder as well as bit depth. A proper MP3 vs lossless test will require you to encode everything yourself using a known encoder, rather than casual comparisons of stuff downloaded from the internet. I personally find great difficulty in distinguishing between MP3 and CD at bitrates above 240 kbps, below that yes it is possible.
It's possible to train the ear to recognise lossy artefacts - for MP3 just listen very carefully to the high frequencies for signs of "phasing." For Ogg Vorbis it's ringing, for WMA it's an increase in low level noise and some "flattening" of the dynamics. As you up the bitrates, these artefacts become less and less and eventually there are no obvious clues.
Edits: 03/03/10
"The experimenter knew what the answers were, and was subconsciously conveying the answers through body language to "successful" subjects."
Also, if you have multiple subjects taking a test in the room at the same time, they can be potentially influenced by each other's body language. As a result, their perceptions can not be considered statistically independent, which means that the statistical power of an experiment is substantially reduced (depending on the numbers listening at once).
"The M&M paper is interesting, but wasn't the paper I was thinking of."
Check the attached link. Perhaps this was the paper you were trying to remember.
Tony Lauck
"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar
.
Also, if you have multiple subjects taking a test in the room at the same time, they can be potentially influenced by each other's body language.
And if they're suspended by their ankles from an elastic chord over a vat of boiling acid, they find it hard to concentrate on the quality of the mp3 sample under test, even in isolation in a darkened room. This may explain why neither scenario is regularly reported in the literature.
Meanwhile, thanks for the link to the AES paper, which I look forward to reading. They seem to have eschewed the acid bath . . . how dull.
D
"What is the procedural merit of an experimenter not knowing what he/she is doing?"
If the person conducting the test/interfacing with the participants does not know which example is being played/seen/tasted when, he or she has no opportunity to influence the results.
P
"What is the procedural merit of an experimenter not knowing what he/she is doing?"
If the person conducting the test does not know which example is being played/seen/tasted when, he or she cannot influence the results.
P
Post a Followup:
| FAQ |
Post a Message! |
Forgot Password? |
|
||||||||||||||
|
||||||||||||||
This post is made possible by the generous support of people like you and our sponsors:
Top of Page ]
[ Contact Us ]
[ Support/Wish List ]
[ Copyright © 2024, Audio Asylum, all rights reserved ]
