An R Script for ABX Testing

I couldn’t find a computer program I liked for doing ABX tests and so ended up writing my own.  The full program listing (in the R language) is given below in case someone finds it useful.

Background

Plenty has been written about ABX testing so I’ll give just a quick précis here.  Suppose you have two different audio sources, A and B, and you think you can hear the difference between them.

Q: How does one establish this hypothesis in a scientific way?

A: With a Statistical Hypothesis Test, of which ABX is just one type.

The test subject is presented with samples of stimuli A, B and X, where X is identical to either A or B: the subject must decide which.  On each trial X is randomly set to either A or B.  Several trials are run (say N=20) and the number (n) of correct responses is tallied.

The null hypothesis here is that the subject cannot discriminate between A and B, in which case their responses are indistinguishable from random guessing: the observed number of correct responses is then a random variable sampled from a binomial distribution.  Under our null hypothesis the probability of getting at least n correct guesses in N trials isSo you calculate p using this formula.  If p is sufficiently low (say, less than 0.05) then the probability of having achieved the correct responses by sheer guessing is very low, giving you strong statistical evidence for rejecting the null hypothesis (hence concluding that the test subject actually can discriminate between A and B).  If p isn’t low then you have no grounds to reject the null hypothesis — all you can do is run more trials until you get a low p or until the test subject gives up and admits they’re guessing.

There are some gotchas, like having to guard carefully against “tells” such as switching noises or unmatched volume levels.  Only then can you be sure the subject is actually comparing A and B and not just some quirk in the apparatus.

The Code

My R script presents a simple gui that looks like this:

The Results display can optionally be hidden to avoid bias.  Samples A and B can be anything that can be turned on/off using system commands (customizable in the script itself).  One could use this to present any stimuli, not just audio.  I’ve been using it to switch between outputs in mpd, to test for audibile differences between dsp configurations.  Here’s the full code:

# abx_test.R
# Richard Taylor 4.9.2013
# R script to run a randomized A/B/X test with a simple GUI

#####################################################################
# You need to edit these two functions, presumably using the "system"
# function to run something to switch audio streams from A to B or
# vice versa.  Here I use 'mpc' to switch between outputs in mpd.

debug <- TRUE;        # show gory details on the command-line? 
showresults <- TRUE;  # let subject know how they're scoring? 

mpdoutputA <- 4;  # indices of mpd outputs to A/B/X
mpdoutputB <- 3;
mpdvol <- 85;     # mpd volume level to enforce to ensure level-matching

switchtoA <- function() {
  # ...do something here to switch to sample A...
  system(sprintf("mpc -q pause; mpc -q disable %i; mpc -q enable %i; mpc -q play",
                 mpdoutputB, mpdoutputA ) );
  system(sprintf("mpc -q volume %i", mpdvol));
}

switchtoB <- function() {
  # ...do something here to switch to sample B...
  system(sprintf("mpc -q pause; mpc -q disable %i; mpc -q enable %i; mpc -q play",
                 mpdoutputA, mpdoutputB ) );
  system(sprintf("mpc -q volume %i", mpdvol));
}

#####################################################################
# Don't need to edit anything below here...

playA <- function(panel) {
  if (debug) cat("playing A...\n");
  switchtoA();
  rp.text.change(panel, "nowplaying", "now playing: A" );
  panel
}

playB <- function(panel) {
  if (debug) cat("playing B...\n");
  switchtoB();
  rp.text.change(panel, "nowplaying", "now playing: B" );
  panel
}

playX <- function(panel) {
  if (debug) cat("playing X =",panel$xis,"...\n");
  switch( panel$xis,
    A = switchtoA(),
    B = switchtoB() );
  rp.text.change(panel, "nowplaying", "now playing: X" );
  panel
}

# function that gets called when subject declares "X is A" or "X is B":
declareX <- function( panel, my.decl ) {
  # increment number of trials:
  panel$ntrials <- panel$ntrials + 1;

  if (panel$xis == my.decl) {  # declaration was correct
    if (debug) cat("correct.\n");
    panel$ncorrect <- panel$ncorrect + 1;
  } else {                     # declaration was wrong
    if (debug) cat("wrong.\n");
  }

  # print updated report...
  # If the test subject were just guessing, then ncorrect would be a
  # sample from a binomial distribution, for which the probability
  # of ncorrect or more correct in a sample of size ntrials is:
  pval <- 0.5^panel$ntrials * sum(choose(panel$ntrials, panel$ncorrect:panel$ntrials));
    cat( sprintf("got %i / %i correct (p = %0.4f)\n",
        panel$ncorrect,panel$ntrials,pval) );

    if (showresults)
      rp.text.change(panel, "t1", sprintf("%i / %i correct\np = %0.4f",
        panel$ncorrect,panel$ntrials,pval) );

  # update trial counter:
  rp.text.change(panel, "counter", sprintf("TRIAL #%i",panel$ntrials+1) );

  # choose a new X:
  panel$xis <- sample(c("A","B"),1);
  if (debug) cat("\ngot new X =",panel$xis,"\n");

  panel
}

declareA <- function(panel) {
  if (debug) cat("subject declared A.\n");
  panel <- declareX( panel, "A" );
  panel
}

declareB <- function(panel) {
  if (debug) cat("subject declared B.\n");
  panel <- declareX( panel, "B" );
  panel
}

# choose the initial setting for X:
xval <- sample(c("A","B"),1)
if (debug) cat("\ngot initial X =",xval,"\n");

# Create the rpanel gui:
require(rpanel);
panel <- rp.control(title="A/B/X tester", xis=xval, ntrials=0, ncorrect=0 );
rp.text(panel,"\n", name="sp1", pos="top");
rp.text(panel,"TRIAL #1",name="counter", width=20,
        font="Arial", foreground="white", background="black", pos="top")
rp.text(panel,paste("\nStep 1: X is the same as either A or B. You decide which.\n",
                    "Use the buttons below to sample as many times as you wish.\n"),
                    name="s1", pos="top");
rp.button(panel, action = playA, title = "play A ...", pos="top" );
rp.button(panel, action = playB, title = "play B ...", pos="top" );
rp.button(panel, action = playX, title = "play X ...", pos="top" );
rp.text(panel,"now playing: NA",name="nowplaying", width=20,
        font="Arial", foreground="white", background="navy", pos="top")
rp.text(panel,paste("\nStep 2: Record your response below. Repeat from Step 1.\n"),
                    name="s2", pos="top");
rp.button(panel, action = declareA, title = "\"X is A!\"", pos="top" );
rp.button(panel, action = declareB, title = "\"X is B!\"", pos="top" );
if (showresults) {
    rp.text(panel,"\nResults:", name="s3", pos="top");
    rp.text(panel,"0 / 0 correct\np = NA", width=20,
            font="Arial", foreground="white", background="navy",name="t1", pos="top");
}
rp.text(panel,"\n", name="sp2", pos="top");

Leave a Reply

Your email address will not be published. Required fields are marked *