| squishyball(1) | Xiph Evaluation Tools | squishyball(1) |
squishyball - perform sample comparison testing on the command line
squishyball [options] fileA [fileB [fileN...]] [> results.txt]
squishyball is a simple command-line utility for performing double-blind A/B, A/B/X or X/X/Y testing on the command line. The user specifies two input files to be compared and uses the keyboard during playback to flip between the randomized samples to perform on-the-fly comparisons. After a predetermined number of trials, squishyball prints the trial results to stdout and exits. Results (stdout) may be redirected to a file without affecting interactive use of the terminal.
squishyball can also be used to perform casual, non-randomized comparisons of groups of up to ten samples; this is the default mode of operation.
A/B testing randomizes the order of two input samples and presents them, unnamed, as sample 'A' and sample 'B'. In each trial the user selects A or B as the preferred sample. The samples are then re-randomized for the next trial. This test is useful for establishing relative or preferred quality between two samples.
A/B/X presents two input samples, unrandomized, as sample 'A' and sample 'B'. A third sample 'X' is chosen randomly from either 'A' or 'B'. In each trial, the user selects A or B as the sample believed to be the same as X. X is then re-randomized for the next trial. This test is useful for determining if any differences are audible between two samples and to what confidence level.
Note that because the A and B samples are not randomized (they are presented in the order given on the command line as per standard industry practice), an A/B/X test does not eliminate ordering bias. A stronger version of this test that randomizes all samples is the X/X/Y test below.
Casual comparison mode does not randomize the input samples or perform multiple trials. It simply provides a convenient way to rapidly flip back and forth within a group of up to ten samples.
X/X/Y testing is a form of A/B/X testing in which the order of all samples is randomized and the position of the 'X' sample is not known ahead of time to be in the third position. In each trial, the user selects which of sample 1, 2 or 3 is believed to be the sample that is different from the other two. This test is useful for determining if any differences are audible between two samples and to what confidence level. It is a stronger version of the A/B/X test that eliminates sample order bias.
squishyball 'reconciles' files to identical channel ordering, length and bit-depth before playback begins so that CPU and memory resource usage during playback should be identical for all samples. When 24-bit playback is available and at least one sample is 24-bit or greater (ie, 32-bit or float), all samples are converted/promoted to 24 bits. If 24-bit playback is unavailable, all samples are demoted to 16 bits. Note that Opus and Vorbis files are both considered to be natively float formats.
squishyball checks files for clipping at load time. By default, squishyball will automatically normalize all float inputs by the amount needed to avoid clipping any one. Automatic normalization can be disabled with the -N option. Integer samples are checked for clipping heuristically; two or more consecutive full-range values in a channel count as clipped. Out-of-range integer values cannot be recovered; in this case, squishyball issues a warning and performs no normalization based on the integer clipping.
Downmixing samples to mono with -1 or stereo with -2 will also likely require normalization to avoid clipping; as above, squishyball will automatically normalize all inputs by the amount necessary to avoid clipping in any one unless -N is specified.
Down-conversions of uncompressed and lossless samples (WAV, AIF[C], FLAC, SW) to 16-bit are dithered using a simple white TPDF. Lossy-encoded samples (Vorbis and Opus) are dithered to 16-bit only if one or more uncompressed/lossless inputs are also being dithered. Normalization also triggers dithering of all input samples (uncompressed, lossless and lossy) upon conversion to 16 bit.
-D overrides the default behavior and forces unconditional dithering of all 16-bit down-conversions. Similarly, -t forces unconditional rounded truncation in all cases, disabling dither completely.
Conversions to 24-bit are never dithered.
Many modern audio playback systems (such as PulseAudio or the ALSA 'default' device) give no means of determining if the requested playback paramters are actually being used by the hardware, or if the audio system is helpfully converting everything to some other supported depth/rate. When using these systems, squishyball has no way of knowing if 16-/24-bit playback or sample rate is being honored. Automatic conversion can affect audible playback quality; be careful to verify actual system behavior.
Silent Mode smoothly transitions between samples. It allows the most direct comparison between signals without any intervening auditory distraction. However, the temporary combination of different signals may cause unintended cancellation and comb-filtering effects that can give away the 'unknown' sample just as a 'pop' from an instantaneous transition would.
Mark Mode quickly fades to silence before flipping to another sample, marking the transition. Because the samples never overlap, crosslap artifacts cannot contaminate trial results. However, the audible dip between samples may distract from listening, potentially making it slightly more difficult to detect legitimate artifacts.
Beep Mode is similar to mark mode but adds a soft 'beep' to mark where the transition occurs. It makes the transition point especially obvious. It does not crosslap the samples; one sample is faded completely before the second is mixed in as in mark mode.
Monty <monty@xiph.org>
abx-comparator(1), rateit(1), ogg123(1), oggdec(1), opusdec(1), flac(1)
| 2013 November 9 | Xiph.Org Foundation |