sc_stats(1)

WORK IN PROGRESS
 RELEASED FOR Hub-5NE '97 Eval NAME sc_stats - SCLITE's Statistical System Comparison Program

NOTE: This manual page was created automatically from HTMl pages in the sclite/doc directory. This manual page does not include output file examples. The author suggests using a HTML browser for reading the sclite documentation.

SYNOPSIS sc_stats OPTIONS

DESCRIPTION .PP sc_stats is a program which statistically compares the performance of two or more Automatic Speech Recognition (ASR) systems which have been run on identical test data. Sc_stats is part of the NIST SCTK Scoring Tookit. The program reads alignments generated by sclite, and produces summary reports , graphs and/or compares the systems using any number tests of significant differences.

INPUT ALIGNMENTS
OUTPUT REPORTS
STATISTICAL TESTS
REVISION HISTORY
BUGS/COMMENTS
    Please contact Jon Fiscus at NIST with any bug reports or comments at the email address jonathan.fiscus@nist.gov or by phone, (301)-975-3182. Please include the version number of rover,
Sc_stats Commandline Options

The commandline options for sc_stats can be broken into four categories: Input File Options:
    -p,
Output Options:
    -e, -n, -O,
Report Generation Options:
    -r
Statistical Test Options:
    -t -v -u -g

Input File Options:
    These options control/define the input to sc_stats. Input must come from stdin and the -p option must be used. (Forcing the user to use the -p option enables future expandability while maintaining backward compatability.)
    -p
        Alignments are read from 'stdin' as input to sc_stats. The format of the input must be in the "sgml" output format, created either by '-o sgml' or by piped input from another sctk utility.
Output Options:
    -e desc
        Description of the ensemble of hyp files.
    -O output_dir
        Writes all output files into output_dir. Defaults to the hypfile's directory
    -n name
        Writes all multiple hypothesis file reports to files beginning with 'name'. Using '-' writes to stdout. Default: 'Ensemble'
Report Generation Options:
    -g
        Generate per speaker range graphs, based on the formula defined by '-f'. The reports are written to files whose root name begins with the values defined by '-n'. There are two graphs produced, one showing speaker performance variability across systems and the second showing system performance variablity for across speakers.

        - The 'range' graphs are an ASCII representation of the of the variablity in error rates for a given speaker. The graph is sorted be the mean of statistic computed for each speaker. EXAMPLE

         - The 'grange' graph is a gnuplot version of the same data ploted in 'range. There are two sets of files created. The first set, which is called '*.grange.spk.plt' and data files respectively for the speaker performance variability across systems graph. The second set, which is called '*.grange.sys.plt' and data files respectively for the system performance variability across speakers graph. EXAMPLE

         - The 'grange2' graph is similar to the 'grange' graph except that each systems speaker word error scores are identified by a unique symbol. EXAMPLE
    -r
        Example Example Example Example Example Example Produce no output reports, Default.
Statistical Test Options:
    -t [ mcn | mapsswe | sign | wilc | anovar | std4 ]
        Perform the McNemar Test. Perform the Matched Pairs Sentence Segment Word Error Test Perform the Sign Test Perform the Wilcoxon Signed Rank Test Perform the Analysis of Variance by Rank Test This is a shorthand notation to do the 'standard' four tests: mcn, mapsswe, wilc and sign.
    -v
        For each test performed on a pair of systems files, output a detailed analysis.
    -u
        Rather than creating a comparison matrix for each test, unify statistical test results into a single comparison matrix
    -f [ E | R | W ]
        Use the identified formula for statistical tests: sign, wilcoxon and anovar tests. The formulas are:
         E -> Percentage Word Error
         R -> Percentage Words Correctly Recognized
         E -> Percentage Word Accuracy By default 'E'