Inter-transcriber Reliability

Choose By Word to compare aligned IPA words or By Tier to compare each complete IPA Actual tier. Then select the transcribers included in the report.

Phone agreement and Levenshtein distance are always reported. The optional metrics are consonant agreement, vowel agreement, diacritic agreement, and tone-number agreement; consonant and vowel agreement are selected by default. The diacritic options can retain all diacritics, ignore them all, or ignore only/except a selected set.

Select Next to generate the report. Phon creates a table for every selected pair of transcribers. Each row represents an IPA word or complete tier, according to the selected domain, and includes the chosen agreement metrics. Each table ends with its metric averages, and a summary at the beginning collects the pairwise averages.

Agreement is the average of percent correct in both directions because percent correct is not commutative. For example, phone agreement is the mean of the percent phones correct from transcriber 1 to transcriber 2 and from transcriber 2 to transcriber 1.

Use the report toolbar or its context menu to save the HTML report or export report tables.