LING83800: Inter-annotator agreementInter-Annotator Agreement • October 3rd, 2021
When our models consist of supervised machine learning trained on human annotations, agree- ment between multiple human annotators imposes a upper bound for performance, if only be- cause it is difficult if not impossible to measure system performance exceeding that of human annotators. And, when human annotators do not show substantial agreement, this suggests the annotation task may be ill-defined, or too challenging.