The bibliographic entry was discovered and compiled by a machine, and published without a person checking it against the publication. The publication itself is the authors' own work — this badge says nothing about it.
Posthoc verification and the fallibility of the ground truth
ding2022posthocThe published version and its preprint. ding2021posthoc is arXiv:2106.07353v1 of 2 June 2021; this is the version of record, in the Proceedings of the First Workshop on Dynamic Adversarial Data Collection, 2022, same three authors and same title. Created from the doi recorded on the preprint's note by an earlier reading, not from a reconstruction. The corpus has no link kind for preprint-and-published -- issue 122 -- so distinct-from carries it, as it does for the two Piedeleu records.
Abstract
Classifiers commonly make use of pre-annotated datasets, wherein a model is evaluated by pre-defined metrics on a held-out test set typically made of human-annotated labels. Metrics used in these evaluations are tied to the availability of well-defined ground truth labels, and these metrics typically do not allow for inexact matches. These noisy ground truth labels and strict evaluation metrics may compromise the validity and realism of evaluation results. In the present work, we discuss these concerns and conduct a systematic posthoc verification experiment on the entity linking (EL) task. Unlike traditional methodologies, which asks annotators to provide free-form annotations, we ask annotators to verify the correctness of annotations after the fact (i.e., posthoc). Compared to pre-annotation evaluation, state-of-the-art EL models performed extremely well according to the posthoc evaluation methodology. Posthoc validation also permits the validation of the ground truth dataset. Surprisingly, we find predictions from EL models had a similar or higher verification rate than the ground truth. We conclude with a discussion on these findings and recommendations for future evaluations.
A copy is held
pdf, 656.7 kB. Not published — it may be under
copyright. The facts and links here are.
How it got here
How it got here
agent via bibtex
Added
2026-08-05 00:00 UTC
Not denied by
a person 2026-08-16 16:18 UTC
Cite it as
@unpublished{ding2021posthoc,
title = {Posthoc verification and the fallibility of the ground truth},
author = {Yifan Ding and Nicholas Botzer and Tim Weninger},
year = {2021},
publisher = {arXiv},
doi = {10.48550/arxiv.2106.07353},
}