Reading Notes

Personal reading notes. Original paper: arXiv:2606.07936v2.

arxiv_id: 2606.07936v2

The Illusion of the Gold Standard: A Large-Scale Analysis of Human Evaluation Protocols for Long-Text Generation

Background and Motivation

With the widespread adoption of Large Language Models (LLMs), long-text or open-ended generation tasks have become mainstream in Natural Language Processing (NLP) research. In these scenarios, automated evaluation metrics often perform poorly; therefore, human evaluation is still regarded as the “gold standard” for measuring the quality of generated text, especially in high-stakes domains such as healthcare, science, and law. However, the paper points out that the reliability and reproducibility of current human evaluation procedures depend on transparent and well-documented protocols, details which are frequently missing in existing practices.

The authors argue that the lack of standardization and operationalization limits the effectiveness of evaluations, the robustness of conclusions, and comparability across studies. Furthermore, as the self-evaluation capabilities of models improve, human evaluation is increasingly being supplemented or replaced by LLM judges. In this context, human evaluation also bears the meta-evaluation role of assessing the performance of LLM judges. Based on this, the authors shift their research perspective to the *CL research community, aiming to understand reporting practices regarding human evaluation and critique the deficiencies in current practices.

Research Methods and Standard Definitions

To systematically examine reporting norms within the community, the authors defined a framework containing 20 reportable criteria. These criteria are based on scientific reproducibility literature and cover core aspects such as task definition, annotation operationalization, annotator information, and data analysis and interpretation.

Data Collection and Analysis Process

The paper conducted a large-scale analysis of over 9,100 papers published at *CL conferences between 2023 and 2025. The research focused specifically on papers involving medium-to-long natural language generation, as these tasks impose a heavy burden on human evaluation and lack clear, reproducible automated evaluation metrics.

Regarding data sampling, the authors sampled 356 papers from over 1,800 papers that involved long-text generation and included human evaluation. Among these, 284 papers were fully manually annotated, while the rest were annotated with LLM assistance. This hybrid approach aimed to balance the depth and breadth of the analysis.

Annotation Tools and Protocols

During the codebook development process, the authors iteratively developed evaluation criteria, ultimately determining 37 questions, of which 20 are core reportable criteria. To execute this large-scale analysis, the research team designed a specialized annotation interface.

The figure above shows a screenshot of the annotation interface used by the research team in Google Sheets, which includes questions related to documentation. This interface supports annotators in systematically checking whether each paper reports key details such as evaluation guidelines, annotator demographic information, and sample sizes. Through this structured annotation process, the authors were able to quantify the compliance level of the current community regarding the reporting of human evaluation protocols.

Key Findings

Through the analysis of a large number of papers, the authors revealed obvious defects in the documentation of human evaluation protocols, with widespread under-reporting in key aspects.

Specific Manifestations of Under-reporting

The paper analyzed the reporting ratios of the 20 core criteria. While most papers reported evaluation dimensions, some annotator information, and the number of annotated samples, there is significant under-reporting in other aspects of the evaluation research design. For example:

• Evaluation Guidelines and Rationale: Only about half of the papers included guidelines for human evaluation tasks or provided rationale for the dimensions being evaluated.
• Statistical Rigor: Good practices such as calculating sample sizes using power analysis or reporting statistical confidence are extremely rare.
• Annotator Information: Reporting on annotator inclusion/exclusion criteria, recruitment platforms, and demographic information is also incomplete.

Shift in Evaluation Paradigms

The study also found a notable trend: in recent years, the proportion of papers using human evaluation for long-text generation assessment is decreasing, while the proportion of papers using LLM judges for evaluation seems to be increasing. This change raises further thoughts on the role of human evaluation in meta-evaluation.

Conclusion and Recommendations

Based on the above findings, the authors believe that the ambiguity in current human evaluation practices leads to misunderstandings regarding what is being measured, how it is measured, who contributes to the judgments, and how judgments are interpreted. To support more transparent and reproducible reporting in future research, the paper proposes actionable recommendations.

The authors emphasize that human judgment remains a foundational part of evaluation methods and should be subject to the same rigor and standards as other research methods used by the community. By adopting the defined 20 reporting criteria, the research community can improve the reproducibility of human evaluation research, reduce the “illusion of the gold standard,” and ensure that evaluation results truly reflect model performance.

The analysis code and annotation dataset are publicly available, aiming to promote the community’s continued attention to and improvement of evaluation protocol transparency.