Reporting standards for LLM evaluations: a brief review

Background

Evals carry real weight. They compare models, support deployment decisions, and sometimes ground safety claims. In our earlier commentary on eval reporting standards, we argued that this weight justifies reporting norms closer to those used in mature empirical fields.

As our earlier brief review of LLM evals in health research noted, the applied literature is uneven. Healthcare reviews found the field still dominated by exam-style tasks, with limited use of real patient data and no shared evaluation methodology. That gap may partly explain why the clearest LLM-specific reporting statements have come from medicine, where journal reporting norms were already more developed.

Objective

To briefly map the guidance currently available for designing, reporting, and interpreting LLM evaluations, and to identify where consensus appears strongest and where it remains absent.

Methods

Narrative review of venue guidance, standards-body documents, benchmark frameworks, and domain-specific reporting statements, plus methodological papers on statistical inference in NLP and LLM evaluation. This is not a systematic review and does not estimate adherence rates.

Findings

LLM evaluation is not without guidance, but no common reporting standard covers the full range of capability, safety, and deployment evals. The clearest general-purpose norms come from venue checklists. Formal statistical guidance is starting to emerge from methodological papers and standards bodies. Domain-specific reporting statements are most developed in health fields.

Table 1. Current sources of guidance for LLM evaluation reporting

Venue checklists and documentation

Venue checklists are probably the closest thing to shared practice in general LLM research. The ACL Rolling Review (ARR) checklist asks authors to report data documentation, split sizes, compute budget, hyperparameter search, descriptive statistics, test and validation results, and metric implementation details (ACL Rolling Review). NeurIPS asks for a reproducibility path, experimental settings, and properly defined uncertainty quantification with stated assumptions and variance sources. Both venues frame these as transparency tools, and both explicitly state that negative answers are not grounds for rejection.

ARR also points authors to data statements, model cards, and datasheets for datasets. BenchmarkCards was proposed to standardize benchmark documentation itself, covering objectives, data sources, and limitations. A large share of confusion in LLM evaluation starts one step before model reporting: readers often cannot determine what a benchmark measures or what population it represents.

Statistical and methodological guidance

The methodological literature is less thin than common practice might imply. Dror et al. (2018) argued that significance testing in NLP was often ignored or misused and offered a practical protocol for selecting appropriate tests. Dodge et al. (2019) argued that test-set scores alone cannot settle model comparisons and that validation performance during development should also be reported. NLPStatTest pushed further, incorporating effect sizes and power analysis.

Miller (2024) made the same point in LLM terms: evals are experiments, and the field has largely skipped the statistical infrastructure other empirical sciences take for granted. NIST AI 800-3 goes further, arguing that benchmarks require explicit estimands and valid uncertainty quantification, and showing how item-difficulty estimation and variance decomposition can enrich benchmark summaries beyond single-score reporting.

Domain-specific reporting standards

Medicine has produced the most developed LLM-specific reporting statements. TRIPOD-LLM offers a consensus-based checklist with 19 items and 50 subitems for biomedical LLM studies (Collins et al. 2024). MI-CLEAR-LLM followed from observed variability in how LLM accuracy studies reported model identity, query date, and stochasticity handling. ELEVATE-GenAI covers health economics and outcomes research. QUEST focuses on human evaluation, where adjudication and scoring procedures often matter as much as the model being tested.

The broader medical AI literature has followed a similar pattern. Liu et al. (2024) identified 26 AI-specific reporting guidelines in medicine, though they vary in scope and quality. Reviews of LLM evaluation in health likewise note that no common evaluation methodology has taken hold, which suggests that even the most active part of this reporting space is still early.

Remaining gaps

None of the sources we reviewed works as a general standard across capability, safety, and deployment evals. The gaps are real: no field-wide minimum exists for repeated runs, uncertainty reporting, or power analysis. Individual papers still handle stochasticity, prompt sensitivity, and scorer reliability very differently from one another. Benchmark documentation is often thin, which makes external validity hard to judge even when the headline score is reported carefully.

The result is partial standardization, not mature measurement science. The field currently expects authors to report what they did. It is less consistent about asking whether the resulting estimate is precise enough to act on or stable enough to generalize.

Interpretation

The current state of LLM eval reporting is less a coherent standard and more a stack of partial solutions: venue checklists set a floor for transparency, benchmark frameworks like HELM improve comparability, papers by Dror et al. and Dodge et al. and NIST AI 800-3 give tools for inference, and domain-specific statements like TRIPOD-LLM fill in the detail for high-stakes settings. The field appears to be building toward something. Whether these pieces will connect into a more unified practice is still unclear.

Limitations

This is a brief narrative review, not a systematic one. It maps the current guidance structure rather than estimating adherence rates or quantifying reporting quality across the published literature. Some relevant documents are recent or methodological rather than universally adopted standards. We have not verified that this review is complete.

Conclusion

LLM evaluation is not without reporting guidance, but it still lacks a mature common standard. The more accurate description is probably fragmented progress. General evals lean on venue checklists and benchmark frameworks; statistical inference draws on methodological papers and NIST AI 800-3; high-stakes domains like healthcare are developing their own reporting statements. The work ahead is mainly integration: connecting these pieces into something more coherent and more explicitly statistical.