LLM evals in epidemiology, biostatistics, and health research: a brief review

Background

Large language models (LLMs) are now evaluated across many health-related tasks, but the literature is uneven. Bedi et al. (2025) systematically reviewed 519 studies and found that 44.5% focused on medical knowledge or licensing-style questions, 84.2% focused on question answering, 95.4% used accuracy as the main metric, and only 5.0% used real patient-care data.

That matters for epidemiology, biostatistics, and health research. In these fields, practical value depends less on exam-like recall and more on valid reasoning, transparent methods, reproducibility, and reliable error detection.

Objective

To briefly review what has actually been evaluated, where the published evidence is strongest, what recent preprints add, and which limits still prevent strong claims about real-world readiness.

Methods

Narrative review.

Findings

What has actually been evaluated?

Most published evaluations still measure constrained tasks: multiple-choice questions, short-form question answering, or rubric-based judgments rather than end-to-end research workflows (Bedi et al., 2025; Lee et al., 2024).

Foundational biomedical benchmarks such as PubMedQA helped standardize comparison, but they mainly test answer selection from article abstracts rather than full evidence synthesis, causal inference, or statistical workflow execution (Jin et al., 2019, preprint).

Where the published evidence is strongest

The published evidence shows that current evaluation practice remains narrow. Bedi et al. (2025) documented a benchmark-heavy literature with limited real-world data, and Lee et al. (2024) found no clear evaluation framework and frequent reliance on small query counts, single measurements, and limited secondary analyses.

Tam et al. (2024) reviewed 142 studies and proposed the QUEST framework for human evaluation, emphasizing quality of information, understanding and reasoning, expression style and persona, safety and harm, and trust and confidence.

For biostatistical practice specifically, Grambow et al. (2025) are informative because they are closer to real work. Among 69 respondents from three biostatistics units at two academic medical centers, 44/69 (63.8%) reported using LLMs, but 29/41 (70.7%) reported significant errors, including incorrect code, statistical misinterpretation, and hallucinated functions.

What the preprints suggest

Recent preprints suggest a useful shift toward more domain-specific evaluation, but they should be interpreted cautiously. Arora et al. (2025, preprint) introduced HealthBench, a broad health benchmark designed to measure performance and safety in more open-ended health interactions. Harris et al. (2025, preprint) introduced PubHealthBench for UK government public-health information and highlighted a gap between multiple-choice and free-form answering, which is relevant to epidemiology and public-health communication.

Other preprints extend evaluation into specialized research tasks. Lu et al. (2025, preprint) introduced StatEval for statistics questions. Bonzi et al. (2025, preprint) introduced CareMedEval for biomedical critical appraisal and reasoning grounded in scientific papers. He et al. (2025, preprint) studied whether LLMs can identify CONSORT adherence in randomized trials. Li et al. (2025, preprint) benchmarked meta-analysis data extraction and reported that automation remains challenging, with performance that may be acceptable for some low-risk subtasks but not for unsupervised extraction of all key study data.

Main limits of current evals

Four limits recur across the literature. First, task selection is narrow: question answering and exam-style prompts dominate, while protocol design, causal reasoning, bias assessment, statistical troubleshooting, and full evidence synthesis are less often tested (Bedi et al., 2025; Lee et al., 2024).

Second, evaluation metrics are narrow: accuracy is common, but calibration, uncertainty, fairness, safety, and downstream decision quality are much less often measured (Bedi et al., 2025). Third, study designs are often weak for deployment claims: many studies use small numbers of questions, few repeated runs, and limited external validation (Lee et al., 2024).

Fourth, benchmark success may overstate readiness. Constrained formats can look strong even when open-ended answers, statistical interpretation, or workflow integration remain brittle (Harris et al., 2025, preprint; Grambow et al., 2025).

Interpretation

The most defensible near-term use case is assisted work with human verification, not autonomous epidemiologic or biostatistical reasoning. The evaluation question should shift from “can the model answer this benchmark item?” to “does the workflow remain valid when the model is used here, by this team, under these constraints?”

Limitations of this brief review

This is a brief, narrative review rather than a systematic review. It is meant to map the current evaluation landscape, not to estimate pooled performance. The literature is also moving quickly, and several domain-specific sources are still preprints.

Conclusion

The published evaluation literature is still dominated by narrow benchmarks, while the most relevant domain-specific evaluations are only beginning to emerge. For now, the evidence supports cautious, supervised use for bounded tasks, especially drafting, coding assistance, and information retrieval, but it does not support strong claims of robust, unsupervised reliability in core research workflows.