About the benchmark
LingxiDiagBench evaluates LLM agents on Chinese psychiatric differential diagnosis across 12 ICD-10 categories. It pairs LingxiDiag-16K (16,000 synthetic EMR-aligned dialogues matching real clinical distributions) with LingxiDiag-Clinical (1,709 verified cases from Shanghai Mental Health Center), and measures models on two paradigms:
- Static — diagnosis from fixed consultation transcripts at 2 / 4 / 12-class granularity, plus doctor next-question prediction (BLEU / ROUGE-L / BERTScore).
- Dynamic — the model leads a live multi-turn consultation against a simulated patient agent under one of three strategies: Free-form (base, no scaffolding), APA-Phased (v2, five-phase APA protocol without retrieval) and APA-Phased + RAG (v3, five-phase + FAISS retrieval over a clinical-guideline PDF + CoT — the default). Five clinical dimensions are rated 1–6 by an LLM-as-judge ensemble; the post-dialogue diagnosis is then scored at 2 / 4 / 12-class.
Model requirement for APA-Phased + RAG (v3): the candidate must be able to emit <box>...</box>-wrapped reasoning between turns — v3 parses that span as the doctor's multi-step reasoning. Models without this output capability will degrade silently to free-form behavior on the v3 track. Free-form and APA-Phased (v2) have no such constraint. v3 对模型有隐性要求:必须能在轮次间输出 <box>...</box> 包裹的推理段;不支持的模型在 v3 track 会静默退化为自由问诊。
How to get a model on the leaderboard
Evaluation runs entirely on our infrastructure. The real-clinical data and the hidden test split never leave our servers; the synthetic LingxiDiag-16K validation split is itself a public Hugging Face dataset (XuShihao6715/LingxiDiag-16k) and is reproducible locally. For now, every cell is handled by contacting us. Send us an OpenAI-compatible endpoint or OpenRouter id and we run the full evaluation, then publish the result after review.
Please include:
- Model / method name & version — 模型或方法名称及版本
- An OpenAI-compatible endpoint or OpenRouter id — 接口地址或 OpenRouter id
- Which track(s) & dataset(s) you want evaluated — 评测哪个 track / 数据集
- Contact person & institution — 联系人与机构
Self-serve results upload on the public synthetic split. The LingxiDiag-16K validation inputs are already on Hugging Face (XuShihao6715/LingxiDiag-16k); we plan to open a self-serve flow on top of that — run your model locally and upload a fixed-format predictions JSON, and we score it against held-out ground truth on our side and publish the comparison automatically, no need to contact us. The real clinical data and the hidden test split are never released, which keeps the official numbers trustworthy. 自助评测将基于已公开的 LingxiDiag-16K 合成数据:你本地跑完上传 predictions,我们在内网对照标准答案打分并自动上榜对比;真实临床数据与隐藏测试集始终不外发。