Deploying a good clinical LLM requires more than just acing closed-ended medical QA exams. It needs to be safe, ethical, comprehensive in its responses, and capable of reasoning and tackling complex medical tasks. The MEDIC framework aims to provide a transparent and comprehensive evaluation of LLM performance across various clinically relevant dimensions.
Disclaimer: It is important to note that the purpose of this evaluation is purely academic and exploratory. The models assessed here have not been approved for clinical use, and their results should not be interpreted as clinically validated. The leaderboard serves as a platform for researchers to compare models, understand their strengths and limitations, and drive further advancements in the field of clinical NLP.
🟦 🏥 | 2124 | +18/-20 | 8.95 | +0.02/-0.01 |
🟦 🏥 | 2124 | +18/-20 | 8.95 | +0.02/-0.01 | false | false | ? | instruction-tuned | Original | cc-by-nc-4.0 | 1149 | 122.61 | 2024-10-25 07:09:19+00:00 |
🟦 🏥 | 1.02 | +0.005/-0.006 |
🟦 🏥 | 1.02 | +0.005/-0.006 | 1.01 | 1.02 | 1.02 | 1.03 | 1.03 | 1.05 | 1.01 | 1.04 | 1.02 | false | false | ? | instruction-tuned | Original | cc-by-nc-4.0 | 1149 | 122.61 | 2024-11-28 04:57:07+00:00 |
- Coverage: Measures how thoroughly the summary covers the original document. A higher score means the summary includes more details from the original.
- Conformity: Also called the non-contradiction score, this checks if the summary avoids contradicting the original document. A higher score means the summary aligns better with the original.
- Consistency: Measures the level of non-hallucination, or how much the summary sticks to the facts in the document. A higher score means the summary is more factual and accurate.
- Conciseness: Measures how brief the summary is. A higher score means the summary is more concise. A negative score means the summary is longer than the original document.
- Overall Score: The average of coverage, conformity, consistency, and the harmonic mean of coverage and conciseness (if both are positive, otherwise 0).
🟦 🏥 | 87.82 | 64.31 | 95.78 | 99.61 | -218.26 |
🟦 🏥 | 87.82 | 64.31 | 95.78 | 99.61 | -218.26 | false | false | ? | instruction-tuned | Original | cc-by-nc-4.0 | 1980 | 122.61 | 2024-10-22 23:04:13+00:00 |
- Coverage: Measures how thoroughly the summary covers the original document. A higher score means the summary includes more details from the original.
- Conformity: Also called the non-contradiction score, this checks if the summary avoids contradicting the original document. A higher score means the summary aligns better with the original.
- Consistency: Measures the level of non-hallucination, or how much the summary sticks to the facts in the document. A higher score means the summary is more factual and accurate.
- Overall Score: The average of the above three scores.
🟦 🏥 | 95.14 | 86.25 | 97.42 | 98.33 |
🟦 🏥 | 95.14 | 86.25 | 97.42 | 98.33 | false | false | ? | instruction-tuned | Original | cc-by-nc-4.0 | 1149 | 122.61 | 2024-10-24 06:24:59+00:00 |
🟢 🏥 | 94.23 | 89.33 | 97.45 | 95.92 |
🟢 🏥 | 94.23 | 89.33 | 97.45 | 95.92 | false | false | ? | instruction-tuned | Original | cc-by-nc-4.0 | 1980 | 122.61 | 2024-10-22 23:04:13+00:00 |
🟢 🏥 | 80.02 | 86.97 | 65.33 | 70.95 | 79.26 | 91.95 | 80.6 | 86.38 |
🟢 🏥 | 80.02 | 86.97 | 65.33 | 70.95 | 79.26 | 91.95 | 80.6 | 86.38 | false | false | ? | instruction-tuned | Original | cc-by-nc-sa-4.0 | 1417 | 70.55 | 2024-11-11 13:58:37+00:00 |
About
The MEDIC Leaderboard evaluates large language models (LLMs) on various healthcare tasks across five key dimensions. Designed to bridge the gap between stakeholder expectations and practical clinical applications, the MEDIC framework captures the interconnected capabilities LLMs need for real-world use. Its evaluation metrics objectively measure LLM performance on benchmark tasks and map results to the MEDIC dimensions. By assessing these dimensions, MEDIC aims to determine how effective and safe LLMs are for real-world healthcare settings.
Evaluation Categories
Close-ended Questions
This category measures the accuracy of an LLM's medical knowledge by having it answer multiple-choice questions from datasets like MedQA, MedMCQA, MMLU, MMLU Pro, PubMedQA, USMLE and Toxigen.
We used the Eleuther AI's Evaluation Harness framework, which focuses on the likelihood of a model generating each proposed answer rather than directly evaluating the generated text itself. We modified the framework's codebase to provide more detailed and relevant results. Rather than just calculating the probability of generating answer choice labels (e.g., a., b., c., or d.), we calculate the probability of generating the full answer text.
Open-ended Questions
This category assesses the quality of the LLM's reasoning and explanations. The LLM is tasked with answering open-ended medical questions from various datasets:
Each question is presented to the models without special prompting to test their baseline capabilities. To compare models, we use a tournament-style approach. A judge (Llama3.1 70b Instruct) evaluates pairs of responses to the same question from different models. To eliminate position bias, each comparison is performed twice with reversed response positions. If the winner changes when positions are swapped, we consider the responses too close and declare a tie. After multiple comparisons, we calculate win rates and convert them to Elo ratings to rank the models. It's important to note that this evaluation only assesses the quality of response writing, not medical accuracy. To properly evaluate clinical accuracy, a thorough study involving real healthcare professionals would be necessary.
Medical Safety
Medical Safety category uses the "med-safety" benchmark dataset, which consists of 4500 scenarios presenting harmful medical requests. These scenarios cover all nine principles of medical ethics as defined by the American Medical Association (AMA). In this evaluation, an LLM acts as a judge. Each response from the evaluated model is assigned a "harmfulness" score on a predefined scale. A low score means the model refused the harmful request, while a high score means it complied with the request.
Medical Summarization
This category evaluates the LLM's ability to summarize medical texts, with a focus on clinical trial descriptions from ClinicalTrials.gov. The dataset consists of 1629 carefully selected clinical trial protocols with detailed study descriptions (3000-8000 tokens long). The task is to generate concise and accurate summaries of these protocols.
It uses a novel "cross-examination" framework, where questions are generated from the original document and the LLM's summary to assess the scores of the summary. The four key scores calculated are:
- Coverage: Measures how thoroughly the summary covers the original document. A higher score means the summary includes more details from the original.
- Conformity: Also called the non-contradiction score, this checks if the summary avoids contradicting the original document. A higher score means the summary aligns better with the original.
- Consistency: Measures the level of non-hallucination, or how much the summary sticks to the facts in the document. A higher score means the summary is more factual and accurate.
- Conciseness: Measures how brief the summary is. A higher score means the summary is more concise. A negative score means the summary is longer than the original document.
Note Generation
This category assesses the LLM's ability to generate structured clinical notes from doctor-patient conversations. It uses the same cross-examination framework as Medical Summarization across two datasets:
ACI-Bench: A comprehensive collection designed specifically for benchmarking clinical note generation from doctor-patient dialogues. The dataset contains patient visit notes that have been validated by expert medical scribes and physicians.
SOAP Notes: Using the test split of the ChartNote dataset containing 250 synthetic patient-doctor conversations generated from real clinical notes. The task involves generating notes in the SOAP format with the following sections:
- Subjective: Patient's description of symptoms, medical history, and personal experiences
- Objective: Observable data like physical exam findings, vital signs, and diagnostic test results
- Assessment: Healthcare provider's diagnosis based on subjective and objective information
- Plan: Treatment plan including medications, therapies, follow-ups, and referrals
Currently, the benchmark supports evaluation for models hosted on the huggingface hub and of decoder type. It doesn't support adapter models yet but we will soon add adapters too.
Submission Guide for the MEDIC Benchamark
First Steps Before Submitting a Model
1. Ensure Your Model Loads with AutoClasses
Verify that you can load your model and tokenizer using AutoClasses:
from transformers import AutoConfig, AutoModel, AutoTokenizer
config = AutoConfig.from_pretrained("your model name", revision=revision)
model = AutoModel.from_pretrained("your model name", revision=revision)
tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)
Note:
- If this step fails, debug your model before submitting.
- Ensure your model is public.
2. Convert Weights to Safetensors
Safetensors is a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the Extended Viewer!
3. Complete Your Model Card
When we add extra information about models to the leaderboard, it will be automatically taken from the model card
4. Select the correct model type
Choose the correct model cateogory from the option below:
- 🟢 : 🟢 pretrained model: new, base models, trained on a given text corpora using masked modelling or new, base models, continuously trained on further corpus (which may include IFT/chat data) using masked modelling
- ⭕ : ⭕ fine-tuned models: pretrained models finetuned on more data or tasks.
- 🟦 : 🟦 preference-tuned models: chat like fine-tunes, either using IFT (datasets of task instruction), RLHF or DPO (changing the model loss a bit with an added policy), etc
5. Select Correct Precision
Choose the right precision to avoid evaluation errors:
- Not all models convert properly from float16 to bfloat16.
- Incorrect precision can cause issues (e.g., loading a bf16 model in fp16 may generate NaNs).
- If you have selected auto, the precision mentioned under
torch_dtypeunder model config will be used.
6. Medically oriented model
If the model has been specifically built for medical domains i.e. pretrained/finetuned on significant medical data, make sure check the Domain specific checkbox
7. Chat template
Select this option if your model uses a chat template. The chat template will be used during evaluation.
- Before submitting, make sure the chat template is defined in tokenizer config.
Upon successful submission of your request, your model's result would be updated on the leaderboard within 5 working days!
main | false | instruction-tuned | auto | Original | FINISHED | CHAT TEMPLATE MISSING | CHAT TEMPLATE MISSING | CHAT TEMPLATE MISSING | CHAT TEMPLATE MISSING |
main | false | instruction-tuned | bfloat16 | Original | PENDING | FINISHED | FINISHED | FINISHED | FINISHED |
✉️✨ Submit your model here!
Is your model medically oriented?
Is your model a chat model?