How to Evaluate Medical Image Segmentation Models So Reviewers Trust Your Results

Medical image segmentation is one of the most popular thesis and paper topics in healthcare AI. U-Net and its many variants, transformer-based models and newer foundation models make it easier than ever to train a network that outlines tumours, organs or lesions. Getting a high number in a results table is the easy part. Proving that the number is honest, reproducible and clinically meaningful is where many manuscripts run into trouble with reviewers. This guide walks through the evaluation decisions that matter most, so your segmentation results stand up to scrutiny.

1. Split your data by patient, not by image

The single most common problem in medical imaging papers is data leakage. A CT or MRI scan contains many slices, and a patient may have several scans or several 2D crops. If slices from the same patient appear in both the training and the test set, the model can recognise anatomy it has effectively already seen, and the test score becomes optimistic.

• Split at the patient level (or the study level if each patient has one study), and do it before any augmentation or patch extraction.
• Keep a truly held-out test set that is never used for tuning hyperparameters, choosing epochs or selecting thresholds. Use a separate validation split or cross-validation for those choices.
• State the split clearly in the methods: number of patients and images in each set, and how the split was made. If you used a public dataset with an official split, say so and use it.

2. Choose metrics that match the clinical question

The Dice similarity coefficient is the default metric, and you should usually report it, but Dice alone hides important failure modes. It rewards overlap, so a model can achieve a respectable Dice while producing jagged boundaries or small false-positive islands far from the target.

• Dice and IoU (Jaccard) measure overlap. They are closely related, so reporting both adds little new information; pick one as primary and explain why.
• Boundary metrics such as the Hausdorff distance (often the 95th percentile, HD95) or the average surface distance tell you how far the predicted contour strays from the reference. They matter when the outline itself is used, for example in radiotherapy planning.
• Detection-style measures such as lesion-wise sensitivity and false positives per scan are useful when the task involves many small lesions, where one missed lesion barely changes Dice but matters clinically.
• Volume agreement (for example, the difference between predicted and reference volumes) is relevant when the downstream use is measurement.

Community guidance such as the Metrics Reloaded recommendations is worth reading before you finalise your metric set, because it explains which metrics suit which problem types.

3. Decide how to handle empty cases and averaging

Two details that are often left unstated can change your headline number noticeably. First, what happens when the reference mask is empty (no lesion present) and the prediction is also empty? Dice is mathematically undefined there, and different code libraries treat it differently. Second, are you averaging per image, per patient, or pooling all pixels into one global score? Each choice answers a different question. Report your choice explicitly, and keep it consistent across all compared methods.

4. Report variability, not just averages

A single mean Dice hides how the model behaves on difficult cases. Reviewers increasingly expect:

• Mean with standard deviation, or median with interquartile range when the distribution is skewed.
• Box plots or per-case scatter plots so readers can see outliers.
• Results over several training runs with different random seeds, or across cross-validation folds, rather than one lucky run.
• A paired statistical test (for example, the Wilcoxon signed-rank test on per-case scores) when you claim that your method beats a baseline, plus a confidence interval for the difference.

5. Compare against fair, strong baselines

Comparing a carefully tuned new model against a baseline trained with default settings is a frequent reviewer complaint. Give baselines the same preprocessing, data split, augmentation budget and training effort. Self-configuring frameworks such as nnU-Net are widely used as strong reference points in segmentation research, and including one is often a good way to show that your gains are real. If you copy baseline numbers from another paper, make sure the data split and metric definitions are identical, and say so.

6. Ablations that explain your contribution

If your model adds an attention block, a new loss function and a post-processing step, an ablation study should show what each component contributes. Remove one element at a time, keep everything else fixed, and report the same metrics. This turns "our model is better" into "this specific idea helps, and here is the evidence", which is far more convincing.

7. Test generalisation honestly

Medical images vary between hospitals, scanner vendors, acquisition protocols and patient populations. A model trained and tested on one centre's data may perform worse elsewhere. Where possible, evaluate on an external dataset from a different source. If you cannot, describe this clearly as a limitation rather than implying broad clinical readiness.

8. Show qualitative results, including failures

Figures with the image, the reference mask and the prediction overlaid help readers judge quality. Include typical cases and some of the worst cases, with a short note on why the model struggled (low contrast, small targets, artefacts, unusual anatomy). Honest failure analysis builds trust and often gives you a strong discussion section.

9. Make it reproducible

State the framework and version, image preprocessing (resampling, intensity normalisation, cropping), augmentation, loss, optimiser, learning-rate schedule, number of epochs, hardware, inference settings and any post-processing. Share code and trained weights if your institution and data agreements allow it. Reporting guidelines for AI in medical imaging, such as the CLAIM checklist, are a useful way to check nothing important is missing.

A quick pre-submission checklist

• Patient-level split with no overlap, and a test set untouched during tuning.
• An overlap metric plus at least one boundary or detection metric suited to the task.
• Empty-case handling and averaging method stated.
• Variability, multiple seeds or folds, and paired statistical tests for comparisons.
• Fairly trained baselines and a clear ablation study.
• External validation, or an honest limitation statement.
• Qualitative figures including failure cases, and full reproducibility details.