This webinar series by the SIG for Challenges aims to educate medical imaging researchers on how to successfully participate in and conduct challenges. A particular focus will be placed on issues of quality control and validation using appropriate metrics from the organizers' and participants’ points of view.

(1) How to win a challenge (July 2022)

The recording of the webinar can be found online under this link.

Invited speakers:

Dr. Fabian Isensee has consistently enabled the translation of state-of-the-art algorithms into real-world applications, represented by nnU-Net, the de-facto standard for segmentation in the medical domain. The methods he developed have won multiple international segmentation competitions.

Dr. James Howard is an academic cardiologist who has published numerous research papers using AI to interpret X-rays, cardiac ultrasound, ECG, MRI and cardiac pressure waveforms. He has entered several Kaggle competitions, including the Deepfake Detection Challenge, where he won a gold medal and a $40,000 prize, against 2200 other teams.

(2) How to run a challenge (October 2022)

The recording of the webinar can be found online under this link.

Challenge platforms:

https://grand-challenge.org/ is an open source platform for running challenges. Recently added features include the possibility for challenge participants to upload algorithms that solve a challenge and give other users access to these algorithms to process their own data. Bram van Ginneken, Kiran Vaidhya Venkadesh and Anindo Saha will present how to use the platform for organizing high-profile challenges. The slides of the talk are available from the following link:  grand-challenge-slides.pdf

https://www.synapse.org/ is an open collaboration platform developed by Sage Bionetworks.  Synapse is the main platform supporting DREAM Challenges (dreamchallenges.org). Jake Albrecht from Sage will present tips for challenge organizers on how to define a successful community challenge, with examples from Synapse. The slides of the talk are available from the following link: synapse-slides.pdf

(3) Metrics Reloaded: From segmentation to calibration (February 2023)

The recording of the webinar can be found online under this link.

Invited speakers:

Paul F. Jaeger is a principal investigator at the Interactive Machine Learning Group at the German Cancer Research Center and Helmholtz Imaging. His research focuses on image analysis algorithms, with a particular focus on human interaction. Paul won numerous international competitions on biomedical image analysis and first-authored relevant contributions to the field in high impact journals and conferences like Nature Methods or ICLR. As founder and organizer of heidelberg.ai, Paul helps to connect over 2000 members of the local AI community at monthly events that attract top international scientists to Heidelberg. For his work, Paul received the “Richtzenhain Award for Translational Cancer Research” as well as the “Roland-Ernst Award for Interdisciplinary Research in Radiology”.

Dr. Annika Reinke joined the division of Intelligent Medical Systems at the German Cancer Research Center (DKFZ) to adapt mathematical concepts to societally relevant topics, like scientific benchmarking and validation. Having published disruptive findings on biomedical image analysis challenges in Nature Communications, she is a founding member of the initiative of Biomedical Image Analysis ChallengeS (BIAS) aiming for bringing biomedical image analysis challenges to the next level of quality. She serves as the secretary of the MICCAI special interest group on biomedical challenges and as an active member and taskforce lead of the MONAI working group on evaluation, reproducibility and benchmarking.

Dr. Florian Buettner is a physicist by training and earned his PhD in physics from the University of London/Institute of Cancer Research in 2011. He then focussed his research efforts on bioinformatics and machine learning at the Helmholtz Zentrum München and the European Bioinformatics Institute in Cambridge. He subsequently transitioned to industry and worked as an expert in artificial intelligence at Siemens AG. Now a professor at Goethe University Frankfurt and the German Cancer Research Center (DKFZ)/German Consortium for Translational Cancer Research (DKTK), Florian is currently doing research at the interface between (single-cell) bioinformatics, machine learning and oncology. In collaboration-driven research, he contributes to developing a better understanding of the molecular heterogeneity of cancer by developing interpretable and trustworthy machine learning methods.

(4) Performance Reporting in Medical Imaging AI: Current Practices, Strength of Outperformance Claims and Areas for Improvement (June 2025)

Registration under this link.

Invited speakers:

Dr. Evangelia Christodoulou holds a background in Mathematics and Biostatistics and completed her PhD in Clinical Prediction Modelling at KU Leuven, Belgium, supervised by Prof. Ben Van Calster,  where she collaborated with oncologists and statisticians to advance methods for validating predictive algorithms. In February 2021, she joined the German Cancer Research Center (DKFZ) in Heidelberg, Germany and was awarded a postdoctoral fellowship in 2022 within the AI Health Innovation Cluster, led by Prof. Dr. Lena Maier-Hein. Her current research focuses on the development of robust and reliable AI-based models for clinical outcome prediction in the context of Surgical Data Science. She also works on methodological contributions that address critical challenges in the validation of AI methods for biomedical imaging analysis, with particular emphasis on model performance uncertainty and dataset size considerations.

Dr. Olivier Colliot is a Research Director at CNRS (Division of Computer Science) and the co-head of the ARAMIS team at the Paris Brain Institute. He also holds a chair at the Paris Institute for Artificial Intelligence (PRAIRIE). He has been working for over twenty years on the design and validation of machine learning approaches to better understand, model, diagnose, predict and prevent brain disorders. He is an Associate Editor of Medical Image Analysis, IEEE Transactions on Medical Imaging and the SPIE Journal of Medical Imaging. His current research has a strong focus on statistical aspects of evaluation and benchmarking of AI models. He is a member of the special interest group on biomedical challenges of the MONAI working group on evaluation, reproducibility and benchmarking.

Acknowledgements

This webinar series was partially initiated by the Helmholtz Association of German Research Centers in the scope of the Helmholtz Imaging Incubator (HI).