The STOIC2021 COVID-19 AI challenge: Applying reusable training methodologies to private data

Challenges drive the state-of-the-art of automated medical image analysis. The quantity of public training data that they provide can limit the performance of their solutions. Public access to the training methodology for these solutions remains absent. This study implements the Type Three (T3) chal...

Full description

Saved in:
Bibliographic Details
Published in:Medical image analysis 2024-10, Vol.97, p.103230, Article 103230
Main Authors: Boulogne, Luuk H., Lorenz, Julian, Kienzle, Daniel, Schön, Robin, Ludwig, Katja, Lienhart, Rainer, Jégou, Simon, Li, Guang, Chen, Cong, Wang, Qi, Shi, Derik, Maniparambil, Mayug, Müller, Dominik, Mertes, Silvan, Schröter, Niklas, Hellmann, Fabio, Elia, Miriam, Dirks, Ine, Bossa, Matías Nicolás, Berenguer, Abel Díaz, Mukherjee, Tanmoy, Vandemeulebroucke, Jef, Sahli, Hichem, Deligiannis, Nikos, Gonidakis, Panagiotis, Huynh, Ngoc Dung, Razzak, Imran, Bouadjenek, Reda, Verdicchio, Mario, Borrelli, Pasquale, Aiello, Marco, Meakin, James A., Lemm, Alexander, Russ, Christoph, Ionasec, Razvan, Paragios, Nikos, van Ginneken, Bram, Revel, Marie-Pierre
Format: Article
Language:English
Subjects:
COVID-19
Machine learning
Medical image analysis challenge
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Challenges drive the state-of-the-art of automated medical image analysis. The quantity of public training data that they provide can limit the performance of their solutions. Public access to the training methodology for these solutions remains absent. This study implements the Type Three (T3) challenge format, which allows for training solutions on private data and guarantees reusable training methodologies. With T3, challenge organizers train a codebase provided by the participants on sequestered training data. T3 was implemented in the STOIC2021 challenge, with the goal of predicting from a computed tomography (CT) scan whether subjects had a severe COVID-19 infection, defined as intubation or death within one month. STOIC2021 consisted of a Qualification phase, where participants developed challenge solutions using 2000 publicly available CT scans, and a Final phase, where participants submitted their training methodologies with which solutions were trained on CT scans of 9724 subjects. The organizers successfully trained six of the eight Final phase submissions. The submitted codebases for training and running inference were released publicly. The winning solution obtained an area under the receiver operating characteristic curve for discerning between severe and non-severe COVID-19 of 0.815. The Final phase solutions of all finalists improved upon their Qualification phase solutions. [Display omitted] •STOIC2021, aimed at detecting severe COVID-19, was organized with 10,724 CT scans.•The T3 challenge format allows training on private data and ensures reusable methods.•CT scans and metadata of 2,000 COVID-19 subjects were released under CC-BY-NC 4.0.•The finalist codebases were released publicly under permissive licenses.
ISSN:1361-8415
1361-8423
1361-8423
DOI:10.1016/j.media.2024.103230