A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis
An automated, accurate method is needed for unbiased assessment quantifying accrual of joint space narrowing and erosions on radiographic images of the hands and wrists, and feet for clinical trials, monitoring of joint damage over time, assisting rheumatologists with treatment decisions. Such a met...
Saved in:
| Published in: | JAMA network open 2022-08, Vol.5 (8), p.e2227423-e2227423 |
|---|---|
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
| Format: | Article |
| Language: | eng |
| Subjects: | |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| recordid |
cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_9425151 |
|---|---|
| title |
A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis |
| format |
Article |
| creator |
Sun, Dongmei Nguyen, Thanh M Allaway, Robert J Wang, Jelai Chung, Verena Yu, Thomas V Mason, Michael Dimitrovsky, Isaac Ericson, Lars Li, Hongyang Guan, Yuanfang Israel, Ariel Olar, Alex Pataki, Balint Armin Stolovitzky, Gustavo Guinney, Justin Gulko, Percio S Frazier, Mason B Chen, Jake Y Costello, James C Bridges, Jr, S Louis |
| subjects |
Algorithms Arthritis, Rheumatoid - diagnostic imaging Arthritis, Rheumatoid - drug therapy Bayes Theorem Crowdsourcing Electronic health records Humans Machine Learning Online Only Original Investigation Reproducibility of Results Rheumatoid arthritis Statistics and Research Methods |
| ispartof |
JAMA network open, 2022-08, Vol.5 (8), p.e2227423-e2227423 |
| description |
An automated, accurate method is needed for unbiased assessment quantifying accrual of joint space narrowing and erosions on radiographic images of the hands and wrists, and feet for clinical trials, monitoring of joint damage over time, assisting rheumatologists with treatment decisions. Such a method has the potential to be directly integrated into electronic health records.
To design and implement an international crowdsourcing competition to catalyze the development of machine learning methods to quantify radiographic damage in rheumatoid arthritis (RA).
This diagnostic/prognostic study describes the Rheumatoid Arthritis 2-Dialogue for Reverse Engineering Assessment and Methods (RA2-DREAM Challenge), which used existing radiographic images and expert-curated Sharp-van der Heijde (SvH) scores from 2 clinical studies (674 radiographic sets from 562 patients) for training (367 sets), leaderboard (119 sets), and final evaluation (188 sets). Challenge participants were tasked with developing methods to automatically quantify overall damage (subchallenge 1), joint space narrowing (subchallenge 2), and erosions (subchallenge 3). The challenge was finished on June 30, 2020.
Scores derived from submitted algorithms were compared with the expert-curated SvH scores, and a baseline model was created for benchmark comparison. Performances were ranked using weighted root mean square error (RMSE). The performance and reproductivity of each algorithm was assessed using Bayes factor from bootstrapped data, and further evaluated with a postchallenge independent validation data set.
The RA2-DREAM Challenge received a total of 173 submissions from 26 participants or teams in 7 countries for the leaderboard round, and 13 submissions were included in the final evaluation. The weighted RMSEs metric showed that the winning algorithms produced scores that were very close to the expert-curated SvH scores. Top teams included Team Shirin for subchallenge 1 (weighted RMSE, 0.44), HYL-YFG (Hongyang Li and Yuanfang Guan) subchallenge 2 (weighted RMSE, 0.38), and Gold Therapy for subchallenge 3 (weighted RMSE, 0.43). Bootstrapping/Bayes factor approach and the postchallenge independent validation confirmed the reproducibility and the estimation concordance indices between final evaluation and postchallenge independent validation data set were 0.71 for subchallenge 1, 0.78 for subchallenge 2, and 0.82 for subchallenge 3.
The RA2-DREAM Challenge resulted in the development of algorithm |
| language |
eng |
| source |
ProQuest - Publicly Available Content Database |
| identifier |
ISSN: 2574-3805 |
| fulltext |
fulltext |
| issn |
2574-3805 2574-3805 |
| url |
http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-05-23T02%3A42%3A33IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_pubme&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=A%20Crowdsourcing%20Approach%20to%20Develop%20Machine%20Learning%20Models%20to%20Quantify%20Radiographic%20Joint%20Damage%20in%20Rheumatoid%20Arthritis&rft.jtitle=JAMA%20network%20open&rft.au=Sun,%20Dongmei&rft.aucorp=RA2-DREAM%20Challenge%20Community&rft.date=2022-08-01&rft.volume=5&rft.issue=8&rft.spage=e2227423&rft.epage=e2227423&rft.pages=e2227423-e2227423&rft.issn=2574-3805&rft.eissn=2574-3805&rft_id=info:doi/10.1001/jamanetworkopen.2022.27423&rft_dat=%3Cproquest_pubme%3E2736877742%3C/proquest_pubme%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c454t-c569c26758e5b1f72e10e0ba95805476329b2171f9ebe208da4f478f7d4001623%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2736877742&rft_id=info:pmid/36036935 |
| container_title |
JAMA network open |
| container_volume |
5 |
| container_issue |
8 |
| container_start_page |
e2227423 |
| container_end_page |
e2227423 |
| fullrecord |
<record><control><sourceid>proquest_pubme</sourceid><recordid>TN_cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_9425151</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2736877742</sourcerecordid><originalsourceid>FETCH-LOGICAL-c454t-c569c26758e5b1f72e10e0ba95805476329b2171f9ebe208da4f478f7d4001623</originalsourceid><addsrcrecordid>eNpdkU1v1DAQhi0EotXSv4AsuHDZxZ9xwgFptYUWtBWigrPlJJONl6wdbKfV_nscWqrSk235mdG88yD0hpIVJYS-35uDcZBuffjlR3ArRhhbMSUYf4ZOmVRiyUsinz-6n6CzGPeEEEYorwr5Ep3wgvCi4vIUHdd4E_xtG_0UGut2eD2OwZumx8njc7iBwY_4Kr-tA7wFE9wMXfkWhjgj3yfjku2O-Nq01u-CGXvb4K_euoTP86g7wNbh6x6mg0netngdUh9ssvEVetGZIcLZ_blAPz9_-rG5XG6_XXzZrLfLRkiRlo0sqoYVSpYga9opBpQAqU0lczShCs6qmlFFuwpqYKRsjeiEKjvViryugvEF-njXd5zqA7QNuBTMoMdgDyYctTdW___jbK93_kZXgkkqaW7w7r5B8L8niEkfbGxgGLIHP0XNFFGlEpWa0bdP0H3eq8vxMsWLUqlZ1AJ9uKOa4GMM0D0MQ4meJesnkvUsWf-VnItfP47zUPpPKf8DMmepXg</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><isCDI>true</isCDI><recordtype>article</recordtype><pqid>2736877742</pqid></control><display><type>article</type><title>A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis</title><source>ProQuest - Publicly Available Content Database</source><creator>Sun, Dongmei ; Nguyen, Thanh M ; Allaway, Robert J ; Wang, Jelai ; Chung, Verena ; Yu, Thomas V ; Mason, Michael ; Dimitrovsky, Isaac ; Ericson, Lars ; Li, Hongyang ; Guan, Yuanfang ; Israel, Ariel ; Olar, Alex ; Pataki, Balint Armin ; Stolovitzky, Gustavo ; Guinney, Justin ; Gulko, Percio S ; Frazier, Mason B ; Chen, Jake Y ; Costello, James C ; Bridges, Jr, S Louis</creator><creatorcontrib>Sun, Dongmei ; Nguyen, Thanh M ; Allaway, Robert J ; Wang, Jelai ; Chung, Verena ; Yu, Thomas V ; Mason, Michael ; Dimitrovsky, Isaac ; Ericson, Lars ; Li, Hongyang ; Guan, Yuanfang ; Israel, Ariel ; Olar, Alex ; Pataki, Balint Armin ; Stolovitzky, Gustavo ; Guinney, Justin ; Gulko, Percio S ; Frazier, Mason B ; Chen, Jake Y ; Costello, James C ; Bridges, Jr, S Louis ; RA2-DREAM Challenge Community</creatorcontrib><description>An automated, accurate method is needed for unbiased assessment quantifying accrual of joint space narrowing and erosions on radiographic images of the hands and wrists, and feet for clinical trials, monitoring of joint damage over time, assisting rheumatologists with treatment decisions. Such a method has the potential to be directly integrated into electronic health records.
To design and implement an international crowdsourcing competition to catalyze the development of machine learning methods to quantify radiographic damage in rheumatoid arthritis (RA).
This diagnostic/prognostic study describes the Rheumatoid Arthritis 2-Dialogue for Reverse Engineering Assessment and Methods (RA2-DREAM Challenge), which used existing radiographic images and expert-curated Sharp-van der Heijde (SvH) scores from 2 clinical studies (674 radiographic sets from 562 patients) for training (367 sets), leaderboard (119 sets), and final evaluation (188 sets). Challenge participants were tasked with developing methods to automatically quantify overall damage (subchallenge 1), joint space narrowing (subchallenge 2), and erosions (subchallenge 3). The challenge was finished on June 30, 2020.
Scores derived from submitted algorithms were compared with the expert-curated SvH scores, and a baseline model was created for benchmark comparison. Performances were ranked using weighted root mean square error (RMSE). The performance and reproductivity of each algorithm was assessed using Bayes factor from bootstrapped data, and further evaluated with a postchallenge independent validation data set.
The RA2-DREAM Challenge received a total of 173 submissions from 26 participants or teams in 7 countries for the leaderboard round, and 13 submissions were included in the final evaluation. The weighted RMSEs metric showed that the winning algorithms produced scores that were very close to the expert-curated SvH scores. Top teams included Team Shirin for subchallenge 1 (weighted RMSE, 0.44), HYL-YFG (Hongyang Li and Yuanfang Guan) subchallenge 2 (weighted RMSE, 0.38), and Gold Therapy for subchallenge 3 (weighted RMSE, 0.43). Bootstrapping/Bayes factor approach and the postchallenge independent validation confirmed the reproducibility and the estimation concordance indices between final evaluation and postchallenge independent validation data set were 0.71 for subchallenge 1, 0.78 for subchallenge 2, and 0.82 for subchallenge 3.
The RA2-DREAM Challenge resulted in the development of algorithms that provide feasible, quick, and accurate methods to quantify joint damage in RA. Ultimately, these methods could help research studies on RA joint damage and may be integrated into electronic health records to help clinicians serve patients better by providing timely, reliable, and quantitative information for making treatment decisions to prevent further damage.</description><identifier>ISSN: 2574-3805</identifier><identifier>EISSN: 2574-3805</identifier><identifier>DOI: 10.1001/jamanetworkopen.2022.27423</identifier><identifier>PMID: 36036935</identifier><language>eng</language><publisher>United States: American Medical Association</publisher><subject>Algorithms ; Arthritis, Rheumatoid - diagnostic imaging ; Arthritis, Rheumatoid - drug therapy ; Bayes Theorem ; Crowdsourcing ; Electronic health records ; Humans ; Machine Learning ; Online Only ; Original Investigation ; Reproducibility of Results ; Rheumatoid arthritis ; Statistics and Research Methods</subject><ispartof>JAMA network open, 2022-08, Vol.5 (8), p.e2227423-e2227423</ispartof><rights>2022. This work is published under https://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>Copyright 2022 Sun D et al. .</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c454t-c569c26758e5b1f72e10e0ba95805476329b2171f9ebe208da4f478f7d4001623</citedby><cites>FETCH-LOGICAL-c454t-c569c26758e5b1f72e10e0ba95805476329b2171f9ebe208da4f478f7d4001623</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/2736877742?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>230,315,787,791,892,25799,27985,27986,37077,37078,44955</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/36036935$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Sun, Dongmei</creatorcontrib><creatorcontrib>Nguyen, Thanh M</creatorcontrib><creatorcontrib>Allaway, Robert J</creatorcontrib><creatorcontrib>Wang, Jelai</creatorcontrib><creatorcontrib>Chung, Verena</creatorcontrib><creatorcontrib>Yu, Thomas V</creatorcontrib><creatorcontrib>Mason, Michael</creatorcontrib><creatorcontrib>Dimitrovsky, Isaac</creatorcontrib><creatorcontrib>Ericson, Lars</creatorcontrib><creatorcontrib>Li, Hongyang</creatorcontrib><creatorcontrib>Guan, Yuanfang</creatorcontrib><creatorcontrib>Israel, Ariel</creatorcontrib><creatorcontrib>Olar, Alex</creatorcontrib><creatorcontrib>Pataki, Balint Armin</creatorcontrib><creatorcontrib>Stolovitzky, Gustavo</creatorcontrib><creatorcontrib>Guinney, Justin</creatorcontrib><creatorcontrib>Gulko, Percio S</creatorcontrib><creatorcontrib>Frazier, Mason B</creatorcontrib><creatorcontrib>Chen, Jake Y</creatorcontrib><creatorcontrib>Costello, James C</creatorcontrib><creatorcontrib>Bridges, Jr, S Louis</creatorcontrib><creatorcontrib>RA2-DREAM Challenge Community</creatorcontrib><title>A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis</title><title>JAMA network open</title><addtitle>JAMA Netw Open</addtitle><description>An automated, accurate method is needed for unbiased assessment quantifying accrual of joint space narrowing and erosions on radiographic images of the hands and wrists, and feet for clinical trials, monitoring of joint damage over time, assisting rheumatologists with treatment decisions. Such a method has the potential to be directly integrated into electronic health records.
To design and implement an international crowdsourcing competition to catalyze the development of machine learning methods to quantify radiographic damage in rheumatoid arthritis (RA).
This diagnostic/prognostic study describes the Rheumatoid Arthritis 2-Dialogue for Reverse Engineering Assessment and Methods (RA2-DREAM Challenge), which used existing radiographic images and expert-curated Sharp-van der Heijde (SvH) scores from 2 clinical studies (674 radiographic sets from 562 patients) for training (367 sets), leaderboard (119 sets), and final evaluation (188 sets). Challenge participants were tasked with developing methods to automatically quantify overall damage (subchallenge 1), joint space narrowing (subchallenge 2), and erosions (subchallenge 3). The challenge was finished on June 30, 2020.
Scores derived from submitted algorithms were compared with the expert-curated SvH scores, and a baseline model was created for benchmark comparison. Performances were ranked using weighted root mean square error (RMSE). The performance and reproductivity of each algorithm was assessed using Bayes factor from bootstrapped data, and further evaluated with a postchallenge independent validation data set.
The RA2-DREAM Challenge received a total of 173 submissions from 26 participants or teams in 7 countries for the leaderboard round, and 13 submissions were included in the final evaluation. The weighted RMSEs metric showed that the winning algorithms produced scores that were very close to the expert-curated SvH scores. Top teams included Team Shirin for subchallenge 1 (weighted RMSE, 0.44), HYL-YFG (Hongyang Li and Yuanfang Guan) subchallenge 2 (weighted RMSE, 0.38), and Gold Therapy for subchallenge 3 (weighted RMSE, 0.43). Bootstrapping/Bayes factor approach and the postchallenge independent validation confirmed the reproducibility and the estimation concordance indices between final evaluation and postchallenge independent validation data set were 0.71 for subchallenge 1, 0.78 for subchallenge 2, and 0.82 for subchallenge 3.
The RA2-DREAM Challenge resulted in the development of algorithms that provide feasible, quick, and accurate methods to quantify joint damage in RA. Ultimately, these methods could help research studies on RA joint damage and may be integrated into electronic health records to help clinicians serve patients better by providing timely, reliable, and quantitative information for making treatment decisions to prevent further damage.</description><subject>Algorithms</subject><subject>Arthritis, Rheumatoid - diagnostic imaging</subject><subject>Arthritis, Rheumatoid - drug therapy</subject><subject>Bayes Theorem</subject><subject>Crowdsourcing</subject><subject>Electronic health records</subject><subject>Humans</subject><subject>Machine Learning</subject><subject>Online Only</subject><subject>Original Investigation</subject><subject>Reproducibility of Results</subject><subject>Rheumatoid arthritis</subject><subject>Statistics and Research Methods</subject><issn>2574-3805</issn><issn>2574-3805</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNpdkU1v1DAQhi0EotXSv4AsuHDZxZ9xwgFptYUWtBWigrPlJJONl6wdbKfV_nscWqrSk235mdG88yD0hpIVJYS-35uDcZBuffjlR3ArRhhbMSUYf4ZOmVRiyUsinz-6n6CzGPeEEEYorwr5Ep3wgvCi4vIUHdd4E_xtG_0UGut2eD2OwZumx8njc7iBwY_4Kr-tA7wFE9wMXfkWhjgj3yfjku2O-Nq01u-CGXvb4K_euoTP86g7wNbh6x6mg0netngdUh9ssvEVetGZIcLZ_blAPz9_-rG5XG6_XXzZrLfLRkiRlo0sqoYVSpYga9opBpQAqU0lczShCs6qmlFFuwpqYKRsjeiEKjvViryugvEF-njXd5zqA7QNuBTMoMdgDyYctTdW___jbK93_kZXgkkqaW7w7r5B8L8niEkfbGxgGLIHP0XNFFGlEpWa0bdP0H3eq8vxMsWLUqlZ1AJ9uKOa4GMM0D0MQ4meJesnkvUsWf-VnItfP47zUPpPKf8DMmepXg</recordid><startdate>20220801</startdate><enddate>20220801</enddate><creator>Sun, Dongmei</creator><creator>Nguyen, Thanh M</creator><creator>Allaway, Robert J</creator><creator>Wang, Jelai</creator><creator>Chung, Verena</creator><creator>Yu, Thomas V</creator><creator>Mason, Michael</creator><creator>Dimitrovsky, Isaac</creator><creator>Ericson, Lars</creator><creator>Li, Hongyang</creator><creator>Guan, Yuanfang</creator><creator>Israel, Ariel</creator><creator>Olar, Alex</creator><creator>Pataki, Balint Armin</creator><creator>Stolovitzky, Gustavo</creator><creator>Guinney, Justin</creator><creator>Gulko, Percio S</creator><creator>Frazier, Mason B</creator><creator>Chen, Jake Y</creator><creator>Costello, James C</creator><creator>Bridges, Jr, S Louis</creator><general>American Medical Association</general><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>7X7</scope><scope>7XB</scope><scope>8FI</scope><scope>8FJ</scope><scope>8FK</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>FYUFA</scope><scope>GHDGH</scope><scope>K9.</scope><scope>M0S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>7X8</scope><scope>5PM</scope></search><sort><creationdate>20220801</creationdate><title>A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis</title><author>Sun, Dongmei ; Nguyen, Thanh M ; Allaway, Robert J ; Wang, Jelai ; Chung, Verena ; Yu, Thomas V ; Mason, Michael ; Dimitrovsky, Isaac ; Ericson, Lars ; Li, Hongyang ; Guan, Yuanfang ; Israel, Ariel ; Olar, Alex ; Pataki, Balint Armin ; Stolovitzky, Gustavo ; Guinney, Justin ; Gulko, Percio S ; Frazier, Mason B ; Chen, Jake Y ; Costello, James C ; Bridges, Jr, S Louis</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c454t-c569c26758e5b1f72e10e0ba95805476329b2171f9ebe208da4f478f7d4001623</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Algorithms</topic><topic>Arthritis, Rheumatoid - diagnostic imaging</topic><topic>Arthritis, Rheumatoid - drug therapy</topic><topic>Bayes Theorem</topic><topic>Crowdsourcing</topic><topic>Electronic health records</topic><topic>Humans</topic><topic>Machine Learning</topic><topic>Online Only</topic><topic>Original Investigation</topic><topic>Reproducibility of Results</topic><topic>Rheumatoid arthritis</topic><topic>Statistics and Research Methods</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Sun, Dongmei</creatorcontrib><creatorcontrib>Nguyen, Thanh M</creatorcontrib><creatorcontrib>Allaway, Robert J</creatorcontrib><creatorcontrib>Wang, Jelai</creatorcontrib><creatorcontrib>Chung, Verena</creatorcontrib><creatorcontrib>Yu, Thomas V</creatorcontrib><creatorcontrib>Mason, Michael</creatorcontrib><creatorcontrib>Dimitrovsky, Isaac</creatorcontrib><creatorcontrib>Ericson, Lars</creatorcontrib><creatorcontrib>Li, Hongyang</creatorcontrib><creatorcontrib>Guan, Yuanfang</creatorcontrib><creatorcontrib>Israel, Ariel</creatorcontrib><creatorcontrib>Olar, Alex</creatorcontrib><creatorcontrib>Pataki, Balint Armin</creatorcontrib><creatorcontrib>Stolovitzky, Gustavo</creatorcontrib><creatorcontrib>Guinney, Justin</creatorcontrib><creatorcontrib>Gulko, Percio S</creatorcontrib><creatorcontrib>Frazier, Mason B</creatorcontrib><creatorcontrib>Chen, Jake Y</creatorcontrib><creatorcontrib>Costello, James C</creatorcontrib><creatorcontrib>Bridges, Jr, S Louis</creatorcontrib><creatorcontrib>RA2-DREAM Challenge Community</creatorcontrib><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>Health & Medical Collection</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>Hospital Premium Collection</collection><collection>Hospital Premium Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>AUTh Library subscriptions: ProQuest Central</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>Health Research Premium Collection</collection><collection>Health Research Premium Collection (Alumni)</collection><collection>ProQuest Health & Medical Complete (Alumni)</collection><collection>Health & Medical Collection (Alumni Edition)</collection><collection>ProQuest - Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><jtitle>JAMA network open</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Sun, Dongmei</au><au>Nguyen, Thanh M</au><au>Allaway, Robert J</au><au>Wang, Jelai</au><au>Chung, Verena</au><au>Yu, Thomas V</au><au>Mason, Michael</au><au>Dimitrovsky, Isaac</au><au>Ericson, Lars</au><au>Li, Hongyang</au><au>Guan, Yuanfang</au><au>Israel, Ariel</au><au>Olar, Alex</au><au>Pataki, Balint Armin</au><au>Stolovitzky, Gustavo</au><au>Guinney, Justin</au><au>Gulko, Percio S</au><au>Frazier, Mason B</au><au>Chen, Jake Y</au><au>Costello, James C</au><au>Bridges, Jr, S Louis</au><aucorp>RA2-DREAM Challenge Community</aucorp><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis</atitle><jtitle>JAMA network open</jtitle><addtitle>JAMA Netw Open</addtitle><date>2022-08-01</date><risdate>2022</risdate><volume>5</volume><issue>8</issue><spage>e2227423</spage><epage>e2227423</epage><pages>e2227423-e2227423</pages><issn>2574-3805</issn><eissn>2574-3805</eissn><notes>ObjectType-Article-1</notes><notes>SourceType-Scholarly Journals-1</notes><notes>ObjectType-Feature-2</notes><notes>content type line 23</notes><abstract>An automated, accurate method is needed for unbiased assessment quantifying accrual of joint space narrowing and erosions on radiographic images of the hands and wrists, and feet for clinical trials, monitoring of joint damage over time, assisting rheumatologists with treatment decisions. Such a method has the potential to be directly integrated into electronic health records.
To design and implement an international crowdsourcing competition to catalyze the development of machine learning methods to quantify radiographic damage in rheumatoid arthritis (RA).
This diagnostic/prognostic study describes the Rheumatoid Arthritis 2-Dialogue for Reverse Engineering Assessment and Methods (RA2-DREAM Challenge), which used existing radiographic images and expert-curated Sharp-van der Heijde (SvH) scores from 2 clinical studies (674 radiographic sets from 562 patients) for training (367 sets), leaderboard (119 sets), and final evaluation (188 sets). Challenge participants were tasked with developing methods to automatically quantify overall damage (subchallenge 1), joint space narrowing (subchallenge 2), and erosions (subchallenge 3). The challenge was finished on June 30, 2020.
Scores derived from submitted algorithms were compared with the expert-curated SvH scores, and a baseline model was created for benchmark comparison. Performances were ranked using weighted root mean square error (RMSE). The performance and reproductivity of each algorithm was assessed using Bayes factor from bootstrapped data, and further evaluated with a postchallenge independent validation data set.
The RA2-DREAM Challenge received a total of 173 submissions from 26 participants or teams in 7 countries for the leaderboard round, and 13 submissions were included in the final evaluation. The weighted RMSEs metric showed that the winning algorithms produced scores that were very close to the expert-curated SvH scores. Top teams included Team Shirin for subchallenge 1 (weighted RMSE, 0.44), HYL-YFG (Hongyang Li and Yuanfang Guan) subchallenge 2 (weighted RMSE, 0.38), and Gold Therapy for subchallenge 3 (weighted RMSE, 0.43). Bootstrapping/Bayes factor approach and the postchallenge independent validation confirmed the reproducibility and the estimation concordance indices between final evaluation and postchallenge independent validation data set were 0.71 for subchallenge 1, 0.78 for subchallenge 2, and 0.82 for subchallenge 3.
The RA2-DREAM Challenge resulted in the development of algorithms that provide feasible, quick, and accurate methods to quantify joint damage in RA. Ultimately, these methods could help research studies on RA joint damage and may be integrated into electronic health records to help clinicians serve patients better by providing timely, reliable, and quantitative information for making treatment decisions to prevent further damage.</abstract><cop>United States</cop><pub>American Medical Association</pub><pmid>36036935</pmid><doi>10.1001/jamanetworkopen.2022.27423</doi><oa>free_for_read</oa></addata></record> |