Understanding metric-related pitfalls in image analysis validation

Autor: Reinke, Annika, Tizabi, Minu D., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Kavur, A. Emre, Rädsch, Tim, Sudre, Carole H., Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Blaschko, Matthew, Buettner, Florian, Cardoso, M. Jorge, Cheplygina, Veronika, Chen, Jianxu, Christodoulou, Evangelia, Cimini, Beth A., Collins, Gary S., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Glocker, Ben, Godau, Patrick, Haase, Robert, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Isensee, Fabian, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Karthikesalingam, Alan, Kenngott, Hannes, Kleesiek, Jens, Kofler, Florian, Kooi, Thijs, Kopp-Schneider, Annette, Kozubek, Michal, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rafelski, Susanne M., Rajpoot, Nasir, Reyes, Mauricio, Riegler, Michael A., Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, van Smeden, Maarten, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Wiesenfarth, Manuel, Yaniv, Ziv R., Jäger, Paul F., Maier-Hein, Lena

Publikováno v: Nature methods, 1-13 (2024)

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in im

Externí odkaz: http://arxiv.org/abs/2302.01790

Common Limitations of Image Processing Metrics: A Picture Story

Autor: Reinke, Annika, Tizabi, Minu D., Sudre, Carole H., Eisenmann, Matthias, Rädsch, Tim, Baumgartner, Michael, Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Bankhead, Peter, Benis, Arriel, Blaschko, Matthew, Buettner, Florian, Cardoso, M. Jorge, Chen, Jianxu, Cheplygina, Veronika, Christodoulou, Evangelia, Cimini, Beth, Collins, Gary S., Engelhardt, Sandy, Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Glocker, Ben, Godau, Patrick, Haase, Robert, Hamprecht, Fred, Hashimoto, Daniel A., Heckmann-Nötzel, Doreen, Hirsch, Peter, Hoffman, Michael M., Huisman, Merel, Isensee, Fabian, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Karthikesalingam, Alan, Kavur, A. Emre, Kenngott, Hannes, Kleesiek, Jens, Kleppe, Andreas, Kohler, Sven, Kofler, Florian, Kopp-Schneider, Annette, Kooi, Thijs, Kozubek, Michal, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moher, David, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Noyan, M. Alican, Petersen, Jens, Polat, Gorkem, Rafelski, Susanne M., Rajpoot, Nasir, Reyes, Mauricio, Rieke, Nicola, Riegler, Michael, Rivaz, Hassan, Saez-Rodriguez, Julio, Sánchez, Clara I., Schroeter, Julien, Saha, Anindo, Selver, M. Alper, Sharan, Lalith, Shetty, Shravya, van Smeden, Maarten, Stieltjes, Bram, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Wiesenfarth, Manuel, Yaniv, Ziv R., Jäger, Paul, Maier-Hein, Lena

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performan

Externí odkaz: http://arxiv.org/abs/2104.05642

The MCC-F1 curve: a performance evaluation technique for binary classification

Autor: Cao, Chang, Chicco, Davide, Hoffman, Michael M.

Many fields use the ROC curve and the PR curve as standard evaluations of binary classification methods. Analysis of ROC and PR, however, often gives misleading and inflated performance evaluations, especially with an imbalanced ground truth. Here, w

Externí odkaz: http://arxiv.org/abs/2006.11278

Exploratory analysis of genomic segmentations with Segtools

Autor: Buske Orion J, Hoffman Michael M, Ponts Nadia, Le Roch Karine G, Noble William

Publikováno v: BMC Bioinformatics, Vol 12, Iss 1, p 415 (2011)

Abstract Background As genome-wide experiments and annotations become more prevalent, researchers increasingly require tools to help interpret data at this scale. Many functional genomics experiments involve partitioning the genome into labeled segme

Externí odkaz: https://doaj.org/article/edbe4b569e4c44a4904402059df85823

Sensitive and reproducible cell-free methylome quantification with synthetic spike-in controls

Autor: Wilson, Samantha L., Shen, Shu Yi, Harmon, Lauren, Burgener, Justin M., Triche, Tim, Jr., Bratman, Scott V., De Carvalho, Daniel D., Hoffman, Michael M.

Publikováno v: In Cell Reports Methods 19 September 2022 2(9)