Biomedical challenges have become the de facto standard for benchmarking biomedical image analysis algorithms. While the number of challenges is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) We present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) We release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed within this work. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.
翻译:生物医学挑战已成为生物医学图像分析算法基准制定工作的实际标准。虽然挑战数量在稳步增加,但为确保这些国际竞赛的高质量设计、执行和报告投入的努力却少得令人吃惊。具体地说,文献中几乎没有注意在不确定性情况下的成果分析和可视化。鉴于这些缺点,本文件的贡献有两个方面:(1) 我们提出一套方法,以全面分析和直观地分析单一任务和多任务挑战的结果,并将其应用于若干模拟和实际生活中的挑战,以表明其具体的强项和弱点;(2) 作为这项工作的一部分,我们公布了开放源框架的挑战,以便能够迅速和广泛采用本文件中提议的方法。我们的方法提供了一种直观的方法,使人们能够对算法的相对和绝对性表现获得重要的洞察力,而通常应用的直观化技术无法揭示这一点。在这项工作中进行的实验证明了这一点。我们的框架因此可以成为分析和直观地分析生物医学图像分析领域及范围以外的挑战结果的重要工具。