Justin P. Haldar

Purpose The performance of modern image reconstruction methods is commonly judged using quantitative error metrics like root mean squared‐error and the structural similarity index, which are calculated by comparing reconstructed images against fully sampled reference data. In practice, the reference data will contain noise and is not a true gold standard. In this work, we demonstrate that the “hidden noise” present in reference data can substantially confound standard approaches for ranking different image reconstruction results. Methods Using both experimental and simulated k‐space data and several different image reconstruction techniques, we examined whether there was correlation between performance metrics obtained with typical noisy reference data versus those obtained with higher‐quality reference data. Results For conventional performance metrics, the reconstructions that matched best with the higher‐quality reference data were substantially different from the reconstructions that matched best with typical noisy reference data. This leads to suboptimal reconstruction results if the performance with respect to noisy reference data is used to select which reconstruction methods/parameters to employ. These issues were reduced when employing alternative error metrics that better account for noise. Conclusion Reference data containing hidden noise can substantially mislead the ranking of image reconstruction methods when using conventional error metrics, but this issue can be mitigated with alternative error metrics.