Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation.
Translational psychiatry · 2026
Abstract originale
BACKGROUND: Brain-wide association effects are commonly overestimated in psychiatric neuroimaging research, leading to low replicability and generalizability of findings, and ultimately to poor translation to clinical utility. Cross-validation is currently not standardly used in univariate neuroimaging analyses but could pose a potential counteracting measure for this problem. METHODS: We systematically evaluated the benefit of a cross-validation procedure to expose inflated effect size estimates in mass-univariate voxel-based analysis, using body mass index (BMI) as an example predictor for gray matter. A total of n = 3401 MRI scans from a psychiatric cohort were used to investigate the impact of applying cross-validation on effect size estimates across a range of sample sizes and across different localizations with a range of underlying pooled effect sizes. RESULTS: BMI was negatively associated with gray matter density with a maximum underlying effect size of R²p = 0.036 (in the full sample of n = 3401). However, without cross-validation, effects were substantially overestimated in small samples, with effect sizes ranging up to R²p = 0.535. Even with an underlying null-effect, effect size estimates up to R²p = 0.429 were reached in samples with n = 25. Applying cross-validation revealed that these inflated effects in small samples did not generalize to novel data. A minimum sample size of n = 100 was required for largest brain-wide effects to start generalizing, while samples with n = 400 were needed for smaller effects to generalize. This 'point of initial generalizability' was not reached for underlying null effects, even with the maximum available sample of n = 3401. CONCLUSIONS: Results suggest that cross-validation is an effective measure to expose overestimated effects in small samples and thus could help to counteract low replicability and generalizability in brain-wide mass-univariate association studies. Open-source code is provided for practical implementation.
Parole chiave
Leggi il testo integrale originale
Testo estratto dalla pubblicazione: sono esclusi immagini, media, supplementi e parti soggette a diritti distinti.
Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation
Background
Brain-wide association effects are commonly overestimated in psychiatric neuroimaging research, leading to low replicability and generalizability of findings, and ultimately to poor translation to clinical utility. Cross-validation is currently not standardly used in univariate neuroimaging analyses but could pose a potential counteracting measure for this problem.
Methods
We systematically evaluated the benefit of a cross-validation procedure to expose inflated effect size estimates in mass-univariate voxel-based analysis, using body mass index (BMI) as an example predictor for gray matter. A total of n = 3401 MRI scans from a psychiatric cohort were used to investigate the impact of applying cross-validation on effect size estimates across a range of sample sizes and across different localizations with a range of underlying pooled effect sizes.
Results
BMI was negatively associated with gray matter density with a maximum underlying effect size of R²p = 0.036 (in the full sample of n = 3401). However, without cross-validation, effects were substantially overestimated in small samples, with effect sizes ranging up to R²p = 0.535. Even with an underlying null-effect, effect size estimates up to R²p = 0.429 were reached in samples with n = 25. Applying cross-validation revealed that these inflated effects in small samples did not generalize to novel data. A minimum sample size of n = 100 was required for largest brain-wide effects to start generalizing, while samples with n = 400 were needed for smaller effects to generalize. This ‘point of initial generalizability’ was not reached for underlying null effects, even with the maximum available sample of n = 3401.
Conclusions
Results suggest that cross-validation is an effective measure to expose overestimated effects in small samples and thus could help to counteract low replicability and generalizability in brain-wide mass-univariate association studies. Open-source code is provided for practical implementation.
Introduction
Neuroimaging methods such as magnetic resonance imaging (MRI) have been used for several decades now to gain insights into the neurobiological underpinnings of psychological phenotypes. A plethora of scientific publications describes imaging-derived biomarkers of mental health disorders and perpetuates the hope of utilizing such methods to aid clinical decision-making [1]. However, recently the validity of findings involving relationships between neuroimaging and psychological phenotypes have been questioned due to accumulating reports of low replicability [2–6]. While such a replication crisis is not specific to neuroimaging research (e.g., for the field of psychology see Open Science Collaboration [7]), its contributing factors may be particularly potent in this research domain due to high analytic flexibility in combination with a large number of statistical tests, which is likely to amplify publication bias [8–10]. Relatedly, replicability is undermined by small samples [11] which lead to highly unstable effect size estimates [12–15]. Of note, such estimates are not systematically biased in terms of their direction and in principle would, on average, converge to the true effect if reported without selection. However, the greater sampling variability from small samples leads to more extreme values which leads to an overestimation of absolute effect size magnitude (independent of the direction of effects [4]). Filtering results for statistical significance further aggravates this problem, as only the most extreme estimates pass the threshold, resulting in systematic overestimation of reported effects. Paradoxically, this bias becomes stronger the more conservative the significance threshold, because increasingly extreme estimates are required to survive thresholding [4, 12].
As studies particularly in neuroimaging research frequently use small samples [16, 17], this likely further contributes to low replicability. Recently, Marek et al. demonstrated that even without publication bias, associations between psychological and brain phenotypes are not replicable unless thousands of participants are included in the analysis [4]. This seminal study has led to a vibrant discussion regarding replicability, sample size and effect size in the neuroimaging domain [1, 3, 18–21].
In summary, the outlined evidence emphasizes an urgent need for solutions to increase the replicability of psychiatric neuroimaging findings. One possible solution that has been suggested repeatedly is to validate brain effects in independent data via cross-validation [19, 22, 23]. While cross-validation methods are standardly used in multivariate brain analyses to identify and counteract overfitting and assure generalizability of models [24–27], they are rarely ever used on univariate brain effects, likely also because common neuroimaging analysis software packages do not offer options to conduct cross-validation. However, specifically in the domain of neuroimaging research, cross-validation methods may be useful even for univariate models as the outlined overestimation of effects can be seen as an overfitting of models in the face of high analytic flexibility in combination with a high number of tests in this domain. Thus, we investigate the utility of cross-validation for accurate estimation of effect size in univariate voxel-based brain analysis. Body mass index (BMI) is used as an example predictor due to previous reports of good replicability of BMI with brain structure, as well as its pivotal relevance for various mental disorders [28–31]. Due to reports of relatively large effect size estimates of the association between BMI and brain structure even in large samples (maximum effect size corresponding to approximately 2.7% explained variance [30]) this predictor is suitable to investigate the impact of cross-validation for a broader range of effect size, as compared to other predictors where effect size estimates across brain modalities have been shown to be considerably smaller [4, 32]. In order to evaluate the relevance of cross-validation for generalizability of effects sizes as a function of the sample size, we conducted our analyses across different sample sizes.
In light of the current lack of brain analysis software implementations regarding this matter, we provide open-source Python code to enable other researchers to apply cross-validation for voxel-based brain analysis.
Method
Participants
A total of n = 3401 participants were included from three independent German cohorts: the Marburg-Münster Affective Disorders Cohort Study (MACS; n = 1655), the Münster Neuroimaging Cohort (MNC; n = 722) and the BiDirect cohort (n = 1024). All three cohorts include individuals with major depressive disorder (MDD) and healthy controls (HC) free from any lifetime mental disorder diagnoses. General study methods, exclusion criteria and cohort characteristics were comprehensively described elsewhere (MACS: [33, 34]; MNC: [35, 36]; BiDirect: [37]). See supplements for an additional description of data exclusion steps and sample characteristics specific to the current analysis.
Measures and procedure
BMI was calculated based on self-reported height and weight of participants. T1-weighted high-resolution anatomical brain images were acquired using 3 T MRI scanner in all three studies. For the MACS sample two different MRI scanners were used at the recruitment sites in Marburg (Tim Trio, Siemens, Erlangen, Germany; combined with a 12-channel head matrix Rx-coil) and Münster (Prisma, Siemens, Erlangen, Germany; combined with a 20-channel head matrix Rx-coil). Data from the MNC and BiDirect samples were acquired using the same Gyroscan Intera scanner (later with Achieva update; Philips Medical Systems, Best, The Netherlands). Image preprocessing was conducted using the CAT12-toolbox [38] (https://neuro-jena.github.io/cat/) using default parameters for all samples. Briefly, images were bias-corrected, tissue classified, and normalized to MNI-space using linear (12-parameter affine) and non-linear transformations, within a unified model including high-dimensional geodesic shooting normalization [39]. The modulated gray matter images were smoothed with a Gaussian kernel of 8 mm FWHM. Absolute threshold masking with a threshold value of 0.2 was used for all second level analyses as recommended for VBM analyses (https://neuro-jena.github.io/cat12-help/). Image quality was assessed by visual inspection as well as by using the check for homogeneity function implemented in the CAT12 toolbox.
Statistical analysis
We investigated the association between BMI and gray matter brain structure using an established mass-univariate voxel-based morphometry (VBM) approach. To that end, BMI was used as a predictor in a general linear model (GLM) to predict voxel-wise gray matter density. The following nuisance parameters were included in the model: age, sex, total intracranial volume (TIV) and four dummy-coded scanner variables to control for scanner hardware differences. Two-sided whole brain effects of BMI were tested and partial R2 (R2p) was used as a measure of effect size. To assess the impact of sample size and cross-validation on the estimation of voxel-based effect sizes the following steps were undertaken (for a schematic overview see Fig. 1):Classical non-cross-validated analysis was conducted in the maximum available sample without bootstrapping (n = 3401). The resulting voxel-wise explained variance of BMI was used as an estimate for the realistic underlying effect size (in the following referred to as the reference effect size). Subsequently, voxels were selected covering a range of different representative effect sizes, including the voxel with the largest brain-wide reference effect size. In addition, the voxel with the smallest brain-wide reference effect size was selected to investigate effect size estimation based on an underlying null effect. An uncorrected significance threshold of p < 0.001 and extent threshold of k > 200 was used for visualization purposes of brain-wide associations.
Based on the findings of Marek et al. [4] we subsequently investigated the influence of sample size on the non-cross-validated effect sizes. Sample size was manipulated using samples of 18 different sizes: n = 25, 35, 50, 70, 100, 150, 200, 300, 400, 600, 800, 1000, 1300, 1600, 2000, 2400, 2900, 3401. For each preselected voxel a bootstrapped resampling distribution of effect size was obtained containing k = 500 bootstrap runs per sample size. For each sample size the mean effect size was calculated across all bootstrap runs, as well as a 95% confidence interval (CI).
Finally, the impact of cross-validation on effect size estimates in the selected voxels was investigated across samples created with the resampling procedure described above. To this end, 5-fold cross-validation was applied by estimating linear effects using the GLM described above within respective train sets and then applying the resulting linear coefficients to the respective test sets. To this end the resulting beta coefficients yielded by model estimation within the train set was applied to the design matrix (individual predictor values) of the test set (i.e., data new to the trained linear model). The predictive value of the BMI predictor was evaluated by calculating the explained variance (based on residuals) with and without BMI as a predictor in the model. Then the mean R2p for BMI was calculated across the five test sets. This was used to quantify the extent to which the linear voxel-wise coefficients for BMI obtained from the train sets explain variance within unknown test data (i.e., generalization of effects to unseen data). Note that while R2p usually has a range from 0–1, it can become negative (and even exceed -1) in this case due to the application of linear coefficients to unseen data which can result in an effect size lower than prediction by the mean within the same sample (normally the baseline reference corresponding to an effect size R2p = 0). A point of initial generalizability was defined as the minimum sample size necessary to achieve positive effect sizes in the test sets (i.e., the lower bound of the 95% CI above R2p = 0). This point of initial generalizability can be interpreted as the minimum sample size needed for linear effects of a given effect size to generalize to unknown samples in a 5-fold cross-validation framework. The 5-fold cross-validation was chosen over a higher number of folds to be able to simulate very small (but commonly used) samples sizes. In order investigate the impact of this cross-validation on effect size estimation across different sample sizes we applied the same bootstrap approach described above also to cross-validation-based effect size estimates, resulting in resampling distributions of average test set effect sizes for each sample size (95% CI were based on these bootstrap runs). To allow this combination of cross-validation and bootstrapping, we first conducted the 5-fold split of the total sample and then performed the bootstrapped resampling for each sample size by drawing with replacement from each split separately – thus only allowing replacement within one split but not across splits. This is necessary to avoid data leakage between train and test sets.
Notably, the reference estimate derived from the full sample is not independent of the effect size estimates obtained in the resampling analyses. Within the present framework it serves as a proxy for the underlying population effect, while the resampled subsets represent draws from this population to examine how effect size estimates behave across sample sizes. All analyses were conducted in Python (version 3.9.12) using the nilearn package (version 0.9.1) for voxel-based GLMs and k-Fold from the sklearn package (version 1.1.1) for cross-validation. The complete analysis code is provided online (https://osf.io/cy7fp/?view_only=a10fd0ee7b914f50820b5265f65f0cdb).
Results
Associations between BMI and gray matter – identification of reference effect sizes in n = 3401
The association between BMI and gray matter density in a non-cross-validated standard analysis is shown in Fig. 2a and supplementary Figure S1. Liberal thresholding at uncorrected p < 0.001, k > 200 resulted in widespread clusters across the brain. Effect size in significant voxels ranged up to R2p = 0.036 with predominantly negative associations between BMI and gray matter (Figure S1). Peak voxel coordinates of significant clusters covering a wide range of effect sizes, as well as the voxel with the brain-wide smallest effect size were selected for further analysis (see Fig. 2a). The selected voxels from largest to smallest effects were located in 1) the cerebellum (R2p = 0.036, p < 0.0001, x-24, y-72, z-60), 2) the posterior medial orbitofrontal cortex (mOFC; R2p = 0.024, p < 0.0001, x-4, y25, z-30), 3) the thalamus (R2p = 0.016, p < 0.0001, x6, y-12, z10), 4) the anterior mOFC (R2p = 0.005, p < 0.0001, x4, y69, z-6) and 5) the calcarine (R2p < 0.0001, p = 0.999, x-3, y-80, z12).
The impact of sample size on non-cross-validated effect size estimation
Effect size was systematically overestimated in small samples in the non-cross-validated analysis. Averaged across all bootstrapped samples with size n = 25, effect size was inflated approximately 0.06 R2p units for all five voxels, resulting in a 2.5-fold inflation for the voxel with the largest reference effect size (R2p = 0.091 instead of R2p = 0.036) and 11.9-fold inflation for the voxel with a reference effect size of R2p = 0.005. Maximum effect size estimates went as high as R2p = 0.734 in samples of n = 25. Even the voxel with the brain-wide lowest reference effect size (null-effect) reached a maximum effect size of up to R2p = 0.429 (average R2p = 0.056) in these samples with n = 25. Average estimates of effect size (as well as maximum estimates) decreased rapidly in a nonlinear fashion with increasing sample size.
Inspection of partial correlation coefficients (thus retaining the direction of effects) revealed substantial sign (type S error) and magnitude (type M error) instability of effects across bootstrapped samples, particularly in small sample sizes. For the voxel with the largest brain-wide reference effect size (negative association of r = -0.19 in full sample), associations ranging from a negative correlation r = -0.73 to a positive correlation r = 0.60 were found in samples with n = 25 (for presentation of directional effects see supplementary Figure S2).
Detailed summary statistics for effect size estimates across sample sizes and voxels are presented in supplementary Tables S2–6. While the average effect size estimate across bootstrapped samples may be informative, it should be noted that the distribution of estimates was highly skewed with some extreme outliers. This distribution across single runs from the resampling procedure is further visualized in supplementary Figure S3.
The impact of cross-validation on effect size estimation
On average test set effect size estimates were descriptively lower compared to non-cross-validated effect sizes across all sample sizes and all voxels (this was true for 93.83% of bootstrapped samples across all sample sizes). This disparity was particularly strong in small samples where non-cross-validated effect sizes were largest while test set effect sizes were mostly negative (indicating no generalization of linear effects to unknown samples).
A point of initial generalizability was reached after n = 100 for the largest reference effect size, while larger samples were needed to reach this point for voxels with smaller effect size: n = 100 for R2p = 0.024, n = 150 for R2p = 0.016, and n = 400 for R2p = 0.005. For the voxel with the minimum brain-wide effect size this point of initial generalizability was never reached, as expectable for an underlying null-effect. In fact, for this voxel even the mean and upper bound of the 95% CI were always below an effect size of zero at any sample size, when applying cross-validation, indicating effective protection against false-positives (see supplementary Tables S2–6).
Estimates derived from non-cross-validated analysis and from test sets using cross-validation approached convergence in larger samples. In samples with n = 3401, non- cross-validated and test set effect sizes differed only marginally (average difference across voxels: R2p = 0.001). However, full convergence was never reached between non-cross-validated and test set effect size estimates, meaning that confidence intervals did not overlap, even at largest sample sizes. Detailed descriptive statistics for effect size estimates across sample sizes, voxels and cross-validation are given in supplementary Tables S2–6.
Discussion
Using BMI as an example predictor variable, we investigated the utility of cross-validation for the accurate estimation of effect sizes of univariate brain-wide associations. Replicating previous findings [4], we find that effect size estimates are highly overestimated and unreliable in small samples if no cross-validation is applied. Furthermore, we demonstrate that cross-validation can be used to reveal poor generalizability to new data of these overestimated effects in small samples (indicated by negative effect size estimates). In larger samples, cross-validation-derived test set effect sizes start becoming positive, suggesting a point of initial generalizability that can be interpreted as the minimum sample size necessary to obtain model estimates that have any generalizability value in independent data. Importantly, sample sizes of several hundreds of participants may be sufficient to accurately estimate brain-wide associations, given sufficiently large underlying effect sizes. The cross-validation approach could facilitate a differentiation between artificially inflated large effects and robust ‘true’ large effects. The standardly implementation of cross-validation to assess the generalizability of brain-wide effects should be considered in addition to conventional reporting of significance and traditional effect size estimates. The utilization of cross-validation to facilitate generalizability of neuroimaging findings has been previously extensively investigated in the domain of multivariate machine learning analysis [40–42]. Furthermore, the application of cross-validation has been repeatedly demanded also in the context of univariate MRI analysis for example in the context of counteracting double-dipping and circular inference [22, 43, 44] but also more generally to assess the out-of-sample predictive utility of given effects and to counteract non-replicability [19, 23]. However, to the best of our knowledge, we are the first to systematically investigate the utility of cross-validation for effect size estimation in brain-wide univariate analysis and to provide a concrete tangible framework for others to apply to their analysis.
The presented overestimation of effect sizes in small samples falls in line with previous findings in the literature [12–15]. Marek et al. [4] demonstrated that even the largest brain-wide associations of compound cognitive and psychopathological variables with brain structure and function corresponded to less than 0.5% explained variance, which was inflated to up to approximately 40% explained variance in small samples. The authors conclude that thousands of individuals are needed for robust and replicable estimates in this research domain. While these findings are striking and important, the authors’ conclusion has been questioned by various scholars. The main critique is that true effect sizes may be considerably higher under ideal conditions leading to smaller sample sizes necessary for accurate estimation of effects [45]. In a study of over 1800 adults, Winter et al. [32] investigated univariate brain-wide, cross-modal brain differences between HC and lifetime MDD individuals and conclude that largest effect sizes go up to 1.7% explained variance. However, they further report that comparisons of HC with chronic or acute MDD individuals yield higher effects sizes of up to 2.7% explained variance (in n > 900). Further, studies suggest that effect sizes are larger in other diagnostic groups such as psychotic disorders [46]. Similarly, Repple et al. [27] present transdiagnostic structural connectome alterations, with largest effect sizes found in patients with schizophrenia, possibly rendering smaller samples sufficient to detect effects.
Our findings expand the results by Marek et al. [4] to a voxel-based analysis framework and to a broader range of effect sizes. Importantly, we provide evidence that for larger effect sizes – which can occur under specific conditions as outlined above – hundreds of participants could suffice to accurately estimate linear brain-wide effects. When smaller effect sizes are assumed, our findings are highly comparable to the findings by Marek et al. as we similarly find that an underlying reference effect size (based on the pooled sample) of approximately ~0.5% explained variance requires very large samples to obtain robust estimates of effects, although cross-validation may enable the robust detection of accurate estimates already in somewhat smaller samples (n > 400). The ongoing debate surrounding effect size and sample size in the neuroimaging domain stresses the importance of consistent reporting of effect sizes in publications, as well as interpreting effect sizes in the context of sample size.
While the above considerations could guide sample size planning of neuroimaging studies, a problem arises when the ‘true’ effect size is unknown (and effect size inflations in small samples make it particularly difficult to estimate this solely from existing literature). How can researchers know if an obtained effect size is accurate or inflated? Using a cross-validation framework, we demonstrate that the application of identified linear effects to unseen data can be utilized to identify inflated, non-generalizable effect sizes. Notably, this procedure can be applied in smaller studies to verify if a putative large effect size is robust and may warrant the use of a smaller sample. Importantly, our findings are in line with low test set performance merely reflecting an underpowered sample and not necessarily meaning that an effect does not exist [47]. In other words, even substantial effects of 3.6% explained variance can barely be differentiated from null effects in small samples based on cross-validation-derived effect sizes alone. However, the relative congruence between non-cross-validated and cross-validation-derived effect sizes can provide a strong argument for robust and replicable linear associations with realistic effect size estimates. Thus, calculating non-cross-validated and cross-validation-derived effect sizes combined with a thorough inspection of the similarity between the resulting estimates could be informative. Notably, while cross-validation decreases the probability of effect size inflation, it does not eliminate it. Even unrealistically high effect size estimates can in rare cases generalize to unseen test set data, particularly in small samples, as shown by some extreme outliers in our results.
While it is difficult to deduct a clear recommendation for sample sizes based on our cross-validation analyses, we propose that a point of initial generalizability can be defined as the sample size at which linear effects begin to explain variance in unseen data (in our framework defined as explanatory value > 0 in 97.5% of bootstrapped samples). Importantly, analogous to statistical significance, this criterion does not imply practical or clinical utility. In other words, it is arguably up for debate and strongly dependent on the research question whether a model should explain 1, 5 or 10% of variance in new data to be regarded as useful. However, we argue that a point of initial generalizability can be an informative starting point to address the important question of generalizability. This aspect could expand the traditional power analysis framework by the question of generalizability in addition to significance. In other words, while traditional power analysis answers the question of how large a sample needs to be for a given effect to become significant, our results open up the possibility for defining sample sizes necessary for linear effects to generalize to unseen data. Of note, similar concepts have been previously described, such as the point of stability (sample size where correlation estimates stabilize [15]). While conceptually related, the distinction is that the point of initial generalizability explicitly expresses the out-of-sample predictive utility.
Empirically, test set performances never reached the pooled-sample underlying effect size. This reflects a well-known pessimistic bias of k-fold cross-validation, because models are trained on only a subset of the data and therefore tend to show slightly lower out-of-sample performance than models fitted to the full dataset [48]. However, in large samples effect sizes became highly stable and barely differed whether cross-validation was applied or not. This finding implicates that if several thousands of individuals are available (e.g., due to consortia data), within sample k-fold cross-validation may barely alter the result, regardless of the underlying reference effect size of the pooled sample.
The current study entails several important limitations. Firstly, the concept of generalizability as used in the presented framework refers to predictive performance in unseen individuals within the same dataset rather than to generalizability across independently collected datasets. Thus, the point of initial generalizability should be interpreted as reflecting internal out-of-sample prediction within the studied population rather than external replication across different samples. Secondly, it is unclear whether our findings can be generalized to other cross-validation methods. While 5-fold cross-validation was chosen for the current analysis so that small samples of n = 25 could be included, other splits (e.g., 10-fold) could in principle also be suitable to be applied in voxel-based univariate brain analysis. While a systematic comparison of different cross-validation methods for the application to univariate voxel-based analysis is beyond the scope of this work, this may be a fruitful task for future studies. Further, it is open for discussion to what extent our findings are generalizable to other MRI modalities, such as functional MRI and parcellation-based brain analysis, as well as to predictors of other domains. We believe that generalizability to other voxel-based imaging modalities and other predictors is given, as the general pattern of results should not be specific to VBM effects of BMI. Regarding other predictors, there is no reason that they should show patterns qualitatively differing from BMI. Based on previous investigations and our results, the true underlying effect size is the main factor driving effect size overestimation and thus also the potential benefit of cross-validation. Regarding other imaging modalities, parcellation-based brain analysis approaches could also profit from cross-validation. However, cross-validation could possibly be particularly beneficial in settings that are at higher risk for overfitting statistical models. Such risk is likely to be higher in more complex models, and in settings with higher analytical flexibility and a higher number of statistical tests (the latter being higher in voxel-based as compared to parcellation-based analysis). Interestingly, cross-validation does not seem to ultimately protect from an inflation of effect size in all settings. It has been shown that in complex multivariate modelling of voxel-based associations with psychopathology, even cross-validation-based performance measures are inflated in underpowered studies [49]. In the same vein, we demonstrated that even cross-validation-derived effect sizes can be inflated in small samples although the probability of an overestimation of effects is lower as compared to non-cross-validated estimates (as outlined above).
In summary the utilization of cross-validation can contribute two major benefits: 1) identify if a study is underpowered and corresponding effect size estimates are likely to be highly inflated and 2) corroborate the accuracy of an effect size estimate of a sufficiently powered study (also for underlying null-effects). Thus, we propose that cross-validation procedures should be applied to foster replicability of neuroimaging research and facilitate accurate estimation of effect sizes. Further, we provide concrete steps for an application of cross-validation to mass-univariate voxel-based analysis, as well as corresponding open-source Python code.
Supplementary information
References
1.Nour MM Liu Y Dolan RJ Functional neuroimaging in psychiatry and the case for failing better Neuron 2022 110 2524 44 10.1016/j.neuron.2022.07.005 35981525 Nour MM, Liu Y, Dolan RJ. Functional neuroimaging in psychiatry and the case for failing better. Neuron. 2022;110:2524–44. 10.1016/j.neuron.2022.07.005 35981525
2.Boekel W Wagenmakers EJ Belay L Verhagen J Brown S Forstmann BU A purely confirmatory replication study of structural brain-behavior correlations Cortex 2015 66 115 33 10.1016/j.cortex.2014.11.019 25684445 Boekel W, Wagenmakers EJ, Belay L, Verhagen J, Brown S, Forstmann BU, et al. A purely confirmatory replication study of structural brain-behavior correlations. Cortex. 2015;66:115–33. 10.1016/j.cortex.2014.11.019 25684445
3.Genon S Eickhoff SB Kharabian S Linking interindividual variability in brain structure to behaviour Nat Rev Neurosci 2022 23 307 18 10.1038/s41583-022-00584-7 35365814 Genon S, Eickhoff SB, Kharabian S. Linking interindividual variability in brain structure to behaviour. Nat Rev Neurosci. 2022;23:307–18. 10.1038/s41583-022-00584-7 35365814
4.Marek S Tervo-Clemmens B Calabro FJ Montez DF Kay BP Hatoum AS Reproducible brain-wide association studies require thousands of individuals Nature 2022 603 654 60 10.1038/s41586-022-04492-9 35296861 Marek S, Tervo-Clemmens B, Calabro FJ, Montez DF, Kay BP, Hatoum AS, et al. Reproducible brain-wide association studies require thousands of individuals. Nature. 2022;603:654–60. 10.1038/s41586-022-04492-9 PMC8991999 35296861
5.Goltermann J Winter NR Meinert S Grotegerd D Kraus A Flinken K Gray matter correlates of childhood maltreatment lack replicability in a multi-cohort brain-wide association study Nat Commun 2025 16 8290 10.1038/s41467-025-62374-w 40957871 Goltermann J, Winter NR, Meinert S, Grotegerd D, Kraus A, Flinken K, et al. Gray matter correlates of childhood maltreatment lack replicability in a multi-cohort brain-wide association study. Nat Commun. 2025;16:8290 10.1038/s41467-025-62374-w PMC12441139 40957871
6.Dannlowski U Winter NR Meinert S Grotegerd D Kraus A Flinkenflügel K The neuroanatomy of depression: weak but replicable effects in 4021 individuals from three clinical cohorts Neuropsychopharmacology 2026 10.1038/s41386-026-02491-2 42432162 Dannlowski U, Winter NR, Meinert S, Grotegerd D, Kraus A, Flinkenflügel K, et al. The neuroanatomy of depression: weak but replicable effects in 4021 individuals from three clinical cohorts. Neuropsychopharmacology. 2026. 10.1038/s41386-026-02491-2 PMC13598069 42432162
7.Open Science Collaboration Estimating the reproducibility of psychological science Science 2015 349 aac4716 10.1126/science.aac4716 26315443 Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349:aac4716 10.1126/science.aac4716 26315443
8.Ioannidis JPA Why most published research findings are false PLoS Med 2005 2 e124 10.1371/journal.pmed.0020124 16060722 Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2:e124 10.1371/journal.pmed.0020124 PMC1182327 16060722
9.Botvinik-Nezer R Holzmeister F Camerer CF Dreber A Huber J Johannesson M Variability in the analysis of a single neuroimaging dataset by many teams Nature 2020 582 84 88 10.1038/s41586-020-2314-9 32483374 Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature. 2020;582:84–88. 10.1038/s41586-020-2314-9 PMC7771346 32483374
10.Jennings RG Van Horn JD Publication bias in neuroimaging research: implications for meta-analyses Neuroinformatics 2012 10 67 80 10.1007/s12021-011-9125-y 21643733 Jennings RG, Van Horn JD. Publication bias in neuroimaging research: implications for meta-analyses. Neuroinformatics. 2012;10:67–80. 10.1007/s12021-011-9125-y PMC4368431 21643733
11.Turner BO Paul EJ Miller MB Barbey AK Small sample sizes reduce the replicability of task-based fMRI studies Commun Biol 2018 1 62 10.1038/s42003-018-0073-z 30271944 Turner BO, Paul EJ, Miller MB, Barbey AK. Small sample sizes reduce the replicability of task-based fMRI studies. Commun Biol. 2018;1:62 10.1038/s42003-018-0073-z PMC6123695 30271944
12.Button KS Ioannidis JPA Mokrysz C Nosek BA Flint J Robinson ESJ Power failure: why small sample size undermines the reliability of neuroscience Nat Rev Neurosci 2013 14 365 76 10.1038/nrn3475 23571845 Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14:365–76. 10.1038/nrn3475 23571845
13.Lane DM Dunlap WP Estimating effect size: bias resulting from the significance criterion in editorial decisions Br J Math Stat Psychol 1978 31 107 12 10.1111/j.2044-8317.1978.tb00578.x Lane DM, Dunlap WP. Estimating effect size: bias resulting from the significance criterion in editorial decisions. Br J Math Stat Psychol. 1978;31:107–12. 10.1111/j.2044-8317.1978.tb00578.x
14.Maxwell SE Kelley K Rausch JR Sample size planning for statistical power and accuracy in parameter estimation Annu Rev Psychol 2008 59 537 63 10.1146/annurev.psych.59.103006.093735 17937603 Maxwell SE, Kelley K, Rausch JR. Sample size planning for statistical power and accuracy in parameter estimation. Annu Rev Psychol. 2008;59:537–63. 10.1146/annurev.psych.59.103006.093735 17937603
15.Schönbrodt FD Perugini M At what sample size do correlations stabilize? J Res Pers 2013 47 609 12 10.1016/j.jrp.2013.05.009 Schönbrodt FD, Perugini M. At what sample size do correlations stabilize? J Res Pers. 2013;47:609–12. 10.1016/j.jrp.2013.05.009
16.Elliott ML Knodt AR Ireland D Morris ML Poulton R Ramrakha S What is the test-retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis Psychol Sci 2020 31 792 806 10.1177/0956797620916786 32489141 Elliott ML, Knodt AR, Ireland D, Morris ML, Poulton R, Ramrakha S, et al. What is the test-retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis. Psychol Sci. 2020;31:792–806. 10.1177/0956797620916786 PMC7370246 32489141
17.Szucs D Ioannidis JP Sample size evolution in neuroimaging research: an evaluation of highly-cited studies (1990–2012) and of latest practices (2017-2018) in high-impact journals Neuroimage 2020 221 117164 10.1016/j.neuroimage.2020.117164 32679253 Szucs D, Ioannidis JP. Sample size evolution in neuroimaging research: an evaluation of highly-cited studies (1990–2012) and of latest practices (2017-2018) in high-impact journals. Neuroimage. 2020;221:117164 10.1016/j.neuroimage.2020.117164 32679253
18.Bandettini PA Gonzalez-Castillo J Handwerker D Taylor P Chen G Thomas A The challenge of BWAS: unknown unknowns in feature space and variance Med 2022 3 526 31 10.1016/j.medj.2022.07.002 35963233 Bandettini PA, Gonzalez-Castillo J, Handwerker D, Taylor P, Chen G, Thomas A, et al. The challenge of BWAS: unknown unknowns in feature space and variance. Med. 2022;3:526–31.10.1016/j.medj.2022.07.002 35963233
19.Rosenberg MD Finn ES How to establish robust brain–behavior relationships without thousands of individuals Nat Neurosci 2022 25 835 7 10.1038/s41593-022-01110-9 35710985 Rosenberg MD, Finn ES. How to establish robust brain–behavior relationships without thousands of individuals. Nat Neurosci. 2022;25:835–7. 10.1038/s41593-022-01110-9 35710985
20.Spisak T Bingel U Wager T Multivariate BWAS can be replicable with moderate sample sizes Nature 2023 615 E4 E7 10.1101/2022.06.22.497072 36890392 Spisak T, Bingel U, Wager T. Multivariate BWAS can be replicable with moderate sample sizes. Nature. 2023;615:E4–E7. 10.1101/2022.06.22.497072 PMC9995263 36890392
21.Tervo-Clemmens B Marek S Chauvin RJ Van AN Kay BP Laumann TO Reply to: multivariate BWAS can be replicable with moderate sample sizes Nature 2023 615 E8 E12 10.1101/2022.06.22.497072 36890374 Tervo-Clemmens B, Marek S, Chauvin RJ, Van AN, Kay BP, Laumann TO, et al. Reply to: multivariate BWAS can be replicable with moderate sample sizes. Nature. 2023;615:E8–E12. 10.1101/2022.06.22.497072 PMC9995264 36890374
22.Kriegeskorte N Lindquist MA Nichols TE Poldrack RA Vul E Everything you never wanted to know about circular analysis, but were afraid to ask J Cereb Blood Flow Metab 2010 30 1551 7 10.1038/jcbfm.2010.86 20571517 Kriegeskorte N, Lindquist MA, Nichols TE, Poldrack RA, Vul E. Everything you never wanted to know about circular analysis, but were afraid to ask. J Cereb Blood Flow Metab. 2010;30:1551–7. 10.1038/jcbfm.2010.86 PMC2949251 20571517
23.Klapwijk ET van den Bos W Tamnes CK Raschle NM Mills KL Opportunities for increased reproducibility and replicability of developmental neuroimaging Dev Cogn Neurosci 2021 47 100902 10.1016/j.dcn.2020.100902 33383554 PMC7779745 Klapwijk ET, van den Bos W, Tamnes CK, Raschle NM, Mills KL, et al. Opportunities for increased reproducibility and replicability of developmental neuroimaging. Dev Cogn Neurosci. 2021;47:100902 10.1016/j.dcn.2020.100902 PMC7779745 33383554
24.Redlich R Almeida JR Grotegerd D Opel N Kugel H Heindel W Brain morphometric biomarkers distinguishing unipolar and bipolar depression: a voxel-based morphometry-pattern classification approach JAMA Psychiatry 2014 71 1222 30 10.1001/jamapsychiatry.2014.1100 25188810 Redlich R, Almeida JR, Grotegerd D, Opel N, Kugel H, Heindel W, et al. Brain morphometric biomarkers distinguishing unipolar and bipolar depression: a voxel-based morphometry-pattern classification approach. JAMA Psychiatry. 2014;71:1222–30. 10.1001/jamapsychiatry.2014.1100 PMC5538312 25188810
25.Redlich R Opel N Grotegerd D Dohm K Zaremba D Burger C Prediction of individual response to electroconvulsive therapy via machine learning on structural magnetic resonance imaging data JAMA Psychiatry 2016 73 557 64 10.1001/jamapsychiatry.2016.0316 27145449 Redlich R, Opel N, Grotegerd D, Dohm K, Zaremba D, Burger C, et al. Prediction of individual response to electroconvulsive therapy via machine learning on structural magnetic resonance imaging data. JAMA Psychiatry. 2016;73:557–64. 10.1001/jamapsychiatry.2016.0316 27145449
26.Schaffer C Selecting a classification method by cross-validation Mach Learn 1993 13 135 43 10.1023/A:1022639714137 Schaffer C. Selecting a classification method by cross-validation. Mach Learn. 1993;13:135–43. 10.1023/A:1022639714137
27.Repple J Gruber M Mauritz M de Lange SC Winter NR Opel N Shared and specific patterns of structural brain connectivity across affective and psychotic disorders Biol Psychiatry 2022 93 178 86 10.1016/J.BIOPSYCH.2022.05.031 36114041 Repple J, Gruber M, Mauritz M, de Lange SC, Winter NR, Opel N, et al. Shared and specific patterns of structural brain connectivity across affective and psychotic disorders. Biol Psychiatry. 2022;93:178–86. 10.1016/J.BIOPSYCH.2022.05.031 36114041
28.McWhinney SR Brosch K Calhoun VD Crespo-Facorro B Crossley NA Dannlowski U Obesity and brain structure in schizophrenia – ENIGMA study in 3021 individuals Mol Psychiatry 2022 27 3731 7 10.1038/s41380-022-01616-5 35739320 McWhinney SR, Brosch K, Calhoun VD, Crespo-Facorro B, Crossley NA, Dannlowski U, et al. Obesity and brain structure in schizophrenia – ENIGMA study in 3021 individuals. Mol Psychiatry. 2022;27:3731–7. 10.1038/s41380-022-01616-5 PMC9902274 35739320
29.Opel N Redlich R Grotegerd D Dohm K Heindel W Kugel H Obesity and major depression: Body-mass index (BMI) is associated with a severe course of disease and specific neurostructural alterations Psychoneuroendocrinology 2015 51 219 26 10.1016/j.psyneuen.2014.10.001 25462895 Opel N, Redlich R, Grotegerd D, Dohm K, Heindel W, Kugel H, et al. Obesity and major depression: Body-mass index (BMI) is associated with a severe course of disease and specific neurostructural alterations. Psychoneuroendocrinology. 2015;51:219–26. 10.1016/j.psyneuen.2014.10.001 25462895
30.Opel N Thalamuthu A Milaneschi Y Grotegerd D Flint C Leenings R Brain structural abnormalities in obesity: relation to age, genetic risk, and common psychiatric disorders Mol Psychiatry 2020 26 4839 52 10.1038/s41380-020-0774-9 32467648 Opel N, Thalamuthu A, Milaneschi Y, Grotegerd D, Flint C, Leenings R, et al. Brain structural abnormalities in obesity: relation to age, genetic risk, and common psychiatric disorders. Mol Psychiatry. 2020;26:4839–52. 10.1038/s41380-020-0774-9 PMC8589644 32467648
31.Bond DJ Ha TH Lang DJ Su W Torres IJ Honer WG Body mass index-related regional gray and white matter volume reductions in first-episode mania patients Biol Psychiatry 2014 76 138 45 10.1016/j.biopsych.2013.08.030 24120119 Bond DJ, Ha TH, Lang DJ, Su W, Torres IJ, Honer WG, et al. Body mass index-related regional gray and white matter volume reductions in first-episode mania patients. Biol Psychiatry. 2014;76:138–45. 10.1016/j.biopsych.2013.08.030 24120119
32.Winter NR Leenings R Ernsting J Sarink K Fisch L Emden D Quantifying deviations of brain structure and function in major depressive disorder across neuroimaging modalities JAMA Psychiatry 2022 79 879 88 10.1001/jamapsychiatry.2022.1780 35895072 Winter NR, Leenings R, Ernsting J, Sarink K, Fisch L, Emden D, et al. Quantifying deviations of brain structure and function in major depressive disorder across neuroimaging modalities. JAMA Psychiatry. 2022;79:879–88. 10.1001/jamapsychiatry.2022.1780 PMC9330277 35895072
33.Kircher T Wöhr M Nenadic I Schwarting R Schratt G Alferink J Neurobiology of the major psychoses. a translational perspective on brain structure and function: the FOR2107 consortium Eur Arch Psychiatry Clin Neurosci 2019 269 949 62 10.1007/s00406-018-0943-x 30267149 Kircher T, Wöhr M, Nenadic I, Schwarting R, Schratt G, Alferink J, et al. Neurobiology of the major psychoses. a translational perspective on brain structure and function: the FOR2107 consortium. Eur Arch Psychiatry Clin Neurosci. 2019;269:949–62. 10.1007/s00406-018-0943-x 30267149
34.Vogelbacher C Möbius TWD Sommer J Schuster V Dannlowski U Kircher T The marburg-münster affective disorders cohort study (MACS): a quality assurance protocol for MR neuroimaging data Neuroimage 2018 172 450 60 10.1016/j.neuroimage.2018.01.079 29410079 Vogelbacher C, Möbius TWD, Sommer J, Schuster V, Dannlowski U, Kircher T, et al. The marburg-münster affective disorders cohort study (MACS): a quality assurance protocol for MR neuroimaging data. Neuroimage. 2018;172:450–60. 10.1016/j.neuroimage.2018.01.079 29410079
35.Opel N Redlich R Dohm K Zaremba D Goltermann J Repple J Mediation of the influence of childhood maltreatment on depression relapse by cortical structure: a 2-year longitudinal observational study Lancet Psychiatry 2019 6 318 26 10.1016/S2215-0366(19)30044-6 30904126 Opel N, Redlich R, Dohm K, Zaremba D, Goltermann J, Repple J, et al. Mediation of the influence of childhood maltreatment on depression relapse by cortical structure: a 2-year longitudinal observational study. Lancet Psychiatry. 2019;6:318–26. 10.1016/S2215-0366(19)30044-6 30904126
36.Dannlowski U Kugel H Grotegerd D Redlich R Opel N Disadvantage of social sensitivity: interaction of oxytocin receptor genotype and child maltreatment on brain structure Biol Psychiatry 2016 80 398 405 10.1016/j.biopsych.2015.12.010 26858213 Dannlowski U, Kugel H, Grotegerd D, Redlich R, Opel N, et al. Disadvantage of social sensitivity: interaction of oxytocin receptor genotype and child maltreatment on brain structure. Biol Psychiatry. 2016;80:398–405. 10.1016/j.biopsych.2015.12.010 26858213
37.Teismann H Wersching H Nagel M Arolt V Heindel W Baune BT Establishing the bidirectional relationship between depression and subclinical arteriosclerosis - rationale, design, and characteristics of the BiDirect Study BMC Psychiatry 2014 14 174 10.1186/1471-244X-14-174 24924233 Teismann H, Wersching H, Nagel M, Arolt V, Heindel W, Baune BT, et al. Establishing the bidirectional relationship between depression and subclinical arteriosclerosis - rationale, design, and characteristics of the BiDirect Study. BMC Psychiatry. 2014;14:174 10.1186/1471-244X-14-174 PMC4065391 24924233
38.Gaser C, Dahnke R, Thompson PM, Kurth F, Luders E CAT – A computational anatomy toolbox for the analysis of structural MRI Data. GigaScience. 2024;13:giae049. 10.1093/gigascience/giae049 PMC11299546 39102518
39.Ashburner J Friston KJ Diffeomorphic registration using geodesic shooting and Gauss-Newton optimisation Neuroimage 2011 55 954 67 10.1016/j.neuroimage.2010.12.049 21216294 Ashburner J, Friston KJ. Diffeomorphic registration using geodesic shooting and Gauss-Newton optimisation. Neuroimage. 2011;55:954–67. 10.1016/j.neuroimage.2010.12.049 PMC3221052 21216294
40.Pereira F Mitchell T Botvinick M Machine learning classifiers and fMRI: a tutorial overview Neuroimage 2009 45 199 209 10.1016/j.neuroimage.2008.11.007 PMC2892746 19070668 Pereira F, Mitchell T, Botvinick M. Machine learning classifiers and fMRI: a tutorial overview. Neuroimage. 2009;45:199–209. 10.1016/j.neuroimage.2008.11.007 PMC2892746 19070668
41.Varoquaux G Raamana PR Engemann DA Hoyos-Idrobo A Schwartz Y Thirion B Assessing and tuning brain decoders: cross-validation, caveats, and guidelines Neuroimage 2017 145 166 79 10.1016/j.neuroimage.2016.10.038 27989847 Varoquaux G, Raamana PR, Engemann DA, Hoyos-Idrobo A, Schwartz Y, Thirion B, et al. Assessing and tuning brain decoders: cross-validation, caveats, and guidelines. Neuroimage. 2017;145:166–79. 10.1016/j.neuroimage.2016.10.038 27989847
42.Varoquaux G Cross-validation failure: small sample sizes lead to large error bars Neuroimage 2018 180 68 77 10.1016/j.neuroimage.2017.06.061 28655633 Varoquaux G. Cross-validation failure: small sample sizes lead to large error bars. Neuroimage. 2018;180:68–77. 10.1016/j.neuroimage.2017.06.061 28655633
43.Vul E Harris C Winkielman P Pashler H Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition Perspect Psychol Sci 2009 4 319 24 10.1111/j.1745-6924.2009.01132.x 26158964 Vul E, Harris C, Winkielman P, Pashler H. Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect Psychol Sci. 2009;4:319–24. 10.1111/j.1745-6924.2009.01132.x 26158964
44.Kriegeskorte N Simmons WK Bellgowan PSF Baker CI Circular analysis in systems neuroscience: the dangers of double dipping Nat Neurosci 2009 12 535 40 10.1038/nn.2303 19396166 Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI. Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci. 2009;12:535–40. 10.1038/nn.2303 PMC2841687 19396166
45.DeYoung CG Sassenberg TA Abend R Allen TA Beaty RE Bellgrove MA Reproducible between-person brain-behavior associations do not always require thousands of individuals PsyArXiv 2022 5 47 55 DeYoung CG, Sassenberg TA, Abend R, Allen TA, Beaty RE, Bellgrove MA, et al. Reproducible between-person brain-behavior associations do not always require thousands of individuals. PsyArXiv. 2022;5:47–55.
46.Hettwer MD, Larivière S, Park BY, van den Heuvel OA, Schmaal L, Andreassen OA, et al. Coordinated cortical thickness alterations across six neurodevelopmental and psychiatric disorders. Nature Communications. 2022;13:6851. 10.1038/s41467-022-34367-6 PMC9652311 36369423
47.Helwegen K Libedinsky I Heuvel MPV Statistical power in network neuroscience Trends Cogn Sci 2023 23 282 301 10.1016/j.tics.2022.12.011 36725422 Helwegen K, Libedinsky I, Heuvel MPV. Statistical power in network neuroscience. Trends Cogn Sci. 2023;23:282–301. 10.1016/j.tics.2022.12.011 36725422
48.Hastie T, Tibshirani R, Friedman J The elements of statistical learning. 2nd ed. Springer; 2009.
49.Flint C Cearns M Opel N Redlich R Mehler DMA Emden D Systematic misestimation of machine learning performance in neuroimaging studies of depression Neuropsychopharmacology 2021 46 1510 7 10.1038/s41386-021-01020-7 33958703 Flint C, Cearns M, Opel N, Redlich R, Mehler DMA, Emden D, et al. Systematic misestimation of machine learning performance in neuroimaging studies of depression. Neuropsychopharmacology. 2021;46:1510–7. 10.1038/s41386-021-01020-7 PMC8209109 33958703
Come citare questo studio
Goltermann J, Winter NR, Gruber M, Fisch L, Richter M, Grotegerd D, Dohm K, Meinert S, Leehr EJ, Böhnlein J, Kraus A, Thiel K, Winter A, Flinkenflügel K, Leenings R, Barkhau C, Ernsting J, Berger K, Minnerup H, Straube B, Alexander N, Jamalabadi H, Stein F, Brosch K, Wroblewski A, Thomas-Odenthal F, Usemann P, Teutenberg L, Pfarr JK, Jansen A, Nenadić I, Kircher T, Gaser C, Opel N, Hahn T, Dannlowski U. (2026). Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation.. Translational psychiatry. DOI: 10.1038/s41398-026-04529-x
Consulta la pubblicazione originale ↗