Handling Incomplete Multimodal Data in ASD Diagnosis: A Masked Autoencoder Approach With Sex-Stratified Fusion.
Brain and behavior · 2026
Abstract originale
PURPOSE: Autism spectrum disorder (ASD) diagnosis remains challenging due to heterogeneous phenotypic presentations and the inherent complexity of neurodevelopmental patterns. While multimodal approaches combining functional magnetic resonance imaging (fMRI) and behavioral data have shown promise, existing methods suffer from three critical limitations: (1) inability to handle missing modalities in clinical settings, (2) lack of interpretability for clinical decision support, and (3) failure to account for sex-based neurobiological differences. METHOD: We propose a novel Missing-Modality-Robust Multimodal Fusion Framework (MMR-MF3) that addresses these challenges through three key innovations. First, we introduce a masked autoencoder (MAE)-based imputation module that reconstructs missing modalities during inference, enabling robust prediction even with incomplete data. Second, we design a transformer-based cross-modal attention mechanism with modality-type embeddings that captures complex interactions between brain connectivity and behavioral phenotypes. Third, we implement sex-stratified classification heads that account for neurobiological sex differences, improving diagnostic accuracy across demographic groups. FINDING: Our framework achieves state-of-the-art performance on Autism Brain Imaging Data Exchange (ABIDE) I and II datasets, with AUC of 0.89 for males and 0.86 for females. Critically, we validate our approach using leave-one-site-out (LOSO) cross-validation to ensure generalizability across the highly heterogeneous multisite ABIDE data. Extensive ablation studies demonstrate the contribution of each component, while SHapley Additive exPlanations (SHAP)-based explainability analysis reveals clinically meaningful biomarkers. CONCLUSION: The proposed framework represents a significant advancement toward clinically deployable ASD diagnostic tools.
Parole chiave
Leggi il testo integrale originale
Testo estratto dalla pubblicazione: sono esclusi immagini, media, supplementi e parti soggette a diritti distinti.
Handling Incomplete Multimodal Data in ASD Diagnosis: A Masked Autoencoder Approach With Sex‐Stratified Fusion
Abstract
Purpose
Autism spectrum disorder (ASD) diagnosis remains challenging due to heterogeneous phenotypic presentations and the inherent complexity of neurodevelopmental patterns. While multimodal approaches combining functional magnetic resonance imaging (fMRI) and behavioral data have shown promise, existing methods suffer from three critical limitations: (1) inability to handle missing modalities in clinical settings, (2) lack of interpretability for clinical decision support, and (3) failure to account for sex‐based neurobiological differences.
Method
We propose a novel Missing‐Modality‐Robust Multimodal Fusion Framework (MMR‐MF3) that addresses these challenges through three key innovations. First, we introduce a masked autoencoder (MAE)‐based imputation module that reconstructs missing modalities during inference, enabling robust prediction even with incomplete data. Second, we design a transformer‐based cross‐modal attention mechanism with modality‐type embeddings that captures complex interactions between brain connectivity and behavioral phenotypes. Third, we implement sex‐stratified classification heads that account for neurobiological sex differences, improving diagnostic accuracy across demographic groups.
Finding
Our framework achieves state‐of‐the‐art performance on Autism Brain Imaging Data Exchange (ABIDE) I and II datasets, with AUC of 0.89 for males and 0.86 for females. Critically, we validate our approach using leave‐one‐site‐out (LOSO) cross‐validation to ensure generalizability across the highly heterogeneous multisite ABIDE data. Extensive ablation studies demonstrate the contribution of each component, while SHapley Additive exPlanations (SHAP)‐based explainability analysis reveals clinically meaningful biomarkers.
Conclusion
The proposed framework represents a significant advancement toward clinically deployable ASD diagnostic tools.
Keywords: autism spectrum disorder, explainable AI, fMRI, graph neural networks, masked autoencoder, multimodal fusion, sex‐stratified classification, transformer
A masked‐autoencoder‐based framework fuses fMRI connectivity and phenotypic data via a cross‐modal transformer and sex‐stratified classification heads to diagnose ASD even with missing modalities. On ABIDE I+II, it achieves AUC 0.876 (fivefold CV) and 0.854 (LOSO across 36 sites), with MAE imputation providing a +0.029 AUC benefit and improved SHAP‐based interpretability.
1.Introduction
Autism spectrum disorder (ASD) is a complex neurodevelopmental condition characterized by challenges in social communication and restricted, repetitive patterns of behavior (Lord et al., 2018). With a global prevalence of approximately 1 in 36 children (Maenner et al., 2023), early and accurate diagnosis is critical for timely intervention. However, current diagnostic practices rely primarily on behavioral observations and clinical interviews, which are time‐consuming, subjective, and require specialized expertise that is often unavailable (Volkmar et al., 2014).
Neuroimaging, particularly functional magnetic resonance imaging (fMRI), offers objective biomarkers that could complement behavioral assessments (Uddin et al., 2013). Resting‐state fMRI (rs‐fMRI) studies have identified atypical functional connectivity patterns in individuals with ASD, particularly in the default mode network (DMN), salience network, and fronto‐parietal networks (Herbert et al., 2011). These findings suggest that neuroimaging‐based approaches could provide valuable diagnostic information.
1.1.Challenges in Multimodal ASD Diagnosis
Despite promising results, several challenges hinder the clinical translation of multimodal ASD diagnostic systems:
Challenge 1: Missing modalities. In real‐world clinical settings, it is common to encounter incomplete data. Patients may not undergo fMRI scanning due to cost, contraindications, or noncompliance, particularly in pediatric populations. Existing multimodal methods typically require all modalities to be present, making them impractical for clinical deployment (Parisot et al., 2017).
Challenge 2: Lack of interpretability. Deep learning models, while powerful, often function as “black boxes.” For clinical acceptance, diagnostic systems must provide explanations that clinicians can understand and trust (Holzinger et al., 2019). Understanding which brain regions and behavioral features contribute to predictions is essential for biological validation and treatment planning.
Challenge 3: Sex differences. ASD exhibits significant sex differences in prevalence (male‐to‐female ratio of approximately 4:1) and phenotypic presentation (Loomes et al., 2017). Females with ASD often present with different symptom profiles and may be underdiagnosed due to “camouflaging” behaviors (Hull et al., 2017). Most existing methods do not account for these differences, potentially leading to biased performance across sex groups.
1.2.Contributions
To address these challenges, we propose Missing‐Modality‐Robust Multimodal Fusion Framework (MMR‐MF3), a comprehensive framework with the following contributions:
(1)Missing modality robustness: We introduce a novel masked autoencoder (MAE)‐based imputation module that learns to reconstruct missing modalities from available ones, enabling robust predictions with incomplete data. We evaluate robustness under both random and structured missingness patterns.
(2)Cross‐modal fusion: We design a transformer‐based fusion mechanism with modality‐type embeddings that captures complex interactions between fMRI connectivity patterns and behavioral features.
(3)Sex‐stratified classification: We implement separate classification heads for males and females, accounting for neurobiological sex differences and improving diagnostic accuracy across demographic groups. We compare against alternative strategies for handling sex differences.
(4)Clinical interpretability: We integrate SHapley Additive exPlanations (SHAP)‐based explainability analysis, with clear methodology stating that attribution is computed on the latent fused representation, and validate findings against prior neuroimaging literature.
(5)Rigorous generalization validation: We conduct leave‐one‐site‐out (LOSO) cross‐validation in addition to fivefold CV to demonstrate that performance generalizes across the heterogeneous ABIDE sites and is not inflated by site‐specific artifacts.
2.Related Work
2.1.Neuroimaging‐Based ASD Diagnosis
Early neuroimaging studies for ASD diagnosis primarily relied on hand‐crafted features from structural MRI (Ecker et al., 2010) and rs‐fMRI (Anderson et al., 2011). These approaches achieved moderate accuracy but required extensive domain expertise for feature engineering.
Deep learning methods have shown superior performance by learning hierarchical representations directly from raw data. Convolutional neural networks (CNNs) have been applied to brain images (Keskin et al., 2019), while graph neural networks (GNNs) have gained popularity for modeling brain connectivity networks (Parisot et al., 2017; Ktena et al., 2017). GNNs naturally represent the brain as a graph where nodes correspond to brain regions and edges represent functional connectivity.
2.2.Multimodal Fusion for ASD
Multimodal approaches that combine neuroimaging with behavioral data have emerged as a promising direction. Heinsfeld et al. (2018) proposed a deep learning model combining fMRI and phenotypic data. Li and Fan (2018) developed a multimodal framework using both structural and functional MRI.
Recent work has explored advanced fusion strategies. X. Wang et al. (2022) proposed attention‐based fusion for ASD diagnosis using ABIDE I+II. Eslami et al. (2023) developed ASDFusion, a transformer‐based multimodal approach achieving strong performance on ABIDE. However, these methods typically assume all modalities are available and do not handle missing data scenarios.
2.3.Missing Modality Handling
Several approaches have been proposed for handling missing modalities:
Imputation‐based methods estimate missing values using statistical techniques (Van Buuren and Groothuis‐Oudshoorn, 2011) or deep learning (Yoon et al., 2018). However, these methods are typically applied as preprocessing steps and do not integrate imputation with the downstream task.
Modality‐invariant representations learn shared latent spaces across modalities (Y. Wang et al. 2020). While elegant, these approaches may lose modality‐specific information crucial for diagnosis.
Partial observation methods adapt model architecture to handle missing inputs (Ma et al. 2018). Our MAE‐based approach combines the benefits of imputation and task‐specific learning.
2.4.Explainable AI in Neuroimaging
Explainability is crucial for clinical acceptance. SHAP (Lundberg and Lee 2017) has been widely adopted for interpreting machine learning models in neuroimaging (Rosenkrantz et al. 2020). Recent work has applied SHAP to identify brain regions associated with ASD (Thomas et al. 2020).
However, most explainability studies focus on unimodal models. Multimodal explainability, particularly attribution to specific modalities, remains underexplored. Recent work by X. Zhang et al. (2022) has begun addressing multimodal explainability in neuroimaging contexts.
2.5.Sex Differences in ASD
Sex differences in ASD have received increasing attention. Neuroimaging studies have identified sex‐specific brain connectivity patterns in ASD (Floris et al. 2018). Behavioral studies have documented different phenotypic presentations between males and females (Hull et al. 2017).
Recent diagnostic approaches have begun incorporating sex‐stratified analysis. Zheng et al. (2022) demonstrated improved ASD classification by modeling sex differences explicitly. Our work extends this by integrating sex‐stratified classification within a multimodal missing‐modality‐robust framework.
3.Methodology
3.1.Problem Formulation
We formulate ASD diagnosis as a multimodal binary classification problem. Let D={(xi(f),xi(p),yi,si)}i=1N be a dataset of N subjects, where
xi(f)∈Rdf represents fMRI connectivity features,
xi(p)∈Rdp represents phenotypic/behavioral features,
yi∈{0,1} is the diagnosis label (0: TD, 1: ASD),
si∈{M,F} denotes biological sex.
During inference, some modalities may be missing. We define a mask vector mi=[mi(f),mi(p)]∈{0,1}2 indicating the availability of each modality. Our goal is to predict yi given available modalities and their masks.
3.2.Overview of MMR‐MF3
Figure 1 illustrates the overall architecture of our proposed framework. MMR‐MF3 consists of five main components:
(1)Modality‐specific encoders: GNN encoder for fMRI and MLP encoder for phenotypic data.
(2)MAE imputation module: Reconstructs missing modalities from available ones.
(3)Cross‐modal transformer: Fuses information across modalities with attention.
(4)Sex‐stratified heads: Separate classifiers for male and female subjects.
(5)Explainability module: SHAP‐based attribution analysis on latent representations.
3.3.fMRI Encoding with GNN
We represent fMRI data as a brain connectivity graph G=(V,E,A), where
V={v1,…,vn} represents n brain regions (nodes),
E represents functional connections (edges),
A∈Rn×n is the adjacency matrix from functional connectivity.
We use graph isomorphism network (GIN) (Xu et al. 2019) as our GNN encoder due to its expressive power:
(1)hv(l+1)=MLP(l)(1+ε(l))·hv(l)+∑u∈N(v)hu(l)where hv(l) is the feature of node v at layer l, N(v) denotes neighbors of v, and ε(l) is a learnable parameter.
After L GIN layers, we apply global mean pooling to obtain the graph‐level representation:
(2)h(f)=1n∑v∈Vhv(L)
3.4.Phenotypic Data Encoding
Phenotypic features include demographic information (age, IQ) and behavioral assessment scores (ADOS, SRS, etc.). All phenotypic features are z‐score normalized before encoding to ensure equal contribution to the reconstruction loss. We encode these using a multi‐layer perceptron:
(3)h(p)=MLPpheno(x(p))=σW3σW2σW1x(p)+b1+b2+b3where σ denotes the GELU activation function (Hendrycks and Gimpel 2016).
3.5.MAE‐Based Missing Modality Imputation
Our MAE module reconstructs missing modalities using available ones. The key insight is that fMRI connectivity patterns and behavioral phenotypes are correlated, allowing cross‐modal reconstruction.
3.5.1.Masking Strategy
During training, we randomly mask modalities with probability pmask:
(4)x∼(m)=[MASK]ifm(m)=0x(m)otherwisewhere m∈{f,p} denotes the modality.
3.5.2.MAE Architecture
The MAE consists of an encoder and decoder, as shown in Figure 2.
Encoder: The encoder processes available modalities through modality‐specific encoders:
(5)z(m)=Encoderm(x∼(m)),formwherem(m)=1Modality embeddings: We add learnable modality‐type embeddings to distinguish between modalities:
(6)z(m)=z(m)+e(m)where e(m)∈Rd is the embedding for modality m.
Decoder: The decoder reconstructs missing modalities:
(7)x^(m)=Decoderm(z(f),z(p)),formwherem(m)=0
3.5.3.Reconstruction Loss
The reconstruction loss is computed only for masked modalities. Since all phenotypic features are z‐score normalized, the L2 loss equally weights all features regardless of their original scale:
(8)Lrecon=∑m:m(m)=0∥x^(m)−x(m)∥22
3.6.Cross‐Modal Transformer Fusion
After imputation, we fuse information from both modalities using a transformer encoder with cross‐modal attention, as illustrated in Figure 3.
3.6.1.Input Representation
We create a sequence of modality tokens:
(9)Z=[z(f);z(p);zcls]∈R3×dwhere zcls is a learnable classification token.
3.6.2.Multi‐Head Self‐Attention
The Transformer applies multi‐head self‐attention:
(10)MHSA(Z)=Concat(head1,…,headh)WOwhere each attention head is computed as:
(11)headi=softmaxZWiQ(ZWiK)TdkZWiVThe attention mechanism naturally captures cross‐modal interactions, with attention weights indicating the importance of each modality for the prediction.
3.6.3.Feed‐Forward Network
After attention, a position‐wise feed‐forward network is applied:
(12)FFN(Z)=GELU(ZW1+b1)W2+b2
3.6.4.CLS Token Aggregation
The final classification token zcls(L) from the last transformer layer serves as the fused representation:
(13)hfused=zcls(L)
3.7.Sex‐Stratified Classification
To account for sex differences, we use separate classification heads for male and female subjects:
(14)y^=σ(Wmalehfused+bmale)ifs=Mσ(Wfemalehfused+bfemale)ifs=Fwhere σ is the sigmoid function.
This design allows the model to learn sex‐specific decision boundaries while sharing the feature extraction backbone.
3.8.Training Strategy
3.8.1.Joint Loss Function
We optimize a joint loss combining classification and reconstruction:
(15)L=Lcls+λLreconwhere Lcls is the binary cross‐entropy loss:
(16)Lcls=−ylog(y^)+(1−y)log(1−y^)and λ is a hyperparameter balancing the two objectives.
3.8.2.Random Modality Masking
During training, we randomly mask modalities to simulate missing data scenarios:
Both modalities available: probability pboth
Only fMRI: probability pfonly
Only phenotypic: probability pponly
This ensures the model learns to handle all missing‐modality combinations.
3.9.Explainability With SHAP
We use SHAP (Lundberg and Lee 2017) to provide interpretable explanations. SHAP values are computed on the latent fused representation
hfused (128‐dimensional), not on raw input features. This attribution reveals which dimensions of the learned representation contribute most to predictions.
To validate clinical interpretability, we additionally compute Kernel SHAP on the original phenotypic features and GNN node attention scores for fMRI regions, then correlate these with the latent space attributions.
3.9.1.Global Feature Importance
For global interpretability, we compute SHAP values for all latent dimensions across the test set:
(17)ϕj=1Ntest∑i=1Ntestϕj(i)where ϕj(i) is the SHAP value of latent dimension j for sample i.
3.9.2.Per‐Modality Attribution
We aggregate SHAP values by modality to understand the contribution of each data type:
(18)Φ(m)=∑j∈Fm|ϕj|where Fm is the set of latent dimensions primarily associated with modality m (determined by analyzing decoder gradients).
3.9.3.Individual Explanations
For clinical decision support, we provide individualized explanations showing which latent features contributed most to a specific prediction, as visualized in Figure 4.
4.Experimental Setup
4.1.Dataset
We evaluate our method on the Autism Brain Imaging Data Exchange (ABIDE) I and II datasets (Di Martino et al. 2014; Cameron Craddock et al. 2018). These publicly available datasets aggregate rs‐fMRI and phenotypic data from multiple international sites.
4.1.1.Data Acquisition
ABIDE I includes data from 17 international sites, while ABIDE II includes 19 sites. Both datasets provide the following:
rs‐fMRI scans (TR varies by site: 1.5–3.0s).
T1‐weighted anatomical scans.
Comprehensive phenotypic information including age, sex, IQ, and diagnostic instruments.
4.1.2.Preprocessing Pipeline
We follow a standardized preprocessing pipeline using the Configurable Pipeline for the Analysis of Connectomes (C‐PAC) (Craddock et al. 2014). The pipeline includes:
(1)Slice timing correction
(2)Motion correction (realignment)
(3)Skull stripping
(4)Co‐registration to MNI152 template
(5)Nuisance regression (motion parameters, CSF, white matter)
(6)Band‐pass filtering (0.01–0.1 Hz)
(7)Spatial smoothing (FWHM = 6mm)
4.1.3.Functional Connectivity Extraction
We extract functional connectivity using the Craddock 200 (CC200) atlas (Craddock et al. 2012), which parcellates the brain into 200 regions of interest (ROIs). For each ROI, we compute the mean time series and calculate Pearson correlation between all pairs of regions, resulting a 200×200 connectivity matrix. We use the upper triangular elements (19,900 features) as our fMRI input.
4.1.4.Harmonization
To address site effects, we apply ComBat harmonization (Johnson et al. 2007):
(19)xijharmonized=xij−α^i−Xijβ^iσ^iσ^i∗+α^i∗+Xijβ^i∗where α^i, β^i, and σ^i are estimated site‐specific parameters, and starred quantities represent pooled estimates.
4.1.5.Class Balancing
Due to class imbalance, we apply BorderlineSMOTE (Han et al. 2005) to the training fold only (after cross‐validation splitting) to avoid data leakage:
(20)xnew=xi+δ·(xnn−xi)where xnn is a randomly selected nearest neighbor from the minority class borderline instances and δ∈[0,1] is a random number.
SMOTE timing: SMOTE is applied after cross‐validation fold splitting, exclusively on the training set. The validation and test sets remain unmodified. For the high‐dimensional fMRI features, we apply SMOTE in the 64‐dimensional latent space (after GNN encoding) rather than the original 19,900‐dimensional space to avoid the curse of dimensionality.
4.2.Implementation Details
4.2.1.Model Architecture Hyperparameters
4.2.2.Training Configuration
We train our model using the AdamW optimizer (Loshchilov and Hutter 2019) with:
Learning rate: 10−4 with cosine annealing
Weight decay: 10−5
Batch size: 32
Maximum epochs: 200 with early stopping (patience = 20)
Dropout: 0.3
4.2.3.Missing Modality Simulation
During training, we simulate missing modalities with the following distribution:
Both modalities: 60%
fMRI only: 20%
Phenotypic only: 20%
4.3.Evaluation Metrics
We evaluate performance using the following:
AUC‐ROC: Primary metric, area under the receiver operating characteristic curve
Accuracy: TP+TNTP+TN+FP+
F1‐Score: 2·Precision·RecallPrecision+Recall
Sensitivity: TPTP+FN
Specificity: TNTN+FP
4.4.Cross‐Validation Strategy
We employ two complementary cross‐validation strategies to ensure rigorous evaluation:
1. Fivefold site‐stratified cross‐validation: We use fivefold stratified cross‐validation, ensuring:
Balanced class distribution in each fold.
Balanced sex distribution in each fold.
No site leakage (subjects from the same site in the same fold).
2. LOSO cross‐validation: To demonstrate generalizability across the heterogeneous ABIDE sites, we conduct LOSO cross‐validation where:
Each of the 36 ABIDE sites is held out as a test set once.
The model is trained on the remaining 35 sites.
Results are aggregated across all 36 hold‐out iterations.
LOSO is the gold standard for ABIDE evaluation as it prevents the model from learning site‐specific MRI artifacts (scanner effects, protocol differences) rather than true ASD biomarkers.
4.5.Statistical Testing
We perform McNemar's test (McNemar 1947) to compare model predictions:
(21)χ2=(b−c)2b+cwhere b and c are the counts of discordant predictions.
We also report 95% confidence intervals using bootstrapping with 1000 iterations (Tables 1 and 2).
TABLE 1Dataset description: ABIDE I and II combined.
Characteristic ASD TD Total
Overall
N 1089 1124 2213
Age (years) 14.2 ± 7.8 14.8 ± 8.1 14.5 ± 7.9
Male/female 920/169 892/232 1812/401
ABIDE I
N 539 573 1112
Sites 17 17 17
Age (years) 17.1 ± 8.3 17.3 ± 8.5 17.2 ± 8.4
ABIDE II
N 550 551 1101
Sites 19 19 19
Age (years) 11.4 ± 5.9 12.2 ± 6.2 11.8 ± 6.0
TABLE 2Model architecture hyperparameters.
Component Configuration
GNN encoder (fMRI)
Architecture GIN
Hidden dimensions [128, 128, 64]
Number of layers 3
Dropout 0.3
MLP encoder (phenotypic)
Hidden dimensions [64, 32]
Activation GELU
Dropout 0.3
MAE module
Latent dimension 64
Decoder layers 2
Masking probability 0.2
Transformer fusion
Hidden dimension 128
Number of layers 4
Number of heads 8
FFN dimension 512
Dropout 0.1
Classification heads
Hidden dimensions [64, 32]
Output dimension 1
5.Results and Discussion
5.1.Overall Performance
Table 3 presents the performance comparison between our proposed MMR‐MF3 and baseline methods using fivefold cross‐validation.
TABLE 3Performance comparison (fivefold CV).
Method AUC Acc F1 Sen Spe
fMRI (GNN) 0.812 0.754 0.741 0.728 0.779
Pheno (MLP) 0.798 0.739 0.726 0.712 0.765
Concat. 0.841 0.772 0.761 0.748 0.795
SDAE (Heinsfeld et al. 2018) 0.854 0.784 0.773 0.761 0.806
Ours 0.876 0.801 0.792 0.782 0.819
Note: CIs for our method: AUC [0.868, 0.884], Acc [0.792, 0.810], F1 [0.783, 0.801], Sen [0.772, 0.792], and Spe [0.810, 0.828].
Our method achieves the highest AUC of 0.876, significantly outperforming unimodal approaches (fMRI‐only: 0.812, phenotypic‐only: 0.798) and simple concatenation fusion (0.841). The improvement over the best baseline (SDAE: 0.854) is statistically significant (p<0.001, McNemar's test).
5.2.LOSO Validation
Table 4 presents the critical LOSO validation results.
TABLE 4Leave‐one‐site‐out (LOSO) cross‐validation results.
Method AUC Acc F1 Sen Spe
fMRI (GNN) 0.791 0.738 0.726 0.714 0.762
Concat. fusion 0.823 0.756 0.745 0.734 0.778
SDAE (Heinsfeld et al. 2018) 0.834 0.768 0.757 0.746 0.789
MMR‐MF3 (Ours) 0.854 0.782 0.772 0.761 0.802
Note: LOSO validation across 36 ABIDE sites demonstrates generalizability. Performance drop from fivefold CV (0.876–0.854) indicates model learns true ASD biomarkers rather than site artifacts.
Under LOSO cross‐validation, our method achieves AUC of 0.854, demonstrating strong generalizability across sites. The modest drop from fivefold CV (0.876–0.854) indicates that our performance is not inflated by site‐specific artifacts. This is critical validation for any ABIDE‐based study, as it shows the model learns true ASD biomarkers rather than scanner‐specific patterns.
5.3.Sex‐Stratified Performance and Baseline Comparisons
Table 5 shows performance stratified by sex, including comparisons with alternative strategies for handling sex differences.
TABLE 5Sex‐stratified performance comparison.
Method AUC Acc F1 Sen Spe
Male subjects (N= 1812)
Non‐stratified 0.871 0.798 0.789 0.778 0.817
Sex as covariate 0.868 0.796 0.787 0.776 0.815
Separate models 0.854 0.787 0.778 0.768 0.805
Sex‐stratified (ours) 0.891 0.815 0.807 0.798 0.831
Female subjects (N = 401)
Non‐stratified 0.832 0.761 0.748 0.734 0.787
Sex as covariate 0.839 0.768 0.756 0.743 0.792
Separate models 0.821 0.754 0.742 0.729 0.778
Sex‐stratified (ours) 0.858 0.789 0.778 0.768 0.809
Note: Comparison with alternative strategies for handling sex differences. Sex‐stratified heads outperform both adding sex as a covariate and training completely separate models.
Our sex‐stratified approach achieves AUC of 0.891 for males and 0.858 for females, compared to 0.871 and 0.832 for the non‐stratified version. The improvement is more pronounced for females, suggesting that accounting for sex differences is particularly beneficial for this underrepresented group.
Comparison with alternative strategies:
Sex as covariate: Adding sex as an input feature achieves AUC 0.868 (males) and 0.839 (females), underperforming our stratified approach.
Separate unimodal models: Training separate models per sex without multimodal fusion achieves AUC 0.854 (males) and 0.821 (females), demonstrating the value of both multimodal fusion and sex‐stratified heads.
5.4.Missing Modality Robustness
Table 6 evaluates robustness to missing modalities under random masking.
TABLE 6Missing modality robustness analysis (random masking).
Available modalities AUC Acc F1 Δ AUC
Both (fMRI + phenotypic) 0.876 0.801 0.792 —
fMRI only 0.812 0.754 0.741 −0.064
Phenotypic only 0.798 0.739 0.726 −0.078
With MAE imputation
fMRI only (imputed phenotypic) 0.845 0.778 0.768 −0.031
Phenotypic only (imputed fMRI) 0.834 0.769 0.758 −0.042
Note: MAE imputation enables graceful degradation. With imputation, single‐modality performance approaches the unimodal baseline (fMRI: 0.812, phenotypic: 0.798).
Our MAE‐based imputation enables graceful degradation when modalities are missing. With only fMRI, we achieve AUC of 0.812; with only phenotypic data, AUC of 0.798. This is comparable to unimodal models trained specifically for those modalities, demonstrating effective cross‐modal imputation.
5.5.Structured Missingness Evaluation
To evaluate robustness under realistic clinical missingness patterns (which are often nonrandom), we conduct structured missingness experiments:
Table 7 shows performance under three clinically relevant missingness scenarios:
(1)Age‐biased missingness: fMRI missing for subjects under 12 years (young children have higher scan failure rates due to anxiety/motion).
(2)Sex‐biased missingness: fMRI missing for all female subjects (evaluates worst‐case scenario for the underrepresented group).
(3)Site‐biased missingness: fMRI missing for five randomly selected sites (simulates resource‐limited settings).
TABLE 7Structured missingness evaluation (clinically realistic scenarios).
Missingness pattern AUC Acc F1 N
No missingness (baseline) 0.876 0.801 0.792 2213
Age‐biased
fMRI missing (age < 12) 0.838 0.769 0.758 1847
Sex‐biased
fMRI missing (all female) 0.821 0.754 0.742 1812
Site‐biased
fMRI missing (five random sites) 0.834 0.768 0.757 1904
Note: Performance under clinically realistic nonrandom missingness patterns. Even in worst‐case scenarios (all females missing fMRI), MAE imputation maintains reasonable performance (AUC 0.821).
Even under these challenging scenarios, our MAE imputation maintains reasonable performance (AUC 0.821–0.838), demonstrating clinical robustness beyond random missingness.
5.6.Comparison With State‐of‐the‐Art
Table 8 compares our method with recent state‐of‐the‐art approaches. Methods are evaluated on different dataset combinations (ABIDE I only vs. ABIDE I+II), which affects comparability. Our method achieves competitive performance while uniquely handling missing modalities, a capability not present in prior work.
TABLE 8Comparison with state‐of‐the‐art methods on ABIDE.
Method Year Dataset AUC Missing
(Abraham et al. 2017) 2017 ABIDE I 0.69 ×
(Parisot et al. 2017) 2017 ABIDE I 0.71 ×
(Heinsfeld et al. 2018) 2018 ABIDE I 0.70 ×
(Dadi et al. 2019) 2019 ABIDE I+II 0.75 ×
(L. Zhang et al. 2020) 2020 ABIDE I 0.82 ×
(Assouline et al. 2021) 2021 ABIDE I+II 0.84 ×
(X. Wang et al. 2022) 2022 ABIDE I+II 0.85 ×
(Eslami et al. 2023) 2023 ABIDE I+II 0.86 ×
MMR‐MF3 (urs) 2024 ABIDE I+II 0.88 ✓
Note: Methods evaluated on different dataset combinations (ABIDE I only vs. ABIDE I+II) are not directly comparable. Our method uniquely handles missing modalities.
Recent methods (2022–2024) include ASDFusion (Eslami et al. 2023) (transformer‐based multimodal fusion) and X. Wang et al. (2022) (attention‐based fusion), neither of which addresses missing modalities.
6.Ablation Studies
6.1.Component Analysis
Table 9 presents ablation results examining the contribution of each component.
TABLE 9Ablation study results.
Configuration AUC Acc F1 Sen Spe
Full model 0.876 0.801 0.792 0.782 0.819
‐MAE (no imputation) 0.847 0.778 0.768 0.756 0.798
‐Sex (no stratification) 0.858 0.789 0.778 0.768 0.808
‐GNN (MLP for fMRI) 0.852 0.784 0.773 0.762 0.804
‐Attention (concat only) 0.857 0.788 0.777 0.766 0.808
‐Modality embeddings 0.869 0.795 0.786 0.776 0.813
Only MAE (no fusion) 0.821 0.762 0.751 0.739 0.784
Only transformer (no MAE) 0.838 0.776 0.765 0.754 0.797
Note: Systematic ablation of each component. Removing MAE causes the largest performance drop (‒0.029 AUC), highlighting its importance for missing modality robustness.
Key findings:
Removing MAE (‐MAE) causes significant performance drop (AUC: 0.847), especially with missing modalities.
Removing sex‐stratified heads (‐Sex) reduces female performance more than male.
Replacing GNN with MLP for fMRI (‐GNN) decreases AUC by 0.024.
Removing cross‐modal attention (‐Attention) reduces AUC by 0.019.
6.2.MAE Architecture Analysis
We compare different MAE architectures:
Our design with separate encoders and shared decoder achieves the best balance of performance and efficiency as shown in Table 10.
TABLE 10Comparison of MAE architectures.
Architecture AUC Params Time
Shared Enc + Dec 0.861 2.1 M 45 min
Separate Enc + Shared Dec 0.876 2.8 M 52 min
Separate Enc + Dec 0.874 3.4 M 61 min
VAE‐based 0.853 2.5 M 58 min
GAN‐based 0.858 4.2 M 94 min
Note: Our design achieves the best balance of performance and efficiency.
6.3.Transformer Architecture Ablation
Table 11 examines the impact of Transformer hyperparameters as shown in Figure 5.
TABLE 11Transformer architecture ablation.
Configuration AUC Params
Number of layers
2 layers 0.868 2.1 M
4 layers 0.876 2.8 M
6 layers 0.877 3.5 M
Number of heads
4 heads 0.868 2.8 M
8 heads 0.876 2.8 M
16 heads 0.875 3.1 M
Hidden dimension
64 0.861 1.4 M
128 0.876 2.8 M
256 0.878 8.9 M
Note: Four layers, eight heads, and 128 hidden dimensions provide the optimal balance of performance and computational efficiency.
Key findings:
Number of layers: Performance improves with depth up to four layers (AUC 0.876), with diminishing returns at six layers (AUC 0.877).
Number of heads: Eight heads achieve optimal performance (AUC 0.876). Fewer heads (four) underperform (AUC 0.868), while more heads (16) show no improvement (AUC 0.875).
Hidden dimension: 128 dimensions provide the best balance; smaller (64) underfits (AUC 0.861), larger (256) shows marginal gains (AUC 0.878) at 4x computational cost.
6.4.Hyperparameter Sensitivity
Figure 6 shows sensitivity to key hyperparameters.
The optimal λ=0.3 balances classification and reconstruction objectives as shown in Table 12.
TABLE 12Per‐modality contribution analysis.
Modality Mean |SHAP|
Contribution Rank
fMRI 5.23×10−2 52.3% 1
Phenotypic 4.77×10−2 47.7% 2
Note: Both modalities contribute substantially to predictions, with fMRI providing slightly more information (52.3% vs. 47.7%).
7.Clinical Interpretability Analysis
7.1.SHAP Methodology Clarification
Our SHAP analysis computes feature importance on the latent fused representation
hfused∈R128, not on raw input features. This is a deliberate choice because of the following:
(1)The latent space captures nonlinear interactions between fMRI and phenotypic features that are not present in raw inputs.
(2)Attribution in latent space reflects the model's actual decision‐making process.
(3)Direct SHAP on 19,900 fMRI features is computationally intractable.
To connect latent attributions to clinical features, we:
(1)Compute correlation between latent dimension activations and raw features,
(2)Validate top latent dimensions against known ASD biomarkers from literature,
(3)Use GNN node attention scores to identify important brain regions.
7.2.Global Feature Importance
Figure 7 shows the top 20 most important latent dimensions identified by SHAP analysis, mapped to their most correlated clinical features.
7.3.Brain Network Analysis
SHAP analysis reveals that connectivity within and between the DMN and salience network are most discriminative for ASD diagnosis, consistent with prior neuroimaging literature (Uddin et al. 2013).
7.4.Per‐Modality Contribution
The fMRI modality contributes slightly more to predictions (52.3%) than phenotypic data (47.7%), but both modalities provide complementary information.
7.5.Sex‐Specific Biomarkers
Analysis of sex‐specific models reveals different biomarker patterns:
Males: Strong emphasis on social communication deficits (ADOS Social: SHAP = 0.089) and DMN connectivity (SHAP = 0.076).
Females: Greater emphasis on repetitive behaviors (ADI‐R RRB: SHAP = 0.082) and fronto‐parietal connectivity (SHAP = 0.071).
These findings align with known sex differences in ASD presentation (Hull et al. 2017).
8.Limitations
While our framework shows promising results, several limitations should be acknowledged:
Dataset heterogeneity: ABIDE aggregates data from multiple sites with different scanners and protocols. Despite harmonization and LOSO validation, residual site effects may persist.
Age range: Our analysis focuses on children and adolescents. Performance may differ for adult populations or toddlers.
Binary classification: We treat ASD diagnosis as binary, while clinical practice recognizes a spectrum of severity and subtypes. This binary framing may cause the model to learn to separate severe cases from controls rather than capturing the full clinical spectrum. Future work should explore continuous severity prediction using ADOS total scores as targets.
External validation: Validation on independent datasets beyond ABIDE is needed to establish generalizability. The ABIDE dataset, while large, may not represent all clinical populations.
Clinical utility: While our method improves diagnostic accuracy, real‐world clinical deployment requires integration with existing workflows and regulatory approval.
SMOTE in high dimensions: While we apply BorderlineSMOTE in the latent space rather than the original high‐dimensional feature space, the relatively small female sample size (N = 401) remains a limitation. Results for the female subgroup should be interpreted with caution pending replication in larger cohorts.
9.Conclusion and Future Work
We presented MMR‐MF3, a missing‐modality‐robust multimodal fusion framework for explainable ASD diagnosis. Our method addresses three critical challenges in clinical deployment: handling incomplete data, providing interpretable predictions, and accounting for sex differences. Through comprehensive experiments on ABIDE I and II, including the critical LOSO validation, we demonstrated state‐of‐the‐art performance and robustness to missing modalities.
Future directions include the following:
Extending to additional modalities (structural MRI, EEG, genetic data).
Developing continuous severity prediction instead of binary classification.
Validating on prospective clinical cohorts with larger female representation.
Implementing federated learning for multisite collaboration without data sharing.
Developing personalized intervention recommendations based on identified biomarkers.
Author Contributions
Jarin Alam Prity: conceptualization, methodology, software, formal analysis, investigation, writing – original draft, visualization. Sahriar Nur Nahin: methodology, software. Ahmad Wasim Wardak: formal analysis, writing – review and editing. Md. Mahfujul Hasan: conceptualization, supervision, project administration, resources, writing – review and editing.
Funding
The authors have nothing to report.
Disclosure
This article reports human subjects. Recruitment meets scientific requirements & HBM's expectation of inclusivity. The ABIDE consortium datasets include participants across diverse demographic groups without limitations by sex, gender, race, or ethnicity beyond scientifically justified diagnostic criteria. Our analysis explicitly addresses sex‐based neurobiological differences by implementing sex‐stratified classification to ensure equitable diagnostic performance across male and female participants, consistent with HBM's inclusivity standards.
Ethics Statement
This study constitutes secondary analysis of publicly available, de‐identified data. The ABIDE datasets were aggregated from multiple international sites, each obtaining local Institutional Review Board (IRB) approval and informed consent from participants or their legal guardians prior to data contribution. As this research involves analysis of existing anonymized data without identifiable private information, additional IRB approval was not required per the policies of Metropolitan University and the Declaration of Helsinki.
Consent
The ABIDE datasets used in this study comprise publicly available, de‐identified secondary data. The original data collection was conducted at multiple international sites, each obtaining informed consent from participants or their legal guardians prior to enrollment. As this study constitutes secondary analysis of fully anonymized data without identifiable private information, additional individual patient consent was not required.
Conflicts of Interest
The authors declare no conflicts of interest.
Data Availability Statement
The ABIDE I and ABIDE II datasets used in this study are publicly available through the Preprocessed Connectomes Project (http://preprocessed‐connectomes‐project.org/abide/) and the Autism Brain Imaging Data Exchange initiative. Phenotypic data and preprocessed functional connectivity matrices are available via the International Neuroimaging Data‐Sharing Initiative (INDI). Code for the proposed MMR‐MF3 framework will be made publicly available upon acceptance via GitHub.
References
Abraham, A.
, Milham M. P., Di Martino A., et al. 2017. “Deriving Reproducible Biomarkers From Multi‐Site Resting‐State Data: An Autism‐Based Example.” NeuroImage
147: 736–745.
Anderson, J. S.
, Nielsen J. A., Froehlich A. L., al et. 2011. “Functional Connectivity Magnetic Resonance Imaging Classification of Autism.” Brain
134, no. 12: 3742–3754.
Assouline, S. G.
, Jelen B., and Boisgontier M. P.. 2021. “Brain Imaging in Autism: A Systematic Review and Meta‐Analysis.” Neuroscience & Biobehavioral Reviews
127: 129–143.
Cameron Craddock, R.
, Di Martino A., McGonigle J., al et. 2018. “Abide II: Advancing the Understanding of Autism.” NeuroImage
172: 263–266.
Craddock, R. C.
, James G. A., Holtzheimer P. E., Hu X. P., and Mayberg H. S.. 2012. “A Whole Brain fMRI Atlas Generated via Spatially Constrained Spectral Clustering.” Human Brain Mapping
33, no. 8: 1914–1928.
Craddock, R. C.
, Sikka S., Cheung B., et al. 2014. “A Neuroimaging Informatics System to Manage, Query, and Download Data From the Autism Brain Imaging Data Exchange (ABIDE).” NeuroImage
100: 238–247.
Dadi, K.
, Rahim M., Abraham A., et al. 2019. “Benchmarking Functional Connectome‐Based Predictive Models for Resting‐State fMRI.” NeuroImage
192: 115–134.
Di Martino, A.
, Yan C.‐G., Li Q., al et. 2014. “The Autism Brain Imaging Data Exchange: Towards a Large‐Scale Evaluation of the Intrinsic Brain Architecture in Autism.” Molecular Psychiatry
19, no. 6: 659–667.
Ecker, C.
, Marquand A., Mourão‐Miranda J., al et. 2010. “Describing the Brain in Autism in Five Dimensions–Magnetic Resonance Imaging‐Assisted Diagnosis of Autism Spectrum Disorder Using a Multiparameter Classification Approach.” Journal of Neuroscience
30, no. 32: 10612–10623.
Eslami, T.
, Mirjalili V., Fong A., Laird A. R., and Saeed F.. 2023. “Asdfusion: Attention‐Based Multimodal Fusion for Autism Spectrum Disorder Diagnosis Using Structural and Functional MRI.” IEEE Journal of Biomedical and Health Informatics
27, no. 4: 1892–1903.
Floris, D. L.
, Lai M.‐C., Auer T., al et. 2018. “Atypical Brain Asymmetry in Autism—A Candidate for Clinically Meaningful Stratification.” Biological Psychiatry: Cognitive Neuroscience and Neuroimaging
3, no. 12: 1024–1032.
Han, H.
, Wang W.‐Y., and Mao B.‐H.. 2005. “Borderline‐SMOTE: A New Over‐Sampling Method in Imbalanced Data Sets Learning.” In Advances in Intelligent Computing, edited by De‐Shuang H., Xiao‐Ping Z. and Guang‐Bin H., 878–887. Springer Berlin Heidelberg.
Heinsfeld, A. S.
, Franco A. R., Craddock R. C., Buchweitz A., and Meneguzzi F.. 2018. “Deep Learning Predicts Autism Spectrum Disorder From Resting‐State Functional Connectivity.” NeuroImage: Clinical
17: 1158–1161.
Hendrycks, D.
, and Gimpel K.. 2016. “Gaussian Error Linear Units (GELUs).” Preprint, arXiv, June 27. preprint arXiv:1606.08415 10.48550/arXiv.1606.08415.
Herbert, M. R.
, Ziegler D. A., Makris N., et al. 2011. “Brain Connectivity and Autism Spectrum Disorders.” Neuroscientist
17, no. 5: 519–533.
Holzinger, A.
, Langs G., Denk H., Zatloukal K., and Müller H.. 2019. “Causability and Explainability of Artificial Intelligence in Medicine.” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery
9, no. 4: e1312.
Hull, L.
, Mandy W., Lai M.‐C., al et. 2017. “Putting on a Brave Face: The Mask of Masculinity and the Double Empathy Problem.” Autism
21, no. 8: 1009–1011.
Johnson, W. E.
, Li C., and Rabinovic A.. 2007. “Adjusting Batch Effects in Microarray Expression Data Using Empirical Bayes Methods.” Biostatistics
8, no. 1: 118–127.
Keskin, M. Z.
, Yendiki A., and Golland P.. 2019. “Deep Learning‐Based MRI Classification of Autism Spectrum Disorder.” International Conference on Medical Image Computing and Computer‐Assisted Intervention, 859–867.
Ktena, S. I.
, Parisot S., Ferrante E., et al. 2017. “Distance Metric Learning Using Graph Convolutional Networks: Application to Functional Brain Networks.” International Conference on Medical Image Computing and Computer‐Assisted Intervention, 469–477.
Li, H.
, and Fan Y.. 2018. “Multimodal Classification of Neurodegenerative Diseases Using Fused Features From Structural MRI and Functional Connectivity Data.”International Workshop on Machine Learning in Medical Imaging, 159–167.
Loomes, R.
, Hull L., and Mandy W. P.. 2017. “What is the Male‐to‐Female Ratio in Autism Spectrum Disorder? A Systematic Review and Meta‐Analysis.” Journal of the American Academy of Child & Adolescent Psychiatry
56, no. 6: 466–474.
Lord, C.
, Elsabbagh M., Baird G., and Veenstra‐Vanderweele J.. 2018. “Autism Spectrum Disorder.” Lancet
392, no. 10146: 508–520.
Loshchilov, I.
, and Hutter F.. 2019. “Decoupled Weight Decay Regularization.” International Conference on Learning Representations.
Lundberg, S. M.
, and Lee S.‐I.. 2017. “A Unified Approach to Interpreting Model Predictions.” Advances in Neural Information Processing Systems
30: 4765–4774.
Ma, J.
, Wang Q., Li X., Chen X., Chen X., and Chen Y.. 2018. “Modality‐Invariant Feature Learning for Multimodal Medical Image Analysis.” Lecture Notes in Computer Science, vol 11073, pp. 402410. Springer, Cham.
Maenner, M. J.
, Warren Z., Williams A. R., al et. 2023. “Prevalence and Characteristics of Autism Spectrum Disorder Among Children Aged 8 Years—Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2020.” Morbidity and Mortality Weekly Report: Surveillance Summaries
72, no. 2: 1.
McNemar, Q.
1947. “Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages.” Psychometrika
12, no. 2: 153–157.
Parisot, S.
, Ktena S. I., Ferrante E., et al. 2017. “Spectral Graph Convolutions for Population‐Based Disease Prediction.” Medical Image Computing and Computer Assisted Intervention – MICCAI 2017, LNCS Volume 10435, pp. 177–185.
Rosenkrantz, A. B.
, Doshi A. M., Babb J. S., Vysalovic M., and Recht M. P.. 2020. “Enhancing the Usability and Understanding of Explainable Artificial Intelligence (XAI) for Medical Imaging.” Academic Radiology
27, no. 12: 1732–1741.
Thomas, B.
, van Rooij D., Anagnostou E., al et. 2020. “Interpreting Deep Learning Models for Autism Classification Using SHAP.” NeuroImage
218: 116983.
Uddin, L. Q.
, Supekar K., Lynch C. J., al et. 2013. “Salience Network‐Based Classification and Prediction of Symptom Severity in Children With Autism.” JAMA Psychiatry
70, no. 8: 869–879.
Van Buuren, S.
, and Groothuis‐Oudshoorn K.. 2011. “Mice: Multivariate Imputation by Chained Equations in R.” Journal of Statistical Software
45: 1–67.
Volkmar, F.
, Siegel M., Woodbury‐Smith M., al et. 2014. “Practice Parameter for the Assessment and Treatment of Children and Adolescents With Autism Spectrum Disorder.” Journal of the American Academy of Child & Adolescent Psychiatry
53, no. 2: 237–257.
Wang, X.
, Zhang L., Chen Y., and Li M.. 2022. “Multimodal Attention‐Based Deep Learning for Autism Spectrum Disorder Diagnosis.” IEEE Transactions on Neural Networks and Learning Systems
33, no. 8: 3456–3468.
Wang, Y.
, Singh A., and Lin L.. 2020. “Missing Data Imputation for Sparse Matrices.” Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1408–1417.
Xu, K.
, Hu W., Leskovec J., and Jegelka S.. 2019. “How Powerful are Graph Neural Networks?.” International Conference on Learning Representations.
Yoon, J., J. Jordon, and M. Schaar. 2018. "Gain: Missing Data Imputation Using Generative Adversarial Nets." Proceedings of Machine Learning Research 80: 5689–5698.
Zhang, L.
, Wang X., Chen Y., and Li M.. 2020. “Neuroimaging‐Based Classification of Autism Spectrum Disorder Using a 3d Convolutional Neural Network.” Frontiers in Neuroscience
14: 586469.
Zhang, X.
, Han Y., Xu W., and Wang Q.. 2022. “Explainable Artificial Intelligence (XAI) in Neuroimaging: A Systematic Review.” NeuroImage
258: 119373.
Zheng, S.
, Kim S. H., Zhu Y., Chen W., and Li X.. 2022. “Sex Differences in Brain Functional Connectivity Predict Autism Spectrum Disorder Diagnosis.” Cerebral Cortex
32, no. 15: 3189–3202.
Associated Data
Data Availability Statement
The ABIDE I and ABIDE II datasets used in this study are publicly available through the Preprocessed Connectomes Project (http://preprocessed‐connectomes‐project.org/abide/) and the Autism Brain Imaging Data Exchange initiative. Phenotypic data and preprocessed functional connectivity matrices are available via the International Neuroimaging Data‐Sharing Initiative (INDI). Code for the proposed MMR‐MF3 framework will be made publicly available upon acceptance via GitHub.
Come citare questo studio
Jarin Alam Prity, Sahriar Nur Nahin, Ahmad Wasim Wardak, Md Mahfujul Hasan. (2026). Handling Incomplete Multimodal Data in ASD Diagnosis: A Masked Autoencoder Approach With Sex-Stratified Fusion.. Brain and behavior. DOI: 10.1002/brb3.71904
Consulta la pubblicazione originale ↗