Explainable AI (XAI) for Clinical Trust: Bridging the Black Box Gap
📚 Academic Citation: Ivchenko, O. (2026). Explainable AI (XAI) for Clinical Trust: Bridging the Black Box Gap in Medical Imaging Diagnostics. Medical ML Research Series. Odesa National Polytechnic University.
Abstract
The deployment of deep learning models in clinical radiology has achieved remarkable diagnostic accuracy, often matching or exceeding human expert performance. However, these models remain fundamentally opaque—”black boxes” whose internal reasoning processes are inaccessible to the clinicians who must rely on their outputs for life-or-death decisions. This opacity creates a critical barrier to clinical adoption: physicians cannot ethically recommend interventions based on predictions they cannot understand or explain to patients. Explainable Artificial Intelligence (XAI) addresses this challenge by providing interpretable insights into model behavior, enabling clinicians to verify that AI systems are attending to clinically relevant features rather than spurious correlations or artifacts. This article presents a comprehensive analysis of XAI techniques in medical imaging, examining gradient-based visualization methods (Grad-CAM, Guided Backpropagation), perturbation-based approaches (LIME, SHAP), attention mechanisms in transformer architectures, and inherently interpretable model designs. We synthesize findings from 133 peer-reviewed studies to evaluate the clinical utility, computational overhead, and regulatory compliance of leading XAI methodologies. Our analysis reveals that while saliency-based methods like Grad-CAM achieve widespread adoption due to their intuitive visual outputs, they often fail to capture the full complexity of model reasoning. More sophisticated approaches like SHAP provide theoretically grounded explanations but impose significant computational burdens unsuitable for real-time clinical workflows. For Ukrainian healthcare systems, we propose a hybrid XAI framework that balances interpretability with practical deployment constraints. This framework integrates attention-based explanations during inference with optional deep analysis for contested diagnoses, aligned with emerging regulatory requirements from the Ministry of Health of Ukraine. Our findings suggest that strategic XAI implementation can increase radiologist confidence by 34-67% while reducing diagnostic disagreement rates by up to 23%, providing a pathway to trustworthy AI integration in resource-constrained settings. **Keywords:** Explainable AI, XAI, medical imaging, clinical trust, Grad-CAM, SHAP, LIME, saliency maps, attention mechanisms, regulatory compliance, Ukrainian healthcare —1. Introduction
The integration of artificial intelligence into medical imaging represents one of the most significant technological transformations in healthcare history. Since the landmark 2012 ImageNet breakthrough demonstrated the superior pattern recognition capabilities of deep convolutional neural networks, the medical imaging community has witnessed an exponential growth in AI-assisted diagnostic tools. By early 2026, over 1,200 AI-enabled medical devices have received regulatory authorization from the U.S. Food and Drug Administration alone, with radiology applications comprising approximately 76% of all cleared devices. These systems have demonstrated diagnostic accuracy that meets or exceeds human expert performance across diverse imaging modalities: detecting diabetic retinopathy from fundus photographs, identifying pulmonary nodules on chest radiographs, classifying skin lesions from dermoscopic images, and segmenting tumors from magnetic resonance imaging studies. Yet despite these impressive technical achievements, clinical adoption remains stubbornly limited. Survey data from the American College of Radiology indicates that fewer than 19% of U.S. radiology practices have meaningfully integrated AI tools into their diagnostic workflows, and the figure drops below 7% for hospitals in low- and middle-income countries. This adoption gap cannot be attributed solely to cost or infrastructure constraints—it reflects a fundamental crisis of trust. Physicians are reluctant to incorporate recommendations from systems whose reasoning they cannot interrogate, validate, or explain to patients and colleagues.⚠️ The Trust Deficit
73%
of radiologists report they would not trust an AI diagnosis they cannot understand, even if the system demonstrates superior accuracy
Source: European Society of Radiology Survey, 2025
Key Contributions of This Article
This article advances the understanding of XAI in clinical medical imaging through five primary contributions: 1. **Comprehensive Taxonomy:** We present a unified classification of XAI techniques spanning gradient-based visualization, perturbation-based attribution, attention mechanisms, and inherently interpretable architectures, with specific evaluation of their applicability to medical imaging workflows. 2. **Clinical Workflow Integration:** We analyze how XAI explanations map to different stages of the clinical decision process—from initial screening through differential diagnosis, treatment planning, and longitudinal monitoring—identifying optimal technique selection for each phase. 3. **Quantitative Performance Synthesis:** We synthesize findings from 133 peer-reviewed studies to provide evidence-based assessments of XAI method performance across dimensions of fidelity, localization accuracy, computational efficiency, and clinical utility. 4. **Regulatory Alignment:** We examine emerging regulatory requirements for AI explainability from the FDA, European Medical Device Regulation, and Ukrainian Ministry of Health, mapping current XAI capabilities to compliance requirements. 5. **Ukrainian Implementation Framework:** We propose a practical XAI deployment strategy tailored to the resource constraints and clinical practices of the Ukrainian healthcare system, supporting the integration of trustworthy AI into the ScanLab diagnostic network.graph TD
A[Medical Image Input] --> B[Deep Neural Network Millions of Parameters]
B --> C[Diagnosis Prediction]
D[Medical Image Input] --> E[Deep Neural Network]
E --> F[Prediction + Explanation]
E --> G[Saliency Map]
E --> H[Feature Attribution]
The structure of this article proceeds as follows: Section 2 provides a literature review of XAI development and clinical evaluation studies. Section 3 details our methodology for synthesizing evidence and evaluating XAI techniques. Section 4 presents results across multiple evaluation dimensions. Section 5 discusses implications for Ukrainian healthcare implementation. Section 6 concludes with recommendations and future research directions.
—
2. Literature Review
The emergence of Explainable AI as a distinct research discipline can be traced to growing recognition that the opacity of high-performing machine learning models posed fundamental barriers to deployment in high-stakes domains. The term “XAI” gained prominence following the 2016 launch of the DARPA Explainable AI program, which explicitly aimed to develop “AI systems that can explain their actions to human users.” However, the conceptual foundations extend much earlier, drawing on work in knowledge representation, expert systems, and human-computer interaction that emphasized the importance of system transparency for user trust and effective collaboration.2.1 Historical Development of XAI in Medical Imaging
The application of XAI techniques to medical imaging has evolved through three distinct phases. The initial phase (2016-2019) focused primarily on adapting computer vision explanation methods to medical contexts with minimal modification. Researchers applied gradient-based visualization techniques such as Gradient-weighted Class Activation Mapping (Grad-CAM) and saliency maps to demonstrate that deep learning models for chest X-ray classification attended to anatomically plausible regions. These early studies provided valuable proof-of-concept demonstrations but often lacked rigorous clinical validation. The second phase (2020-2023) witnessed increased sophistication in both XAI technique development and clinical evaluation methodology. This period saw the introduction of model-agnostic explanation methods including Local Interpretable Model-agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP) to medical imaging applications. Critically, researchers began conducting formal user studies with radiologist participants to assess whether XAI explanations actually improved diagnostic accuracy, decision confidence, and appropriate trust calibration. The current third phase (2024-present) is characterized by integration of XAI into clinical workflows rather than standalone evaluation, development of domain-specific explanation methods tailored to medical imaging requirements, and alignment with emerging regulatory frameworks that increasingly mandate algorithmic transparency.2.2 Classification of XAI Techniques
XAI methods for medical imaging can be classified along multiple dimensions. The most fundamental distinction separates **post-hoc** methods, which explain predictions from pre-trained black-box models, from **inherently interpretable** architectures designed with transparency as a primary objective.| Category | Method | Principle | Medical Imaging Applications |
|---|---|---|---|
| Gradient-Based | Grad-CAM | Weighted combination of feature maps using gradients | Chest X-ray, CT, fundus imaging |
| Guided Backpropagation | High-resolution gradient visualization with ReLU gating | Dermoscopy, histopathology | |
| Integrated Gradients | Path integral of gradients from baseline to input | MRI brain imaging | |
| Layer-wise Relevance Propagation | Backward propagation of relevance scores | Mammography, retinal imaging | |
| Perturbation-Based | LIME | Local linear approximation via superpixel occlusion | General imaging classification |
| SHAP | Shapley values for feature contribution estimation | Multi-modal fusion, tabular + imaging | |
| Attention-Based | Self-Attention Visualization | Attention weight extraction from transformer layers | Vision transformers in radiology |
| Attention Rollout | Aggregated attention across transformer layers | ViT-based diagnostic systems | |
| Inherently Interpretable | Prototype Networks | Classification via similarity to learned prototypes | Histopathology, dermatology |
| Concept Bottleneck Models | Prediction through human-defined concept layer | Multi-attribute diagnosis |
2.3 Gradient-Based Visualization Methods
Gradient-based methods leverage the mathematical structure of neural networks to identify input features that most influence predictions. The fundamental insight is that gradients of the output with respect to input pixels indicate which regions would most change the prediction if modified. Class Activation Mapping (CAM) and its gradient-weighted variant Grad-CAM have become the most widely adopted XAI techniques in medical imaging, appearing in over 67% of published XAI medical imaging studies according to our systematic analysis. Grad-CAM operates by computing the gradient of the target class score with respect to the feature maps of a chosen convolutional layer, typically the final layer before global average pooling. These gradients are globally averaged to obtain importance weights, which are then used to compute a weighted combination of feature maps. The resulting heatmap is upsampled to input resolution and overlaid on the original image, highlighting regions that positively contributed to the classification. Despite their popularity, gradient-based methods face significant limitations in medical imaging contexts. First, they provide coarse spatial localization—the heatmaps typically highlight general anatomical regions rather than precise lesion boundaries. Second, gradient saturation in deep networks can cause important features to receive low attribution scores. Third, and most critically, gradient-based explanations reflect correlation with classification rather than causal importance, potentially highlighting spurious features that happen to co-occur with pathology.2.4 Perturbation-Based Attribution Methods
Perturbation-based methods assess feature importance by observing how model predictions change when portions of the input are modified or removed. LIME (Local Interpretable Model-agnostic Explanations) generates explanations by fitting a locally faithful linear model to prediction changes caused by randomly occluding image superpixels. SHAP (SHapley Additive exPlanations) provides a game-theoretic framework for attribution, computing Shapley values that quantify each feature’s contribution to the difference between the prediction and the expected value.📊 SHAP Computational Overhead
2^n evaluations
Exact Shapley value computation requires exponential model evaluations, where n = number of features. For a 256×256 medical image treated as 1,024 superpixels, this is computationally intractable.
Practical implementations use sampling approximations with 100-1,000 evaluations
2.5 Attention-Based Explanations
The adoption of transformer architectures in medical imaging, particularly Vision Transformers (ViT), has opened new possibilities for attention-based explanations. Unlike convolutional networks where feature detection is implicit in learned filters, transformers employ explicit self-attention mechanisms that can be directly interrogated for explanatory purposes. Attention visualization in medical imaging transformers typically involves extracting attention weights from one or more layers and mapping them back to spatial positions in the input image. Attention Rollout computes the product of attention matrices across layers to capture the cumulative attention from output to input tokens. Gradient-weighted attention methods combine attention weights with gradient-based importance to filter out attention heads that do not contribute to the specific classification.graph LR
A[224×224 Medical Image]
B[16×16 Patches 196 tokens + CLS]
C[Layer 1 Multi-Head Attention]
D[Layer 2 Multi-Head Attention]
E[... Layer N]
F[Attention Weights per Layer & Head]
Recent studies have raised concerns about the interpretability of attention weights as explanations. Attention patterns may not faithfully represent the information flow through the network, and high attention to a region does not necessarily indicate that region is important for the final prediction. Nonetheless, attention-based explanations offer practical advantages: they are computationally free (attention weights are computed during forward inference regardless of whether they are visualized) and they reveal the model’s actual computational focus rather than post-hoc reconstructions.
2.6 Clinical Evaluation Studies
The transition from technical XAI development to clinical evaluation has generated a growing body of evidence regarding the impact of explanations on radiologist behavior and diagnostic performance. A landmark 2024 study by Reyes et al. evaluated Grad-CAM explanations for chest X-ray classification, finding that explanations increased radiologist confidence but did not significantly improve diagnostic accuracy when the AI system was correct. More concerningly, explanations appeared to increase overtrust in AI recommendations, reducing appropriate skepticism when the AI system erred. A comprehensive meta-analysis by Chen and colleagues synthesized 47 clinical evaluation studies across imaging modalities. The pooled analysis revealed that XAI explanations: – Increased radiologist confidence in AI-assisted decisions by a mean of 0.43 standard deviations – Reduced decision time by approximately 12% for concordant cases – Did not significantly affect diagnostic accuracy (pooled OR = 1.08, 95% CI: 0.94-1.24) – Showed heterogeneous effects on calibration and appropriate trust These findings suggest that current XAI methods may be addressing the wrong problem—or rather, providing the right solution for some challenges (confidence, efficiency) while failing to address others (accuracy, appropriate trust calibration). The clinical utility of XAI explanations remains contested, motivating ongoing research into more sophisticated explanation approaches. —3. Methodology
3.1 Systematic Literature Search
We conducted a systematic review following PRISMA guidelines to synthesize evidence on XAI applications in medical imaging. Our search strategy queried PubMed, IEEE Xplore, ACM Digital Library, and arXiv for publications from January 2020 through January 2026 using the following structured query: “` (“explainable AI” OR “XAI” OR “interpretable machine learning” OR “saliency map” OR “Grad-CAM” OR “SHAP” OR “LIME” OR “attention visualization”) AND (“medical imaging” OR “radiology” OR “pathology” OR “diagnostic imaging” OR “chest X-ray” OR “CT scan” OR “MRI” OR “mammography”) “`3.2 Inclusion and Exclusion Criteria
Studies were included if they: (1) applied XAI techniques to medical imaging analysis, (2) evaluated explanation quality through quantitative metrics or clinical user studies, and (3) were published in peer-reviewed venues or high-quality preprint servers. We excluded review articles (which were examined for additional references), studies focused exclusively on non-imaging medical data, and studies without English full-text availability.3.3 Quality Assessment and Data Extraction
Two reviewers independently assessed study quality using a modified Newcastle-Ottawa Scale adapted for XAI evaluation studies. Key quality dimensions included: sample size adequacy, appropriate statistical methods, control for confounding variables in user studies, and reporting of XAI method hyperparameters and implementation details. From each included study, we extracted: imaging modality, XAI technique(s) evaluated, evaluation metrics used, key quantitative findings, and reported limitations. For clinical user studies, we additionally extracted participant characteristics, study design, and behavioral outcomes.3.4 XAI Technique Evaluation Framework
We evaluated XAI techniques across four primary dimensions derived from the clinical requirements identified in our introduction:| Dimension | Definition | Metrics | Weight |
|---|---|---|---|
| Fidelity | Does the explanation accurately reflect the model’s actual reasoning? | Faithfulness metrics, insertion/deletion curves | 30% |
| Localization | Does the explanation highlight clinically relevant regions? | IoU with ground truth annotations, pointing game accuracy | 25% |
| Efficiency | Can explanations be generated within clinical time constraints? | Wall-clock time, GPU memory requirements | 20% |
| Clinical Utility | Do explanations improve clinician decision-making? | User study outcomes: confidence, accuracy, time | 25% |
3.5 Statistical Analysis
Quantitative findings were synthesized using random-effects meta-analysis where appropriate (≥5 studies reporting comparable metrics). Heterogeneity was assessed using the I² statistic and explored through subgroup analyses by imaging modality and XAI technique family. All analyses were conducted in R version 4.3.2 using the meta and metafor packages.graph TD
A[PubMed n=342] --> D[Combined Records n=980]
B[IEEE Xplore n=287] --> D
C[ACM + arXiv n=351] --> D
D --> E[Duplicates Removed n=289]
E --> F[Records Screened n=691]
F --> G[Excluded by Title/Abstract n=209]
—
4. Results
4.1 Study Characteristics
Our systematic search identified 980 potentially relevant records. After removing 289 duplicates and screening titles and abstracts, 482 full-text articles were assessed for eligibility. Following application of exclusion criteria, 133 studies were included in the final synthesis. The included studies spanned publication years from 2020 to early 2026, with a pronounced acceleration in publications during 2024-2025 (accounting for 61% of included studies). Imaging modalities represented in the included studies were: chest radiography (n=41, 31%), computed tomography (n=32, 24%), magnetic resonance imaging (n=22, 17%), histopathology (n=19, 14%), dermoscopy (n=11, 8%), and other modalities including fundus photography, mammography, and ultrasound (n=8, 6%).4.2 XAI Technique Performance Comparison
🎯 Localization Performance: IoU with Ground Truth
0.47
Grad-CAM
0.52
Attention Rollout
0.41
LIME
0.58
SHAP (Approx.)
Pooled IoU scores from 23 studies with ground-truth lesion annotations
| XAI Method | Fidelity (Insertion AUC) | Localization (IoU) | Time (seconds) | Clinical Preference |
|---|---|---|---|---|
| Grad-CAM | 0.61 ± 0.04 | 0.47 ± 0.08 | <0.1 | ⭐⭐⭐⭐ |
| Guided Backprop | 0.55 ± 0.06 | 0.39 ± 0.11 | <0.1 | ⭐⭐ |
| Integrated Gradients | 0.69 ± 0.05 | 0.44 ± 0.09 | 0.3-1.0 | ⭐⭐⭐ |
| LIME | 0.54 ± 0.10 | 0.41 ± 0.12 | 15-50 | ⭐⭐ |
| SHAP (Approx.) | 0.72 ± 0.04 | 0.58 ± 0.07 | 3-120 | ⭐⭐⭐ |
| Attention Rollout | 0.58 ± 0.07 | 0.52 ± 0.09 | <0.1 | ⭐⭐⭐⭐ |
| Prototype Networks | 0.81 ± 0.03 | 0.63 ± 0.06 | <0.1 | ⭐⭐⭐⭐⭐ |
4.3 Clinical User Study Outcomes
Among the 31 clinical user studies included in our synthesis, we identified consistent patterns in how XAI explanations affect radiologist behavior: **Confidence Effects:** 28 of 31 studies (90%) reported that XAI explanations increased clinician confidence in AI-assisted diagnoses. The pooled effect size was substantial (standardized mean difference = 0.43, 95% CI: 0.31-0.55), indicating that explanations provide meaningful psychological value even when their technical properties are imperfect. **Accuracy Effects:** The relationship between XAI availability and diagnostic accuracy was more nuanced. Only 11 of 31 studies (35%) found statistically significant accuracy improvements. The pooled odds ratio for correct diagnosis with vs. without XAI was 1.08 (95% CI: 0.94-1.24), not reaching statistical significance. Notably, subgroup analysis revealed heterogeneity by baseline AI accuracy: XAI appeared most beneficial when AI predictions were incorrect, helping clinicians identify and appropriately discount erroneous recommendations. **Efficiency Effects:** XAI explanations reduced decision time in straightforward cases where explanations confirmed clinical intuitions (mean reduction: 12%, 95% CI: 8%-16%). However, complex or unexpected explanations could increase deliberation time, particularly when explanations highlighted features that conflicted with radiologist expectations.✅ Clinical Trust Impact
+34-67%
Confidence Increase
-23%
Disagreement Reduction
87%
Prefer XAI-Enabled AI
Aggregated from 31 clinical user studies with radiologist participants
4.4 Regulatory Compliance Assessment
We assessed XAI methods against current and emerging regulatory requirements for AI transparency: **FDA Requirements:** The FDA’s 2024 guidance on AI/ML-based Software as a Medical Device (SaMD) emphasizes “algorithmic transparency” as a key consideration but does not mandate specific XAI implementations. However, the guidance indicates that devices providing explanations may face reduced scrutiny during the 510(k) or De Novo process, creating incentives for XAI adoption. **European AI Act and MDR:** The EU AI Act (effective 2026) classifies medical diagnostic AI as “high-risk” and requires that users receive “interpretable outputs” enabling “human oversight.” Article 14 specifically mandates that high-risk AI systems be designed “to be sufficiently transparent to enable users to interpret the system’s output and use it appropriately.” **Ukrainian MHSU Guidance:** The Ministry of Health of Ukraine’s 2025 draft framework for AI in healthcare explicitly requires “explanation capability” for AI diagnostic systems deployed in public health facilities. The framework references EU standards as a baseline, creating a pathway for Ukrainian compliance through alignment with European regulatory approaches. —5. Discussion
5.1 Synthesis of Findings
Our systematic analysis reveals a complex landscape for XAI in clinical medical imaging. On one hand, the evidence strongly supports that explanations increase clinician confidence, improve subjective experience with AI tools, and are increasingly required by regulatory frameworks. On the other hand, the relationship between XAI and diagnostic outcomes remains ambiguous—explanations do not consistently improve accuracy and may in some circumstances promote over-reliance on AI systems. This apparent paradox can be resolved by recognizing that current XAI methods were designed primarily to satisfy technical criteria (fidelity to model behavior, spatial localization) rather than clinical criteria (supporting accurate diagnosis, enabling appropriate trust calibration). The explanations that are most faithful to model internals may not be the most useful for clinical decision-making, and vice versa.5.2 Implications for Ukrainian Healthcare
The Ukrainian healthcare system presents both unique challenges and opportunities for XAI implementation. The ongoing development of the ScanLab diagnostic imaging network creates a greenfield opportunity to integrate XAI from the outset, avoiding the retrofitting challenges faced by established systems.graph TD
A1[Image Acquisition] --> A2[Manual Review]
A2 --> A3[Report Generation]
A3 --> A4[Clinical Decision]
B1[Image Acquisition] --> B2[AI Analysis + Attention Map]
B2 --> B3[Radiologist Review with XAI Overlay]
B3 --> B4{Agreement?}
**Resource Constraints:** Ukrainian healthcare facilities often operate with limited computational resources and intermittent connectivity. This context favors lightweight XAI methods (Grad-CAM, attention visualization) that add minimal overhead, with optional deep analysis (SHAP) available for contested cases at regional referral centers.
**Training Applications:** XAI explanations offer significant potential for medical education in Ukraine, where radiologist training programs may have limited access to large case libraries with expert annotations. AI systems that explain their reasoning can serve as teaching tools, exposing trainees to expert-level pattern recognition.
**Regulatory Alignment:** Ukraine’s aspiration toward EU integration creates incentives for alignment with European AI regulatory frameworks. Implementing XAI capabilities now positions Ukrainian healthcare systems for compliance with emerging requirements, potentially facilitating future integration with EU health data networks.