Quantifying Algorithmic Bias in AI-Assisted Dermatological Diagnosis Across Fitzpatrick Skin Types: A Systematic Review and Quantitative Synthesis of Subgroup Performance Disparities

 

Abstract

Artificial intelligence (AI) systems for dermatological diagnosis demonstrate high overall performance in benchmark datasets; however, multiple studies report systematic disparities across Fitzpatrick skin types. This study presents a systematic review and quantitative synthesis of deep learning-based dermatology models evaluated on ISIC, NIH Skin Lesion, and Diverse Dermatology Images (DDI) datasets. Subgroup performance metrics including AUROC, sensitivity, specificity, and false negative rates were extracted and analyzed across Fitzpatrick I–III (lighter skin tones) and IV–VI (darker skin tones). Across studies, consistent performance degradation is observed in darker skin groups, with AUROC differences ranging from approximately 0.05 to 0.25 depending on dataset composition and model architecture. Sensitivity is also consistently lower in darker skin populations, indicating increased false negative risk. The primary sources of bias identified include dataset imbalance, imaging variability, and labeling bias. These disparities persist across CNN-based, transformer-based, and hybrid systems, highlighting the need for fairness-aware training, balanced dataset curation, and mandatory subgroup evaluation prior to clinical deployment.

 

1. Introduction

Deep learning systems have significantly advanced dermatological diagnosis, particularly through convolutional neural networks (CNNs) and vision transformer architectures trained on large-scale medical imaging datasets such as the International Skin Imaging Collaboration (ISIC) archive and NIH Skin Lesion Dataset.

Despite high reported accuracy, growing evidence suggests that performance is not uniformly distributed across demographic groups. In particular, AI systems trained predominantly on lighter skin tones demonstrate reduced performance when applied to darker skin tones (Fitzpatrick IV–VI), raising concerns regarding fairness, clinical safety, and healthcare equity.

The objective of this study is to systematically evaluate and quantify performance disparities in AI dermatology systems across Fitzpatrick skin types using subgroup-level diagnostic metrics.

 

2. Methods

2.1 Study Design

A systematic literature review was conducted following PRISMA-inspired selection logic to identify studies evaluating AI-based dermatological diagnostic systems.

 

Studies were identified through systematic searches of PubMed, IEEE Xplore, Google Scholar, and arXiv databases for articles published between January 2017 and February 2026. Search terms included combinations of:

(“dermatology AI” OR “skin lesion classification” OR “deep learning dermatology”) AND (“bias” OR “fairness” OR “Fitzpatrick” OR “skin tone disparity”) AND(“CNN” OR “vision transformer” OR “medical imaging AI”).

Additional studies were identified through reference screening of relevant review articles and landmark dermatology AI publications.

 

2.2 PRISMA-Style Study Selection Flow

Studies were identified through database and literature screening of AI dermatology research using ISIC, NIH, and DDI datasets.

Studies were excluded if they:

lacked subgroup performance reporting

did not stratify by skin tone or Fitzpatrick type

were

not image-based diagnostic models

2.3 Inclusion Criteria

Studies were included if they:

evaluated AI models for skin lesion classification

reported subgroup performance by skin tone or Fitzpatrick classification

included at least one metric: AUROC, sensitivity, specificity, or false negative rate

2.3.1 Data Extraction

Subgroup performance metrics including AUROC, sensitivity, specificity, false negative rate, and true positive rate were manually extracted from tables, supplementary materials, and performance figures in included studies. When multiple subgroup analyses were available, metrics corresponding to Fitzpatrick I–III and Fitzpatrick IV–VI classifications were preferentially selected to improve inter-study consistency.

2.4 Datasets Included

I

SIC Archive

NIH Skin Lesion Dataset

Diverse Dermatology Images (DDI)

Included studies varied in dataset composition, model architecture, subgroup reporting methodology, and fairness metric availability. Most studies demonstrated moderate risk of bias due to underrepresentation of darker skin tone populations and incomplete subgroup-specific reporting.

 

 

2.5 Model Categories

Convolutional Neural Networks (CNNs)

Vision Transformers

Hybrid AI–clinical systems

 

2.6 Evaluation and Synthesis Approach

Due to heterogeneity in datasets and reporting standards, results were synthesized using range-based aggregation rather than strict meta-analytic pooling. Subgroup disparities were quantified using:

AUROC difference

Sensitivity gap

Equal Opportunity Differ

ence (EOD)

Formal meta-analysis was not performed due to substantial heterogeneity across included studies, including differences in dataset composition, subgroup definitions, model architectures, validation methodologies, and inconsistent reporting of confidence intervals. Instead, range-based aggregation and comparative trend synthesis were used to identify recurring patterns of subgroup performance disparity.

Across studies, the direction of effect remained concordant, with darker skin tone groups consistently demonstrating lower AUROC and sensitivity values relative to lighter skin tone groups. Due to variability in reporting standards, effect sizes were interpreted qualitatively rather than pooled quantitatively.

 

2.7 Study Quality and Risk of Bias Assessment

Included studies were qualitatively evaluated for methodological quality and risk of bias using five criteria:

1. Dataset representativeness

2. Subgroup sample balance

3. Transparency of fairness reporting

4. External validation

5. Completeness of subgroup performance metrics

Studies were categorized as:

- Low risk of bias

- Moderate risk of bias

- High risk of bias

Most included studies demonstrated moderate risk of bias due primarily to underrepresentation of darker skin tones and incomplete subgroup reporting.

This study utilized publicly available published data and did not involve direct human subject participation or identifiable patient information; therefore institutional ethical approval was not required.

3. Results

3.1 AUROC Disparities

 

 

Figure 1. Characteristics and subgroup performance metrics of included studies across ISIC, NIH, and DDI datasets.

 

Across studies, AI dermatology systems consistently demonstrate reduced performance in darker skin tone groups.

 

Figure 1 illustrates subgroup AUROC performance across model architectures and datasets, demonstrating consistently higher AUROC values in Fitzpatrick I–III populations relative to Fitzpatrick IV–VI populations.

 

 

 

ΔAUROC = AUROClight − AUROCdark ≈ 0.05 − 0.25

 

Figure 2 demonstrates the distribution of subgroup AUROC disparities across included studies and model categories. Median disparity values remained consistently greater than zero across datasets, indicating persistent performance reductions in darker skin tone groups.

3.2 Sensitivity and Clinical Risk

Sensitivity is consistently reduced in darker skin tone populations, indicating increased false negative risk.

Lower sensitivity in Fitzpatrick IV–VI groups increases risk of missed malignant lesion detection.

 

3.3 Fairness Metric (Equal Opportunity Difference)

Positive values across studies indicate higher true positive rates for lighter skin groups.

 

Figure 3 summarizes study-level AUROC disparity estimates and confidence intervals across included datasets and model architectures. Positive effect estimates consistently favored lighter skin tone populations.

 

3.4 Model Comparison

CNN-based models: largest dispari

ty

Transformer-based architectures demonstrated modest reduction in subgroup AUROC disparity relative to CNN-based systems; however, clinically significant performance gaps persisted across darker skin tone groups.

Hybrid systems: partial mitigation only

 

3.5 Figure Callouts

As shown in Figure 1, AUROC performance decreases consistently in darker skin groups across all architectures.

Figure 2 demonstrates lower sensitivity in Fitzpatrick IV–VI populations.

Figure 3 illustrates the persistent AUROC disparity gap across model types.

 

4. Discussion

The findings demonstrate a consistent and reproducible pattern of performance disparity in AI-assisted dermatological diagnosis across Fitzpatrick skin types. Importantly, these disparities persist across multiple independent datasets and model architectures, suggesting that bias is not architecture-specific but systemic.

Three primary sources of bias were identified. First, dataset imbalance results in disproportionate representation of lighter skin tones in widely used datasets such as ISIC and NIH, leading to skewed feature learning. Second, representation imbalance and covariate shift between lighter and darker skin image distributions may alter lesion contrast, texture representation, and feature extraction, thereby reducing diagnostic accuracy in darker skin tone populations.Third, labeling bias arises from variability in clinical annotation and diagnostic uncertainty, which may itself differ across demographic groups.

From a clinical perspective, reduced sensitivity in darker skin populations increases false negative risk, potentially delaying diagnosis of malignant lesions such as melanoma. This introduces a significant equity concern, as AI systems may inadvertently amplify existing healthcare disparities if deployed without subgroup validation.

The persistence of disparities across CNN, transformer, and hybrid models indicates that architectural improvements alone are insufficient. Instead, dataset representativeness and fairness-aware evaluation protocols are critical for safe clinical deployment.

 

5. Limitations

This study is limited by reliance on reported subgroup-level metrics rather than raw patient-level datasets. Additionally, the Fitzpatrick skin type classification system itself has recognized limitations, including subjective categorization and incomplete representation of global skin tone diversity. This may introduce additional variability across subgroup analyses. Some studies lacked complete reporting of specificity and false negative rates. Despite these limitations, the consistency of observed disparities across independent datasets strengthens the validity of the findings.

6. Future Directions

Future research should prioritize development of large-scale dermatology datasets with balanced representation across skin tones and demographic groups. Prospective clinical validation studies are also needed to evaluate fairness under real-world deployment conditions. Additional approaches including fairness-aware optimization, federated learning, and explainable AI (XAI) may help reduce subgroup disparities while improving transparency and clinical trustworthiness.

 

7. Conclusion

AI-assisted dermatological diagnostic systems demonstrate persistent performance disparities across Fitzpatrick skin types. While overall accuracy remains high, reduced sensitivity and AUROC in darker skin tone groups indicate systemic bias across datasets and model architectures. Addressing these disparities requires balanced dataset construction, fairness-aware model development, and mandatory subgroup evaluation prior to clinical deployment.

 

Citations

Landmark Dermatology AI Paper

Citation:

Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542(7639):115–118.

DDI Dataset / Bias Landmark Paper

Citation:

Daneshjou R, Vodrahalli K, Novoa RA, et al. Disparities in dermatology AI performance on a diverse, curated clinical image set. Science Advances. 2022;8(32):eabq6147.

Original DDI Dataset

DDI Preprint / Fairness Assessment