SkinGPT-4 provides a generalizable foundation for fair and customizable skin disease classification models

Summary: SkinGPT-4 is a large vision-language model used to interpret skin disease images. Preliminary evaluations on the open-sourced SCIN dataset have observed skin tone biases in SkinGPT-4 performance. Here, we leveraged the SkinGPT-4 backbone to develop finetuned models for custom skin disease classification tasks and explored bias mitigation strategies. Six-hundred cases with uniform skin tone representation were queried from the SCIN dataset containing common skin diseases, such as Tinea and Urticaria, for model training. Customized image classification models were designed by attaching a learnable multilayer perceptron (MLP) head to the SkinGPT-4 vision encoder. Hyperparameters such as learning rate, batch size, and MLP-depth were tuned, and model performance was evaluated across skin condition pairs with similar presentations. Performance metrics, such as AUROC and demographic parity, were measured. Across skin disease pairs with similar appearances, our models achieved an average F1, precision, and AUROC of 0.75, 0.78, and 0.78, respectively. The average demographic parity and the largest difference between distinct skin tones were observed as 0.75 and 0.21, respectively. In our best model, group-stratified demographic parity scores of 0.83, 0.83, 0.76, 0.89, 0.90, and 0.90 were achieved across skin tone categories in the Fitzpatrick scale from 1-6, respectively, indicating robust fairness. Similar performances were observed when adapting our base model to disease triplets, providing evidence of generalizability. This study demonstrated the efficacy of training accurate and fair machine learning models using SkinGPT-4 for custom skin disease classification tasks. Our study hopes to improve the clinical application of AI in dermatology, particularly regarding demographic biases. Benjamin Liu¹, Ryan Bui^{1, 6}, Peter Wang^{2, 1}, Adnan Ahmed^{3, 1}, Derek Jiu^{4, 1}, Kiran Nijjer¹, Kevin Zhu^{5, 1}, Lilly Zhu¹ 1. Stanford University, Stanford, CA, United States. 2. University of Waterloo, Waterloo, ON, Canada. 3. York University, Toronto, ON, Canada. 4. St John's School, Houston, TX, United States. 5. University of California Berkeley, Berkeley, CA, United States. 6. University of California Irvine, Irvine, CA, United States. Bioinformatics, Computational Biology, and Imaging