Establishment of dermatopathology image encyclopedia DermpathNet using artificial intelligence empowered workflow

Summary: Abstract Body: Accessing high-quality open-access dermatopathology image databases for learning and cross-referencing poses a common challenge for clinicians and dermatopathology trainees. While existing platforms like VisualDx offer collections, they often require subscriptions which limit accessibility. Here, we employed an artificial intelligence (AI)-enabled workflow to curate and categorize images from the PubMed Central (PMC) database, covering 166 benign and malignant cutaneous neoplasms. We aimed to establish a comprehensive dermatopathology database for educational, cross-referencing, and machine-learning purposes. Our workflow involved retrieving full-length articles from PMC using specific keywords, extracting relevant images, and classifying them using a novel hybrid method. This approach combined deep learning-based image modality classification with figure caption analyses. Validation on 651 manually annotated images demonstrated the robustness of our workflow, with precision rates of 87.02% for the deep learning approach, 84.44% for the keyword-based retrieval method, and 92.64% for the hybrid approach. To enhance accessibility, we developed a new website DermpathNet featuring a fully annotated image database of over 7,000 images. The website organizes images based on their ontological relations and offers a user-friendly search bar for rapid retrieval of specific diagnoses. Finally, we conducted an open-ended challenge study to assess the performance of AI algorithms, including GPT-4v, on the retrieved image dataset. Our analysis revealed limitations in current AI image analysis algorithms, with a zero F1-score in the open-ended setting. Additionally, existing AI algorithms may rely on non-image features to arrive at inaccurate diagnoses. These findings underscore the current challenges in AI-assisted image analysis and need for future development. Ziyang Xu¹, Mingquan Lin², Yiliang Zhou², Zihan Xu², Shane Meehan³, Seth Orlow¹, Alexandra Flamm¹, Ata S. Moshiri¹, Yifan Peng² 1. Dermatology, NYU Langone Health, New York, NY, United States. 2. Department of Population Health, Weill Cornell Medicine, New York, NY, United States. 3. Dermatology, Mount Sinai Health System, New York, NY, United States. Bioinformatics, Computational Biology, and Imaging