Overcoming Challenges in Evaluating a Pramana Scanner for Cervical Cytology ThinPrep Specimens: Lessons Learned and Solutions Explored

Summary: Introduction: Evaluation of novel artificial intelligence (AI) algorithms for pathology requires high-quality scanned images. Compared with histology, the attributes of cytological specimens create unique scanning challenges. From past experience, investment in scanner validation minimizes downstream variables when evaluating AI algorithm performance; however, there is no standardized guidance. Mayo Clinic implemented the Pramana scanner to evaluate Techcyte's AI-assisted cervical ThinPrep screening tool. Here, we report our process and the challenges faced when establishing a scanner to evaluate an AI cytology algorithm. Materials and Methods: A total of 112 slides were collected from leftover clinical specimens, scanned in batches of 5-20, and examined by cytologists with experience in the development of AI algorithms. Close collaboration with Pramana scanner technicians was necessary to achieve diagnostic quality. After which, a ""Golden Set"" of 10 slides was selected for ongoing quality assurance (QA). QA slides were scanned, examined, and annotated by a cytopathologist for final scanner optimization prior to AI algorithm implementation. Results: With an initial failure rate of 10%, the most common issues were missing cells/fields, inability to locate cells, and focal plane errors. Qualitative attributes such as thick cellularity and double-thick slides required adjustments. The most significant technical issue involved specimens with scant cellularity, which required fiducial markings to force scan area. Slide-handling issues due to misaligned labels or unlabeled slides hindered slide gripper functionality. After adjustments, the failure rate improved to 2%. Conclusions: Scanner parameters optimized for histology are not useful for cytology. A separate cytology scanner profile that addresses color, brightness, contrast, focal plane, and scan area is necessary. Considerations, such as user roles, barcode compatibility, and local storage management, are essential for operational efficiency. Addressing scanner challenges in the implementation phase lays a solid foundation for exploring AI tools to enhance the diagnostic accuracy and workflow efficiency.