Inter-rater Reliability Severity Grading of Digital Images for Radiation Dermatitis: Secondary Analysis of a Prospective Pilot Trial

Summary: With the expansion of telehealth, there is an increased demand for virtual evaluations of skin conditions or dermatitis, which has an important utility for cancer patients undergoing radiotherapy. Radiation dermatitis is the most common acute adverse effect of radiation treatments. In this study, we evaluated the robustness of severity grading of digital images from a prospective pilot trial of head and neck (HN) cancer patients. The study results will guide the development of a future trial assessing inter-rater agreement of severity grading of photographs taken by patients on their mobile device through the MyCap mobile application. Between 2020 and 2022, 24 patients diagnosed with HN cancer were enrolled in an open-label randomized trial assessing a keratin-based topical cream for radiation dermatitis. All patients were treated with a minimum radiation dose of 60 Gy. Dermatitis assessments and digital images of treated HN areas were obtained using a digital camera in the clinic room under ambient lighting by clinical trial nurses. Images were collected at the following timepoints: prior to treatment, at weekly on-treatment visits, and 1-month post-radiation therapy completion. A total of 398 images were collected and used for severity grading of radiation dermatitis. Eight providers in the radiation oncology department at a single academic center independently reviewed the images and in random order. The clinicians were blinded to the timepoint the image was taken during the treatment course. All grading was performed according to the CTCAE 5.0. Graders had the option of selecting “unable to determine” to omit grading of an image. Inter-rater reliability was assessed with Kendall’s coefficient of concordance (Kendall’s W). Subsequent sub-analyses to assess inter-rater reliability by week of treatment were completed. The Kendall’s W between all eight raters is 0.622 (p-value < 0.0001). After stratification by week of treatment (Table 1), inter-rater reliability was worst when assessing pre- and post-op digital images, Kendall’s W = 0.401 and 0.383 respectively. The inter-rater reliability trend showed improvement with each week of radiation treatment. Overall, there was moderate agreement and inter-rater reliability with severity grading of radiation dermatitis utilizing digital images. We suspect the lower agreement in pre-op and post-op periods is due to skin changes described in the CTCAE that are difficult to discern without an image of a patient’s baseline available, such as xerosis. This study was limited due to the lack of standardized protocols for digital image collection. In summary, digital images are a reliable method for evaluating radiation dermatitis based on CTCAE criteria. Further studies are needed to establish standardized protocols to utilize digital images for radiation dermatitis assessments to improve inter-rater reliability. Nga T. Nguyen, MD, MPH (Presenting Author) - Atrium Health Wake Forest Baptist; Ariel Choi, MD (Co-Author) - Atrium Health Wake Forest Baptist; Sarah Glynn, DO (Co-Author) - Atrium Health Wake Forest Baptist; Amanda Goetz, MD (Co-Author) - Atrium Health Wake Forest Baptist; Claire Lanier, MD (Co-Author) - Atrium Health Wake Forest Baptist; John Petersen, DO (Co-Author) - Atrium Health Wake Forest Baptist; Patrick Young, DO (Co-Author) - Atrium Health Wake Forest Baptist; Bart Frizzell, MD (Co-Author) - Atrium Health Wake Forest Baptist; Luke Burnett, PhD (Co-Author) - Keranetics; Karen Winkfield, MD, PhD (Co-Author) - Vanderbilt University Medical Center; Ryan Hughes, MD (Co-Author) - Atrium Health Wake Forest Baptist