Original Research

Artificial Intelligence vs Medical Providers in the Dermoscopic Diagnosis of Melanoma

Author and Disclosure Information

Early diagnosis of melanoma drastically reduces morbidity and mortality; however, most skin lesions are not initially evaluated by dermatologists, and some patients may require a referral. This study sought to determine the performance of an artificial intelligence (AI) application in classifying lesions as benign or malignant to determine whether AI could assist in screening potential melanoma cases. One hundred dermoscopic images (80 benign nevi and 20 biopsy-verified malignant melanomas) were assessed by an AI application as well as 23 dermatologists, 7 family physicians, and 12 primary care mid-level providers. The AI’s high accuracy and positive predictive value (PPV) demonstrate that this AI application could be a reliable melanoma screening tool for providers.

Practice Points

  • Artificial intelligence (AI) has the potential to facilitate the diagnosis of pigmented lesions and expedite the management of malignant melanoma.
  • Further studies should be done to see if the high diagnostic accuracy of the AI application we studied translates to a decrease in unnecessary biopsies or expedited referral for pigmented lesions.
  • The large variability of formal dermoscopy training among board-certified dermatologists may contribute to the decreased ability to identify pigmented lesions with dermoscopic imaging compared to AI.


 

References

The incidence of skin cancer continues to increase, and it is by far the most common malignancy in the United States. Based on the sheer incidence and prevalence of skin cancer, early detection and treatment are critical. Looking at melanoma alone, the 5-year survival rate is greater than 99% when detected early but falls to 71% when the disease reaches the lymph nodes and 32% with metastasis to distant organs.1 Furthermore, a 2018 study found stage I melanoma patients who were treated 4 months after biopsy had a 41% increased risk of death compared with those treated within the first month.2 However, many patients are not seen by a dermatologist first for examination of suspicious skin lesions and instead are referred by a general practitioner or primary care mid-level provider. Therefore, many patients experience a longer time to diagnosis or treatment, which directly correlates with survival rate.

Dermoscopy is a noninvasive diagnostic tool for skin lesions, including melanoma. Using a handheld dermoscope (or dermatoscope), a transilluminating light source magnifies skin lesions and allows for the visualization of subsurface skin structures within the epidermis, dermoepidermal junction, and papillary dermis.3 Dermoscopy has been shown to improve a dermatologist’s accuracy in diagnosing malignant melanoma vs clinical evaluation with the unaided eye.4,5 More recently, dermoscopy has been digitized, allowing for the collection and documentation of case photographs. Dermoscopy also has expanded past the scope of dermatologists and has become increasingly useful in primary care.6 Among family physicians, dermoscopy also has been shown to have a higher sensitivity for melanoma detection compared to gross examination.7 Therefore, both the increased diagnostic performance of malignant melanoma using a dermoscope and the expanded use of dermoscopy in medical care validate the evaluation of an artificial intelligence (AI) algorithm in diagnosing malignant melanoma using dermoscopic images.

Triage (Triage Technologies Inc) is an AI application that uses a web interface and combines a pretrained convolutional neural network (CNN) with a reinforcement learning agent as a question-answering model. The CNN algorithm can classify 133 different skin diseases, 7 of which it is able to classify using dermoscopic images. This study sought to evaluate the performance of Triage’s dermoscopic classifier in identifying lesions as benign or malignant to determine whether AI could assist in the triage of skin cancer cases to shorten time to diagnosis.

Materials and Methods

The MClass-D test set from the International Skin Imaging Collaboration was assessed by both AI and practicing medical providers. The set was composed of 80 benign nevi and 20 biopsy-verified malignant melanomas. Board-certified US dermatologists (n=23), family physicians (n=7), and primary care mid-level providers (n=12)(ie, nurse practitioners, physician assistants) were asked to label the images as benign or malignant. The results from the medical providers were then compared to the performance of the AI application by looking at the sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV). Statistical significance was determined with a 1 sample t test run through RStudio (Posit Software, PBC), and P<.05 was considered significant.

Performance of the AI Application Compared With Practicing Medical Providers

Results

The AI application performed extremely well in differentiating between benign nevi and malignant melanomas, with a sensitivity of 80%, specificity of 95%, accuracy of 92%, PPV of 80%, and NPV of 95% (Table 1). When compared with practicing medical providers, the AI performed significantly better in almost all categories (P<.05)(Figure 1). With all medical providers combined, the AI had significantly higher accuracy, sensitivity, and specificity (P<.05). The accuracy of the individual medical providers ranged from 32% to 78%.

. Performance of artificial intelligence (AI)(Triage Technologies Inc) vs medical providers in differentiating benign nevi vs malignant melanoma.

FIGURE 1. Performance of artificial intelligence (AI)(Triage Technologies Inc) vs medical providers in differentiating benign nevi vs malignant melanoma.

Compared with dermatologists, the AI was significantly more specific and accurate and demonstrated a higher PPV and NPV (P<.05). There was no significant difference between the AI and dermatologists in sensitivity or labeling the true malignant lesions as malignant. The dermatologists who participated had been practicing from 1.5 years to 44 years, with an average of 16 years of dermatologic experience. There was no correlation between years practicing and performance in determining the malignancy of lesions. Of 14 dermatologists, dermoscopy was used daily by 10 and occasionally by 3, but only 6 dermatologists had any formal training. Dermatologists who used dermoscopy averaged 11 years of use.

The AI also performed significantly better than the primary care providers, including both family physicians and mid-level providers (P<.05). With the family physicians and mid-level provider scores combined, the AI showed a statistically significantly better performance in all categories examined, including sensitivity, specificity, accuracy, PPV, and NPV (P<.05). However, when compared with family physicians alone, the AI did not demonstrate a statistically significant difference in sensitivity.

Pages

Recommended Reading

Hair Repigmentation as a Melanoma Warning Sign
MDedge Dermatology
Symmetric Palmoplantar Papules With a Keratotic Border
MDedge Dermatology
Study highlights potential skin cancer risk of UV nail polish dryers
MDedge Dermatology
What happens to melanocytic nevi during laser hair removal?
MDedge Dermatology
Study suggests narrow excision margins safe in early melanoma resection
MDedge Dermatology
Pembrolizumab monotherapy effective for rare melanoma
MDedge Dermatology
USPSTF releases updated recommendations on skin cancer screening
MDedge Dermatology
Acute Onset of Vitiligolike Depigmentation After Nivolumab Therapy for Systemic Melanoma
MDedge Dermatology
‘Exciting’ results for cancer vaccine plus pembro in melanoma
MDedge Dermatology
Guidelines for assessing cancer risk may need updating
MDedge Dermatology