Retrospective Analysis and Prospective Validation of AI-Based Software for Intracranial Hemorrhage Detection in a High-Volume Trauma Center Scientific Reports

Retrospective peer review

The data set for the retrospective review included 2916 CT scans of the head with corresponding reports. 20 negative CT scans per report were reported by Aidoc. Ten of the 20 flags were false positives caused by artifacts (n=6), calcifications (n=2), cerebrum falx (n=1), or right sinus (n=1). One indicator was equivocal because the hyperdense focus could be related to a small contusion or hyperdense lesion. Nine were judged positive for ICH by the examining neuroradiologist; two were stealth reformats, with bleeding described in a separate report; three were haemorrhages consistent with the expected postoperative aspects. Of the other four scans positive for ICH, two had follow-up imaging where the hemorrhage had resolved, and two had no follow-up imaging. One of the misses was a small area of ​​subarachnoid hemorrhage and the other was a very small extra-axial hemorrhage (Fig. 2).

Two missed ICH detected by Aidoc. The key slices [right] and the saliency map generated by Aidoc [left] missed ICH identified by the audit; (A) right lower frontal lobe subarachnoid hemorrhage; and (B) a trace of extra-axial blood in the left parietal region.

With the exception of the reported scans which were stealth and postoperative images, the emergency department was contacted with the results and addenda were issued for all reports that missed an ICH. The emergency department contacted the two patients who did not have follow-up imaging. Both patients showed no long-term sequelae.

Prospective validation

For prospective validation, a total of 1446 cranial CT scans were analyzed by Aidoc for the presence of ICH. The reviewing neuroradiologist identified 212 of them as positive for ICH; a prevalence of 14.7%. The prevalence of ICH in the emergency cohort was 6.3% (56/884), inpatients 30.3% (141/456) and outpatients 13.4% (13/97 ).

Aidoc reported 220 as ICH-positives, of which 180 (81.81%) were true positives. Of the 1226 scans that were not flagged by Aidoc, 30 (2.5%) were false negatives.

The diagnostic accuracy of the software for all cases was as follows: sensitivity 85.7% (95% CI 80.3–90.2%); specificity 96.8% (95% CI 95.6–97.6%); PPV 81.8% (95% CI 76.8–86.0%), NPV 97.6% (95% CI 96.6–98.2%). Diagnostic accuracy stratified by location is shown in Table 1. Specificity and NPV were consistent between different locations. Sensitivity and PPV were highest in the inpatient cohort (90.1% and 93.4%), followed by emergency (82.1% and 68.8%), then outpatient (53.8%). % and 53.8%).

The false positive scans had one or more features misidentified as ICH. The majority of these features were normal or calcified cerebral falxes (9/40, 22.5%), followed by artifacts (7/40, 17.5%), postoperative dural thickening (6/40, 15 .0%), meningioma (4/40, 10%), normal or calcified choroid (2/40, 5.0%), vessels (2/40, 5.0%), other calcifications (2/40, 5.0%), developmental venous abnormality (1/40, 2.5%), cavernoma (1/40, 2.5%), encephalomalacia (1/40, 2.5%), hyperdense tumor (1/ 40, 2.5%), colloid cyst (1/40, 2.5%), dural venous sinus (1/40, 2.5%), C1-C2 pannus (1/40, 2.5%) , tentorium (1/40, 2.5%).

Nineteen of the 30 false negatives were subacute ICH and 11 were acute ICH. The majority were subdural hemorrhage (12/30, 40%), followed by subarachnoid hemorrhage (6/30, 20%), postoperative extra-axial hemorrhage (4/30, 13%), intraparenchymal hemorrhage (3/30, 10%), other extra-axial hemorrhages (3/30, 10%) and basal ganglia hemorrhages (2/30, 7%). The inpatient and outpatient cohorts had a higher rate of subacute failure (10/13 and 4/6, respectively). Six acute and five subacute ICH were missed in the emergency cohort.

Lead Time (TAT)

The turnaround time (TAT) for the post-implementation dataset described above was compared to a pre-implementation dataset, which included 1628 CT scans of the head; 1469 (90.2%) were negative for ICH and 159 (9.8%) were positive for ICH. Both datasets had a similar proportion of emergency, inpatient, and outpatient cases. The inpatient cohort had the highest incidence of ICH (18% before, 31% after), followed by outpatients (9% before, 13% after) and then emergency patients (5% before, 6% after) .

For all negative ICH scans, the mean pre-implementation TAT was 90.9 (SD 279.8) minutes and the mean post-implementation TAT was 133.2 (SD 442.9) minutes. For all ICH-positive scans, the mean pre-implementation TAT was 66.7 (SD 41.5) minutes and the post-implementation TAT was 80.0 (SD 54.25) minutes. The TAT stratified by emergency, inpatient, and outpatient is shown in Table 2.

There was a decrease in TAT for positive ICH scans in the emergency and outpatient cohorts of 3.7 min (-5.1%) and 9.9 min (-14.2%), respectively. There was an increase in TAT for ICH-positive scans in hospitalized patients of 22.6 min (35.6%). The difference in TAT for all cases and hospitalized cases was statistically significant (P= 0.017 and P= 0.003). The difference in TAT for the ED and outpatient cohorts was not statistically significant (P= 0.59 and P= 0.07).

Of 49 consulting radiologists and registrars, 26 responded to the survey. Three radiologists used Aidoc at 100% of their reporting time; three 75%; four 50%; seven 25% and nine 0%.