A total of 3,812 female participants were recruited for this research (≥ 18 years old). All participants attended the Breast Cancer Foundation (FUCAM A.C.) in Mexico City between June 2018 and April 2021 to receive screening mammography or complementary tests due to suspicion of BC.

Exclusion criteria in the trials included pregnancy, previous BC diagnosis, currently undergoing BC treatment, breast implants, partial or total mastectomy, breast surgery or surgery in the thorax area within the previous two years, and/or a breast biopsy in the previous three months. During the test, patients were excluded if: they did not receive mammography after the initial infrared test; they only received a breast ultrasound, tests reported them as breast imaging reporting and data system (BI-RADS) 3,4 or 5 without follow-up at FUCAM A.C. to confirm BC; they did not receive radiologic or histopathologic evaluations, or infrared tests were poorly taken.

This research was approved by the Research, Bioethics, and Biosafety Committee from FUCAM A.C. All participants signed informed consent.

Prior to the infrared test, each patient answered a questionnaire about their family history, symptoms, previous activities, and conditions that affect their body temperatures, such as drug and/or stimulant intake, and current menstrual cycle phase (Supplementary material). The infrared tests were performed before the mammography, ultrasound, and/or biopsy, following the guidance of the American Academy of Thermology [13], with the allowed adjustments (Figure 1). Three basal images were taken in the first image set: frontal, 30° to the right of midline, and 30° to the left. The second image set called the dynamic test, was performed to identify which areas retained a higher temperature after cooling the breast, armpit, and neck areas using disposable hypoallergenic wet tissues. Following 1 min of rest after cooling to allow the remaining liquid to evaporate, the second set of images was taken from the same positions as the first image set (Figure 2). For patients with recent mammography or ultrasound, the infrared test was performed between 24 h and 30 days after those tests.

The six infrared images and the clinical data made up a registry for each patient. Registries were stored on a software database made for this purpose. Infrared tests were performed by four nurses trained to use the infrared camera and software. The infrared camera used was a FLIR^® T650sc, with an infrared resolution of 640 pixels × 480 pixels, thermal sensitivity of < 20 mK (0.02°C), and accuracy of ± 1°C.

The research study was divided into two phases: (1) the development of the AI algorithms, and (2) clinical validation. In the second phase, a double-blind design was implemented, where the infrared test operator did not know the mammography’s results, and the mammography technicians and radiologists did not know the infrared test’s results.

To find the best AI model for these trials, several AI algorithms were developed and tested. The algorithms developed included architectures based on neural networks (multi-layered, convolutional, and deep learning), support vector machines (SVMs), and K-nearest neighbors. These AI algorithms were supplied with a subset of patient registries as well as the mammograms’ BI-RADS score (mammograph evaluations performed by certified radiologists from FUCAM A.C.). The number of patient registries needed to obtain an efficient AI algorithm that could distinguish between patients with or without malignancy was defined by theoretical saturation. The AI algorithms were designed to generate a binary classification: zero (0) for test results that were “non-suspicious” of BC and one (1) for test results “suspicious” of BC. The 6 categories of the BI-RADS [14] scale which was used to define the BC probability of each patient are shown in Table 1.

The subset of patient registries supplied to each AI algorithm was labeled as “without BC” if they belonged to patients with mammogram BI-RADS1,2, and “with BC” if they belonged to patients with mammogram BI-RADS 3,4, or 5 where BC was confirmed by biopsy. In order to obtain an AI algorithm with high BC sensitivity, the registries from patients with BI-RADS 3,4, or 5 that had a follow-up biopsy result of benign pathology, and BI-RADS 3 cases that were re-evaluated afterward as BI-RADS 2 (mammography false positives), were not supplied to the AI algorithms.

To evaluate and compare the efficiency of each AI algorithm and identify the best model, a subset of patient registries was kept aside from the development process described above. This “validation subset” was used to calculate the effectiveness metrics: sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), area under the curve (AUC), and F₁ score for each algorithm. For this research, only the algorithms that showed a sensitivity equal to or superior to 95% with an F₁ score superior to 0.75 were kept.

In the second phase, a new group of participants underwent infrared-AI and mammography screening at FUCAM A.C. between September 2020 and April 2021. The findings from the infrared-AI software were consigned into individual reports according to the classification of the American Academy of Thermology based on the scale proposed by Villa Marie and Marseille [15]; when the AI algorithm outputs a “non-suspicious” (0) test result, it was reported as a Th1 or Th2. When the AI algorithm outputs a test result “suspicious” of BC (1), it was reported as a Th3, Th4, or Th5.

The sample size for this phase was 337 patients; this was calculated using the equation on studies for diagnostic tests proposed by Pedraza and Raad [16], with α at 5% and β at 10%. The sensitivity for the mammography applied in the development phase is π₁ (sensitivity from the mammography applied in the development phase), and π₂ (sensitivity from the AI algorithm validation subset) is the sensitivity from the AI algorithm validation subset.

Descriptive statistics analysis was applied to clinical variables obtained from clinical files, radiological and histopathological reports, as well as patient questionnaires completed prior to the infrared tests. Chi [2] and Fishers’ exact tests were used to analyze the correlation between the infrared test results and the clinical variables. The statistical analyses were carried out using the software EPI Info™ 6.0 version [Centers for Disease Control and Prevention (CDC), Atlanta, USA] and web statistical calculator SISA (Daan Uitenbroek, Hilversum, Netherlands) [17].

For validation of the infrared-AI software, the algorithm’s effectiveness metrics were calculated and compared with the same scores calculated from the mammography evaluations and biopsy. A two-tailed P ≤ 0.05 was considered statistically significant.

AI algorithms developed in this phase were supplied with 2,699 patient registries, of which 2,609 were mammography BI-RADS 1,2 (patients without BC) and 90 were BI-RADS 3–5 with biopsy that confirmed BC. The 289 patient registries with BI-RADS 3–5 whose biopsy indicated benign pathology, or who were re-evaluated as BI-RADS 2 after the follow-up, were not supplied to the AI algorithms (see Figure 3). The AI algorithm with the best performance during this testing was a convolutional neural network (CNN) based on the residual net 50 (ResNet 50), with additional layers connected to an output artificial neuron (Figure 4).

The validation subset used for this algorithm was comprised of 72 patient registries, including 34 patients with BC and 38 patients without BC. The effectiveness metrics for the mammography used during phase 1 and the validation subset are shown in Table 3.

During phase 2, additional 277 registries were generated. Together with the patient registries used with the AI algorithm validation subset, this resulted in a total of 349 that were used to validate the clinical effectiveness of the infrared-AI software. Of these, 39 corresponded to patients with confirmed BC diagnoses. The effectiveness metrics of the infrared-AI software at detecting BC cases were compared against reference mammography of the same 349 patients (Table 4).

A statistical correlation was not found between the infrared-AI software test results and patient BMI (P = 0.697) or breast density (P = 0.081), nor with the presence of conditions that alters body temperature (P = 0.287). No other clinical variables showed a statistical correlation with the test results from the infrared-AI software (Table 5).

The use of AI in breast image classification has been tested in various research studies to classify mammograms [26] as well as infrared images. In 2018, Gogoi et al. [27] revised different AI algorithms for the analysis of breast infrared images, identifying a sensitivity superior to 80% in most of them, in comparison with a sensitivity of 98% in their AI method based on polynomial SVM. Zuluaga-Gomez et al. [28] developed a CNN that classified infrared breast images from a database of 57 patients; by using “data augmentation” and “hyperparameters tuning”, their AI algorithm obtained a sensitivity of 92% and an accuracy of 94%.

It is important to note that Zuluaga-Gomez et al. [28] and other works used publicly available databases for their research, mainly the Database for Mastology Research (DMR) database (287 patients) [29], and the Department of Biotechnology, Tripura University and Jadavpur University (DBT-TU-JU) database (100 patients) [30]. The present research study was made in a specialized medical center for breast pathologies (FUCAM A.C.) which allowed the research team to build a large proprietary database of infrared images (20,022 images from 3,337 patients) with respective clinical, radiological, and histopathological patient information. Under the scope of some recent review publication works [31, 32], this research is one of the largest combining AI and thermography.

Additionally, unlike many studies, this research was not limited to the evaluation of an AI algorithm but also carried out clinical validation of the software itself. In a similar clinical trial, Rassiwala et al. [33] performed infrared, clinical exploration, radiologic and histopathologic tests with 1,008 female patients. Their methodology for analyzing thermal images was based on the temperature gradient calculation, and those with test results of delta T (ΔT) > 2.5 were classified as abnormal. They reported a sensitivity of 97.6%, specificity of 99.17%, PPV of 83.67%, and NPV of 99.89%.

The low capacity of the infrared-AI software to differentiate between benign and malignant pathologies, with a specificity of 72%, could be due to the exclusion of 289 patients with benign pathology from the AI algorithm’s development. This decision was made due to the research’s principal goal, which was to increase the algorithm’s BC sensitivity. Future studies could investigate whether introducing benign pathology to the AI algorithm development increases specificity without reducing sensitivity. In addition, the CNN architecture used in the infrared-AI software increases its efficiency when more data is provided [34], therefore continuing to collect data could improve the clinical effectiveness of the infrared-AI software.

The authors acknowledge a few methodological limitations of this research. Firstly, the improbable 100% sensitivity and NPV for the used mammography are likely due to an inability to measure false negatives from FUCAM A.C.’s mammography. Similarly, the research’s scope did not include a third diagnosis test for instances where the infrared-AI software detected a suspicious case, but the mammography did not. A third diagnosis test was not implemented here because it was considered unethical to perform a biopsy on patients when the standard BC screening technique considered the patient not to be at risk of malignancy.

In the future, the infrared-AI software could be evaluated with a larger sample of patient registries and following a trial design that removes the methodological limitations described above. In addition, future work could evaluate the efficiency and cost-effectiveness of a BC screening strategy that integrates the infrared-AI software with mammography and ultrasound for females aged 40 years and above, and one that integrates the infrared-AI software with ultrasound for younger females.

In this work, an infrared-AI software developed to detect BC from infrared breast thermography images (infrared-AI software) showed high sensitivity (94.87%) and high NPV (99.12%), but lower specificity (72.26%) and PPV (30.08%) compared to mammography, the standard BC screening technique. The infrared-AI software is therefore proposed as a complementary tool for BC screening, but not for definitive diagnosis. Increasing the amount and range of patient data that the AI algorithm uses to build its predictive model, as well as the amount of patient data available for clinical validation, will improve its efficiency, accuracy, and potential value as a complementary BC screening tool.