Patients

We investigated a random sample of 55 general practitioner-referred patients evaluated in SUs across three general hospitals, selected from the prospectively collected consecutive Haaglanden SU database (April 2015–December 2022), the same cohort as in our previous study [4]. This database comprises patients with documented T-LOC who were referred for evaluation and were assessed by the same syncope-expert in a dedicated SU following a standardized protocol (as described in the following section).

SU evaluation

The study protocol was approved by the Medical Ethics Review Committee Leiden/The Hague and Delft (METC nr: 18‑061). All participants provided written informed consent. GPT-5 was evaluated under controlled conditions at the Amsterdam University Medical Center, ensuring patient safety and data privacy. Ethical approval for the use of GPT-5 with anonymized data was obtained, including General Data Protection Regulation compliance and provisions for human oversight (METC, Amsterdam, nr: 2024.0124).

All patients were first evaluated in the SU by the same syncope-expert, using a standardized diagnostic work-up protocol (see Figure 1). The initial diagnostic assessment was performed in accordance with guideline-based syncope evaluation and is hereafter referred to as core evaluation (CE). CE consisted of thorough history-taking, physical examination, an active standing test, and a 12-lead ECG. Extended evaluation (EE) comprised CE plus transthoracic echocardiography (TTE) and exercise ECG (XECG) whenever feasible, with additional investigations (e.g., Holter monitoring and head-up tilt testing [HUTT]) performed only when clinically indicated, at the discretion of the syncope-expert (Figure 1). Findings from the evaluation were recorded in a fixed, structured format across all cases to minimise interpretative variation. The clinical report was limited to a structured summary of observed clinical and test data; no ‘impression/assessment’ or diagnostic interpretation was provided to GPT-5. In cases of diagnostic uncertainty, multidisciplinary consultation was readily available.

Display full size

Figure 1. Diagnostic flowchart. Phase-1: Core evaluation (CE) in the syncope unit by the syncope-expert and GPT-5. Phase-2: Follow-up of 1.5 years. Phase-3: Diagnostic adjudication by an expert committee after 1.5 years of follow-up, serving as the reference standard diagnosis, resulting in the final diagnosis. In addition to adjudication, diagnostic precision and diagnostic safety were also assessed.

Follow-up

The syncope-expert and GPT-5 made a diagnosis after the CE and after the EE, using identical information. Thereafter, we followed the patients systematically for eighteen months and gathered information regarding T-LOC episodes and diagnostic changes. Structured questionnaires captured all recurrent syncope, further diagnostic testing, hospital admissions, and any change in diagnosis following the SU visit (Figure 1). Follow-up consult was performed at six and eighteen months. When appropriate, data were verified by additional phone consultation.

After follow-up, an expert panel of three physicians (FJdL, JSYdJ, TTB) with expertise in syncope reviewed all cases using all available data, without reference to the prior diagnostic conclusion. GPT-5 was evaluated in a non-interactive, case-based format and could not ask clarifying questions, request additional information, or interact with patients or clinicians. Both the final adjudication panel and GPT-5 were blinded to the diagnostic labels under comparison: the adjudication panel did not receive the initial diagnosis of the syncope-expert or GPT-5 diagnoses, and GPT-5 did not receive the syncope-expert diagnosis or the final adjudicated diagnosis. They reached a ‘final diagnosis’ following the same protocol as in prior studies [4, 22]. Cases in which no etiological final diagnosis could be supported after complete follow-up were classified as ‘unexplained T-LOC’, consistent with guideline-based syncope practice. The final diagnosis was employed as the reference standard to evaluate diagnostic precision and safety, as outlined in the following section (Figure 1). Separate clinical reports were drafted following the CE and EE, each encompassing the data corresponding to that particular assessment. An anonymized example of the structured CE/EE report template is provided in Supplementary material 1.

Risk-stratification

Risk features were defined and categorized using the 2018 version of the ESC syncope guidelines [1]. Specifically, low-risk, high-minor, and high-major risk features were evaluated across history, physical examination, and ECG. Minor high-risk features were subsequently interpreted according to the conditional ESC criteria: minor high-risk features related to the syncopal event were classified as clinically high-risk only when accompanied by structural heart disease and/or an abnormal ECG, whereas minor high-risk electrocardiographic features were classified as clinically high-risk only when the history was consistent with arrhythmic syncope (see Table 1) [23]. For each SU letter, we assessed whether these features were explicitly documented. This assessment was performed to evaluate the completeness of reporting; letters were not excluded on the grounds of missing information, and no information was added.

Instructions to LLM

LLMs such as GPT-5 are a specific type of AI based on transformer architectures, trained on massive text corpora to predict the next word (token) in a sequence; by repeating this prediction step, they can generate, summarize, and reformulate clinical text and answer free-text questions in a probabilistic, pattern-based manner [17–19].

To facilitate this analysis, a single pre-specified structured prompt was applied to all cases. It was developed based on the 2018 ESC syncope guideline and its web-based Practical Instructions [1, 24]. Prompt design followed the RICCE framework (Role, Instructions, Context, Constraints, and Examples), as previously applied in syncope-related GPT-4o evaluation, with the aim of improving standardization and reproducibility of model input [4]. GPT-5 was allowed to indicate diagnostic uncertainty and to state when the available information was insufficient to support a clear diagnosis. The prompt had been developed and tested previously on separate cases not included in the present study and was fixed before analysis. The same prompt was then applied across all study cases and across all five runs per case [10, 25, 26]. No additional model training or fine-tuning was performed prior to this analysis. GPT-5 was provided with the clinical report after the CE and after EE. Building on our prior study with GPT-4o using SU referral letters, we applied the same evaluation framework to the newer GPT-5 model, expected to perform better with richer, guideline-concordant SU data (CE and EE).

Analysis

We used several methods to assess the diagnostic performance of GPT-5 compared to the syncope-expert. In accordance with the Haaglanden database, the syncope-expert documented a single diagnosis for each patient rather than a differential diagnosis comprising multiple aetiologies. Outcomes were diagnostic yield, the diagnostic inclusion rate, and for GPT-5 the diagnostic precision score (DPS).

Diagnostic yield was defined as the proportion of cases in which any etiological diagnosis (reflex syncope, orthostatic hypotension (OH), cardiac syncope, or other specified cause) or ‘unexplained T-LOC’ was assigned by the syncope-expert or GPT-5 after CE and after EE. Yield therefore reflects etiological classification rather than direct physiological observation during the event.

The diagnostic inclusion rate quantified whether the final diagnosis appeared anywhere within GPT-5’s differential diagnosis list (coverage), whereas the DPS (see next section) quantified the conciseness of that list by rewarding correct inclusion and penalizing over-inclusive differentials (precision).

The DPS was calculated using the methodology previously described [4]. In short: DPS rewards inclusion of the correct final diagnosis (+1 point) while penalizing over-inclusive differentials through a fixed deduction per incorrect diagnosis. For this we calculated a penalty score to determine the DPS for GPT-5 after both the CE and EE. For example, if GPT-5 provided the largest possible list of six differential diagnoses, including the final diagnosis and five incorrect ones, the penalty score for each incorrect diagnosis would be 1 ÷ 5 = 0.20. The DPS for that case would then be: +1 − (5 × 0.20) = 0. If the final diagnosis was not included at all, no point was awarded, but the penalty score is still applied to all diagnoses in the list, resulting in a negative DPS of (5 × 0.20) = –1.00. This method ensures that concise, accurate diagnostic reasoning was rewarded, while excessive or unfocused differential lists were penalized accordingly (see also Supplementary Table S1).

Diagnostic safety

To assess diagnostic safety, we analyzed a subgroup of cases with a final diagnosis of cardiac syncope. Diagnostic safety was defined as how often the first diagnosis of cardiac syncope (the most alarming cause of T-LOC) was correctly identified by the syncope-expert and GPT-5 (CE and EE), compared to the final diagnosis. Secondly, all cases with a final cardiac diagnosis were traced back to their initial diagnostic classification to quantify how often cardiac syncope was missed at the first diagnostic assessment of the syncope-expert and GPT-5.

Within-case consistency of GPT-5

Per case, a single GPT-5 analysis was initially planned and executed. In one case, an additional analysis was performed inadvertently. The discrepant result prompted a protocol amendment. Five independent GPT-5 analyses (with the same prompt) per case were generated for all cases, and for both the CE and EE phases using identical case input, the same prompt template, and the same model parameters as in the primary analysis [4].

Within-case diagnostic consistency across the five runs was assessed by counting, for each run, how often the first diagnosis of the differential list was replicated. For discordant cases (≥ 2 different first diagnoses), specific diagnosis combinations were enumerated (e.g., OH with reflex, cardiac with reflex, OH with cardiac, etc.) to determine in which diagnostic domains the model showed inconsistency or mismatch. Also, we recorded the proportion of cardiac diagnoses besides reflex and OH that appeared at least once in the differential diagnosis.

Outcomes for comparison with the final diagnosis were summarized at two levels.

• 1.

  Case level: For each case, the number of runs was counted in which the first diagnosis of the differential diagnosis matched the final diagnosis (0/5–5/5).

• 2.

  Group level: All individual runs were pooled per phase (CE and EE, 54 cases × 5 analyses = 270 runs per phase), and the proportion of runs where the first differential diagnosis matched the final diagnosis was calculated.

Determinants and diagnostic inconsistency

We evaluated the determinants and their association with GPT-5’s diagnostic inconsistency for CE and EE, including demographics; characteristics of the syncopal event before, during, and after the episode; vital signs; results of additional diagnostic tests; risk scores; and features of the syncope referral letter, as detailed in Supplementary Table S2.

Statistical analysis

Continuous variables are presented as means with standard deviations (SD) or medians with IQR, depending on the distribution of the data. Categorical variables are presented as counts and percentages. Diagnostic performance is reported as a percentage of total cases, representing the proportion of correct classifications.

The diagnostic correctness of GPT-5 and the syncope-expert (each assessed against the final diagnosis) a paired exact McNemar test was used; results are reported as a matched-pairs odds ratio (OR) with 95% confidence interval (CI), separately for CE and EE.

A Sankey plot was generated to visually represent the flow of data and the distribution of diagnostic outcomes between the syncope-expert and GPT-5, highlighting the transitions and associations across the various stages of the study.

Univariable logistic regression was performed to evaluate the association between each candidate factor and the AI outcome; results are presented as OR with 95% CIs and P-values.

Risk-stratification

All 55 clinical reports after CE by the syncope-expert contained documentation of the syncopal event, physical examination, past medical history, and ECG findings. Across these domains, risk features were recorded per patient and event (Table 1) [23].

• Syncopal event: 76 low-risk; 62 high-risk (48 minor, 14 major).

• Physical examination: 45 low-risk; 10 high-risk.

• Past medical history: 60 low-risk; 6 high-risk.

• ECG findings: 43 low-risk; 13 high-risk (3 minor, 10 major).

Of the 48 minor high-risk features related to the syncopal event, thirteen met clinically relevant high-risk criteria (structural heart disease (n = 5) and high-risk ECG abnormalities (n = 8)). No cases fulfilled the minor ECG criteria of recurrent falls presumed to be arrhythmogenic in origin [23].

Diagnostic phases

Diagnostic yield

The diagnostic yield (i.e., the production of any diagnosis including “unexplained T-LOC”) of the syncope-expert was 94% for both the CE and EE assessments (Figure 2). The diagnostic yield of GPT-5 was 100% for the CE and 96% for the EE assessments (Figure 2). Diagnostic yield was prespecified as the production of any diagnostic label and was analyzed separately from final-diagnosis inclusion and the DPS, which are reported below.

Display full size

Figure 2. Diagnostic yield, inclusion rate and diagnostic precision score. The colored and yellow bars together represent the diagnostic yield of syncope-expert and GPT-5. The purple bars show the inclusion rate for syncope-expert, and the blue bars show the inclusion rate for GPT-5. The inclusion rate is defined as the proportion of final diagnosis that was included in the list of the differential diagnosis of GPT-5. The yellow bar: the proportion of diagnosis that did not included the final diagnosis. The red bar shows the diagnostic precision score: (%). CE: core evaluation; EE: extended evaluation.

Diagnostic inclusion rate (syncope-expert and GPT-5)

One case was lost to follow-up, leaving 54 cases for the analysis of the diagnostic inclusion rate and the DPS. The syncope-expert included the final diagnosis in 67% (n = 36) of cases in both the CE and EE protocols. GPT-5 included the final diagnosis in 52% (n = 28) of CE cases and 57% (n = 31) of EE cases (Figure 2). No significant difference between the syncope-expert and GPT-5 was observed in CE (exact McNemar p = 0.057) or EE (p = 0.302). In CE (n = 54), the syncope-expert alone included the final diagnosis in 11 cases, whereas GPT-5 was correct in three cases (matched-pairs OR 0.27, 95% CI 0.05–1.03); in EE (n = 54) these were 10 vs 5 (OR 0.50, 95% CI 0.13–1.61).

DPS (GPT-5)

The maximum number of diagnoses provided by GPT-5 was six. Consequently, the penalty-score for each incorrect diagnosis in the list of differential diagnoses was 0.20 (see for explanation also in Materials and methods section and Supplementary Table S1). Using this methodology, we calculated the DPS for every case. The cumulative DPS in the CE phase was −2% (mean −0.03, SD 0.54). After EE information, cumulative DPS was −1% (mean −0.01, SD 0.49) (see Figure 2).

Diagnostic safety (syncope-expert and GPT-5)

To quantify diagnostic safety, GPT-5 assigned a cardiac diagnosis in 17% (n = 9) of 54 cases after CE. This matched the final adjudicated diagnosis in 1 of 9 cases, corresponding to a positive predictive value of 11%. The remaining cases had a final diagnosis of unexplained T-LOC (44%, n = 4), OH (22%, n = 2), and reflex syncope (22%, n = 2) (Figure 3). After EE GPT-5 assigned a cardiac diagnosis in 9% (n = 5) of 54 cases. In 20% (1/5), this matched the final diagnosis, corresponding to a positive predictive value of 20%. The remaining cases had a final diagnosis of unexplained T-LOC (60%, n = 3) and OH (20%, n = 1) (Figure 3).

Display full size

Figure 3. Diagnostic flowchart of the diagnosis from the syncope-expert compared to the final diagnosis. Diagnostic flowchart comparing the diagnostic classification of the syncope-expert with the final diagnosis. The left panel shows the diagnosis after the CE evaluation based on only history-taking, physical examination, ECG, and active standing test. The middle panel presents the EE diagnosis based on CE with additional testing (echocardiography, exercise stress testing, Holter monitoring, and tilt-table test). The right panel shows the final diagnosis. Lines indicate the diagnostic concordance or shift between diagnoses. FND: functional neurological disorder; CE: core evaluation: EE: extended evaluation.

The diagnoses established by the syncope-expert during the CE phase remained unchanged after the subsequent EE evaluation in all cases. A cardiac cause was assigned in 20% of cases (11/54). In 27% (3/11), this diagnosis matched the final diagnosis. In the remaining cases, the syncope-expert either withheld a diagnosis (46%, 5/11), misclassified the case as OH (18%, 2/11), or as reflex syncope (9%, 1/11) (Figure 4).

Display full size

Figure 4. Diagnostic flowchart of GPT-5-based classification compared to the final diagnosis. Diagnostic flowchart comparing the GPT-5-based classification with the final diagnosis. The left panel displays GPT-5’s diagnostic classification based on information of the clinical report of CE: containing only the guideline-based initial evaluation (i.e., history-taking, physical examination, ECG, and active standing test). The middle panel shows GPT-5’s diagnostic classification based on information of the clinical report containing the guideline-based CE + additional testing: EE: Additional diagnostic tests included: echocardiography, exercise stress testing, Holter monitoring, and tilt-table testing. The right panel shows the final diagnosis. Lines indicate the diagnostic concordance or shift between diagnosis. FND: functional neurological disorder; CE: core evaluation: EE: extended evaluation.

Secondly, we examined how cases that were ultimately adjudicated as cardiac syncope were initially classified. Four of 54 patients had a final diagnosis of cardiac syncope. Of these four cases, GPT-5 classified 75% (3/4) as reflex syncope and 25% (1/4) as cardiac in both the CE and EE (Figure 3). The syncope-expert diagnosed 75% (3/4) of cases as cardiac syncope, and 25% (1/4) as OH (Figure 4) after both the CE and EE.

Diagnostic consistency

Across five GPT-5 runs per case on CE data, a single consistent diagnosis was provided in 69% (n = 37) of cases, while 30% (n = 16) yielded two different diagnoses, and 2% (n = 1) yielded three. In these 17 cases with discordant diagnoses among the different runs the different combinations of first diagnoses in the list were paired for each case: OH + reflex (47%, n = 8), cardiac + reflex (24%, n = 4), OH + cardiac (12%, n = 2), functional neurological disorder + reflex (6%, n = 1), traumatic + cardiac (6%, n = 1), and cardiac + OH + reflex (6%, n = 1). In total, a cardiac diagnosis was part of the differential diagnosis (after 5 runs) at least once in eight of the seventeen (47%) cases where GPT-5 was inconsistent, see Figure 5A.

Display full size

Figure 5. Diagnostic consistency. Within-case diagnostic consistency of GPT-5 across five independent runs per case in patients for CE (A) and for EE (B). In each panel, the stacked bar on the left shows the proportion of cases with 1, 2, or > 2 distinct diagnoses across the five runs, while dashed connectors from the inconsistent segments link to the corresponding diagnostic combinations on the right. All values are displayed as percentages with case counts. CE: core evaluation; EE: extended evaluation.

Across five GPT-5 runs on EE data (n = 54), a single consistent diagnosis was provided in 74% (n = 40), while 24% (n = 13) yielded two different diagnoses, and 2% (n = 1) yielded three. The combination of proposed diagnoses in these fourteen discordant cases was: OH + reflex (64%, n = 9), cardiac + reflex (21%, n = 3), OH + cardiac (7%, n = 1), and cardiac + reflex + unexplained T-LOC (7%, n = 1). In these 14 inconsistent cases, a cardiac diagnosis appeared at least once in the differential diagnosis during the five runs for five patients (36%), see Figure 5B.

Diagnostic consistency and agreement with the final diagnosis were summarized at two levels:

As shown in Figure 6, cases were categorized according to the number of GPT-5 runs (0–5/5) in which the final diagnosis was ranked first in the differential list.

Display full size

Figure 6. Reproducibility of GPT-5 diagnoses across multiple runs. Distribution of cases according to the number of five independent GPT-5 runs (all runs with identical clinical reports) in which the final diagnosis was ranked first in the differential diagnosis list. Bars illustrate whether the final diagnosis was never, in some, or in all runs ranked first, separately for the initial ESC guideline–based evaluation without additional testing (CE) and for the CE including additional diagnostic testing (EE). CE: core evaluation; EE: extended evaluation.

Analysis of the CE phase revealed that the final diagnosis was never ranked first across the five runs in 35% of cases (n = 19), whereas it achieved consistent first-rank status in 41% of cases (n = 22). In the remaining 24% (n = 13) of cases, the final diagnosis was ranked first in all 1/5, 2/5, and 3/5 runs in 7% (n = 4), and 4/5 runs in 2% (n = 1) of cases, respectively. When all 270 CE runs were pooled, the final diagnosis occupied the first position in 51% (n = 138) of runs (Figure 6).

In the EE group, the final diagnosis was never ranked first (0/5) in 39% (n = 21) of cases and was ranked first in all five runs (5/5) in 44% (n = 24) of cases. In the remaining 17% (n = 9) of cases, the final diagnosis was ranked first in 1/5, 2/5, 3/5, and 4/5 runs in 4% (n = 2), 2% (n = 1), 7% (n = 4), and 4% (n = 2) of cases, respectively. Overall, across 270 EE runs, the final diagnosis was ranked first in 53% (n = 144) of runs (Figure 6).

Determinants of GPT-5 diagnostic inconsistency

In univariable logistic regression analyses, in the CE data: total peri-procedural factors (OR 3.24, 95% CI 1.20–8.69; P = 0.020) and fatigue (OR 11.20, 95% CI 1.28–97.71; P = 0.029) were significantly associated with GPT-5 inconsistency, while medical history (including risk-factors and medication) and total word count in the SU history showed a trend towards an association with GPT-5 inconsistency (OR 1.009, 95% CI 1.000–1.018; P = 0.059 and OR 1.008, 95% CI 1.000–1.015; P = 0.054, respectively) (Supplementary Table S2).

EE: pain after syncope (OR 0.15, 95% CI 0.03–0.82; P = 0.029) and injury related to syncope (OR 0.13, 95% CI 0.03–0.72; P = 0.019) were inversely associated with GPT-5 inconsistency, while total post-procedural factors and total peri-procedural factors showed trends towards inverse and positive associations with GPT-5 inconsistency, respectively (OR 0.78, 95% CI 0.61–1.00; P = 0.050 and OR 2.51, 95% CI 0.97–6.49; P = 0.058) Supplementary Table S2).

Diagnostic performance

Diagnostic yield (defined as the production of any diagnosis, including “unexplained T-LOC”) was reported as a measure of diagnostic completeness rather than correctness. GPT-5 yield was 100% after CE, and 96% after EE, versus 94% for the syncope-expert in both phases. Conversely, GPT-5 included the final diagnosis in 52% (CE) and 57% (EE) versus 67% for the syncope-expert in both phases. Outcomes are similar to previous research, where GPT achieved 54% diagnostic accuracy (here: diagnostic inclusion rate), versus 75% for specialists [27]. This comparison is contextual only, because GPT-4o was not re-tested on the present dataset under identical conditions. Therefore, the present study cannot determine whether GPT-5 provides incremental diagnostic improvement over GPT-4o. There was no statistically significant difference between GPT-5 and the syncope-expert, despite the syncope-expert demonstrating higher diagnostic precision than GPT-5.

Additional cardiac function tests syncope-expert

Among the 55 patients, additional testing (mainly TTE and exercise ECG) was frequent, but never changed the syncope-expert’s initial diagnosis. Prior work from de Jong et al. [11] showed that 93% of final diagnoses were already established after the guideline initial evaluation. Additional testing never changed these diagnoses, underscoring that a diagnosis is predominantly determined by first-line clinical information rather than routine downstream testing [13]. In the present study, the syncope-expert’s core diagnosis remained unchanged after TTE and exercise ECG, whereas GPT-5’s performance showed modest improvement only after access to additional clinical information. This is consistent with the relatively favorable cardiac health profile of the cohort (Table 2), implying a low pre-test probability for clinically relevant structural heart disease. Consequently, routine TTE and exercise ECG provided no incremental diagnostic yield beyond the CE, emphasizing that additional investigations should be reserved for clearly indicated scenarios rather than performed to “complete” the diagnostic work-up. This is in accordance with the ESC syncope guidelines [1], supports indication driven testing [28–32]. We conclude that thorough history-taking, physical examination, active standing test and ECG remain the cornerstone of T-LOC assessment, as also emphasized in the ESC syncope guidelines [1].

Risk-stratification and diagnostic safety

Across all 55 cases evaluated in the SU, documentation of the core domains was complete and standardized (event history, physical examination including vital signs and active standing test, relevant medical history/medication, and ECG), such that every patient met the minimum prerequisites for a valid and reproducible risk-assessment [1, 2, 13, 22, 24, 33–35]. This contrasts with our earlier work in which referrals from primary care frequently lacked essential baseline information, highlighting substantial scope for improvement in primary care through a systematic CE and structured reporting. For secondary care, these results emphasize the need to further standardize SU practice (uniform intake proforma and explicit ECG interpretation), to transparently link risk-features to management decisions (admission vs outpatient care and targeted downstream testing), and to provide consistent feedback to primary caretakers to improve the quality of future referrals and preserve SU capacity for patients in whom specialist assessment offers true incremental value [12, 36–38].

Despite all data being available, safety diverged on cardiac syncope diagnosis detection: GPT-5 diagnosed a cardiac cause in only (1/4), whereas the syncope-expert scored (3/4). GPT-5 misclassifications skewed to the more benign reflex or OH, risking false reassurance. Additional EE data did not improve discrimination, indicating that limitations in reasoning rather than lack of appropriate data caused this pattern [4]. The syncope-expert’s willingness to defer a diagnosis in the case of clinical uncertainty likely functions as a safety buffer against early closure. Obviously, missing cardiac causes of T-LOC may induce risks of harm that potentially could have been prevented [39–41].

Effect of additional diagnostic testing on diagnostic performance of GPT-5

More comprehensive clinical data (thorough history-taking, active standing test, ECG and standard additional diagnostic tests after guideline-based CE) only modestly improved GPT-5’s diagnostic performance (52% to 57%), suggesting that complete data alone are insufficient for diagnostic evaluation, without mechanisms that embed pathophysiological priors or calibrated uncertainty. Likewise, targeted instruction by prompting GPT-5 with contemporary syncope guidelines [1] and web practical instruction [24] yielded only limited gains in diagnostic performance, underscoring again that access to codified knowledge alone does not replace internalized clinical reasoning [4].

DPS, interpretability and trustworthiness of LLM outputs

In contrast to the clearly positive DPS achieved by human clinicians in the same dataset [4], GPT-5 remained negative (near zero) in both phases, mainly due to penalties because of a large number of differential diagnoses [42]. We speculate that GPT-5’s differential diagnoses approximate an educated guess, with substantial probability retained on incorrect options. LLMs generate comprehensive differentials while their internal ranking/probability structure is not transparent and may be poorly calibrated [42–44]. This is consistent with previous research [4]. Although GPT-5 can generate plausible diagnostic explanations, these should be interpreted as probabilistic post-hoc narratives rather than as evidence of traceable causal or clinical pathophysiological reasoning. This distinction is clinically relevant, as prior vignette-based work with GPT-4 showed that access to an LLM did not necessarily improve physicians’ diagnostic reasoning, despite superior standalone model performance [20]. The explanatory depth required for clinical use is task dependent. For screening, triage, or documentation-support tasks, transparent feature ranking or identification of missing information may be sufficient. In contrast, diagnostic classification of T-LOC and treatment-directed decisions, particularly when cardiac syncope is possible, require causal and pathophysiological reasoning, calibrated uncertainty, and the ability to defer a diagnosis when the available evidence is insufficient. We argue that any clinical use of GPT-5 would require auditable evidence trails, calibrated confidence estimates, and an explicit capacity to abstain or defer a definite diagnosis in high-risk cases when multiple options are still open [45].

Implications for LLM integration in clinical pathways

From the above, it follows that to date autonomous GPT-5 use in syncope pathways is not yet justified. The speed, breadth, and documentation inside clinician-led, protocol-bound support by LLM models with guardrails (e.g., mandatory escalation on cardiac flags or low confidence) could be employed within clinical decision making. We suggest that the use of LLM models should be restricted to the support of human clinical decision making. An example of pragmatic use of LLMs’ could be a role to improve clinician diagnostic reasoning, triage or documentation augmentation with human oversight and verification for high-risk phenotypes [26, 46]. In addition, practical implementation depends not only on the availability of explanations, but also on their timing and cognitive burden. In time-sensitive clinical settings, explanations that are generated too late, or that require extensive clinician interpretation, may have limited practical value. Similarly, overly long or non-prioritized explanation reports may increase cognitive load and obscure the key diagnostic uncertainty or safety signal. Therefore, LLM-generated explanations should be concise, prioritized, and actionable, highlighting the information that changes risk assessment or management rather than providing exhaustive narrative justification.

Fivefold reanalysis

The fivefold reanalysis extends the primary single-output evaluation by assessing whether GPT-5 produced stable diagnostic conclusions when identical clinical information was presented repeatedly. Across five re-analyses per case, EE resulted in limited improvement: a fully correct first diagnosis was established in 41% after one run and increased to 44% after multiple runs. The presence of the final diagnosis anywhere in the five runs increased from 51% to 53%, insufficient for dependable, patient-level decisions. In the clinically important subgroup with final adjudicated cardiac syncope, GPT-5 selected the final diagnosis in 1 of 4 patients, whereas the syncope-expert did so in 3 of 4 patients. Given the very small number of final cardiac syncope cases, this observation should not be interpreted as a precise estimate of diagnostic sensitivity or safety. Conversely, the safety analysis also showed false-positive cardiac classification: after CE, only 1 of 9 GPT-5 cardiac classifications matched the final adjudicated diagnosis, and after EE, only 1 of 5 matched the final adjudicated diagnosis. Thus, the safety limitation was bidirectional, comprising both under-recognition of final cardiac syncope and over-classification of cardiac etiology in patients with non-cardiac final diagnoses. These findings reinforce the need for clinician oversight, particularly for potentially high-risk cardiac etiology, and do not support the use of GPT-5 as a stand-alone diagnostic tool in syncope evaluation.

Within-case reliability (fivefold reanalysis)

Only 69% of the assessments by GPT-5 in the CE phase and 74% in the EE phase were stable across five runs [47]. Between 16% and 32% of diagnoses varied despite identical inputs, putting serious limitations on single-run outputs. Discordant outputs clustered along hemodynamic phenotypes (orthostatic vs reflex), indicating that the boundaries used by the model are not strict and have limited pathophysiological basis.

The generation of multiple, internally inconsistent diagnosis by GPT-5 is inherently unsafe. Diagnosing the same event simultaneously as a benign (e.g., orthostatic or reflex) and as high-risk cardiac syncope invites anchoring to the reassuring explanation, diminishes the perceived urgency of the “cardiac” signal, and may lead to inappropriate de-escalation of monitoring and work-up.

The particular implication of the above is that the syncope-expert currently outperforms AI for safety-critical detection and governance. Therefore, it is crucial to maintain human oversight, use multi-run consensus, escalate on any cardiac “hit” (“one-vote-cardiac”), and abstain when unanimity is absent [45, 48]. Until sensitivity and stability improve, the present findings support the use of GPT-5 only as supervised decision support within syncope-expert-led pathways.

From univariable logistic regression analyses of predefined clinical determinants, GPT-5’s diagnostic behavior appeared to vary systematically with the clinical context of the case and with the amount of structured clinical information available at the time of assessment. The overall burden of documented determinants at presentation was significantly associated with the model’s diagnostic output, whereas determinants recorded after the event showed only a borderline association. Taken together, these findings suggest that GPT-5’s diagnostic output was more strongly influenced by syncopal episodes described with numerous or more severe symptoms, particularly when multiple clinical factors were present at the time of syncope. In other words, GPT-5 seems to add more value in complex, highly symptomatic presentations than in cases with only few or mild symptoms.

Temperature setting LLM

In this study, we verified that the model was operating with a low temperature setting, which minimizes stochastic variability and promotes reproducible outputs across repeated runs for the same case, and therefore did not adjust this parameter. In LLMs, the temperature controls randomness in token sampling: lower values constrain the model to high-probability continuations and yield more deterministic responses, whereas higher values increase variability of the output given the same input [49, 50].

Strengths and limitations

The principal strength of this analysis is the head-to-head comparison with a syncope-expert, GPT-5, and adjudication, across the predefined CE and EE phases. Because syncope mechanisms are rarely captured with simultaneous ECG and blood pressure at the time of T-LOC, we used a final diagnosis after structured follow-up as the best available reference standard, acknowledging that some cases will remain unexplained despite extensive evaluation. Although the final reference diagnosis was established by an independent expert panel using all available follow-up data, the study was performed in a single-center syncope-unit cohort and used one experienced syncope-expert as the clinical comparator. The adjudication process strengthens the internal validity of the reference standard and reduces the risk of confirmation bias, but it does not fully address external generalizability. Diagnostic performance may differ across centers, referral pathways, patient case mix, and clinician expertise. Therefore, external validation in larger multicenter cohorts, preferably including multiple independent syncope-experts, is required before these findings can be generalized to broader syncope-care settings. The small number of cardiac outcomes limits precision for safety-critical estimates; however, this mirrors real-world prevalence and probably preserves external validity. Second, we did not perform formal test–retest assessment of the syncope-expert index rater, precluding direct comparison of within-rater variability between the syncope-expert and GPT-5. Third, although a single GPT-5 run per case was prespecified, an inadvertent repeat run in one case yielded discrepant output, underscoring potential stochastic variability in LLM-based classification and the need for prespecified strategies (e.g., deterministic settings or multi-run consensus) in future evaluations. Accordingly, we performed fivefold re-analysis, demonstrating within-case variability that would not be detected in single-run designs. GPT-5 narrows the diagnostic performance gap with expert SU workflows on aggregate metrics yet underperforms where safety matters most, that is, the case of cardiac syncope. Diagnostic safety in this study was operationalized as identification of cardiac syncope; we did not assess potential harm from overdiagnosis or unnecessary management triggered by incorrect non-cardiac classifications. Moreover, GPT-5 could not ask for more information and depended solely on the report provided by the syncope-expert. Thereby, the full capabilities of AI may not have been evaluated. Future work should prioritize hybrid architectures that fuse LLMs’ outputs with pathophysiological inference, calibrated uncertainty and abstention, and prospectively validated escalation rules for cardiac red-flags. Larger, diverse cohorts and multi-run evaluation protocols with the use of diagnostic adjudication in order to test real life diagnostic performance or accuracy will be essential to quantify variance and to certify dependable behavior at patient level.

Conclusion

GPT-5 demonstrated a high diagnostic yield and diagnostic performance approaching that of the syncope-expert in parts of the structured diagnostic pathway. However, run-to-run inconsistency and the exploratory observation in the small cardiac syncope subgroup underscore the need for clinician oversight, particularly for safety-critical diagnoses. Structured, clinician-led reasoning remains essential in syncope evaluation, especially when cardiac or otherwise high-risk causes must be considered. In our cohort, the addition of TTE and exercise ECG data did not materially improve GPT-5 diagnostic refinement, supporting the continued primacy of guideline-based CE: careful history-taking, physical examination, active standing testing, and 12-lead ECG. Despite the rapid evolution of LLMs, their role in syncope care should remain supportive within clinician-led pathways rather than determinative, unless future systems demonstrate calibrated uncertainty, reliable abstention, stable multi-run consensus, and clinically grounded pathophysiological reasoning.