Research Frontiers | Large Language Models for Longitudinal Symptom Tracking Following Vagus Nerve Stimulator Implantation

Introduction

Following implantation of a vagus nerve stimulation (VNS) device, longitudinal changes in a patient's condition are generally documented in clinical notes. These notes may describe seizure activity, mood, or pain at individual visits. Their unstructured format, however, makes changes over several years difficult to compare systematically.

A study published in JAMIA Open in 2026 evaluated whether a large language model (LLM) could extract symptoms and severity ratings from clinical notes to construct patient-level longitudinal trajectories. Its primary focus was the performance of a clinical information extraction method. The resulting trajectories also provided a preliminary view of variation among patients.

Background

Limitations of clinical documentation

Clinical documentation must distinguish a symptom explicitly recorded as absent from a symptom that is not mentioned. The former provides evidence of an assessment at that visit; the latter provides no information about symptom status. Notes may also document an ongoing symptom without enough detail to grade its severity. Classifying all such entries as symptom absence would introduce error into longitudinal estimates of change.

Standardized scales provide consistent symptom assessments but are not necessarily completed at every routine visit. The investigators therefore examined whether symptoms could be extracted reliably from existing clinical notes and followed over time. This is a necessary methodological step before such records can support more detailed longitudinal analyses.

Methods

Symptom extraction and severity classification

The investigators assembled five datasets comprising 95 patients and 1,427 manually annotated clinical notes. Training data were drawn primarily from COVID-related records, with additional notes from patients receiving VNS or other surgical care. Data were partitioned at the patient level into training (58 patients), validation (8 patients), and test (29 patients) sets, preventing notes from the same patient from appearing in both training and testing. The test set comprised 23 VNS patients and 6 patients who underwent other surgery.

The analysis used a two-stage pipeline. The first stage identified 13 symptom categories and classified each symptom as present, explicitly denied, or not mentioned. For symptoms classified as present, the second stage assigned a severity level of mild, moderate, or severe. Symptoms documented without sufficient information for severity classification received a separate label. In the longitudinal analysis, symptoms that were not mentioned were treated as missing data and excluded from calculations of mean severity. The investigators compared models of different sizes with manual annotations and then organized preoperative and postoperative VNS records by calendar quarter relative to surgery.

Eligibility for the seizure severity trajectory analysis required at least five relevant notes before surgery and five after surgery. Clinically meaningful improvement was defined as a reduction of at least one severity category relative to the preoperative baseline, sustained across two consecutive postoperative quarters containing relevant notes. Quarters without relevant documentation could fall between those two observed quarters. Ten patients undergoing other major surgery without VNS were included for descriptive comparison; six of them were also included in the model test set. The groups were neither randomized nor matched. Specific VNS stimulation parameters were not reported.

Results

Symptom classification performance

For seizure identification in VNS test notes, the best-performing model achieved a weighted F1 score of 0.98 (95% confidence interval, 0.96 to 0.99). The corresponding score in test notes from other surgical patients was 0.91. The F1 score integrates precision and recall. Following task-specific training, overall weighted F1 increased from 0.75 to 0.92, while the proportion of outputs conforming to the prespecified format increased from 80% to 100%. Weighted F1 scores for severity classification across symptoms ranged from 0.76 to 0.96.

These aggregate results require qualification. Weighted F1 is influenced by the frequency of each class and may obscure missed cases of uncommon symptoms. Sensitivity for suicidal ideation in the largest model was 1.00 in VNS notes but 0.80 in notes from other surgical patients. Although the F1 score for self-harm classification reached 0.99, only 2 to 4 positive cases were documented in each group. This small number does not establish reliable detection of this clinically important risk.

Figure 3 presents symptom classification performance stratified by sex and race. Some subgroups included very few patients. Moreover, confidence intervals were estimated by resampling notes without accounting for correlation among repeated notes from the same patient. The subgroup analyses may identify differences for further evaluation, but they do not establish equivalent model performance across demographic groups.

Figure 1. Weighted average performance of fine-tuned LLaMA models of different sizes in symptom classification.

Figure 2. Weighted average performance of the fine-tuned LLaMA-3.3-70B model in symptom severity classification.

Figure 3. Symptom classification performance by race and sex across LLaMA models of different sizes.

Variation in longitudinal trajectories

Longitudinal trajectories were constructed for 23 VNS patients in the test set, but only 8 met the documentation threshold for seizure severity analysis. Four met the prespecified improvement criterion, 2 had greater severity than their preoperative baseline in the last quarter with available records, and 2 did not meet the improvement criterion. Among the four patients classified as improved, the first qualifying time points were postoperative quarters 0, 0, 1, and 2, yielding a median of 0.5 quarters. Quarter 0 indicates that change was documented after surgery within the same calendar quarter. It does not establish an onset of stimulation effect within days or weeks.

In the patient represented in Figure 4, mean severity according to manual annotation decreased from 1.80 before surgery to 0.00 after surgery. A score of 0 indicated that seizures were mentioned but could not be graded from the available text; it did not indicate complete seizure resolution. The decrease may therefore reflect differences in documentation detail and cannot be interpreted directly as a reduction in seizure frequency or clinical burden. Model predictions agreed with manual annotations at most time points, but differed substantially at the final observation: the manual label remained 0, whereas the model classified severity as high. This discrepancy illustrates the need to verify individual observations when interpreting an automated trajectory.

Figure 4. Longitudinal seizure severity trajectory for a representative VNS patient.

Figure 5. Cohort-level heatmap of seizure severity in 8 VNS patients, aligned to the date of surgery.

Trajectories for other symptoms differed in timing. Among 14 patients whose records met the documentation threshold for depressive symptoms, 8 met the improvement criterion. The median time to the first qualifying quarter in these patients was 14 quarters. Because a symptom may not be documented again for an extended period, these estimates represent the first recorded observation of improvement rather than a precise time of physiological onset.

Notes from patients undergoing other surgery did not show the same systematic postoperative decrease. However, the groups were not matched for diagnosis, preoperative symptoms, follow-up duration, or frequency of documentation. Nor was a reliable group-by-time interaction analysis performed. The comparison is therefore descriptive and cannot establish that VNS caused the observed changes.

Figure 6. Longitudinal seizure severity trajectory for a representative NSQIP surgical control patient.

Interpretation

Interpretive and methodological limitations

By organizing information dispersed across clinical notes into quarterly trajectories, the approach made between-patient variation more readily observable. Nevertheless, the analysis was limited to symptoms documented during clinical encounters and could not characterize events between visits. The VNS records came from a single medical center, and severity ratings depended on human interpretation of narrative text. External validation is needed to determine whether similar performance can be achieved in other settings.

The investigators also examined associations among symptoms in a downstream prediction model. The explainability analysis characterized how that model used symptom features; it did not identify the LLM's internal decision process or test biological pathways involving the vagus nerve. These associations may be affected by the rarity of some symptoms and by variation in documentation practices. Resampling notes rather than patients may have produced confidence intervals that were too narrow because individual patients contributed multiple notes. The small demographic subgroups also preclude firm conclusions about model fairness.

Future research

Future multicenter studies could use standardized symptom documentation and compare model-derived trajectories with patient-reported measures, seizure diaries, and objective assessments to determine what information is missing from routine notes. Uncommon but clinically consequential symptoms, including suicidal ideation, require larger validation samples and specific assessment of false-negative rates. Evaluating whether changes associated with VNS exceed those attributable to the natural history of disease or general surgical effects will require comparison groups matched for diagnosis, baseline symptoms, and follow-up intensity, together with prespecified between-group analyses. Such studies would help define the appropriate role of automated record review in clinical follow-up.

These findings are relevant to the broader conversation about tracking responses to vagus nerve stimulation, but they should not be read as evidence for consumer ear-based devices. The study examined an LLM workflow applied to clinical notes from patients who received implanted VNS. It did not test ZenoWell products, home-use taVNS, or a device's ability to improve symptoms. ZenoWell's Luna Plus are noninvasive ear-based stimulation devices built around repeatable routines, with Luna Plus adding optional app-based guidance and session history. Their features and any user-reported changes need to be evaluated on their own terms. Future work should study home-use devices with prospective symptom tracking and clearly defined outcomes.

Reference

Thakkar V, Silverman GM, Putzer M, et al. Longitudinal symptom severity tracking in vagus nerve stimulation patients: a 2-stage LLM-based pipeline with explainable AI. JAMIA Open. 2026;9(5):ooag195. https://doi.org/10.1093/jamiaopen/ooag195