From Manual Triage to Automated Intelligence
Haven Headache's Path to Scalable Patient Care
Haven Headache & Migraine Center
SUMMARY
Haven Headache & Migraine Center is creating virtual-first headache and migraine care, based around a daily headache-tracking messaging system.
Patient messages include everything from symptoms and treatments to medications and logistics, resulting in a “messy” database.
Catching any urgent messages within the noisy data required operational complexity and was stressful for the Haven team.
Given our experience in turning messy datasets into usable information, Hop built an LLM-powered tool to extract key information from patient messages.
Our team was able to secure a high level of accuracy on all tasks of interest, and every emergency is now flagged as needing urgent attention.
The tool allows Haven to gain insight into a patient’s headache patterns, track clinical parameters, and identify when messages require urgent attention, while saving them the time, cost, and operational complexity of scaling their support staff.
THE COMPANY
Haven Headache & Migraine Center is creating virtual-first headache and migraine care designed to put patients in the driver's seat. The company stems from personal experience and passion – Haven's CEO, Izac Ross, has grappled with chronic migraine since he was two. A key element of Haven's virtual-first approach is a daily check-in message, which allows Haven to monitor a patient's progress and make adjustments to their plan. As Haven grows, the daily check-ins will also enable the company to create a proprietary, first-of-its-kind anonymized headache dataset that can be useful for future population health-level research and interventions.
THE CHALLENGE
As part of their remote therapeutic monitoring (RTM) program, Haven messages patients every day to ask about their headaches, including information about headache severity, treatments used, and side effects experienced.
While patients love to communicate with a healthcare provider via messaging – especially because it provides one place to communicate about symptoms and treatments, medications and side effects, and appointment and account logistics – what gets stored in Haven’s database is a jumble of messy messages. Patients respond to questions late (or early, anticipating the daily questions), make typos and send follow-ups correcting them, and comingle information about their symptoms with asks to schedule appointments.
Additionally, some patients message Haven regarding emergencies, and catching and responding to these in time requires additional operational complexity and a lot of added stress.
Making sense of any given patient’s story was an effort-intensive, manual process, and Haven aspired to much more – smoothly automated patient conversations, proactively noticing when patients’ headache patterns change, and efficiently helping their patients enact their care plans. However, they didn’t yet have the in-house expertise to make these goals a reality.
Knowing that Hop Labs has experience turning messy datasets into usable information, Haven turned to Hop for help with extracting key information from patient messages: a) which Haven team to send the message to, b) clinical information about headaches and treatments, and c) level of urgency.
“We needed an AI engineer, and we had budget for it, but we didn’t have the ability to hire nor the timeline to do it effectively. I knew we could trust Hop to fill that gap.”
THE APPROACH
To help Haven with the challenges outlined above, Hop Labs built an LLM-powered tool to extract information from messages sent by patients.
LLMs aren’t without their downsides – they’re expensive, with per-user costs much higher than those of any other software tool. Also, LLMs are notoriously unreliable and prone to hallucinations, stating incorrect information as fact. In order to characterize and mitigate these downsides, Hop developed a detailed evaluation dataset with diverse user inputs to characterize the performance of the system, assessing accuracy, cost, and latency.
One of the main expenses associated with these techniques is the curation of extensive training datasets for models. In order to help Haven prepare for that eventuality, Hop assisted them with standing up a process to collect data labels during the normal course of business. This will allow their nurses to create the training datasets as they go, minimizing the cost of dataset curation.
As operating cost is also a key concern of our client, Hop implemented several approaches, including traditional NLP techniques and caching, to reduce the volume of text passed to the LLM. Putting these improvements in place drove the daily per-user cost down to just cents per day, which is affordable at Haven’s current scale.
Of course, Haven is a fast-growing startup, and within a few years even that low cost will become worth driving down further. Hop has experience with additional techniques that can reduce the expense for some portions of the information extraction task – techniques like fine-tuned embedding models and model routing – but at present the potential savings from implementing them is too small to justify the R&D effort.
THE RESULTS
To assist Haven in making sense of their patient data, Hop Labs delivered a production-ready, LLM-powered tool that processes incoming patient messages. Understanding the specific information contained in those messages allows Haven to gain insight into a patient’s headache patterns over time, track clinical parameters like treatments and side effects, and identify when messages require urgent attention. Additionally, this system saves Haven the time, cost, and operational complexity of scaling their support staff.
“For anyone who needs to supplement their AI expertise or AI feature set, Hop is an amazing solution.”
After extensive experimentation with the LLM tool, Hop was able to secure a high level of accuracy on all tasks of interest. For headache severity messages, patients go almost 300 days without a system error; treatments are correctly identified with a high F1 score of 99.4%, and every emergency is flagged as needing urgent attention.
Now with better insights, Haven can flexibly accommodate messy patient conversation threads – for example, avoiding asking for information a patient has already provided. With structured information about patient symptoms and treatments, Haven is poised to notice when intervention is needed – say, for worrying changes in headache patterns, or escalation or misuse of treatments. And rapid identification of urgent messages empowers Haven’s care team to both respond to emergencies immediately and otherwise focus fully on patient communication without their earlier stress about missing an urgent message.
With this new technology, Haven has not only upleveled the service to their patients, they’ve made a leap toward developing a useful dataset for headache and health research. What was once a messy data challenge is now a strategic asset and streamlined operation, giving Haven a foundation for headache care at scale.
Looking to turn messy datasets into usable information? Contact us to learn how Hop can help.