Insights & Perspectives
Ideas for real-world impact.
Engineer Better Research Results From a Solid Workbench
Treating the process of your work as important as the result will improve the quality of your results. A lot of focus gets put on building the right thing for customers, and rightfully so, but it’s important to remember that we have to first build our workbench. Whether we do that haphazardly or intentionally can have an enormous impact on the quality of our results.
Evaluating the Evaluators: LLM Assessments in Practice
The process of making an LLM app reliable is mostly trial and error, involving spot-checking by the developer, reviews by product owners, and auto-evaluation. At Hop, we feel there’s a rich set of design decisions when it comes to auto-evaluation that aren’t regularly discussed; we share those in this post.
Could You Be Talking to an AI Doctor?
Think back to your last telehealth visit with a doctor. Perhaps your kid had a persistently high fever, or you had worrying chest pain. Are you sure you were interacting with a human? What makes you sure?
Testing Research Code: Is It Worth It?
Machine learning researchers often don’t write tests for their code. They’re not software engineers, and their code needs only to train a model or prove out an experiment. However, at Hop, we’ve found that adding certain kinds of tests can actually accelerate research and increase confidence in results through improving code quality and encouraging reuse.
Why Most LLM App POCs Fail
LLMs aren’t yet widely used as an architectural component in production — the core issue is reliability. Not knowing how to engage with the reliability challenge limits the success of most teams building LLM-powered applications. In our projects at Hop, we’ve developed a relatively uncommon perspective on how to effectively engage with this reliability challenge.
Machine Learning Is About Statistics After All: A Series of Vignettes, Part 1
It’s now possible to train complicated models while understanding very little about how they work. There’s a widespread attitude that it’s enough to know how to code up architectures in PyTorch and correct obscure bugs, and that the math is someone else’s problem. We at Hop put ML models into production, and we’re here to tell you that the math is not someone else’s problem.
Code Quality for Research
Research (and especially applied research of the type that Hop does) is a type of multi-armed bandit problem — one that tries to balance new approaches (exploration) with successful approaches (exploitation). The code quality/technical debt conversation becomes a bit easier to think about if you articulate where on the exploration/exploitation spectrum you currently are.
ML Systems as Decision Factories
If you're an executive with deep domain expertise but no ML background, and you're trying to understand if ML will be useful in your space, it can be useful to think of an ML system as a type of factory. In particular, it's a "decision factory".
Hiring Your Minimum Viable Machine Learning Team
A question we often get from executives exploring machine learning and AI for their organizations is: "What is the minimum viable ML/AI team?" There are likely many right answers, and some industries have unique constraints. However, in our experience, a minimal-but-effective ML/AI team requires a few specific roles to be filled.