Google's Rules of ML applied to LLM features: measure first, AI second
TL;DR
Google's Rules of ML was written for classic ML but fits LLMs precisely: baseline first, launch something simple (even without AI), get the infrastructure right, run in shadow, then expand. LLMs make demos so easy that this discipline matters more.
The team is excited: "let's build an AI feature!". Three months later there's a nice demo, but nobody knows whether it beats the old way, and there's no measurement in place. Google's Rules of Machine Learning (Martin Zinkevich) was written for classic ML, but its first rules apply to today's LLM features with surprising precision. This post picks the most useful rules and "translates" them to the LLM context.
The key rules
| Original rule | LLM-feature version |
|---|---|
| #1 Don't be afraid to launch a product without ML | If rules, templates or plain search solve 80%, launch that first. The LLM comes later. |
| #2 First, design and implement metrics | Measure current behaviour (ticket resolution rate, handling time) before adding AI, so you have a baseline. |
| #3 Choose ML over a complex heuristic | When your rule set has grown to hundreds of hard-to-maintain branches, that's when an LLM really pays off. |
| #4 Keep the first model simple and get the infrastructure right | A simple prompt + full logging + evals + rollback matter more than a "clever" prompt. |
| #5 Test the infrastructure independently from the ML | Test the pipeline (data fetching, API calls, output parsing, retries) with stubbed outputs, independent of model quality. |
| #16 Plan to launch and iterate | Prompts and models will keep changing; design for easy, safe changes from the start. |
A four-step path for LLM features
- Baseline: what the current process produces, how long it takes, what it costs.
- A no-AI or minimal-AI version: simple rules or a basic prompt; build logging, evals and an on/off flag.
- Shadow mode: the AI runs but isn't shown to users; compare its output with real outcomes.
- Gradual launch: enable for a small group, watch the metrics defined in step 1, then expand.
Why the old rules still hold
- The core problem is unchanged: probabilistic systems need measurement; you can't verify them by reading code.
- LLMs make demos too easy: which makes it even easier to skip baselines and infrastructure, exactly the trap Rules of ML warns about.
- Real operating costs: LLMs add token cost and latency; metrics must include both.
Risks and caveats
- Easy-to-measure but wrong metrics: "number of chatbot answers" going up doesn't mean happier customers. Pick metrics close to business goals.
- Skipping the no-AI option: sometimes a better search box is enough, cheaper and more reliable.
- No way back: without a fast kill switch, one bad prompt can affect every user.
How to apply it
- Dev: Build logging, evals and a feature flag before tuning prompts; swap the model for a stub to test the pipeline.
- Tech Lead: Require every AI feature proposal to have baseline metrics and a shadow plan before work starts.
- PM: Ask "how would we solve this without AI?" in every idea review.
Sources & references
- Rules of Machine Learning — Martin Zinkevich, Google for Developers