Kort gezegd: Model drift is when an AI model slowly gives worse or different answers after launch, even though nobody changed its code. The world, the data and the model underneath keep moving. You catch it by running the same tests on a fixed schedule, comparing the results with a baseline and keeping the evidence.
Your AI passed every check on launch day. Six months later, it may not. Model drift rarely makes noise. It shows up as a strange answer to a customer or a rise in support tickets. Or an auditor asks how you know the model still works. If you don’t test it regularly, you find out last.
What is model drift?
Model drift is the gap that grows between how an AI model behaved when you tested it and how it behaves today. The code can be exactly the same. What changes is everything around it. The questions people ask shift. The data the model reads gets updated. The meaning of a good answer moves, for example when a policy or a price changes. And if you use a large language model from a vendor, the vendor can update or replace that model while your app keeps running. Each change is small. Together they add up, and nobody gets an alert. That is why drift needs ongoing monitoring. A test you ran once tells you how the model behaved on that day. It tells you nothing about this week.
There are a few common forms. Data drift means the inputs change: new products, new slang, new types of customer. Concept drift means the right answer changes, while the model still gives the old one. Model updates mean the system underneath you is different, even if the name on the contract is the same.
Why do AI models drift after launch?
AI models drift because they learn from a snapshot of the past, while they work in a present that keeps changing. A model trained or tuned on last year’s questions has never seen this year’s. For apps built on a vendor’s language model there is an extra reason: the model itself can change. Researchers at Stanford and UC Berkeley tested two versions of GPT-4, from March and June 2023, on the same questions. On one task, spotting prime numbers, accuracy fell from 84% to 51%. That is what their study on how ChatGPT’s behaviour changes over time found. Same product name, very different results. Your own setup adds more moving parts. Someone edits a system prompt. A document in your knowledge base gets replaced. A new tool gets connected. Each one can change what the model says.
We see the same pattern that software testing has always shown. Even well-designed AI systems drift and misbehave over time. The difference with normal software is that an AI model can change its behaviour without a new release.
How do you detect model drift?
You detect model drift by asking the model the same set of questions on a fixed schedule and comparing the answers with a baseline. The baseline is the result from a moment when you knew the model behaved well. If the scores drop, or the answers change in ways you didn’t expect, something has drifted. This only works if the test set stays fixed and covers what matters to you. That means accuracy on your core questions, but also refusals, bias, privacy and safety. A test set that changes every run can’t show a trend. Good drift detection also keeps the exact responses, not just a score. When something goes wrong, you want to see what the model actually said, on which date, to which question.
A practical setup has four parts:
- A fixed test set of real questions, including the awkward ones your customers ask.
- A schedule, such as weekly, and an extra run after every change to prompts, data or vendor model.
- Thresholds that tell you when a drop is worth action, so you don’t chase noise.
- A record of every run, with scores and responses, that you can show to an auditor.
How often should you test for drift?
Test as often as the model can change and as often as a mistake would hurt. For a customer-facing chatbot built on a vendor model, weekly is a sensible default, because the vendor can update the model at any time. For a model that makes decisions about people, such as credit or hiring, daily runs or tests after every data update make sense. Under the EU AI Act, providers of high-risk AI systems must run post-market monitoring. That means watching how the system performs after it goes live. The NIST AI Risk Management Framework also treats measuring and tracking AI risk as an ongoing task, not a one-time check.
There is a trade-off. If your AI only drafts internal notes that a person always reviews, weekly automated tests may be more than you need. A manual check each month and a test after every change can be enough. Match the effort to the risk.
What this means for your team
Every AI system drifts, so drift alone says nothing bad about your team. Not knowing about it is what hurts. Customers blame your organisation, not the model, and during an audit missing evidence counts against you. Plan on the assumption that your model will drift, and make sure you are the first to see it.
Start small. Write down 20 to 50 questions that matter most to your business. Run them now and save the answers as your baseline. Then repeat the run on a schedule and after every change.
This is where a testing routine helps. For over thirteen years, Nekst IT, the company behind VeritasAI, has helped organisations deliver software they can trust through testing, validation and quality assurance. VeritasAI runs automated risk and compliance tests on your AI model every week or every day. It compares the results over time and lets you review the exact responses your model gave. You see where the model performs well, where it slips and what to fix.
See where your AI is drifting before your customers do.
Frequently asked questions
What is the difference between data drift and concept drift?
Data drift means the inputs change, such as new types of questions or customers. Concept drift means the correct answer changes, for example after a new policy, while the model keeps giving the old answer. Both make a model less reliable over time.
Can a large language model drift if I never retrain it?
Yes. If you use a vendor’s model, the vendor can update it. Your prompts, documents and connected tools can also change. Any of these can change the answers, even without retraining.
Is monitoring for model drift required by the EU AI Act?
For high-risk AI systems, providers must set up post-market monitoring that tracks how the system performs once it is in use. For other systems the law is less specific. You still need to show your AI is under control if a customer or regulator asks.
How many test questions do I need to detect drift?
There is no fixed number. A focused set of 20 to 50 real questions is a good start, as long as it stays the same between runs. Add questions for bias, privacy and safety, not just accuracy.