Detecting drift in production applications
In the context of LLMs, drift refers to the gradual degradation of its performance over time. This is typically caused by changes in data distributions or user behavior that the model was not originally trained on. Like ML models, the output of generative AI models is dictated by their input. When the input changes, the output from these models can drift from the original intent. This can lead to a degradation of the generative AI application. In extreme cases, it can produce detrimental outcomes in terms of safety, legal, or monetary losses. A proactive drift-monitoring framework is essential for maintaining quality and reliability.
Types of drift in LLMs
Drift can be categorized into two main types, both of which can affect performance: data drift and concept drift.
Data drift refers to a statistical change in the input data the model receives. In LLMs, this is most effectively measured as a shift in the distribution of input prompt embeddings. Data drift occurs when the topics, questions, or language style of user prompts in production begin to differ significantly from the data the model was trained or last evaluated on. For example, a customer service bot for a mobile phone company might experience data drift after the launch of a new flagship phone because the distribution of user queries shifts to this new topic.
Concept drift refers to a more subtle change in the underlying relationship between inputs and the desired outputs. It occurs when user expectations change or when the meaning of concepts evolves over time. For example, in a financial analysis application, the factors that define a "good investment" might change due to new market conditions. The user prompts might look statistically similar (which means that the data drift is low), but the desired answer has changed (which means the concept drift is high). Detecting concept drift is significantly more challenging. It often involves monitoring downstream business metrics and user feedback rather than direct statistical tests on inputs.
Data drift detection
A robust framework for detecting data drift in LLM embeddings should be multi-layered. It should combine efficient statistical methods for initial detection with more sophisticated semantic analysis for interpretation.
Layer 1: Statistical drift on embeddings
This layer serves as the automated, first line of defense. It follows this process:
Establish a baseline – Capture a representative sample of prompt embeddings from a stable period, such as the first month of production, to serve as the reference distribution.
Monitor production data – In real-time or in batches, capture the embeddings of incoming production prompts to create the current distribution.
Compare distributions – Use statistical tests to quantify the distance between the reference and the current distributions. Commonly used drift metrics, such as the Kolmogorov–Smirnov (KS) test
are less effective for generative AI use cases because of the multi-dimensional nature of the LLM embeddings. It's therefore important to use statistics that perform better at measuring drift changes in embedding spaces, such as Wasserstein distance . Alert if a threshold is breached – If the calculated distance exceeds a predefined threshold, an alert is triggered. This alert indicates that significant data drift has occurred.
For more information about statistical detection of drift, see the following resources:
Feature attribution drift for models in production (Amazon SageMaker AI documentation)
Monitor embedding drift for LLMs deployed from Amazon SageMaker JumpStart
(AWS blog post) How A&E Engineering Uses Serverless Technology to Host Online Machine Learning Models
(AWS blog post) Detecting data drift using Amazon SageMaker AI
(AWS blog post) Learning High-Density Regions for a Generalized Kolmogorov-Smirnov Test in High-Dimensional Data
(NeurIPS Proceedings) Kolmogorov-Smirnov (KS) (Amazon SageMaker AI documentation)
Layer 2: Semantic drift using an LLM
A statistical alert indicates that a drift has happened, but it doesn't indicate why. To gain actionable insights, you can use the LLM-as-a-judge approach to analyze and classify the nature of the drift:
Sample the drifted data – When a statistical drift alert is triggered, collect a sample of the prompts from the period that caused the alert.
Perform a semantic analysis and classify the drift – Use a judge LLM with a carefully engineered prompt to compare the drifted prompts to a sample from the reference baseline. The judge's task is to categorize the nature of the change. For example, it might be prompted to classify the primary reason for the drift as "Emergence of a new topic," "Shift in user intent," "Increase in query complexity," or "Change in language style."
Review the results – The results of this classification provide the human team with a clear understanding of the drift's root cause. This classification guides the subsequent improvement actions.
For more information about using an LLM to gain a semantic understanding of detected drift, see the following resources:
