How do you validate an AI tool or machine learning model for GxP use?

1Start from intended use and risk.

An AI model is validated inside a computerised system, so the usual rules apply first. EU GMP Annex 11 asks for risk management through the system’s life cycle, with the extent of validation “based on a justified and documented risk assessment”.¹ What changes with AI is how much the intended use has to say. Draft Annex 22 asks for the task to be described “in detail based on an in-depth knowledge of the process”, with a full characterisation of the input data the model will see, including common and rare variations, limitations and possible biased inputs. A process subject matter expert is responsible for the description, which is approved before acceptance testing starts. Where it helps, the inputs are divided into subgroups, such as accept and reject decisions, sites, equipment or defect types.²

FDA’s January 2025 draft guidance frames the same step as a context of use: the model’s role and scope in answering a question, and what other evidence is used alongside it. Model risk then combines model influence, how much the model’s output counts, with decision consequence, how serious a wrong decision would be, and the credibility evidence needed rises with that risk.³ EMA’s reflection paper of September 2024 expects manufacturers to manage the risks of AI “from early development to decommissioning”.⁴

2Set acceptance criteria before testing.

Draft Annex 22 asks for test metrics suited to the use, such as a confusion matrix, sensitivity, specificity, accuracy, precision or F1 score for a model that classifies product, with acceptance criteria defined by a process expert and approved before testing. The criteria may differ by subgroup, and they must be “at least as high as the performance of the process it replaces”, so that performance has to be known first.² FDA asks for every performance estimate to come with confidence intervals.³ EMA prefers metrics that are insensitive to class imbalance, such as the Matthews correlation coefficient, and says pre-defined performance thresholds related to the context of use support a model’s credibility.⁴

3Keep the test data independent.

The test is only meaningful if the model has never seen the test data. Draft Annex 22 expects technical or procedural controls that keep test data out of development, training and validation, either by collecting it after training or by splitting it off before training starts. Split-off test data sits behind access control and an audit trail, with no copies elsewhere, and the record shows which data was used for testing, when and how often. Staff who have seen the test data should not train or validate the same model; where that is impossible, they work in pairs with a colleague who has not.²

The test data must represent the whole intended use, stratified across subgroups and large enough for adequate statistical confidence, with labels verified to a very high degree of correctness, for example by several experts or by laboratory tests. Generating test data or labels, for example with generative AI, “is not recommended”, and any use of it should be fully justified.² EMA adds that a test set is used once: if the model needs more development, “a completely new and independent test dataset is required”.⁴

4Run and document the test.

In draft Annex 22 the plan sets out the intended use, metrics, acceptance criteria, test data, test script and how the metrics are calculated, and a process expert helps write it.² During testing of a model for critical use, draft Annex 22 asks the system to record which features drove each classification, using techniques such as SHAP, LIME or heat maps, and to have those features reviewed as part of approving the results; it also asks for a confidence score per output and a threshold so that the model classifies only when that is suitable.²

5Keep it validated in use.

Validation does not end at release. Draft Annex 22 puts the model, its system and its process under change control and configuration control before deployment, and asks for regular monitoring of model performance and of drift in the input data.² FDA expects changes to a manufacturing model to go through the change management system of the pharmaceutical quality system, with parts of the credibility assessment repeated as needed.³ EMA says that for uses with high patient risk and high regulatory impact, all non-trivial changes to the software and hardware stack, including key dependencies, need a re-evaluation of performance, and that every model, especially one without a human in the loop, needs a system risk management plan that includes how to suspend or decommission it.⁴

6Generative AI and large language models.

Draft Annex 22 does not cover generative AI and LLMs and says they should not be used in critical GMP applications. In non-critical uses, qualified people are responsible for checking the output.² That may change: EMA’s report of its expert workshop of 30 June and 1 July 2026 says the drafting group has discussed allowing them with a documented, risk-based control strategy, and records the message that human oversight “cannot compensate for inadequate validation”.⁵⁶ FDA asks that where a human is in the loop, evaluation measures the human-AI team, not the model alone.³

7Where the GAMP AI Guide fits.

ISPE published the GAMP Guide: Artificial Intelligence in July 2025, a 290-page guide meant to be used in parallel with GAMP 5 Second Edition, for AI-enabled computerised systems in GxP use.⁷ Its contents follow the system life cycle through concept, project, operation and retirement, covering data splitting, testing, system monitoring and change and configuration management along the way, with further appendices on AI maturity by autonomy and adaptiveness, model categories and explainable AI.⁸ It is an industry guide, not a regulation; the expectations above come from the regulators. For the general lifecycle it builds on, see GAMP 5 Second Edition.

8How it runs in Seal.

In Seal, assurance starts from intended use and function risk, and supplier evidence, the configuration baseline, testing, anomalies and release stay linked, so each change is assessed from what moved (computer system assurance). The validation and AI assurance guide applies the same approach to Neil, Seal’s AI agent: evaluate it on representative jobs with expert-reviewed acceptance criteria, include obsolete procedures, conflicting records, missing units and insufficient evidence, define when Neil should stop or ask, and run the same cases again after a relevant model or configuration change, keeping failures as well as successes. A general model benchmark does not establish suitability for your process.

The guide also treats an AI-generated test as a starting point, to review against the requirement and execute, not as evidence in itself. Neil is not used in GMP execution: it proposes changes, their tests run, and your team releases them.