Can ChatGPT and other generative AI or LLMs be used in GMP work?

1What regulators have said about LLMs.

The clearest statement is in the draft Annex 22 on artificial intelligence, which the European Commission and PIC/S published for consultation from 7 July to 7 October 2025.¹ It covers AI models used in “critical applications with direct impact on patient safety, product quality or data integrity”. It is written for static models with deterministic output, which give identical outputs for identical inputs. Models whose output is probabilistic “might not provide identical outputs”, are not covered and “should not be used in critical GMP applications”.²

Then it names LLMs directly: the document “does not apply to Generative AI and Large Language Models (LLM), and such models should not be used in critical GMP applications.” In non-critical applications, “personnel with adequate qualification and training should always be responsible for ensuring that the outputs from such models are suitable for the intended use”, which the draft calls a human-in-the-loop.²

Annex 22 is still a draft. In May 2026 an HPRA GMP inspector told the QP Forum in Dublin that it is “currently in draft status”, under revision after the 2025 consultation, with “further focused stakeholder consultations” scheduled for 2026.³ On 6 November 2026 the EudraLex Volume 4 page lists Annex 11 in its January 2011 revision and no Annex 22.⁴

2What FDA’s April 2026 warning letter said.

On 2 April 2026 FDA’s Center for Drug Evaluation and Research issued Warning Letter 320-26-58 to Purolea Cosmetics Lab. The firm had told investigators that it used AI agents to create drug product specifications, procedures and master production or control records. FDA wrote: “If you use AI as an aid in document creation, you must review the AI generated documents to ensure they were accurate and actually compliant with CGMP”, and cited the failure as a violation of 21 CFR 211.22(c).⁵ That section makes the quality control unit responsible for approving or rejecting all procedures or specifications that affect the identity, strength, quality and purity of the drug product.⁶

The letter also records that the firm had not validated its process before distributing product, and that it said it was unaware of the requirement because the AI agent had never told it. If the firm resumes drug production and uses AI for CGMP work such as procedures and specifications, FDA wrote, “any output or recommendations from an AI agent must be reviewed and cleared by an authorized human representative of your firm’s QU”.⁵

3Why generative models are treated differently.

Among the risks that NIST’s Generative AI Profile lists as unique to or made worse by generative AI is confabulation: “the production of confidently stated but erroneous or false content”, known colloquially as hallucination. It warns that outputs can include confabulated logic or citations that make a wrong answer look justified, and that people can come to over-rely on generated content, which it calls automation bias.⁷ EMA’s reflection paper on AI makes the same point about medicinal product information: generative language models “are prone to include plausible but erroneous or incomplete output”, so quality review must confirm that generated text is factually and syntactically correct before submission.⁸

The draft Annex 22’s testing requirements, with pre-set acceptance criteria, independent test data, explainability and confidence thresholds, are written for static, deterministic models. Large language models sit outside them, which is why the draft keeps them out of critical applications and relies on qualified people for the rest.²

4Where the main sources stand.

5What to settle before staff use a chatbot.

• Whether the use is GMP at all. When a company says an AI tool is for “business use only”, HPRA expects it to “be sure the system in question does not fall under the remit of GMP”. Inspectors may ask at the opening meeting whether AI is used in any GMP system or process.³
• The agreement with the provider. Annex 11 applies to “all forms of computerised systems used as part of a GMP regulated activities”, and where a third party provides a service “for data processing, formal agreements must exist” that state its responsibilities.¹¹
• Controls in proportion to the risk. ICH Q9(R1) says “the level of effort, formality and documentation of the quality risk management process should be commensurate with the level of risk.”¹²
• Training for the people who use it. In the EU, Article 4 of the AI Act, as amended in July 2026, requires providers and deployers of AI systems to “take measures to support the development of AI literacy of their staff”.¹³

6How it runs in Seal.

Neil, Seal’s AI agent, is built on large language models, and Seal treats it on the draft Annex 22’s terms: Neil is not used in GMP execution. It drafts and proposes; it signs, approves and releases nothing. Each finding links to the record, version and field behind it, and Neil names what the records don’t show rather than filling the gap. It sees what you grant it, never more than the person asking. Each of the jobs Neil does shows the records it works from and the review a person makes.

The validation and AI assurance guide covers what is sent to the model and retained, the evidence to keep for an AI-assisted decision and how to assess a change of model. Under Seal’s agreements, customer data is not used to train the underlying models, and AI can be limited or switched off for the organisation.