Detect and Mitigate Adversarial Prompts in Generative AI Applications
Generative AI applications must check user prompts to catch and block adversarial inputs that try to trick the AI into harmful or unintended behaviour.
Plain language
If your organisation runs a generative AI application (software that creates text, images or code in response to a typed request, like a chatbot), people may deliberately craft tricky inputs to make it misbehave. These tricks, often called prompt injection or jailbreak attempts, can include hidden instructions or strange text added to the end of a request (a "suffix") designed to make the AI ignore its safety rules. This control requires the application to actively examine every user prompt, spot these manipulations, and stop the AI from being fooled into producing sensitive or harmful content.
Framework
ASD Information Security Manual (ISM)
Control effect
Preventative
Classifications
NC, OS, P, S, TS
ISM last updated
Sept 2025
Control Stack last updated
18 June 2026
E8 maturity levels
N/A
Guideline
Guidelines for software developmentTopic
Prompt InjectionOfficial control statement
Generative AI applications evaluate user prompts to detect and mitigate adversarial inputs or suffixes designed to elicit unintended behaviour or assist in the generation of sensitive or harmful content.
Why it matters
If adversarial prompts are not detected, attackers can jailbreak the AI into leaking sensitive data or producing harmful content, exposing the organisation to data breaches and reputational damage.
Operational notes
Adversarial techniques evolve constantly, so keep detection rules current and re-test the application regularly rather than treating prompt screening as a one-time configuration.
Implementation tips
- The development team should add a prompt-screening step in front of the generative AI application that inspects each user request before it reaches the AI model, checking for known manipulation patterns and unusual suffixes that try to override the AI's instructions.
- The IT or AI team should keep a regularly updated list of known adversarial techniques (such as 'ignore previous instructions' phrasing and common jailbreak suffixes) and feed it into the screening tool so newly discovered attacks are caught.
- The development team should configure the application to block, sanitise, or flag suspicious prompts rather than passing them straight to the AI, and return a safe refusal message when a prompt is identified as adversarial.
- The AI team should run regular adversarial testing (sometimes called red teaming), deliberately feeding the application tricky prompts to confirm it detects and refuses attempts to produce sensitive or harmful content.
- The application owner should ensure every blocked or flagged prompt is logged so the security team can review attack attempts, measure how often they occur, and tune the detection rules over time.
Audit / evidence tips
- Askto see how the generative AI application screens incoming user promptsLook atthe design or configuration of the prompt-checking layerGoodshows a documented mechanism that inspects every prompt for adversarial inputs and suffixes before the AI responds
- Askthe list of adversarial patterns or jailbreak techniques the application defends against and when it was last updatedLook atthe update historyGoodshows a maintained, current list rather than a one-off setup left untouched
- Askthe results of recent adversarial testing or red-team exercises against the applicationLook atthe test cases and outcomesGoodshows tricky prompts were tried, the system blocked them, and any failures were fixed
- Askto be shown what happens when an adversarial prompt is submittedLook ata live or recorded demonstrationGoodshows the application detects the attempt, refuses, and does not generate sensitive or harmful content
- Asklogs of detected or blocked prompts and who reviews themLook atrecent log entries and review recordsGoodshows attempts are captured, regularly reviewed, and used to improve detection
Cross-framework mappings
How ISM-1924 relates to controls across ISO/IEC 27001, ISO/IEC 42001, Essential Eight, and ASD ISM.
ISO 27001
| Control | Notes | Details |
|---|---|---|
handshakeSupports(2)expand_less | ||
| Annex A 8.12 | ISM-1924 focuses on preventing prompt injection so the AI does not generate or disclose sensitive or harmful content due to adversarial p... | |
| Annex A 8.16 | ISM-1924 requires generative AI applications to evaluate user prompts to detect and mitigate prompt injection attempts that could cause u... | |
extensionDepends on(2)expand_less | ||
| Annex A 8.26 | ISM-1924 requires generative AI solutions to evaluate user prompts and mitigate adversarial inputs intended to elicit unintended behaviou... | |
| Annex A 8.28 | ISM-1924 requires organisations to build AI applications that can identify and mitigate adversarial prompt content (e.g | |
These mappings show relationships between controls across frameworks. They do not imply full equivalence or certification.
Related ASD ISM controls in Software development
See all Guidelines for software development controls, or browse the full ASD ISM library.