ASD ISM 2114 Monitor baselines for AI application performance
What it says
Baselines of expected behaviour and performance for AI applications are established and monitored for unexpected deviations.
Quoted as published. Everything else on this page is written by Control Stack.
What it means
In plain EnglishEstablish baselines of expected behaviour and performance for AI applications and continuously monitor them to detect unexpected deviations.
This control means recording what "normal" looks like for an AI application (both how it behaves and how it performs) and then watching live activity against those recorded baselines. When the application starts producing different outputs, slows down, or makes more errors than the baseline expects, the deviation is flagged so it can be investigated. It catches model drift, degraded accuracy and abnormal behaviour before they cause harm.
Free. Save your status, tasks and notes.
Why it matters
Without established and monitored baselines, gradual model drift, accuracy degradation, prompt-injection effects or resource abuse in an AI application go unnoticed because there is no "known good" reference to compare against. Deviations such as rising hallucination rates, climbing token consumption or latency spikes are only discovered after they have already produced wrong outputs, breached data or disrupted dependent business processes.
How to implement
For each AI application, define the specific metrics to baseline: behaviour metrics such as output/embedding drift, response accuracy or quality score, error rate and refusal/safety-trigger rate, and performance metrics such as latency, throughput and token (or compute) usage per request.
Capture each metric over a representative period of normal operation and record a baseline as a range or statistical band (for example mean plus tolerance, or expected percentiles) rather than a single fixed number, so normal variation is not flagged.
Feed the live metrics into a monitoring or observability tool and configure it to compare current values against each baseline continuously, with one view per AI application.
Set concrete alert thresholds tied to each baseline (for example trigger when accuracy or quality drops more than a set amount, latency exceeds the baseline percentile, or token usage or error rate breaches the baseline band) and route alerts to the team that owns the application.
Recalibrate the affected baselines whenever the model is retrained, the version or prompt/system message changes, or a data-drift event is confirmed, and record the old and new values so intended shifts are distinguished from unexpected ones.
When an alert fires, run a root-cause investigation (model, data, infrastructure or external interference) and record the deviation and corrective action in a register so repeat deviations are tracked.
Keep it running
Baselines are not set-and-forget: recalibrate them after every model retraining, version upgrade, prompt or system-message change, or confirmed data-drift event, since the previous "normal" no longer applies. Keep a short window of pre-change and post-change metrics so you can distinguish an intended shift from an unexpected one. Tune alert thresholds periodically to reduce false positives while still catching genuine deviations, and review flagged deviations on a defined cadence so trends (not just single spikes) are caught. Retain the raw behaviour and performance telemetry long enough to reconstruct what normal looked like before a suspected deviation.
Evidence and audit
- Check 1 of 5What an auditor asks for
Ask for
Request the baseline definition for a sample AI application.
Look at
Check that it covers both behaviour metrics (such as output drift, error or refusal rate) and performance metrics (such as latency, throughput and token usage), and that each has a documented normal range rather than a single nominal value.
Good evidence
A written baseline for the application lists behaviour and performance metrics, each with an expected range that monitoring can be compared against.
- Check 2 of 5What an auditor asks for
Ask for
Request the monitoring configuration for that application and a demonstration of one metric flowing through it.
Look at
Trace one metric from capture through to its comparison against the baseline, and confirm live metrics are actually compared against the baseline rather than merely collected.
Good evidence
The metric can be followed end to end, and the monitoring tooling evaluates current values against the documented baseline range.
- Check 3 of 5What an auditor asks for
Ask for
Request the alert rules for the application, the deviation register and the relevant monitoring logs.
Look at
Confirm concrete deviation thresholds are configured (for example accuracy below a set percentage, or latency or token usage above the baseline band), then pick a recent flagged deviation and trace it to its investigation and corrective action.
Good evidence
Thresholds are explicit, an alert fired for a real deviation in the logs, and the register shows the deviation was investigated and actioned rather than only logged.
- Check 4 of 5What an auditor asks for
Ask for
Request the baseline change history alongside the application's change or release history.
Look at
Compare the baseline change dates and values against the most recent model retrain, version upgrade or data-drift event.
Good evidence
Baselines were recalibrated after the latest significant change, with dates and values that line up with the release history.
- Check 5 of 5What an auditor asks for
Ask for
Request the AI application inventory and the list of applications currently under baseline monitoring.
Look at
Reconcile the inventory against the monitored set to confirm baselines and monitoring exist for each in-scope AI application, not only one flagship system.
Good evidence
Every in-scope AI application in the inventory has an established baseline and active deviation monitoring, with any gaps explained and tracked.
Mappings
Closest match in ISO/IEC 27001: A.8.16, Monitoring networks and systems for anomalous behaviour, which this control partially meets.
ISO/IEC 27001: 2 mappings
This control partially meets A.8.16.
ISM-2114 requires establishing baselines for AI application behaviour and performance and monitoring for deviations. Annex A 8.16 focuses on monitoring for anomalous behaviour across networks, systems, and applications and aligns with security anomaly detection. ISM-2114 is more specific to baselines and AI deviations, while Annex A 8.16 is broader.
This control partially overlaps A.8.6.
ISM-2114 involves establishing baselines for AI performance and behaviour to monitor deviations. Annex A 8.6 requires monitoring and adjusting resource use, linked with aspects of AI performance baselining, like resource utilisation. Annex A 8.6 focuses on capacity management, while ISM-2114 covers broader AI performance aspects.
ISO/IEC 42001: 1 mapping
This control partially overlaps A.6.2.6.
ISM-2114 necessitates defining and monitoring AI application baselines for unexpected deviations. Annex A 6.2.6 involves documenting AI system operational requirements, such as system and performance monitoring, which forms the governance framework for implementing baseline monitoring. It is broader across AI lifecycle needs, whereas ISM-2114 is focused on baseline monitoring.
Mappings show how controls relate across frameworks. They do not imply full equivalence or certification.