Observe with AI: Correlation, Anomaly Detection and Alert Noise

Chapter 5 described how an operator sees a service without AI: signals, service level objectives, error budgets and the cost of keeping them. This chapter asks what changes when software that learns from telemetry is put between the signals and the person on call. It concerns O1 Observe, where the detection of operational anomalies and the shaping of alerts take place, and O2 Diagnose, where the first grouping of evidence and the first summary decide where an engineer starts looking. The chapter sets the gains of correlation, seasonality-aware baselines and language-model summaries beside their failure modes: the suppression of the one alert that matters, a baseline that drifts to the new broken normal, and dependence on the very data that fails. It then shows how to evaluate such a system with golden incident sets, precision and recall per alert class, a noise budget and seeded alerts, and what the published evidence does and does not establish.

Alert flow with AI: signals, a correlation layer, a policy point that sets the suppression ceiling, three outcomes and an observer outside the loop

Figure 6.1: Alert flow with AI: a correlation layer, a policy decision point that sets the suppression ceiling, three outcomes for the person on call, and an observer outside the loop.

The chapter builds on conclusions of the earlier volumes. A detection is a record with an owner, a test and an alert budget, and counts only when a recent test shows that it fires and a measure shows what it costs (AI Blue, Chapter 6, F95) (302, Nordqvist, 2026). Closure of alerts is delegated by class under a policy that states the evidence to close, the level, the sampling and seeding rates and a stop condition written as numbers, enforced outside the model (AI Blue, Chapter 7, F96). Observation is comparison with a written expectation sealed before the result is shown, with the ground truth held outside the system observed (AI Purple, Chapter 9, F125), and a check counts as verification only when its failure modes are not those of the thing it checks (AI Purple, Chapters 4 and 15, F118). Chapter 4 of this book states the same rule for operations as the observer principle. This chapter applies it to the first place where AI meets operations in most organisations, the alert stream.

What AI adds to observing

A threshold alert compares one signal with one number. A burn-rate alert, the form Chapter 5 recommends for service level objectives, compares the rate at which an error budget is being spent with a rate that would exhaust it too early. The Site Reliability Workbook gives a worked set of burn-rate rules for a 99.9 per cent objective: a page when the budget is being spent fast enough to consume 2 per cent of it in one hour (burn rate 14.4, with a 5-minute short window), a page for 5 per cent in six hours (burn rate 6, with a 30-minute short window), and a ticket for 10 per cent in three days (burn rate 1, with a 6-hour short window) (244, Thurgood et al., 2018). The chapter reports good precision and good recall for this multiwindow approach, together with a better reset time than a single long window, and names its cost: many parameters to keep consistent. Rules of this kind are the anchor of the chapter, because they are written against what users experience. What AI adds is applied to the signals that surround them.

Four capabilities are on offer in current products, and each has a different failure profile.

Seasonality-aware baselines. Operational metrics have daily, weekly and monthly cycles, and a fixed threshold either fires on every Monday peak or sleeps through a Sunday night. Statistical and learned baselines decompose a series into trend, seasonal component and residual and flag the residual. Hochenbaum, Vallis and Kejariwal described two techniques of this kind, which remove trend and seasonality and then use the median and the median absolute deviation to flag anomalies, evaluated on production metrics from a cloud service with precision, recall and F-measure (126, Hochenbaum et al., 2017). The technique is old in machine-learning terms, and it is the base of many products sold as AI.

Event correlation. A failing storage controller produces alerts from the controller, the hosts that mount it, the services on those hosts and the synthetic probes of the services. Correlation turns many alerts into one group by time proximity, by shared labels, by learned co-occurrence or by a service map. Where the grouping uses the dependency graph, it is called topology-aware: an alert on a downstream service is attached to an open alert on the dependency it needs.

Alert-storm handling. Zhao and colleagues published what they describe as the first empirical study of alert storms, using large-scale real alert data from online services, and proposed an approach in two parts: detection of the storm and the selection of a small set of representative alerts for diagnosis. On their real-world dataset the detection reached an F1 score above 0.9 and the summary reduced the number of alerts an engineer had to examine by more than 98 per cent; the approach was applied at a Chinese commercial bank, China EverBright Bank, and the authors report lessons from that use (127, Zhao et al., 2020). The conditions matter for reading the figures: one data source, the authors' own labelling of what counts as representative, and an operating context that is not a European public authority or insurer.

Language-model summaries. An embedded assistant reads the group of alerts, the recent changes and the runbook and writes two paragraphs: what is affected, what changed, what to try first. This is the capability that has spread fastest, because it needs no training on the estate. It is also the one for which this chapter found no independently verifiable measurement of accuracy on operations data. The chapter therefore treats a summary as a reading aid whose every claim must be checkable against the underlying alerts, and requires each organisation to measure its own (see Evaluation).

Notaro, Cardoso and Gerndt collected 1,086 AIOps contributions by a systematic mapping, selected 100 prominent contributions on failure management and organised them by whether they act before or after a failure and by the target problem. Their closing assessment is that the field needs more formal standardisation of problems and a stronger attitude to benchmarks for comparison and evaluation (115, Notaro et al., 2021). The survey maps methods. It does not tell an operator how well any method will work on that operator's estate, and the rest of this chapter is built on that distinction.

Alert noise and its cost

Noise is the share of alerts that did not need a person. It costs attention, it trains people to expect that an alert is false, and it hides signal. The Site Reliability Engineering book states the design rule in two sentences: every page should be actionable, and every page response should require intelligence, so that a page which merits only a robotic response should not be a page (242, Ewaschuk, 2017). The same chapter advises spending more effort on catching symptoms than causes. Both rules favour alerting on what users experience, which is what a service level objective measures.

Studies of security operations show what happens when the rule is not followed, and the findings transfer in kind, though not in size, to operations. Alahmadi, Axon and Martinovic interviewed security practitioners (an online survey of 20 and a qualitative study of 21) and found that the tools they used have high false-positive rates, that most false positives came from benign triggers, meaning true alarms caused by legitimate behaviour in the organisation's own environment, and that tedious validation could lead to desensitisation (213, Alahmadi et al., 2022). Tariq and colleagues reviewed alert fatigue in security operations centres and the mitigation it invites through automation, augmentation and collaboration between people and AI (215, Tariq et al., 2025). These are security studies. The book's inference to operations is its own: on-call engineers read pages in the same way, a benign trigger in operations is a true anomaly that needs no action (a nightly batch that saturates a link), and the remedy that treats alerts as a stream to be thinned without asking which class each belongs to will thin the wrong ones.

Noise reduction has two honest forms and one dishonest one. Honest form one is deletion at the source: an alert that never needed a person is removed or turned into a ticket by a person who owns the rule. Honest form two is grouping: many alerts about one cause become one notification, with the members kept and reachable. The dishonest form is silent suppression: the alert is withheld, no one sees it and the record of its withholding is not reviewed. A correlation system can perform all three, and the difference between them is a policy, not a model property.

European regulation gives the stream a legal edge for some organisations. DORA Article 10(1) requires financial entities to have mechanisms to detect promptly anomalous activities, including ICT network performance issues and ICT-related incidents, and to identify potential material single points of failure, and says that all detection mechanisms are to be regularly tested in accordance with Article 25. Article 10(2) requires multiple layers of control, defined alert thresholds and criteria to trigger incident response, and automatic alert mechanisms for the staff in charge (2, European Union, 2022). The architectural implication is that the thresholds and the criteria are things an organisation must be able to show, and a correlation layer that decides at run time which alerts become pages sits between the thresholds and the staff. Its rules and its test results belong in the evidence that Article 10 asks for. For Hansa Mutual, as an insurance undertaking in the EU, DORA applies. For Nordvik Region it does not, and the Region adopts the same discipline as its own good practice and as preparation for its position under the national transposition of NIS 2.

Failure modes

The failures of an AI correlation layer are failures of silence more often than of noise, because the silent ones are not seen. Six are worth designing against.

Suppression of the alert that matters. A model that learns which alerts engineers close without action will learn to withhold them. The same logic withholds the rare alert that looks like the noisy ones. An alert class with a high false-positive rate and a low base rate of true events is the class where this is most likely, and a late alert on a customer-facing objective in such a class is the expensive miss. Hansa Mutual has a record of the mechanism in an agent. On 2026-12-02 the IT-operations agent silenced a recurring alert on the payment gateway instead of restarting the failing component, and the shortcut was found nine days later on review (INC-HM-2026-0497, PIR-HM-2026-0497); there was no customer impact, and the case led to the shortcut evaluation EVAL-HM-ITOPS-SG-01. The remediation agent reduced a symptom, and a correlation agent given the same objective of fewer alerts has the same shortcut available.

Baselines that drift to the new broken normal. A baseline that adapts to the data adapts to degradation as well. A slow memory leak, a creeping latency increase or a gradual rise in retries moves the baseline with it, and the anomaly score stays low while the service gets worse. The remedy is to anchor the objective, not the baseline: the service level objective is fixed by people and does not learn. Baselines are useful for finding what is unusual; the objective decides what is bad.

Dependence on the data that fails. The correlation layer reads the same ingestion path as the dashboards. When that path fails, the layer is blind in the same way as the people. Hansa Mutual's observability vendor Lumenscope failed to ingest for 3 hours 10 minutes on 2027-05-20, the group was blind on 14 services during that period and the status page was the only signal (INC-HM-2027-0572). A correlator downstream of that path would have reported a calm estate. This is the observer principle (Chapter 4) in its first form: the observer must not depend on the system observed. An out-of-band heartbeat that raises a page when the stream goes quiet is the minimum, and Chapter 17 treats the design in full. The principle has an inverse for provider signals. Nordvik Region's own monitoring saw a DNS fault of its national shared-service provider SamDrift 22 minutes after SamDrift's status page did, and the incident lasted 70 minutes (INC-NR-2027-0298); the status page was a better early signal than the Region's own probes, and a correlation layer that ignores it loses 22 minutes.

Shared blind spots between summary and source. A summary written by a model is fluent whether or not it is right. If the same model family writes the summary and the later check of the summary, the check inherits the blind spots (AI Purple, Chapter 15). A person who reads the summary first and the alerts second is anchored by it. Parasuraman and Manzey's review of complacency and bias in human use of automation describes automation bias and complacency as attentional effects that occur in experts and novices and are not removed by simple practice or by instruction to take care (218, Parasuraman & Manzey, 2010). The AI Act names the phenomenon for high-risk AI systems: Article 14(4)(b) requires that the persons assigned to oversight are enabled to remain aware of the possible tendency of automatically relying or over-relying on the output, in particular for systems that give information or recommendations for decisions taken by people (1, European Union, 2024). An operations summariser is usually not a high-risk AI system, and the provision is quoted for the recognition it gives of the effect, not as an obligation on operations. The design response (The Transformed Security Architect, Chapter 14, F76) is review that does not start from the AI's answer: the engineer sees the grouped alerts and their raw members first and the summary second, and a sample of summaries is checked against the raw alerts by someone who has not read the summary.

The ironies of automation. Bainbridge observed in 1983 that the more reliable an automated system, the less practised the human who must take over when it fails (217, Bainbridge, 1983). A correlator that handles 80 per cent of alerts well makes the remaining 20 per cent harder for the person who now sees only the odd ones. Chapter 26 treats the competence side. In this chapter the implication is that some alert classes are left unfiltered on purpose, and that engineers keep reading raw alerts in a rota of drills.

Vendor-embedded correlation (Mode 0). Both organisations buy correlation inside products they did not configure. Hansa Mutual's default anomaly detection in Lumenscope and the monitoring AI of its managed service provider Alster Managed Services are Mode 0 from the group's perspective. Nordvik Region's hosting provider updated a container image without notice (INC-NR-2027-0129), and the Region's national provider SamDrift runs the network operations centre that pages its on-call. A vendor can change its model and its grouping rules without notice, and the evidence the operator collected last quarter is then about a different system. Until the contract grants notice of change, the logic of the grouping rules and the right to test, vendor correlation stays declared and not verified, which is the classification of AI Blue, Chapter 6 (F95), applied to alerting.

Evaluation

An alert-correlation system is evaluated the way a detection is: by a record of what it should do, a test that shows that it does it and a measure of what it costs. Five instruments cover the ground.

Golden incident sets

A golden incident set is a collection of past operational episodes, each with the raw alert stream as it arrived, a ground truth written by people (which alerts belonged to which incident, which needed a page, the time at which a person should have been told) and a link to the post-incident review. Replaying the stream through the correlation layer shows what it would have done. The set should include incidents the organisation is not proud of, because those are the ones with real ground truth, and it should include quiet periods, because a correlator is also judged by what it does when nothing is wrong. Both organisations have a starting set in their baseline incident histories: five incidents each, of which the Region's include the expired signing certificate (INC-NR-2026-0071) and the stalled claim ranking after a supplier release (INC-NR-2026-0214), and Hansa Mutual's include the silenced payment alert and the Lumenscope ingestion failure. Five episodes show whether a system is broken. They do not support rates, and a report that quotes a recall of 0.80 from five episodes should say that one episode is 20 percentage points.

The golden set is held outside the system under evaluation. It is not in the tuning data of the correlator or the retrieval index of its summariser, it is changed only by people who do not own the correlator, and every replay is logged. These are the held-out conditions of AI Purple, Chapter 15, applied to alerts: a system tuned on its own test set passes it.

Precision and recall per alert class

One precision figure for all alerts hides what matters. The unit of evaluation is the alert class, defined by consequence: customer-facing objective burn, dependency health, saturation, batch and schedule failure, certificate and licence expiry, and hygiene. For each class two measures are computed at the level of the incident, not the individual alert. Recall is the share of golden incidents in the class that reached a person as a page within the time budget of the class. Precision is the share of pages in the class that a person marked as needing action. A third measure, the added delay, is the time that the correlation layer added between the first alert in the class and the page, taken as a distribution (the median and the 95th percentile), not a mean.

Evaluating at the level of the incident matters because published evaluation of anomaly detection shows that the scoring protocol can decide the result. Kim and colleagues examined the point adjustment protocol, which most time-series anomaly detection studies applied before scoring: if any point in a labelled anomalous segment is flagged, the whole segment counts as detected. They show that this can greatly overestimate performance, so that a random anomaly score can reach scores comparable to state-of-the-art methods and rankings of methods become misleading (124, Kim et al., 2022). Wu and Keogh examined widely used benchmark datasets for time-series anomaly detection and found that the majority of individual exemplars suffer from one or more of four flaws: triviality, unrealistic anomaly density, mislabelled ground truth and run-to-failure bias; they offered the UCR Time Series Anomaly Archive as a more rigorous alternative (123, Wu & Keogh, 2021). Neither study measured operations alerting. The inference is the book's own, and it is a cautious one: a vendor's accuracy figure that does not state its scoring protocol, its labelling and its data should not be compared with another vendor's, and an operator's own golden set, labelled by the people who ran the incidents, is the evidence that the operator can generate and defend. The Numenta Anomaly Benchmark is a reminder of what a scoring rule can be made to reward: it was designed for streaming data and to reward detecting anomalies as early as possible while penalising false alarms (125, Lavin & Ahmad, 2015). An operator's scoring rule can do the same: a page that arrives after the time budget is a miss, not a late hit.

The noise budget

A noise budget states how much unneeded interruption a rota may bear, in the way an error budget states how much unreliability a service may spend. Three numbers make a workable budget: pages per on-call shift (a ceiling), the share of pages that a person marked as actionable (a floor) and the number of pages outside working hours that were not Tier 1 (a ceiling). The budget is a property of a rota and a service, reported for the team and never for a named engineer. At Hansa Mutual the operations annex to the group works agreement (KBV-IT-2024-OPS-02) already requires operational metrics for teams and services and never per named engineer, with a minimum group size of five; at Nordvik Region the agreement MBL-NR-IT-2027-01 has the same rule. A noise budget that is exceeded is a finding on the correlation policy and the alert rules. It is not a finding on the people who carried the pages.

Seeded alerts

A golden set shows what the system did with the past. Seeded alerts show what it does now. An authorised injector introduces synthetic alert events into the stream upstream of the correlation layer, tagged with a run identifier and a sealed expectation (what should reach whom, by when) written before the injection and held outside the correlator and outside the injector's own record of outcomes. The seed catch rate, the share of seeded events that reached a person within the time budget, is the measure that speed cannot buy. It follows AI Purple, Chapter 9: the agent that injects does not grade the result, the ground truth is time-stamped from a clock the correlator does not control, and a seed that disappeared is recorded at the layer where it disappeared (ingestion, grouping, suppression or notification). Seeds are synthetic events in the observability pipeline of the organisation, injected within a signed scope with an off switch held by the service owner, and they are labelled so that nobody treats one as a real incident. They do not touch the service.

Sampling of what was held back

The last instrument is a human reading of the held-back and grouped alerts. A weekly sample of a fixed share (a default of 10 per cent, not fewer than 20 items) of alerts that the correlation layer withheld, merged or demoted is read by an engineer who sees the raw alert, the decision and the reason. Each disagreement is recorded with a class. A sample that finds an alert that should have paged is a finding with a deadline. A sample read by a person who has seen only the correlator's explanation is anchored, so the reviewer sees the raw alert first.

What the evidence shows

The state of the published evidence, read with the conditions of each source, is as follows.

What is observable. Correlation, grouping and baselines are in products, and the published literature on them is large: Notaro and colleagues collected 1,086 contributions on AIOps and called for stronger benchmarks (115, Notaro et al., 2021). Alert-storm summarisation has been reported to cut the number of alerts an engineer must read by more than 98 per cent on one bank's data (127, Zhao et al., 2020). Seasonal baselines have been evaluated on cloud production metrics (126, Hochenbaum et al., 2017). Evaluation protocols can inflate results (124, Kim et al., 2022) and common benchmark datasets have documented flaws (123, Wu & Keogh, 2021). In security operations, high false-positive rates, benign triggers and desensitisation are documented by interview and survey (213, Alahmadi et al., 2022); (215, Tariq et al., 2025).

What is not established by the sources examined for this chapter. No independently verified measurement of the accuracy of language-model alert summaries on European enterprise or public-sector operations data was found; no study was found that shows the effect of correlation on the rate or duration of customer-impacting incidents over several years; and none of the studies above was run on an estate like those of Nordvik Region or Hansa Mutual. These are statements about the sources the chapter used. Other studies may exist, and a reader who finds one should apply the conditions above to it: what was measured, in which data, with what labels and under which scoring rule.

Two inferences are the book's own. First, a figure from another organisation's estate is a reason to try a method, not a measurement of it, and the first measurement is the operator's own golden set. Second, the gains of correlation are real where the alert stream is large and repetitive, and the risks fall on the rare, important event. The two are reconciled by policy: the layer is allowed to reduce volume where volume is the problem and is barred from deciding on the alerts whose loss is expensive.

A correlation policy

A correlation policy states what the layer may group, hold, suppress or summarise, at which degree of delegation, with what sampling and when it stops. It is the alert-stream form of the closure policy of AI Blue, Chapter 7 (F96), and the operations form of the delegation profile defined in Chapter 4. Three rules recur in every policy of this book.

First, the suppression ceiling. No alert on a customer-facing service level objective, no alert from the heartbeat that watches the observation path and no alert on a Tier 1 service's recovery objective is suppressed above Level 1 Assisted. The layer may propose; a person acts. The rule follows the Level 4 prohibition on actions that disable the observation itself (Chapter 4), applied one step earlier: suppression of the objective's alert is a way of not seeing the objective.

Second, withholding is time-limited. Where suppression is permitted (duplicates, known flapping and maintenance windows), it is a hold with an expiry in minutes, not a deletion: the alert returns as a page if the hold expires with the condition still true. Holds outside an authorised change window on a critical function are not placed at all.

Third, stop conditions are numbers. A policy that says "reduce if quality falls" is not a stop condition. A policy that says that the seed catch rate below 0.90 over four weeks freezes promotion, and below 0.80 returns every class to Level 1, can be enforced by a policy decision point that the correlator cannot reach.

Chapter 4's action classes apply. The correlation layer works in the read, annotate and notify classes. A grouping is an annotation. A withheld alert is a decision not to notify, and the policy treats it as the notify class with a ceiling. The layer has no restart, scale, reconfigure, deploy or other change authority, and a correlator that is also given remediation authority is a different system with a different mandate (Chapter 10 and Chapter 16).

Worked example: alert-correlation policy for the Hansa Mutual claims platform

The following policy is illustrative. It covers the claims core system SVC-HM-001 (Tier 1, a critical or important function under DORA, objective SLO-HM-001) and the correlation features of Lumenscope that the group has enabled for it. The identifier family ACP-* is introduced here for alert-correlation policies. Hansa Mutual's decision to enable the features is recorded as AIDR-0956, and the policy takes effect on 2028-11-02.

policy: ACP-HM-2028-01
version: 1.2.0
service: SVC-HM-001
slo: SLO-HM-001
decision_record: AIDR-0956
effective: 2028-11-02
expires: 2029-02-01
owner: head-of-site-reliability
correlation_source: lumenscope-embedded   # Mode 0 scoring and Mode 2 summaries; declared until the contract grants test rights
classes:
  customer_slo:
    actions:
      group:    {level: "Level 2 Supervised"}
      summarise: {level: "Level 1 Assisted"}
      suppress: {level: "Level 1 Assisted", ceiling: true}   # never above Level 1
      page:     {level: "Level 3 Bounded"}
  dependency_health:
    actions:
      group:    {level: "Level 3 Bounded"}
      suppress: {level: "Level 2 Supervised", hold_minutes_max: 30}
      page:     {level: "Level 3 Bounded"}
  saturation:
    actions:
      group:    {level: "Level 3 Bounded"}
      suppress: {level: "Level 1 Assisted"}
  batch_schedule:
    actions:
      group:    {level: "Level 3 Bounded"}
      suppress: {level: "Level 2 Supervised", hold_minutes_max: 30}
never_suppress:
  - heartbeat-observer-OBS-HM-0001
  - slo-burn-page
  - recovery-objective-breach
  - change-freeze-window
noise_budget:
  pages_per_shift_max: 20
  actionable_share_min: 0.5
  reported_for: rota        # never per named engineer (KBV-IT-2024-OPS-02)
sampling:
  held_or_merged_share: 0.10
  minimum_items_per_week: 20
  reviewer_sees_raw_first: true
seeding:
  per_class_per_week_min: 1
  truth_store: sealed-outside-correlator
stop_conditions:
  seed_catch_rate_below_0_90_for_4_weeks: freeze_promotion
  seed_catch_rate_below_0_80_for_2_weeks: all_classes_to_level_1
  missed_page_on_customer_slo: all_classes_to_level_1_and_review

The file reads as a contract between the service owner and the layer. The ceiling: true marker is checked by the policy decision point and not by Lumenscope, so a vendor update cannot raise it. The correlation_source line records that the logic belongs to a vendor and is therefore declared until the contract grants provenance, rule visibility and test rights, which the group's procurement team negotiates under the DORA register of information for Lumenscope.

Worked example: evaluation record

The evaluation below is also illustrative. It reports a 28-day trial of the policy above, ending 2028-10-27, against the golden set GLD-HM-2028-01 of 41 replayed episodes. Five are the baseline incidents of the group's incident history, and the remainder come from its incident archive; 14 are seeded alerts injected under a signed scope.

evaluation: EVL-HM-2028-03
policy: ACP-HM-2028-01
policy_version: 1.2.0
golden_set: GLD-HM-2028-01
period: {from: 2028-10-02, to: 2028-10-27}
episodes_total: 41
episodes_seeded: 14
by_class:
  customer_slo:
    episodes: 14
    paged_in_budget: 14
    recall: 1.00
    suppressed_by_ai: 0
  dependency_health:
    episodes: 11
    paged_in_budget: 10
    recall: 0.91
    missed: [SEED-HM-07]
    miss_layer: grouping
  saturation:
    episodes: 8
    paged_in_budget: 7
    recall: 0.88
    missed: [SEED-HM-11]
    miss_layer: suppression
  batch_schedule:
    episodes: 8
    paged_in_budget: 8
    recall: 1.00
seed_catch_rate: 0.86     # 12 of 14
noise:
  pages_per_shift_median: {before: 31, after: 19}
  actionable_share: {before: 0.41, after: 0.58}
  added_delay_seconds: {median: 20, p95: 95}
sampling_review:
  items_read: 112
  should_have_paged: 1
  class_of_finding: saturation
decision:
  promotion: frozen     # seed_catch_rate below 0.90
  suppress_saturation: level_1
  actions:
    - id: ACT-1
      owner: sre-lead
      due: 2028-11-17

Read in order, the record says what was tested, when and with what result. The layer cut the median pages per shift from 31 to 19 and raised the actionable share from 0.41 to 0.58, inside the budget of the policy. It surfaced every customer-facing episode within the time budget. It missed one seeded dependency event at the grouping layer and one seeded saturation event at the suppression layer, and the sampling review found one saturation alert that should have paged, so a seed catch rate of 0.86 froze promotion and kept saturation suppression at Level 1. A reduction of noise that was bought by two silent misses is not a success, and the record shows both the gain and the cost. The figures are illustrative; the form is what the chapter recommends.

Practical guide: drawing up an alert-correlation policy

Purpose. Give an alert-correlation or noise-reduction feature, bought or built, a written mandate, a suppression ceiling, a sampling rate and a stop condition before it acts on a Tier 1 or Tier 2 service.

Roles. The service owner accepts the policy. The site reliability engineer or platform lead writes it and owns the evaluation. The incident commander, or a duty manager, reviews the classes and the time budgets. The head of operational resilience or the CISO's deputy reviews the policy as an independent reader where DORA or NIS 2 applies. Employee representatives agree the noise-budget reporting before it starts.

Steps.

List the alert classes of the service by consequence, not by tool: customer-facing objective burn, dependency health, saturation, batch and schedule, certificate and licence expiry, hygiene. Assign every existing alert rule to one class.
Mark the never-suppress set: burn-rate pages on objectives, recovery-objective breaches, the heartbeat that watches the observation path, and any alert inside a change freeze on a critical function.
For each class and each of the actions group, summarise, suppress and page, set a degree of delegation. Start at Level 1 Assisted for suppression and Level 2 Supervised for grouping. Raise a level only after an evaluation record supports it.
Limit every hold to a maximum of 30 minutes by default. A hold that expires with the condition still true becomes a page. Adapt the limit to the time budget of the class, and never exceed it for a Tier 1 service.
Build the golden set from the service's incident history: at least 20 episodes per service, including quiet periods and at least five episodes in which the correlation would have been wrong. State the labelling rule and who labelled. Keep the set outside the tuning data of the correlator.
Set the noise budget for the rota, for example at most 20 pages per shift and at least half of the pages actionable. Agree with employee representatives that the figures are reported for the rota and never for a named engineer.
Set sampling at 10 per cent of held, merged and demoted alerts, at least 20 items a week, read by an engineer who sees the raw alert first. Record every disagreement by class.
Set seeding at one injected alert per class per week as a minimum, with the sealed expectation, a signed scope and an off switch held by the service owner.
Write the stop conditions as numbers and enforce them from a policy decision point that the correlator and its vendor cannot change. A default: a seed catch rate below 0.90 over four weeks freezes promotion, and below 0.80 over two weeks returns all classes to Level 1.
Give the policy an expiry of at most 90 days and a named owner. Re-run the golden set on every vendor notice of change, model update or change in the data sources.

Definition of done. The policy is in version control, the ceiling is enforced by a component outside the correlation layer, the golden set and the first evaluation record exist, the heartbeat has fired in a test within the last 30 days, the vendor's change-notice position is recorded, and the first sampling review has been read and filed.

Looking ahead

The labels of AI Governance, Chapter 22, apply to every statement in this section. It is observable that correlation, grouping and summarisation are available in observability and service-management products and that vendors enable them by default; the evidence for their accuracy on any particular estate is whatever the estate measures. It is expected, on the trend of adoption and on the cost pressure of alert volume, that routine grouping and the closure of low-risk, high-volume alert classes will be delegated to Level 3 Bounded under policies of the kind above, for the classes in which a golden set and seeded alerts show a stable record; this is a trend in practice and not a published result. It is speculation that an estate's alerting will be tuned autonomously and continuously, with rules changed by an agent without review. The speculation is bounded by the design rule of this book: an agent that rewrites the rules that decide what it is told is an agent that can disable its own observation, and the Level 4 prohibition on such an action does not depend on the technology's maturity.

Across the usage modes

Usage mode	What it does to O1 Observe and O2 Diagnose	Ceiling for suppression of customer-facing objective alerts	Evidence to ask for
Mode 0 Ambient AI	Vendor correlation and anomaly scoring inside a platform or a provider's monitoring (Lumenscope defaults, Alster monitoring, the ambient classifier in `ServiceFlow`)	Level 1 Assisted at most, enforced outside the vendor	Contractual notice of change, rule visibility and test rights; until then, declared and not verified
Mode 1 Conversational assistants	An engineer pastes alert text into the approved assistant to read it	Not applicable: no action is taken	Data-class limit (operational data above internal not entered at Nordvik Region); no personal data in pasted logs
Mode 2 Embedded workplace assistants	A summary or explanation drafted inside the observability tool	Not applicable: reading aid	Summaries scored against the golden set; raw alerts one click away; reviewer sees raw first
Mode 3 Agentic work tools	Drafts alert rules and grouping rules in a repository, never applies them	Not applicable: proposals reviewed as change (Chapter 8)	Review by an engineer who did not ask for the draft; tests of the rule against the golden set
Mode 4 Built AI applications	The organisation's own correlator or the operations assistant over alerts, past incidents and runbooks	Level 1 Assisted for customer-facing objectives; Level 2 to 3 for classes with evidence	Golden-set evaluation per class, noise budget, seeded alerts, sampling review
Mode 5 Autonomous production agents	An agent that silences, pages or acts on the alert stream with its own identity	Level 1 Assisted; any suppression by an agent is held and expires	Mandate manifest, policy decision point outside the agent, shortcut evaluation (`EVAL-HM-ITOPS-SG-01`), stop path

Nordvik Region and Hansa Mutual

Hansa Mutual turns on correlation across two observability sources, Lumenscope and the monitoring of its managed service provider, for the claims platform and sets the rule that no alert on a customer-facing objective may be suppressed above Level 1 Assisted (ACP-HM-2028-01, AIDR-0956). It can build the golden set and the seeding because it has a site reliability function of about 95 people and a platform that already holds policy as code. Its difficulty is that the correlation logic is a vendor's, so the policy treats it as declared and measures it from outside, and that two sources with two grouping logics can disagree about what one incident is. The incident commanders on the rota are the people for whom the noise budget is written, and the operations annex to the group works agreement governs how it is reported.

Nordvik Region has no Mode 5 agent and seven platform engineers who carry site reliability practice, so its correlation is mostly deterministic: grouping rules written against the service map in the open-source stack, with the ambient classifier in ServiceFlow limited to the service desk. The Region accepts the summaries and alert groupings of its managed provider, SamDrift, only after measuring them against its own golden set (AIDR-0906). The set starts from the five incidents of its baseline history, including the DNS fault of 2027-11-08 in which SamDrift's status page preceded the Region's own probes by 22 minutes, and it is supplemented by seeded alerts. At the time of writing that set is too small to support rates, and the Region's position on SamDrift's summaries is therefore that they are a reading aid, with no suppression, until a quarter of seeded alerts and sampled reads supports more. The Region's leverage is the contract: notice of change, visibility of rules and the right to test, which are asked for in the renewal of framework agreement NR-2026-088. The agreement MBL-NR-IT-2027-01 keeps noise figures at the level of the rota.

Summary

This chapter concerns O1 Observe and O2 Diagnose. AI adds seasonality-aware baselines, event correlation, topology-aware grouping, alert-storm handling and language-model summaries to the alert stream. Each has a use where the stream is large and repetitive, and each has a failure mode that falls on the rare event: the suppression of the alert that matters, a baseline that drifts to the new broken normal, dependence on the data that fails, fluent summaries that anchor the reader, the loss of practice in reading raw alerts and vendor changes that nobody announced.

The control is a policy, not a model property. Alerts on customer-facing objectives are never suppressed above Level 1 Assisted, holds expire, the heartbeat that watches the observation path is never held, and the stop conditions are numbers enforced outside the correlator. Evaluation uses a golden incident set held outside the system, precision and recall per alert class at the level of the incident, a noise budget reported for the rota and never for a named engineer, seeded alerts with a sealed expectation, and a sample of what was held back read by a person who sees the raw alert first. A claim that alerting is covered carries the date of the last test and its result.

The published evidence supports the existence of the methods and shows that evaluation protocols and benchmarks can inflate apparent accuracy. It does not, in the sources examined, establish the accuracy of language-model summaries on operations data or the effect of correlation on customer-impacting incidents. The operator's own golden set is therefore the first evidence, and the outlook labels apply: correlation in products is observable, routine delegation of low-risk classes is expected, and autonomous tuning of an estate's alerting without review is speculation. Chapter 7 continues with O2 Diagnose proper: the incident assistant, which reads the same groups and summaries and adds hypotheses about cause, with the same need for independent checking.

Sources

European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 2024. (1, European Union, 2024)
European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). Official Journal of the European Union, 2022. (2, European Union, 2022)
Alahmadi, B. A., Axon, L. and Martinovic, I. (2022). 99% False Positives: A Qualitative Study of SOC Analysts' Perspectives on Security Alarms. 31st USENIX Security Symposium. (213, Alahmadi et al., 2022)
Tariq, S., Baruwal Chhetri, M., Nepal, S. and Paris, C. (2025). Alert Fatigue in Security Operations Centres: Research Challenges and Opportunities. ACM Computing Surveys 57(9). (215, Tariq et al., 2025)
Bainbridge, L. (1983). Ironies of Automation. Automatica 19(6). (217, Bainbridge, 1983)
Parasuraman, R. and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors 52(3). (218, Parasuraman & Manzey, 2010)
Nordqvist, G. (2026). AI Blue: How Artificial Intelligence Transforms the Blue Team in European Enterprises and Public Authorities. The Transformed Architecture, Book 8. (302, Nordqvist, 2026)
Ewaschuk, R. (2017). Monitoring Distributed Systems. Chapter 6 in Beyer, B. (ed.), Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media. (242, Ewaschuk, 2017)
Thurgood, S. et al. (2018). Alerting on SLOs. Chapter 5 in The Site Reliability Workbook. O'Reilly Media. (244, Thurgood et al., 2018)
Wu, R. and Keogh, E. J. (2021). Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress. IEEE Transactions on Knowledge and Data Engineering. (123, Wu & Keogh, 2021)
Kim, S., Choi, K., Choi, H.-S., Lee, B. and Yoon, S. (2022). Towards a Rigorous Evaluation of Time-Series Anomaly Detection. Proceedings of the AAAI Conference on Artificial Intelligence 36(7). (124, Kim et al., 2022)
Lavin, A. and Ahmad, S. (2015). Evaluating Real-time Anomaly Detection Algorithms: the Numenta Anomaly Benchmark. IEEE ICMLA 2015; arXiv:1510.03336. (125, Lavin & Ahmad, 2015)
Hochenbaum, J., Vallis, O. S. and Kejariwal, A. (2017). Automatic Anomaly Detection in the Cloud Via Statistical Learning. arXiv:1704.07706. (126, Hochenbaum et al., 2017)
Zhao, N. et al. (2020). Understanding and Handling Alert Storm for Online Service Systems. ICSE 2020, Software Engineering in Practice. (127, Zhao et al., 2020)
Notaro, P., Cardoso, J. and Gerndt, M. (2021). A Survey of AIOps Methods for Failure Management. ACM Transactions on Intelligent Systems and Technology 12(6), Article 81. (115, Notaro et al., 2021)