Diagnose: Triage, Root-Cause Analysis and the Incident Assistant

An incident is an interval in which a service does not do what its objective promises and the organisation does not yet know why. Diagnosis is the work done inside that interval: deciding that something is wrong, deciding how much it matters, forming explanations, testing them and handing a cause to the people who will act on it. This chapter concerns O2 Diagnose, the function of the operations loop that Chapter 4 defines as finding what is wrong and why. It draws on O1 Observe, whose signals start every diagnosis (Chapters 5 and 6), and it hands over to O3 Change and O4 Restore, whose actions follow from a diagnosis and are treated in Chapters 9 and 10.

The incident assistant is the newest participant in this work. It retrieves past incidents and runbooks, lists what changed before the symptoms began, ranks candidate explanations and drafts status updates. The chapter describes what such an assistant can do, what published studies of automated root-cause analysis report and under which conditions, why the first explanation an assistant offers tends to become the only one the team considers, and how an incident commander stays in command. It ends with the record that makes the assistant's contribution inspectable afterwards.

The diagnosis flow: the read-only incident assistant, the human decision points, the incident record and a stop path outside the platform

Figure 7.1: The diagnosis flow: the read-only incident assistant, the human decision points, the incident record and the stop path outside the assistant platform.

The chapter's conclusion is the book's own, drawn from the evidence below. In diagnosis the assistant is a source of evidence and of hypotheses, and it is never the authority for an action. Its tools are read-only. The person who acts on a diagnosis records a second hypothesis and a test that could refute the first before the action is taken, and the record shows which parts of the work were the assistant's.

Diagnosis before AI

Diagnosis in operations is older than any assistant, and the practice that AI enters is worth stating plainly. The Google site reliability engineering (SRE) literature describes troubleshooting as the hypothetico-deductive method: the responder forms candidate causes from observations and knowledge of the system, then tries to rule them in or out by experiment (131, Jones, 2017). The same chapter lists the pitfalls that experienced responders know well: chasing symptoms that are irrelevant, misreading metrics, proposing improbable theories, assuming that a past cause must recur, and following correlations that arise from coincidence or from a shared cause. It also records, as a productive place to start, the question of what touched the system last, because a working system tends to remain in its current state until an external force such as a configuration change or a shift in the type of load acts on it.

The weight given to change is supported by a measurement of the same community. The SRE book states that SRE has found roughly 70 per cent of outages to be due to changes in a live system (245, Treynor Sloss, 2017). The figure is the authors' own experience at one organisation, reported without a method, and it should be read as direction and not as a constant. It explains why change correlation is the first thing both people and assistants look at, and why Chapter 9 treats the change itself as the thing to make safe.

Incident command

Diagnosis is also a social activity with a structure. The SRE book's chapter on managing incidents separates four responsibilities: incident command, which holds the high-level state of the incident; operational work, in which the operations team is the only group modifying the system; communication; and planning (132, Stribblehill, 2017). It names two failure modes that recur in the literature on incident handling: poor communication, in which nobody knows what coworkers are doing, and freelancing, in which an engineer makes production changes without coordinating with the person in charge of troubleshooting. Its most important instruction to the incident commander is to keep a living incident document.

The SRE Workbook adds practices that bear directly on the assistant. It advises declaring incidents early and often, writing down criteria for what counts as an incident, keeping a working record of debugging and mitigation as the incident proceeds, and, once three or more people are involved, keeping a shared document of working theories, eliminated causes and debugging data (133, Mace et al., 2018). In the workbook's case study a scribe captures actions, owners and timestamps. By default the incident commander assumes every role that has not been delegated. These are the roles into which the assistant is placed, and the placement matters. An assistant with write tools in production is a second party able to modify the system, and it recreates the freelancing failure unless its authority is bounded. In diagnosis its tools are therefore limited to reading telemetry and records and to annotating the incident record, which are the first two of the action classes that Chapter 4 defines. Any action beyond them is a change or a restoration and passes through the runbooks and gates of Chapters 8 to 10.

What the work consists of

Four activities can be separated, and the assistant touches each differently. Triage decides whether a signal is an incident, how severe it is and who is needed. Hypothesis formation turns symptoms and recent changes into candidate explanations. Testing chooses an observation that would distinguish between them. Handover gives the cause, or the best current account of it, to whoever acts: the operations lead for a rollback, the service owner for a fix, the communications lead for a customer message. The NIST incident response profile treats this work as one part of a larger risk management cycle that starts before the incident and continues after it (41, NIST, 2025). That framing is the reason the record in this chapter is built to serve Chapter 12 (post-incident review) as much as the incident in progress.

The ironies of automation

Bainbridge's analysis of industrial automation predicts what follows when a system takes over the easy part of a job. The operator is left with the tasks that the automation cannot do, which are the rarest and hardest, and the skills for those tasks decay with disuse (217, Bainbridge, 1983). If an assistant resolves the routine diagnoses, the incident commander meets only the unusual ones, with less practice in the routine ones that build the pattern knowledge to handle the unusual. The consequence for design is stated in Chapter 26: part of diagnosis is kept in human practice on purpose. Within this chapter it shows up as the rule that a person forms a hypothesis before the assistant's ranking is shown.

The incident assistant

The incident assistant in this book is a Mode 4 built AI application or a Mode 2 embedded workplace assistant, according to where it lives (Mode 2 Embedded workplace assistants, Mode 4 Built AI applications; AI Governance and AI Platform and Security, Chapter 2). It is not a Mode 5 autonomous production agent. It works inside the incident, with the identity of its own use case, read-only tools and a mandate that Chapter 16 specifies. Four capabilities can be distinguished, because each needs a different kind of checking.

Retrieval. The assistant searches past incident records, post-incident reviews, problem records, runbooks and the knowledge base for items similar to the present symptoms. Retrieval gives the responder a precedent: an earlier incident in which a queue setting stalled a service, a runbook for a certificate that expired. Its failure is a precedent that matches on wording and not on mechanism. The check is cheap, because the responder opens the cited record and reads whether the mechanism fits. Retrieval is the capability with the lowest risk and the clearest evidence of benefit, and it is where both organisations start.

Change correlation. The assistant queries the change calendar, the GitOps log and the release records for changes that touched the affected service or its dependencies before the symptoms began, and lists them with times and authors. This is the machine form of what touched it last. Its failure is the correlation that arises by coincidence or from proximity in time. A change just before the alert is not thereby the cause, and a change an hour before may be.

Hypothesis ranking. The assistant combines retrieval, change correlation and the current telemetry into a short list of candidate causes, each with the evidence for it and the evidence against it, and with a rank score. The score is an ordering produced by the model and its scaffolding. It is not a calibrated probability unless the organisation has measured calibration on its own incidents, and the record states which it is. Ranking is the capability that benchmarks measure and the one most exposed to anchoring (below).

Drafting. The assistant drafts status updates, handover summaries and the timeline for the incident record from the bridge discussion and the channel. Its failure is the plausible narrative that is wrong in a detail the reader will not check, and the narrative that records what the assistant concluded as though it were what was observed. A person reads and approves every draft before it leaves the incident channel, and the assistant never writes to customers or to a regulator.

Inputs the assistant should not trust

The assistant reads text that other parties can write: alert annotations, ticket descriptions, log lines, messages from providers. A model that follows instruction-like text in its input can be steered by whoever wrote it. The mechanism and the controls are the subject of Chapter 18. For diagnosis the consequence is the read-only tool surface, which bounds what a steered assistant can do to misdirection and disclosure and not to action. Misdirection remains possible, and it is the reason for the independent evidence required below.

Personal data in the corpus

Incident records, log excerpts and tickets contain personal data, for the Region the special-category data of social services and benefits, for the group the policyholder and claimant data of seven markets. The GDPR requires that personal data be adequate, relevant and limited to what is necessary for the purposes of processing (Article 5(1)(c)) and that the controller and processor apply measures appropriate to the risk (Article 32) (6, European Union, 2016). The implication for the assistant is architectural: the retrieval corpus is built from incident records that are written to a minimised standard, raw payloads and claimant identifiers are excluded at ingestion, and the assistant's own prompts and outputs are logs that fall under the same retention and access rules. Chapter 5 treats personal data in telemetry; this chapter adds that the assistant's context window is another place where it can end up.

What the benchmarks say

Several public studies measure how well language-model systems locate the cause of an operational failure. They are useful to an organisation as a map of difficulty, and they mislead when a single figure is carried over from its conditions to an estate that has none of them. Five studies are summarised here with the conditions that decide what each number means.

The studies

OpenRCA. Xu and colleagues built a benchmark of 335 failures from three enterprise software systems, with more than 68 GB of telemetry in the form of logs, metrics and traces. The model receives a natural-language query and must identify elements of the root cause from the telemetry. The authors report that current models handle only the simplest cases: with the root cause analysis agent (RCA-agent) they designed as a baseline, the best model, Claude 3.5 Sonnet, solved 11.34 per cent of the failures (128, Xu et al., 2025). The models tested were those available in 2024 and the figure describes that generation and that agent scaffold.

RCAEval. Pham and colleagues published 735 failure cases from three microservice systems, covering 11 fault types, with an evaluation environment of fifteen reproducible baselines (129, Pham et al., 2025). The three systems are open benchmark applications of between a dozen and several dozen services, into which the authors injected faults. The baselines are mostly statistical and causal-inference methods and not language models. In the authors' preliminary results on the Train Ticket system, the best average scores were 0.46 and 0.54 on the Avg@5 metric, which they describe as moderate and as leaving ample room for improvement.

Ahmed and colleagues. This study evaluated GPT-3 and GPT-3.5 models on more than 40,000 incidents from Microsoft, recommending root causes and mitigation steps, and added a human evaluation with the actual incident owners (227, Ahmed et al., 2023). Twenty-five incident owners took part, and each scored the model output on a scale of 1 to 5. The mean correctness scores for individual models ranged from 2.28 to 3.16. When the highest score among the several candidate outputs was taken, the mean correctness and readability scores ranged from 3.52 to 4.64, and more than 70 per cent of incident owners gave an overall score of three or above. Owners also reported that the encoder-decoder baselines produced generic recommendations. The result is positive about usefulness as a suggestion and is not a measure of how often a first suggestion is right.

RCACopilot. Chen and colleagues describe an on-call system that matches an incident to a handler by alert type, collects diagnostic information, predicts a root-cause category and produces an explanation. On one year of Microsoft incidents the authors report accuracy of up to 0.766 (130, Chen et al., 2023). The diagnostic information collection component has been in use at Microsoft for more than four years. The task is to predict a category, which is a coarser task than naming the cause of a failure that has no precedent; that comparison is the book's own.

ITBench. Jha and colleagues released a framework of 102 real-world scenarios across three domains. At its ICML 2025 publication, agents built on state-of-the-art models resolved 11.4 per cent of the site reliability engineering scenarios, 25.2 per cent of the compliance and security scenarios and 25.8 per cent of the financial operations scenarios (excluding anomaly detection) (225, Jha et al., 2025).

Reading an accuracy figure

The studies do not agree, and they are not meant to. Table 7.1 states for each what was asked, of what data, and what the result does not show.

Study Task Data and setting Result as reported What it does not show
OpenRCA (2025) Identify root-cause elements from a query 335 failures, 3 enterprise systems, over 68 GB 11.34 % solved by the best model with the baseline agent Newer models or agents; the share of failures that people solve
RCAEval (2025) Locate the root-cause service and indicator 735 injected-fault cases, 3 open benchmark systems Avg@5 of 0.46 and 0.54 for the best baselines on one system Language-model performance; faults that were not injected
Ahmed and colleagues (2023) Recommend root cause and mitigation Over 40,000 Microsoft incidents; 25 owners rated outputs Mean correctness 2.28 to 3.16 per model; over 70 % of owners scored overall usefulness three or above The rate at which a top suggestion is correct
RCACopilot (2023) Predict a root-cause category One year of Microsoft incidents Accuracy up to 0.766 Novel causes outside the known categories
ITBench (2025) Resolve a scenario end to end 102 scenarios 11.4 % of SRE scenarios Performance on a given organisation's estate

Table 7.1: Five studies of automated diagnosis, with the task, the data and what the figure does not show.

Six conditions decide how a figure transfers. The first is the task, which differs between choosing a category, naming a component, giving a reason and resolving the scenario. The second is the data: injected faults on a demonstration application are cleaner and better labelled than the faults of a production estate that has grown for fifteen years. The third is the metric, because top-one accuracy, accuracy among the top five and a human rating of usefulness answer different questions. The fourth is the precedent: a model that has seen similar incidents behaves differently from one that has not, and the authors of RCACopilot and of the Microsoft study worked on historical incidents of one organisation. The fifth is the generation of the model and its scaffold, since results of 2023 and 2025 do not describe models of 2028. The sixth is who judged: the incident owners who rated outputs in the Microsoft study knew the true cause, and an on-call engineer at three in the morning does not.

The book's own conclusion follows from this. A published figure bounds what an organisation can reasonably hope for and says little about what its own assistant will do. The figure that transfers is one measured on the organisation's own closed incidents, by replaying them with the root cause withheld and recording whether the right explanation appears first, whether it appears in the top three and how often a confident wrong explanation is ranked first. Chapter 6 describes golden incident sets for alert correlation, and the same discipline applies here. The replay is repeated whenever the model, the prompt or the corpus changes, because each of those is a change to the instrument.

Outlook

Assistants for retrieval, correlation and drafting in incident tooling are observable: the Microsoft system's collection component has been in production use for more than four years (130, Chen et al., 2023). New benchmark results with newer models are expected, and the right response to each is the reading procedure above. A system that diagnoses novel failures with no person able to verify the explanation is speculation; the benchmarks above report that the systems tested solved a minority of the cases presented to them.

Automation bias and anchoring

When an assistant offers a ranked explanation, the risk is less that it is sometimes wrong than that the team stops considering anything else. Three bodies of research bear on this, and each should be read for what it measured.

What the research measured

Parasuraman and Manzey reviewed empirical studies of complacency and automation bias. They report that automation bias results in omission and commission errors when decision aids are imperfect, that it occurs in both naive and expert participants, that it cannot be prevented by training or instructions, and that it can affect decision making in individuals as well as in teams (218, Parasuraman & Manzey, 2010). Complacency occurs under multiple-task load, when manual tasks compete with the automated task for attention, and cannot be overcome with simple practice. An incident bridge is a multiple-task environment.

Goddard, Roudsari and Wyatt reviewed 74 studies that met their criteria, from 13,821 retrieved papers, on automation bias in clinical decision support (134, Goddard et al., 2012). They identified user factors, attitudes towards the system such as trust and confidence, and environmental factors such as workload, task complexity and time pressure as mediators, and mitigators that include training and emphasising user accountability, and system designs that vary the position of advice, show updated confidence levels and present information and not explicit recommendations. The setting is clinical, and the transfer to operations is the book's inference. The mediators, workload and time pressure, are those of an incident.

Two experiments bear on explanations. Bansal and colleagues found that AI help produced complementary gains over either the human or the AI working alone, but that explanations did not increase those gains. Explanations made people more likely to accept the AI's recommendation whether or not it was correct (135, Bansal et al., 2021). Buçinca, Malaya and Gajos tested three cognitive forcing designs on 199 participants and found that they significantly reduced overreliance compared with two simple explainable-AI approaches, and also that the designs that reduced overreliance most were the least favourably rated by participants (136, Buçinca et al., 2021). The implication is that an assistant that shows its reasoning does not by that act protect its users, and that the controls that work impose a cost that users dislike.

Anchoring was described by Tversky and Kahneman as adjustment from an initial value in numerical estimates, with a bias that persists after adjustment (137, Tversky & Kahneman, 1974). The studies concerned numerical estimation. That a first-ranked hypothesis works as an anchor for a team is the book's hypothesis, in line with the mechanism and with the reports above, and it is tested in the organisation's own seeded exercises and not assumed.

The AI Act names the hazard. Article 14(4)(b) requires that, for high-risk AI systems, the natural persons assigned to oversight be enabled to remain aware of the possible tendency of automatically relying or over-relying on the output of the system (automation bias), in particular where the system gives recommendations for decisions to be taken by natural persons (1, European Union, 2024). Article 14(4) goes on to ask that those persons can correctly interpret the output, can decide not to use the system or to disregard, override or reverse its output, and can interrupt it through a stop button or similar procedure. An operations assistant is in general not a high-risk AI system. A system used as a safety component in the management and operation of critical digital infrastructure appears in Annex III point 2, and whether a given assistant is such a component is a classification decision that the AI Decision Record of the use case states. Where the Article does not apply, it remains a usable checklist of what oversight of a recommending system consists of, and the book uses it as one.

A short debate

The case for concern. An incident is the setting in which every mediator of automation bias is present: workload, time pressure and a stream of competing tasks. A ranked list with a confident top entry removes the effort of generating alternatives, and a team that is tired and wants the incident to end accepts it. The research shows that training and instruction do not prevent the bias and that explanations can raise acceptance of wrong answers. Bainbridge's analysis adds that the people who supervise the assistant have fewer opportunities to practise the diagnoses that it makes routine.

The case for calm. The Microsoft incident owners rated model suggestions as useful, and in some cases they reported a suggestion that pointed to a root cause that was hard to detect and had been found only in the post-incident phase (227, Ahmed et al., 2023). A ranked list with evidence for and against is not a verdict, and responders who work with precedent every day know to check it. Teams already live with the equivalent bias towards the last change and the loudest dashboard. An assistant that lists three candidates with their evidence may widen the field of explanations that a tired team considers.

Where the positions meet. Both accept that suggestion quality varies with the incident, that the rank is not a probability unless calibrated, that the cost of acting on a wrong first hypothesis is highest for actions that are slow to reverse, and that the effect of the assistant on the team is measurable in the team's own exercises. Both accept that the controls which reduce overreliance cost time and goodwill and should be applied in proportion to the consequence of the action.

What can be done. Make the first move a person's, show evidence before verdicts, require a second hypothesis and a refuting test before consequential actions, seed exercises with a wrong but plausible hint and measure how often it is caught, and lower the assistant's level when the catch rate falls. The next subsection sets these out as design.

Designing against anchoring

Five measures follow, each tied to the foundations the book takes over from earlier volumes. The Transformed Security Architect, Chapter 14, concluded that verifying what AI proposes requires review that does not start from the AI's answer, independent reproduction, seeded errors and an oversight-quality metric. AI Blue, Chapter 14, added a critique agent on a different model family for questions and not verdicts, a stop rule that freezes delegation when the catch rate falls, and caps and rotation so that review stays real at volume (300, Nordqvist, 2026); (302, Nordqvist, 2026). AI Governance, Chapters 10 and 21, concluded that human oversight is designed and not declared, and that approvals given in seconds are a signal of rubber-stamping (296, Nordqvist, 2026).

  1. Evidence before verdict. The assistant's first screen for the commander shows what changed, similar past incidents and the telemetry it used. The ranked list is released when the commander and the operations lead have each recorded a hypothesis, or ten minutes after declaration, whichever comes first. The default is the book's own and can be adapted to the incident class; Goddard and colleagues report presentation of information, and not explicit recommendations, as a mitigator in the clinical setting.
  2. Several hypotheses, with evidence against. The list always has at least three candidates, each with the evidence that supports it and the evidence that does not. A single answer is never displayed.
  3. The second-hypothesis rule. Before an action of a class above annotate, notify or scale is taken on the strength of the assistant's top-ranked hypothesis, a person records a second hypothesis and one observation that would refute the first, and the result. The observation must come from a source that the assistant did not select, such as the GitOps log read directly by the operations lead. The rule is a cognitive forcing function in the sense of the experiment above, and its cost is minutes.
  4. Seeded incidents. In game days and replays, the facilitator plants a plausible wrong hint and records whether the commander catches it. A catch rate that falls below the threshold in the delegation profile lowers the assistant's ceiling for ranking to Level 0 Advisory until a new test passes.
  5. Oversight quality, measured for the team. The record carries the share of incidents with a second hypothesis, the share in which the commander departed from the top-ranked hypothesis and the time between the ranking being shown and a consequential action. These are reported by team and service. Under KBV-IT-2024-OPS-02 at Hansa Mutual and MBL-NR-IT-2027-01 at Nordvik Region they are never reported for a named engineer, and Hansa applies a minimum group size of five.

The human decision points

The assistant draws a line across the diagnosis, and decisions on the human side of it are made by named people. Table 7.2 lists them, with the highest level at which the assistant takes part. The levels are Level 0 Advisory to Level 4 Autonomous (AI Red Team, Chapter 6) applied per action class as Chapter 4 defines. Level 4 Autonomous is used for none of these, and the thesis prohibition applies without exception to any action that disables the observation or the stop path.

Decision Holder Assistant's highest level Reason
Open a candidate incident On-call engineer Level 2 Supervised The candidate is a draft record, inert until a person confirms it in one action; it starts no roles, clocks or pages, so the cost of a false candidate is low
Declare an incident and set the severity Incident commander Level 2 Supervised Declaration starts roles, clocks and records
Escalate to a service owner, a provider or management Incident commander Level 1 Assisted Assistant drafts and looks up contacts
Accept a hypothesis as the basis for action Operations lead with the commander Level 0 Advisory Subject to the second-hypothesis rule
Approve a rollback, failover or restoration Named approver per the delegation profile Level 0 Advisory Chapters 9 and 10
Classify the incident for NIS 2 or DORA Named person with legal and risk functions Level 1 Assisted (drafting) Chapters 14 and 23
Stop the assistant Commander, agent operator, platform engineer on call None Stop path outside the assistant's platform

Table 7.2: The human decision points in diagnosis and the assistant's ceiling for each.

Declaring

The SRE Workbook advises declaring early and records that undeclared incidents led to miscommunication, slower identification of the root cause and late involvement of relevant teams (133, Mace et al., 2018). An assistant can lower the cost of declaring by drafting a candidate record with the signal and the likely service, so that the commander has only to confirm. The candidate is a draft: it takes effect only when a person approves it, which is why Level 2 Supervised is the ceiling for the class. It does not declare. Declaration has consequences that are not diagnostic: it assigns roles, starts the record and, for services in scope of the regulations, can start reporting clocks.

Clocks and the first flag

NIS 2 Article 23(4)(a) requires an early warning of a significant incident without undue delay and in any event within 24 hours of becoming aware of it, and an incident notification within 72 hours (3, European Union, 2022). DORA Article 17 requires a financial entity to record all ICT-related incidents, to assign roles and responsibilities for each type of incident, to inform the management body of at least major incidents and to establish response procedures, and Article 18 sets the criteria by which incidents are classified, among them the number of clients affected, the duration, the geographical spread, data losses, the criticality of the services affected and the economic impact (2, European Union, 2022). Whether an organisation became aware of an incident at the moment an assistant first flagged it, or at the moment a person confirmed it, is a legal question for the organisation and its counsel, and this book does not answer it. The architectural consequence is the book's own: the incident record carries a timestamp for the assistant's first flag, for the first human acknowledgement and for the declaration, from a trusted clock, so that whichever interpretation applies can be shown. Classification for regulatory purposes is made by a named person and the assistant at most drafts (AI Blue, Chapter 19, reached the same rule for the security case (302, Nordqvist, 2026); Chapters 14 and 23 here).

From diagnosis to action

A hypothesis is a reason to act and is not an authority to act. The decision to roll back, fail over or restore belongs to the delegation profile of the service and to the runbook that governs the action (Chapter 10). Where an agent at Level 3 Bounded executes a catalogued runbook after its own diagnosis, the observer principle of Chapters 4 and 17 applies: what verifies the diagnosis must be independent of what produced it, and the stop path must work when the platform does not. This chapter limits itself to the human case and to the assistant that cannot act.

The record

An incident record that contains only a timeline cannot be used to learn how the assistant behaved. The record in this book has five parts. The timeline marks the origin of every entry: monitoring, a named person, the assistant or a system. The hypotheses are listed with their status (open, refuted, supported, confirmed), who proposed them, the evidence for and against and the test that changed the status. The assistant's contributions are listed by kind (retrieved, correlated, ranked, drafted) with the identifier and version of the model and prompt, the sources it retrieved, the rank scores it gave and whether each was right, wrong or not used. The decisions carry their holders, times and the delegation profile under which they were taken. The clocks carry one timestamp per event listed above. A wrong ranking is retained in the record and never overwritten by a corrected one, because it is the data from which the next replay is built.

Records are written as files in version control or in the service management tool with an export, so that they can be diffed, queried and handed to a supervisor. Chapter 12 describes how they feed the post-incident review and problem management, and Appendix A holds the fuller schema.

The record as a state machine

A record's state, and not a sentence in a free-text field, is what should gate action. Service management platforms already model the lifecycle this way. The ServiceNow incident state model in the Zurich release documentation lists six states: New, In Progress, On Hold, Resolved, Closed and Canceled. On Hold carries a required reason (Awaiting Caller, Awaiting Change, Awaiting Problem or Awaiting vendor), and Closed follows a period in Resolved once resolution is confirmed (209, ServiceNow, 2026). A ServiceNow community article, which is not product documentation, describes how permitted moves are defined: each transition has an enter condition, a state may have an exit condition, and a model-level common exit condition applies to every transition, all of which must hold for a move (210, Martinez, 2026). Jira Service Management documents a list of default statuses, among them Under investigation, Awaiting CAB approval, Implementing and Completed, but its page gives names and no definitions (211, Atlassian, 2026). The general notation is older: SCXML defines states whose transitions are triggered by events and conditioned by guards (212, World Wide Web Consortium, 2015).

For this book the consequence is design, not description. A state changes only through a named role. Closure is a guard that requires evidence. An agent's permitted transitions are listed in its mandate and enforced as transitions of the tool the organisation already runs, not as a convention. The official documentation consulted for this chapter does not list change states for either product, so the change and closure states of Chapters 8 and 12 are the book's own model, to be mapped onto whatever the tool offers.

Worked example: incident record INC-HM-2028-0871

The record below is illustrative and fictional. It shows a Hansa Mutual incident in which the assistant's retrieval helped, its ranking was wrong and the second-hypothesis rule found the cause.

id: INC-HM-2028-0871          # illustrative
service: SVC-HM-001           # claims core, Tier 1, DORA CIF
slo: SLO-HM-001               # p95 claim registration under 3 s
commander: Tobias Richter
ops_lead: sre-hamburg-day-2
assistant: {use_case: HM-OPS-002, aidr: AIDR-0951, level: 1,
            model: ops-assist-2028.3, prompt: inc-v14, tools: [read]}
clocks:                       # trusted time, +01:00
  first_signal: "2028-11-20T08:41"
  assistant_first_flag: "2028-11-20T08:43"
  human_ack: "2028-11-20T08:44"
  declared: "2028-11-20T08:46"
timeline:
  - {t: "08:41", by: monitoring, e: "SLO burn-rate alert, registration latency"}
  - {t: "08:43", by: assistant, kind: drafted, e: "Candidate incident record drafted (inert until confirmed; human ack 08:44)"}
  - {t: "08:50", by: commander, e: "Hypothesis recorded: database lock waits"}
  - {t: "08:52", by: assistant, kind: retrieved,
     e: "INC-HM-2027-0833, PRB-HM-0019: config push, canary skipped", used: true}
  - {t: "08:53", by: ops_lead, e: "Hypothesis recorded: slow consumers on registration queue"}
  - {t: "08:56", by: assistant, kind: ranked, e: "Ranking released after human hypotheses"}
  - {t: "09:03", by: commander, e: "Proposes rollback of CHG-HM-6041"}
  - {t: "09:03", by: commander, e: "Second-hypothesis rule invoked"}
  - {t: "09:09", by: ops_lead, e: "GitOps log read directly: CHG-HM-6034 at 07:36"}
  - {t: "09:12", by: ops_lead, e: "Queue depth began rising 07:52, before CHG-HM-6041 at 08:25"}
  - {t: "09:15", by: commander, e: "Approves emergency rollback CHG-HM-6043"}
  - {t: "09:36", by: monitoring, e: "p95 below 3 s; probe from independent region"}
hypotheses:
  - {id: H1, text: "DB index maintenance CHG-HM-6041", by: assistant,
     rank: 1, score: 0.71, status: refuted, test: "queue growth predates change"}
  - {id: H2, text: "Claims-service memory pressure", by: assistant,
     rank: 2, score: 0.12, status: refuted}
  - {id: H3, text: "Consumer pool limit CHG-HM-6034", by: assistant,
     rank: 3, score: 0.09, also_by: ops_lead, status: confirmed}
assistant_contributions: {retrieval: useful, correlation: listed_both_changes,
                          ranking: top1_wrong, draft_updates: 2_approved}
decisions: [{what: emergency_rollback, holder: commander, dlg: DLG-HM-OPS-02}]
impact: {sev: 2, start: "08:41", end: "09:36", minutes: 55}

Three features of the record should be copied. The assistant's retrieval and its ranking are marked separately, since the first was useful and the second was wrong, and a single rating of the assistant would hide that. Both changes were listed by the assistant's correlation, and the ranking placed the nearer in time first. The rollback is a person's decision under a named delegation profile.

Practical guide: running an incident with an assistant

Purpose. To keep the commander in command and the record honest when an assistant takes part. Roles. Incident commander, operations lead, scribe (a person, or the assistant's draft approved by the scribe), service owner, agent operator. Defaults below are the book's and must be adapted to the contract, the works agreement and the organisation's risk appetite.

  1. Before the incident. Configure the assistant's tool surface as read and annotate only. Confirm that the stop path, which the commander, the agent operator and the platform engineer on call hold, works from outside the assistant's platform; test it at least quarterly. Confirm that the incident text corpus excludes raw personal data.
  2. At declaration. The commander confirms or rejects the candidate record the assistant opened and records the time. The assistant shows evidence: what changed, similar incidents, the telemetry it used. It does not yet show a ranking.
  3. First hypotheses. The commander and the operations lead each record one hypothesis in the incident document. The ranking is released when both have done so or ten minutes after declaration, whichever comes first.
  4. Read the ranking critically. Check each cited precedent by opening the record and comparing the mechanism. Look for the candidate with the evidence against it that you most want to ignore.
  5. Before any action above annotate, notify or scale. Apply the second-hypothesis rule. A second hypothesis and one refuting observation from a source the assistant did not select are recorded, with the result. The rule takes at most ten minutes. Where a runbook at Level 3 Bounded governs the action, the runbook's own preconditions replace it (Chapter 10).
  6. Approve and act. The named approver under the delegation profile approves; the operations lead acts; an independent probe, not the assistant's reading of the same dashboard, confirms the effect.
  7. After. The scribe marks each assistant contribution as useful, not used or wrong, and the record is attached to the post-incident review (Chapter 12). Add the incident to the replay set.

Definition of done. The record has the five parts above, every consequential action has a second hypothesis and an independent confirmation, and the assistant's wrong contributions are retained.

A fragment of the delegation profile gives the rule an enforceable form:

dlg_fragment: DLG-HM-OPS-02   # illustrative; the full profile is in Chapter 22
assistant: HM-OPS-002
ceilings:
  retrieve: 1
  correlate_changes: 1
  rank_hypotheses: 1
  draft_internal_update: 2
  open_candidate_incident: 2
  declare_incident: 2
  external_communication: 0
display_policy:
  ranking_release: {after_human_hypotheses: 2, or_minutes_from_declaration: 10}
  min_candidates: 3
gate:
  action_classes_requiring_second_hypothesis: [reconfigure, deploy, roll_back, fail_over, delete, restore_over_existing]
  evidence_source_must_differ_from_assistant: true
stop_rule:
  seeded_catch_rate_below: 0.8
  on_breach: {rank_hypotheses: 0}

Across the usage modes

The table gives, for each usage mode, how it appears in diagnosis, the highest level of delegation in the book's profile and the main control.

Usage mode In diagnosis Highest level Main control
Mode 0 Ambient AI Summaries and priority suggestions inside the monitoring and service management tools, and in a provider's own monitoring Not set by the organisation Contract terms for notice and test rights (Chapter 24); the suggestion is never the sole basis for declaration
Mode 1 Conversational assistants An engineer pastes an error message into an approved enterprise assistant Level 0 Advisory Data classes allowed; no incident text above internal
Mode 2 Embedded workplace assistants An incident summariser in the observability platform Level 1 Assisted Summaries read by the scribe before use; the vendor's model changes noticed
Mode 3 Agentic work tools An engineer's tool queries logs and writes search expressions under the engineer's identity Level 1 Assisted Read-only credentials; no action tools in an incident
Mode 4 Built AI applications The incident assistant: retrieval, change correlation, ranking, drafting Level 1 Assisted, Level 2 for drafting and candidate records Read-only tools, second-hypothesis rule, replay and seeded tests
Mode 5 Autonomous production agents An operations agent diagnoses in order to select a runbook Governed by its runbooks (Chapter 10) Independent confirmation of the diagnosis before action; observer principle (Chapter 17)

Table 7.3: Diagnosis across the usage modes.

Nordvik Region and Hansa Mutual

Both organisations met the same question in September 2028, and the assistant is one of the places where their answers differ.

Nordvik Region. The operations assistant NR-OPS-001 (AIDR-0901) is a retrieval tool over runbooks, past incidents and the knowledge base, self-hosted on the Region's reserved capacity and at Level 0 Advisory. The night technical on-call, one of eleven engineers on a weekly rota, and the duty manager who acts as incident commander for Tier 1 incidents (Oskar Berg is one of four) use it to find precedent: the certificate-expiry runbook written after INC-NR-2026-0071, the failover notes of INC-NR-2027-0034 and the review of the DNS fault of INC-NR-2027-0298. It ranks nothing. The reasons are those of the Region's constraints: seven platform engineers who carry SRE practice, no Mode 5 agent and a strong preference for operational data to stay on its own infrastructure. The Region records on each incident the time from declaration to the first relevant document, as a figure for the team. Under MBL-NR-IT-2027-01, any tool that logs engineers' queries is negotiated before it is switched on, and the assistant's query log is therefore held to the incident record and not to a profile of the person. For INC-NR-2027-0298, SamDrift's status page showed the fault 22 minutes before the Region's own monitoring did, which is a lesson for the observer and not for the assistant (Chapter 17). The Region plans as if its Tier 1 services were in scope of NIS 2 as transposed nationally, so the record carries the same three timestamps for the first flag, the first acknowledgement and the declaration.

Hansa Mutual. The incident assistant HM-OPS-002 (AIDR-0951) runs at Level 1 Assisted, with read-only tools over past incidents, runbooks and the change calendar, and the group-wide incident commander rota of 12 trained people uses it. Before AIDR-0951 was approved, the site reliability function replayed 60 closed incidents with the cause withheld; the assistant's first-ranked hypothesis was right in 24 and a correct hypothesis was among its first three in 41 (illustrative figures of the example organisation). Those numbers are the basis of the Level 1 ceiling and of the stop rule's threshold. INC-HM-2028-0871 above shows the pattern the replay predicted: retrieval was valuable, ranking was wrong when two changes were close together in time, and the rule caught it. The commander was Tobias Richter; the stop right for the assistant follows the pattern of the IT-operations agent and is held by the commander, the agent operator and the platform engineer on call. Metrics on oversight quality are reported by team under KBV-IT-2024-OPS-02, with the minimum group size of five. DORA applies to Hansa as an insurance undertaking, so the incident record is also the evidence for Article 17 and for the classification of Article 18.

Where they differ is the ceiling and the evidence behind it. Nordvik chose retrieval because it has little data and few people to run seeded exercises, and it measures what retrieval saves. Hansa chose ranking because it has a replay set and a rota large enough to run seeded exercises. Neither organisation lets the assistant act, and both record its contribution.

Summary

This chapter concerned O2 Diagnose, with inputs from O1 Observe and outputs to O3 Change and O4 Restore. Diagnosis is the hypothetico-deductive work of triage, hypothesis formation, testing and handover, done under the command structure that the SRE literature describes. The incident assistant takes part by retrieving precedent, correlating changes, ranking hypotheses and drafting updates, with read-only tools, and it does so as a source of evidence and not as an authority.

Published studies show a map of difficulty and not a rate. OpenRCA reports 11.34 per cent of 335 failures solved by the best model with its baseline agent, RCAEval's best baselines reach Avg@5 of 0.46 and 0.54 on injected faults, ITBench reports 11.4 per cent of its SRE scenarios resolved, RCACopilot reports accuracy of up to 0.766 on a year of one organisation's incidents, and Microsoft's incident owners found model suggestions useful while rating their correctness at 2.28 to 3.16 on a scale of five. The figure that transfers is the one measured on the organisation's own incidents, repeated whenever the model, the prompt or the corpus changes.

Automation bias cannot be removed by training or instruction, explanations can raise acceptance of wrong answers, and the controls that work cost time. The chapter's design is evidence before verdict, several hypotheses with evidence against, a second hypothesis and a refuting test from an independent source before consequential action, seeded incidents with a stop rule, and oversight quality measured for the team. The decision points are human, declaration and regulatory classification included, and the record marks every assistant contribution, including the wrong ones.

On the outlook labels, assistants for retrieval and drafting in incident tooling are observable, further benchmark results with newer models are expected and diagnosis of novel failures without a person able to verify it is speculation. What an organisation can influence is its own replay set, its seeded exercises and its ceilings. The reader can start on Monday by replaying ten closed incidents with the cause withheld and writing down the result. Chapter 8 turns to O3 Change, where much of the change that diagnosis must explain is now authored by agents.

Sources

  1. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024. (1, European Union, 2024)
  2. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (2, European Union, 2022)
  3. European Parliament and Council of the European Union. Directive (EU) 2022/2555 (NIS 2 Directive). (3, European Union, 2022)
  4. European Parliament and Council of the European Union. Regulation (EU) 2016/679 (General Data Protection Regulation). (6, European Union, 2016)
  5. National Institute of Standards and Technology. Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile. NIST SP 800-61 Rev. 3. (41, NIST, 2025)
  6. Bainbridge, L. (1983). Ironies of Automation. Automatica 19(6). (217, Bainbridge, 1983)
  7. Parasuraman, R. & Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors 52(3). (218, Parasuraman & Manzey, 2010)
  8. Nordqvist, G. (2026). AI Governance. The Transformed Architecture, Book 2. (296, Nordqvist, 2026)
  9. Nordqvist, G. (2026). The Transformed Security Architect. The Transformed Architecture, Book 6. (300, Nordqvist, 2026)
  10. Nordqvist, G. (2026). AI Blue. The Transformed Architecture, Book 8. (302, Nordqvist, 2026)
  11. Xu, J. et al. (2025). OpenRCA: Can Large Language Models Locate the Root Cause of Software Failures? ICLR 2025. (128, Xu et al., 2025)
  12. Pham, L., Zhang, H., Ha, H., Salim, F. & Zhang, X. (2025). RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data. arXiv:2412.17015; WWW '25 Companion. (129, Pham et al., 2025)
  13. Ahmed, T. et al. (2023). Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models. ICSE 2023; arXiv:2301.03797. (227, Ahmed et al., 2023)
  14. Chen, Y. et al. (2023). Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. arXiv:2305.15778. (130, Chen et al., 2023)
  15. Jha, S. et al. (2025). ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks. arXiv:2502.05352. (225, Jha et al., 2025)
  16. Treynor Sloss, B. (2017). Introduction. In Site Reliability Engineering, Google and O'Reilly Media. (245, Treynor Sloss, 2017)
  17. Jones, C. (2017). Effective Troubleshooting. In Site Reliability Engineering, Google and O'Reilly Media. (131, Jones, 2017)
  18. Stribblehill, A. (2017). Managing Incidents. In Site Reliability Engineering, Google and O'Reilly Media. (132, Stribblehill, 2017)
  19. Mace, J. et al. (2018). Incident Response. Chapter 9 of The Site Reliability Workbook, Google and O'Reilly Media. (133, Mace et al., 2018)
  20. Goddard, K., Roudsari, A. & Wyatt, J. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. JAMIA 19(1). (134, Goddard et al., 2012)
  21. Bansal, G. et al. (2021). Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. CHI 2021. (135, Bansal et al., 2021)
  22. Buçinca, Z., Malaya, M. B. & Gajos, K. Z. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proc. ACM Hum.-Comput. Interact. 5(CSCW1). (136, Buçinca et al., 2021)
  23. Tversky, A. & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science 185(4157). (137, Tversky & Kahneman, 1974)
  24. ServiceNow (2026). Incident state model. ServiceNow product documentation, Zurich release. (209, ServiceNow, 2026)
  25. Martinez, D. (2026). Understanding State Models and Transitions. ServiceNow Community blog. (210, Martinez, 2026)
  26. Atlassian (2026). Issue statuses, priorities, and resolutions. Jira Service Management Cloud documentation. (211, Atlassian, 2026)
  27. World Wide Web Consortium (2015). State Chart XML (SCXML). W3C Recommendation. (212, World Wide Web Consortium, 2015)