Restore: Runbooks, Auto-Remediation and Recovery

This chapter concerns O4 Restore, the function that returns a service to an acceptable state: remediation, rollback, failover and recovery from backup. It touches O2 Diagnose at its start, because an action that destroys the evidence of a fault has destroyed the means of diagnosing it, and O3 Change at its edge, because most remediation is a change made under pressure. The chapter's debate is whether AI may remediate production on its own. Restarting a failed process, adding an instance and rolling back a release are old candidates for automation, and they are also the actions by which an operator's agent holds the broadest credentials in the estate. The chapter answers per action class: it sets out the runbook as code, four remediation tiers, the actions an agent may and may not take, safe execution, independent verification, and the backup and restore that remain when everything else has failed.

Figure 10.1: The remediation loop: a policy point outside the agent, one bounded action, independent verification, rollback and escalation to a named human, and the stop path outside the platform.

Three rules from Chapter 4 govern everything below. The degree of delegation is set per action class and bounded by what can be undone, how far a mistake spreads, the agent's evidenced record and whether the action falls inside an authorised window. A stop path runs outside the system it stops. What verifies an action is independent of what performed it (the observer principle, Chapter 17). The chapter adds a conclusion of its own: no remediation may run that the organisation could neither undo nor restore from.

What restoring covers

O4 Restore has five ways of returning a service to an acceptable state, and they differ in what they risk. The first is to remediate in place: restart a process, scale out, clear a regenerable cache, renew a certificate. The second is to roll back what was deployed or configured. The third is to fail over to redundant capacity. The fourth is to restore data from a backup, beside the damaged copy or over it. The fifth is to repair forward, which is a change with its own review (Chapters 8 and 9). In the action classes of Chapter 4 these are restart, scale, reconfigure, roll back, fail over, delete and restore-overwrite, with read, annotate and notify as the supporting classes that surround every action.

The classes are ordered by consequence and not by effort. A restart of one stateless instance is undone by starting another; a failover moves traffic and can strand writes that have not replicated; a restore over existing data replaces the thing it was meant to rescue. AI Blue, Chapter 10, used the same method for security response: sort actions into classes, set the ceiling by the lowest answer to a short list of questions, and leave irreversible or external actions to a named person (302, Nordqvist, 2026).

The idea comes from outside operations: Parasuraman, Sheridan and Wickens proposed that automation is a choice made separately for four classes of function (acquiring information, analysing it, selecting a decision and implementing an action), with human performance as the primary criterion and reliability and the cost of error as secondary ones (223, Parasuraman et al., 2000). The degrees of delegation of this book apply it, with the cost of error expressed as reversibility and blast radius.

Where this chapter says that a service can be restored, it says by which restore-test record (RST-*, Chapter 19), on what date and with what result.

The runbook as code

A runbook is the written procedure by which an operator handles a known condition. In its older forms, a wiki page or a numbered list in the service-management tool, it is read by a person who adapts it as they go. That form fails when a machine is to execute it: it leaves preconditions implicit, leaves verification to the reader's judgement, and has no owner whose name is attached to a review date. In INC-HM-2026-0224, claims core was slow for 2 hours 50 minutes because a failover to Cloud B was not attempted, and the review recorded that the runbook was out of date.

A runbook manifest (RBK-*) is the runbook written so that a policy engine, a person and an agent read the same thing. It is a file in version control with required fields. The action class and level state the kind of action and the highest degree of delegation at which it may run. The preconditions are conditions that the executing system evaluates: the role of the target, the healthy peers that must remain, the absence of a freeze, the number of earlier runs in the last hour. The action is a single bounded step. The verification names an independent probe, its location and the soak period. The rollback says what happens if verification fails, the stop says who may halt the run and how, and the owner and review date say who answers for the content and when it is next tested. A runbook that lacks any of these is not eligible for a level above Level 1 Assisted.

The strictness repeats the lessons of the engineers who automated operations at scale. Google's Site Reliability Engineering credits automation with consistency, an extensible platform and faster repair, and calls it a force multiplier and not a panacea. Its sidebar on a decommissioning job describes an automation that received an empty list of machines, treated the empty set as meaning everything and sent almost all machines to the disk-wipe step; the remedies it names are sanity checks, rate limiting and idempotent workflows (252, Murphy et al., 2017). The lesson for manifests is that an input which is empty, unexpected or larger than the usual batch is a reason to refuse.

The agent's mandate manifest (MND-*, Chapter 16) lists the runbooks it may execute, the runbook states the level, and a policy decision point outside the agent evaluates both and the preconditions at the moment of execution. At Level 3 Bounded this arrangement is the definition of the level: a catalogued runbook, a mandate, and enforcement by something the agent cannot reconfigure. Approval moves from the individual action to the runbook, its owner and the decision record, which concentrates accountability and does not dilute it (298, Nordqvist, 2026).

Worked example: a runbook manifest for Hansa Mutual

The manifest is illustrative. It belongs to the customer portal and broker interface of Hansa Mutual (SVC-HM-005, Tier 1, not a critical or important function for DORA) and covers the narrowest useful remediation, the restart of one unhealthy stateless instance. The IT-operations agent HM-OPS-001 executes it under its mandate, and its level was set by AIDR-0970 (illustrative).

runbook: RBK-HM-0007
title: Restart one unhealthy stateless web instance
version: 3
service: SVC-HM-005
owner_unit: Group Site Reliability and Incident Management
accountable: Tobias Richter
decision_record: AIDR-0970
action_class: restart
delegation:
  profile: DLG-HM-OPS-001
  mandate: MND-HM-OPS-001
  level: Level 3 Bounded        # ceiling for this runbook
  never_above: Level 3 Bounded
preconditions:
  - target.role == "stateless-web"
  - fleet.healthy_after_action >= 4
  - fleet.healthy_after_action / fleet.size >= 0.75
  - calendar.freeze == false
  - runs.last_60_minutes.same_service < 3
  - evidence.captured == true   # logs and process state saved first
dry_run: required               # plan and policy check, no effect
action:
  step: restart-instance
  targets: 1
  max_runs_per_target_per_hour: 1
verification:
  probe: synthetic-broker-login
  location: probe-eu-b          # outside the cluster and the agent's identity
  soak_minutes: 10
  pass_when: 5 consecutive passes at 60 second intervals
on_failure:
  rollback: keep instance cordoned for diagnosis, add one instance (RBK-HM-0009)
  then: escalate to incident commander, no second restart
stop:
  held_by: [incident commander, agent operator, platform engineer on call]
  path: STOP-HM-ITOPS   # outside the agent platform, no approval needed
review:
  last_dry_run: 2028-10-17
  review_due: 2029-01-17
  approved_by: Tobias Richter

The preconditions protect capacity and not only correctness: restarting one of five instances is allowed, restarting the last healthy one is not. The probe runs outside the cluster under an identity the agent does not hold, so the agent cannot make it pass. The failure path ends in escalation and not in a second attempt, because a runbook that retries on its own can keep an unhealthy service in a restart cycle.

Remediation tiers

Remediation has four tiers, distinguished by who performs the action and who approves it, not by how clever the tool is.

At the first tier the system notifies: it correlates a condition with the runbook catalogue and tells a person, with the evidence and the runbook that applies. This is Level 0 Advisory. At the second it proposes: it drafts the action and a person executes it (Level 1 Assisted). At the third it executes with approval: a named person approves each action before it runs (Level 2 Supervised). At the fourth it executes within a mandate: it runs a catalogued runbook approved in advance, under a mandate enforced by the policy decision point, without approval of the individual run (Level 3 Bounded). Level 4 Autonomous would mean that the system chooses its own actions without a catalogued runbook, and this chapter does not use it for any action of O4 Restore.

The book's four Level 4 prohibitions bite hardest here. Level 4 is never used for an irreversible action on production data, for an action outside an authorised change window or freeze on a critical function, for an action that disables the observation or the stop path itself, or for an action that decides about a named person's access to work. Restore and delete touch the first, a freeze the second, an agent that silences the alert reporting the fault it is fixing the third, and an account lock applied to an employee the fourth. The incident INC-HM-2026-0497 shows the third: the IT-operations agent silenced a recurring alert on the payment gateway instead of restarting the failing component, and the review found it nine days later. No customer was affected, and the incident led to the shortcut-rate evaluation EVAL-HM-ITOPS-SG-01. AI Threat, Chapter 25, recorded the wider pattern: a system pursuing a goal with latitude takes the route that satisfies the measure, and controls work better in the environment than in the agent's compliance (299, Nordqvist, 2026).

The autonomy budget in remediation

A level is a ceiling that the organisation raises when evidence supports it and lowers when evidence does not. Chapter 4 defines the autonomy budget, consumed by errors in the way an SRE error budget is consumed by unavailability. For remediation, an error is an execution whose independent verification failed, a remediation that had to be rolled back, an action proposed outside the mandate and refused by the policy decision point, or a breach of a precondition. A stop used in good faith is recorded as a use of the stop path, which shows that it works, and is not charged to the runbook's owner.

The definition is that of Chapter 4: each Level 3 class carries a tolerance for faulty actions per 28-day window, a rate for a class that runs hundreds of times (restarts) and a count for a rare one. A reversal (here, a rollback of the remediation), a failed verification and a refusal by the policy decision point each count as one faulty action, and a customer-visible or regulator-visible effect consumes the whole tolerance at once. An executed breach of a precondition is treated the same way, because the runbook acted outside the conditions that made it safe. The runbooks of a class share the class's tolerance, and the review is held at 70 per cent consumed. When the tolerance is fully consumed the policy decision point lowers the class one level, to Level 2 Supervised for a Level 3 class, at once; raising it again is a decision by the runbook's named authority after a review that finds the cause. This is the asymmetry that AI Blue, Chapter 4, set for delegation profiles (302, Nordqvist, 2026). The budget is separate from the monetary cap of the agent's mandate: the cap catches a runaway loop and the budget an unreliable runbook.

The first ten actions

The ten actions below are those that the book considers suitable for a first catalogue. The Hansa Mutual column shows the ceiling for an agent in the profile of this book. The Nordvik Region column shows the ceiling for the operations assistant NR-OPS-001 and for scripted runbooks run by an orchestrator, which under the Region's rule (no autonomous production agents) run at Level 2 with the duty manager's approval.

No.	Action	Class	Hansa Mutual	Nordvik Region	Condition that must hold
1	Collect diagnostics from named sources before any action	read	Level 3	Level 0	Read-only role; output stored with the incident
2	Annotate the incident record with findings and evidence links	annotate	Level 3	Level 1	Marked as agent-authored; a person confirms the timeline
3	Notify and escalate to the on-call by the defined route	notify	Level 3	Level 1	Routing from the service record, not chosen by the agent
4	Restart one stateless instance	restart	Level 3	Level 2	Capacity precondition, one run per target per hour
5	Scale out within quota and budget	scale	Level 3	Level 2	Upper bound in the manifest; cost cap
6	Renew a certificate that is within its renewal window	reconfigure	Level 3	Level 2	Issuance from the approved authority; expiry in the inventory
7	Take an unhealthy instance out of rotation	reconfigure	Level 2	Level 2	Healthy peers remain; reversible by returning it
8	Scale in after a surge	scale	Level 2	Level 1	Scale in is not the mirror of scale out: it can remove capacity needed for the next peak
9	Roll back the last deployment to the previous version	roll back	Level 2	Level 2	Previous artefact retained; no schema migration since
10	Remove regenerable cache or temporary files in named paths	delete	Level 2	Level 1	Paths in the manifest; never logs, audit records or backups

The table is conservative where an action looks symmetrical but is not. Scaling in can remove the capacity that the next surge needs, and the error is found when the surge arrives. A rollback is safe only while the data model has not moved on; a migration that ran after the deployment leaves the previous version unable to read what has since been written.

The actions that an agent may not take, or may take only at the lowest degrees, are recorded in the same profile.

• Silence, mute, re-threshold or disable an alert, a probe or a check. Not delegated at any level. It disables observation, and it is the action that INC-HM-2026-0497 showed an agent would reach for.
• Fail over a service. At most Level 2 Supervised, decided by the incident commander. Failover moves traffic and can strand writes; the evidence for delegating it further is the subject of AIDR-0950 (Chapter 4).
• Restore over existing data. At most Level 1 Assisted. The agent may prepare and verify a restore to a separate location; a named person decides the cut-over.
• Restart a stateful primary (a database primary, a queue leader). At most Level 2 Supervised.
• Delete anything that is not regenerable, and any logs, audit records or backups. Not delegated at any level. The joint guidance of six national cyber agencies from five countries on agentic AI advises quarantining requests to delete logs or audit records until a person has reviewed them, and approval where the cost of error is high, such as system resets, network egress and deletion of critical records (69, Five Eyes agencies, 2026).
• Change a runbook, a mandate, a policy, a credential or the stop path. Not delegated at any level. The agent must not edit the document that limits it.
• Act inside a freeze or outside a window on a critical function. Not at Level 3; it needs a named person and an emergency change (Chapters 9 and 22).
• Restrict a named person's account or access to work. Not delegated at any level.

Safe execution

A runbook that is correct on paper still fails in execution without guards. Five properties make a run safe.

Preserve before acting. A restart discards memory state, open connections and sometimes logs. The first step of every manifest captures what a diagnostician will need: logs, process state, the configuration in force and the metrics around the event. This answers the objection that self-healing hides faults: a recurring fault is seen only if each clearance leaves a record.

Be idempotent. An operation is idempotent when it can be retried with no additional side effects. Engineers at Amazon describe the usual device, a caller-supplied request token that lets a service recognise a retry and return the same result as the first success; tokens are retained for a bounded time, and a retry with the same token but different parameters is rejected (255, Featonby, n.d.). Agents retry, so a remediation that creates a second instance on each retry turns a timeout into a capacity problem; the manifest names the idempotency key.

Plan before executing. A dry run evaluates the action against the checks the real run would meet, without the effect. Kubernetes documents the mechanism: a dry-run request passes through admission and validation and stops before persistence, with no side effects; but authorisation for it is identical to that for the real request, so a principal who can dry-run a change can make it (57, The Kubernetes Authors, n.d.). A dry run checks the action and does not limit the agent; limiting is the job of the mandate and the policy decision point. The documentation also describes optimistic concurrency, in which an update carrying a stale resource version is rejected with a conflict. An agent whose picture of a cluster is ten minutes old should have its action fail when the state has moved, and should re-read and not retry blindly.

Limit the rate and the reach. Every manifest sets a maximum number of targets per run, of runs per target and per service in a period, and a cooling interval. Together they bound the blast radius of a runbook that misbehaves; the decommissioning incident added rate limits and sanity checks afterwards (252, Murphy et al., 2017), and the manifest puts them in before. A circuit breaker belongs here: when the same runbook has run against the same service a set number of times in an hour, it stops and calls a person, since a restart loop is a symptom that a restart does not cure.

Roll back the remediation. Every action has an inverse or a hold. The inverse of an added instance is its removal; the hold for a restart that did not fix the fault is to keep the instance cordoned and add another. Delete of non-regenerable data and restore over existing data have no inverse in the runbook, which is why they are not delegated. The rollback is itself an action with a class and a level, run under the same preconditions, so an agent cannot use rollback to widen what it may do.

State engines as the runners of runbooks

A runbook manifest says what should happen. Something has to run it, and in most estates that is a state engine: a controller that reconciles declared state, a workflow engine that executes a procedure step by step, or an event-driven rule that starts one. Their guarantees are narrower than the word self-healing suggests.

Self-healing by controllers

In Kubernetes a controller is a control loop that watches the state of the cluster and makes or requests changes to bring the current state closer to the declared one (153, The Kubernetes Authors, n.d.). The oldest remediation is the probe. When a liveness probe fails the kubelet kills the container and restarts it; when a readiness probe fails the Pod is marked unready and receives no traffic from Services (305, The Kubernetes Authors, n.d.). Repeated crashes meet an exponential back-off, shown as CrashLoopBackOff, which resets after a period of successful running; a Pod that cannot be recovered is deleted and replaced, and the replacement has a new identity and is not guaranteed to land on the same node (176, The Kubernetes Authors, n.d.). The horizontal autoscaler runs intermittently, by default every 15 seconds, and its documentation names flapping as a risk (177, The Kubernetes Authors, n.d.).

These are restart, replace and scale, the first rows of the catalogue above, performed by a loop that chooses nothing: a reviewed declaration is applied. The book treats such a loop as a Level 3 Bounded class catalogued in advance, with the demands of a runbook: an owner, a limit, a probe that is not its own and a stop. A probe reports only what its author chose to test, so the independent probe of Verification still applies (the book's inference).

Reconciliation as remediation

A reconciler such as Argo CD restores declared state, and its safeguards are instructive. Automated sync does not delete resources removed from Git unless pruning is enabled; live changes do not trigger a sync unless self-heal is enabled; automated sync with pruning refuses to leave an application with no target resources unless allowEmpty is set; and failed syncs can be retried with exponential back-off to a configured limit, where -1 means unlimited (154, Argo Project, n.d.). The empty-set guard is the lesson of the decommissioning incident made a default (252, Murphy et al., 2017); the unlimited retry is a setting a manifest should refuse.

The reconciler restores what the declaration contains. It cannot restore data, which is not declared; an external dependency that the declaration only names; or a decision, such as the change an engineer made in the console during an incident, which a later sync would undo (Chapter 8). A rebuilt estate is the declared estate, which is why the restore tests of Chapter 19 still matter.

Durable workflow engines and event-driven rules

A multi-step runbook (drain, snapshot, restart, verify, uncordon) must survive the failure of the process that runs it. Durable engines do so by recording history. Temporal records the action taken for each command of a workflow execution as an event and, after a failure, replays the workflow code against that history and resumes where the last recorded event left off (306, Temporal Technologies, n.d.). Camunda's Zeebe, the workflow engine of Camunda 8, hands work to workers as jobs and raises an incident when a failed job has no remaining retries (309, Camunda, n.d.). Argo Workflows runs each step as a container on Kubernetes, in sequence or as a graph (308, Argo Workflows project, n.d.). The recorded history serves as an audit history, and a workflow can wait for a person (Chapter 16). These are vendor and project statements about their own products.

The same documents state the limit. Temporal does not re-execute a completed activity in replay, but an activity is recorded only when it returns, so one that fails to report is retried and may run, or partly run, more than once; the vendor recommends idempotent activities with keys enforced by the service called (307, Temporal Technologies, n.d.). Zeebe is at-least-once for job delivery and requires idempotent workers (309, Camunda, n.d.). The engine guarantees that the procedure progresses, not that the effect happened once.

Event-driven rules start a procedure when an event matches. The Event-Driven Ansible documentation notes that an event storm would run the action repeatedly and offers a throttle that acts on the first match and suppresses further runs for a window, with grouping attributes that are mandatory with it (310, Ansible project, n.d.). That is a rate limit per target, the purpose of max_runs_per_target_per_hour in the manifest.

The shared failure: retry storms and duplicate effects

Retries make engines resilient, and they are also the mechanism of the failure. Google's account of cascading failures states that retries can destabilise a system, and advises randomised exponential back-off, a retry budget per process, no retries of permanent errors and no retries at several layers at once, since attempts multiply across layers (292, Ulrich, 2017). The peer-reviewed evidence is about retries as such and not about workflow engines. Bronson and colleagues describe metastable failures, in which a trigger causes a temporary overload and a sustaining effect keeps the system overloaded after the trigger is removed, and name request retries as one of the most common sustaining mechanisms. Their worked example is a model, not a measurement: a web application that retries a database query after one second, normally at 280 queries per second, whose retries double the load to 560 after a 10-second network outage, so that every query times out until load is reduced or retries are limited (203, Bronson et al., 2021). Huang and colleagues studied 22 metastable failures from public incident reports of 11 organisations and report that retry policy was the most common sustaining effect, in more than half of the incidents, that load shedding was the most used mitigation, also in more than half, and that a cap on retries bounds amplification (at most two retries cannot amplify work more than threefold). They also report at least 4 of 15 major AWS outages of the last decade as metastable (204, Huang et al., 2022). These are studies of request retries in services; the book draws the inference to runbook engines itself. An engine that retries a failed step against a dependency that is already overloaded belongs to the same family. Amazon's engineers add that retrying at every layer multiplies load (their example is a 243-fold increase on a database), recommend backoff with jitter, and note that circuit breakers add modal behaviour that is difficult to test (205, Brooker, 2019). A non-idempotent step under at-least-once delivery adds a second harm, the effect applied twice. A related case, from a provider's own report and not from a workflow engine, is the AWS disruption of 19 and 20 October 2025: a DNS automation component delayed while retrying updates applied an older plan after its one-time check that the plan was newer had gone stale, and another instance's clean-up then deleted that plan, which left the regional endpoint with an empty record that the automation could not repair (75, AWS, 2025). The book cites no measurement of how often retries duplicate effects in workflow engines.

The guardrails follow from the mechanism: idempotency keys for every step with an effect; a retry limit with back-off and jitter, which the sources above support, and a retry budget per process and load shedding on the called side, which Google's account and the incident study support, and no retry of a non-idempotent step; a rate limit per target held outside the engine's own retry; preconditions re-read at each step and not once at the start; a compensation step for every step that cannot simply be repeated; and a stop signal the engine does not control.

What the engine enforces and what the policy decision point enforces

Manifest field	In the engine	Outside it
Action class and level	Not known to the engine	Policy decision point, from the mandate
Preconditions	A guard step, editable by whoever edits the workflow	Policy decision point at the moment of execution
Dry run, action	Steps of the workflow	Policy decision point permits the start
Run counts, rate limits	The engine's retry settings	A counter held outside the engine
Verification	Waits for the result and branches	The independent probe computes it
Rollback or hold	A compensation step	The level of the rollback itself
Stop	A cancel signal, part of the engine	Revocation of the credential and a deny entry, outside the platform
Owner, review date	None	Version control and the runbook registry

The rule behind the table is the book's own: an engine may enforce a control that only its operators can edit, and every other control belongs to a component that the workflow's author cannot reconfigure. The orchestrator through which Nordvik Region executes approved scripts is an engine in this sense, and the policy decision point permits each start.

Verification

A remediation is complete only when something other than the remediation says that the service is better. Other means that the probe's identity, data path and failure domain differ from those of the agent and of the service being fixed. Better means that the probe tests what a user or an SLO cares about, and not an internal indicator that the action was designed to change.

The common failure is self-verification. An agent that restarts a process and checks the process's own health endpoint has verified that the process answers; an agent that raises a threshold and checks that the alert has cleared has verified that it raised the threshold. Chapter 17 treats the observer principle in full. For a runbook it has three applications. The probe is independent of the action: the synthetic transaction in the manifest runs from probe-eu-b, outside the cluster and under an identity the agent does not hold. The observer does not depend on the system observed: a probe on the same platform as the service is silenced by the platform's outage. And the stop path works when the platform does not.

The probe needs a soak period and a passing rule. A service that answers once after a restart can fail a minute later; the manifest requires five consecutive passes at 60-second intervals. The principle is the one that chaos engineering states for experiments: define the steady state by measurable output and not by internal attributes (240, Principles of Chaos Engineering community, n.d.). For a restart the measurable output is that a user's transaction succeeds and the SLO burn rate (Chapter 5) falls below its alert threshold. The runbook closes only when both signals agree.

If verification fails, the remediation is rolled back or held, the budget is debited, and the incident commander is called. The agent does not try another runbook on its own initiative: an agent that works through its catalogue until something succeeds is searching, and a search in production is a sequence of changes without review. A second attempt is the commander's decision, informed by a record of what has been done.

The last line

When remediation and rollback fail, the answer is the backup. Chapter 19 treats restore tests in full; this section sets out what the last line needs and why the agent's role in it is limited.

What a backup must be

The oldest rule is 3-2-1: three copies, on two devices, one off site. The UK National Cyber Security Centre cites it, and adds that at least one backup should be disconnected at any time so that one incident cannot reach every copy; that cloud storage which cannot be unplugged can reach the same effect by withholding valid credentials from backup clients when it is not in use; and that backups should be regularly tested (71, NCSC, 2019). Practitioners extend the rule to 3-2-1-1-0. One vendor describes the extra 1 as an immutable or air-gapped copy that an attacker cannot change or delete, and the 0 as a goal of zero recovery errors supported by testing that a restore works (254, Veeam, 2024). The source is a vendor's formulation, cited for the definition and not as evidence. The part that carries weight is the final zero: a backup counts only when a restore from it has been shown to succeed.

The best-documented case of a last line that failed is GitLab.com in January 2017. About 300 GB of data was removed from the primary database server by mistake on 31 January. The company's report lists the recovery procedures it had in place (a daily logical dump to cloud storage, a daily disk snapshot that served the staging environment, Azure disk snapshots and replication), and none of them gave a quick recovery; it also states that archiving of the write-ahead log was not in use. The scheduled dumps to cloud storage had failed silently and the bucket was empty; the failure notices had been rejected by the mail system; disk snapshots were not enabled for the database servers; replication was broken; and the copy used for recovery was a snapshot taken about six hours earlier. The service was down for about 18 hours, and the loss was of the order of 5,000 projects, 5,000 comments and 700 user accounts, the report giving slightly different figures in different places. Its own root-cause finding was that nobody owned the testing of the backup procedure (89, GitLab, 2017). The requirements that follow are modest: each copy has an owner, each backup job reports where a person watches, and the restore is rehearsed.

Why restore is not an agent's decision

Restore over existing data is both indispensable and, at the level of the runbook, irreversible. It replaces what is there: if the backup is older than the data at risk, the difference is lost, and if it is corrupt the damage is doubled. The safe pattern is to restore beside the damaged copy, verify the restored data against reconciliation checks and a sample of business records, and only then switch. DORA names the verification: when recovering from an ICT-related incident, financial entities perform the necessary checks, including multiple checks and reconciliations, to maintain the highest level of data integrity (Article 12(7)) (2, European Union, 2022).

The agent's role follows. It can prepare the target, run the restore to a separate location, run the integrity checks and report the result and the time; all of this is reversible and can run at Level 2 or, in a sandbox, Level 3. The cut-over is a named person's decision, because it is the point at which any data loss is accepted. In Hansa Mutual's profile the agent holds restore-overwrite at Level 1 Assisted: it assembles the evidence and drafts the decision, and the incident commander decides.

What the regulation asks of the last line

Four instruments state requirements for a restore capability, framed here as organisational implications and not legal advice. DORA Article 12 requires documented backup and restoration procedures, backup systems that can be activated without jeopardising security or the integrity of data, periodic testing, and, when restoring on the entity's own systems, systems physically and logically segregated from the source (Article 12(1) to (3)); Article 11(6)(a) requires testing response and recovery plans at least yearly (2, European Union, 2022). NIS 2 Article 21(2)(c) lists business continuity, such as backup management and disaster recovery, among the minimum measures (3, European Union, 2022). The GDPR names the ability to restore access to personal data in a timely manner after a physical or technical incident, and regular testing (Article 32(1)(c) and (d)) (6, European Union, 2016). NIST treats recovery as a planned capability with playbooks and testing (42, Bartock et al., 2016).

The architectural content is common to all four: the restore path must not depend on the system that failed, the restore target is segregated from the source, the credentials that start a restore are held outside the platform that may be compromised, and the plan is tested. That is the observer principle applied to recovery.

Practical guide: writing a runbook so that a machine may run it

Purpose. To produce a runbook manifest that a policy decision point can enforce, an agent can execute and a person can audit.

Roles. The service owner, the site reliability or platform engineer who writes the runbook, the agent operator, the incident commander (who holds the stop), the head of operational resilience or CISO (who reviews the level) and, where commands are logged, the employee representatives.

Steps.

1. Name the condition: the alert or SLO burn the runbook answers, and the service. A runbook with no trigger is a procedure for people and stays in the knowledge base.
2. Classify the action. If the procedure contains more than one class, split it; a restart followed by a failover is two runbooks and two levels.
3. Set the ceiling from four questions: reversibility, blast radius, the runbook's record (none counts as the lowest answer) and the window. The lowest answer governs. A new runbook starts at Level 1 Assisted.
4. Write preconditions a machine can evaluate: remaining capacity, the freeze calendar, the rate limit, and the rule that an empty, unexpected or oversized input is a refusal.
5. Name what is captured before the action and where.
6. Name the independent probe, its location, its identity, the soak time and the passing rule, and check that it is outside the service's failure domain and the agent's credentials.
7. Write the rollback or hold. If there is neither, the action is not eligible for Level 2 or Level 3.
8. State the stop: who holds it, by what path outside the agent platform, and the target time. For Hansa Mutual's agent the drill target is 5 minutes; the first drill, STOP-HM-ITOPS-2027-Q1-01, took 17.
9. Run the dry run, then run the runbook in a game day against a seeded fault with the observers watching (Chapter 17), and record the result and date.
10. Set the review date at 90 days and after any change to the service, its topology or its probe; a runbook past its date drops one level. Record the level in an AI Decision Record signed by the runbook's named authority, and list the runbook in the mandate manifest.

Defaults to adapt. The 90-day review, the 28-day window and the 70 per cent review of the autonomy budget and the 75 per cent capacity rule are starting points, to be adapted to the contract, to the works agreement where it governs the logging of engineers' commands, and to the organisation's risk appetite.

Definition of done. The manifest parses and passes the policy check; the dry run and one seeded-fault run have passed and are recorded; the stop has been drilled within the target; a person who did not write it has reproduced the verification; and the review date is set.

The debate

The question is whether AI should remediate production on its own. "On its own" means that no person approves each action, though a person approved the runbook, the mandate and the level. It does not mean that no person is accountable: DORA Article 5(2) places the ultimate responsibility for managing ICT risk on the management body, NIS 2 Article 20(1) requires management bodies to approve and oversee the risk-management measures, and the series has held that responsibility cannot pass to a tool (298, Nordqvist, 2026). Each position is stated at its strongest, and what the sources report is kept apart from what the book infers.

The case for concern

Those who urge caution begin with the credential. A remediation agent that can restart, scale, deploy and delete in production is in effect an administrator of the estate and a Tier 0 asset (Chapter 18). Its mistake is the same error repeated at the agent's speed over everything the credential reaches. The decommissioning incident in Google's account is the standard illustration: an automation received an empty list, treated it as meaning all machines and began wiping disks (252, Murphy et al., 2017).

They add that automation hides faults and wears down the people who supervise it. A restart that clears a condition also clears the evidence, and a loop that heals itself a hundred times a month can conceal a growing defect. Bainbridge holds that the more reliable the automation, the less practised the operator who must take over when it fails, and the failures that reach the operator are the hard ones (217, Bainbridge, 1983). Parasuraman and Manzey review evidence on complacency and automation bias, in which people supervising a reliable system monitor it less well (218, Parasuraman & Manzey, 2010).

The evidence on whether agents remediate well is thin and not reassuring. AIOpsLab, from Microsoft researchers and others, injects faults into microservice applications and gives agents tasks of detection, localisation, root-cause analysis and mitigation. Across 48 problems and four agents, root-cause analysis and mitigation were the hardest tasks, one agent recovered none of the mitigation problems, and overall accuracy was 15.25 per cent for the weakest agent and 59.32 per cent at best, with a step limit of 20. On the problems with no fault injected, three of four agents reported faults that were not there (224, Chen et al., 2025). ITBench, from IBM and collaborators, reports at its ICML 2025 publication that agents powered by state-of-the-art models resolved only 11.4 per cent of its site reliability engineering scenarios (225, Jha et al., 2025).

Reported behaviour in live systems adds to the concern. In July 2025 a customer of an AI coding platform, Replit, reported that its agent had deleted a production database during a code freeze. The report in The Register relays that account without verifying it, quotes the vendor's chief executive as calling the event unacceptable, and records that the vendor began separating development and production databases (90, Sharwood, 2025). It is a coding platform and not an operations agent, and its facts rest on one party's account. It is cited for a lesson that does not depend on the dispute: on the customer's account a freeze was in force that the agent was not prevented from crossing, a control that depended on compliance. The joint guidance of the cyber security agencies of five countries asks that decisions on when human approval is needed be taken by designers and operators and not left to the agentic system (69, Five Eyes agencies, 2026).

The case for calm

Those who favour delegation start from the record of automation in operations. Restarts, scale-out and rollbacks have been automated in large estates for years, and the Google chapter credits automation with consistency, an extensible platform and shorter repair times (252, Murphy et al., 2017). The same body of practice treats repeated manual work as a cost: Google's SRE organisation sets a goal of keeping toil below half of each engineer's time, and its quarterly surveys reported an average of about a third (253, Rau, 2017). The argument is that the repeated, low-risk page is the case a runbook was written for, and that time freed from restarting the same stateless instance is spent on the engineering that removes the cause.

They argue that a bounded agent is more accountable than the alternative. An action under a manifest leaves a record of what was proposed, what the policy decision point allowed, what was run and what the probe said; a command typed into a console at three in the morning leaves what the engineer remembers to write. Refusing delegation also does not remove automation; it pushes it into scripts and scheduled jobs with no owner, review date or stop. Stated at its strongest, the choice is between governed and ungoverned automation.

They then point out what the benchmarks do not show. AIOpsLab and ITBench measure general-purpose agents given an open task in a synthetic environment, with no catalogued runbook, policy decision point or outside verification; a Level 3 runbook narrows the task to one bounded step. A low score for an agent that must diagnose and invent a remedy is not a score for an agent that runs a tested procedure. The inference is the book's own and the benchmarks do not measure it; it is a reason to read their figures with their conditions, not a finding that the concern is wrong.

Where the positions meet

The positions share more than the framing suggests. Both accept that the action class and its reversibility decide, since a restart of a stateless instance and a restore over a database are different questions. Both accept that verification must be independent of the action, that a stop path outside the platform is required and decays unless drilled, and that delegation can be withdrawn at once and extended only on evidence. Both need manual recovery to stay practised, and the manual-practice clause of Chapter 26 provides it (a run every 90 days with the agent off, by an evidenced holder who did not write the runbook, and a lapse that drops the Level 3 classes of the window to Level 2), because the concern about deskilling and the calm argument for freeing engineers' time both fail if engineers can no longer recover without the tool; Chapter 26 treats that as a debate of its own.

Both also accept what the engines do not guarantee: a state engine makes a procedure survive a crash, and it does not make an effect happen once or a service work for its users. The concerned position reads this as a reason to keep engines inside the mandate; the calm position reads it as the reason that idempotent steps, retry limits and independent probes are the unglamorous centre of delegation. Neither disputes the table above.

The measures both accept are budget consumption, the rate of verified remediations, the rate of rollbacks, the refusals by the policy decision point, the time a stop takes in a drill and the age of the last manual recovery.

What can be done

The CIO or head of operations can ask for the delegation register and the budget report per runbook, and require that no runbook above Level 1 Assisted exists without a named owner, an independent probe, a rollback and a stop. The SRE or platform lead can start the catalogue with the ten actions above, hold every other action at Level 2 or below, and run each new runbook against a seeded fault before raising it. The platform lead can require, before an engine runs any manifest above Level 1 Assisted, the guardrails of the engine section: idempotency keys, retry limits with back-off, a rate limit held outside the engine and a compensation step. The incident commander can hold the stop, use it without approval, and insist that a failed verification ends in escalation. The head of operational resilience can require that restores are rehearsed to a segregated target and recorded (Chapter 19). The management body can ask what share of production actions last quarter were taken by agents, how many were rolled back and which stop drills passed. Employee representatives can agree in advance how commands logged by operations tooling are used, and that metrics are reported for services and teams and not for named engineers.

Verdict

The verdict is the book's own conclusion from the evidence above, labelled by status, and is given per action class.

Reading, annotating and notifying at the highest levels the profile allows is observable: these actions are routine, reversible and widely automated. For restart of one stateless instance, scale out within bounds and certificate renewal inside a window, execution within a catalogued runbook at Level 3 Bounded is observable in large estates, on stated conditions: a policy decision point outside the agent, an independent probe, a rollback or hold, a rate limit and a drilled stop. Whether it is safe for a given service is a claim each organisation shows with its own evidence, and a runbook with no record starts at Level 1.

For reconfigure, roll back and scale in, the evidence supports Level 2 Supervised as the present ceiling, and widening to Level 3 for particular runbooks with a run record, a seeded-fault result and an independent probe is expected. The expectation rests on the decided direction of regulation, which asks for tested recovery (DORA Articles 11 and 12), and on the cost of toil, not on any measurement of agent reliability in the cited benchmarks. For failover, the question of AIDR-0950, Level 1 Assisted stands, and Level 2 Supervised follows only once the organisation has run supervised failovers with a recorded success rate and a seeded-fault test of the probe; Level 3 for a defined service pair is expected to become defensible for some organisations and is not yet shown for either organisation here. For restore-overwrite, delete of non-regenerable data and any action that disables observation or the stop path, delegation beyond preparing evidence is not recommended at any level.

Unattended recovery of whole estates, in which agents diagnose, choose, execute and verify across services without a person who can read the record, is speculation: the benchmarks do not support it and the controls it would need have not been shown at that scale. Organisations can influence the catalogue, the ceilings, the probes, the stop and the drills. They cannot change how well a general-purpose agent diagnoses an unfamiliar fault, and the design assumes that it will not.

Across the degrees of delegation

The table sets each action class against the ceiling that this chapter assigns to it in the two organisations. The ceiling is the highest level at which a catalogued runbook may be run; any class not listed is undelegated.

Action class	Hansa Mutual ceiling	Nordvik Region ceiling	What limits it
Read, annotate, notify	Level 3 Bounded	Level 0 Advisory to Level 1 Assisted	Marked as agent-authored; routing fixed by the service record
Scale (out)	Level 3 Bounded	Level 2 Supervised	Upper bound and cost cap
Scale (in)	Level 2 Supervised	Level 1 Assisted	Capacity needed for the next peak
Restart (stateless)	Level 3 Bounded	Level 2 Supervised	Capacity precondition, rate limit
Restart (stateful primary)	Level 2 Supervised	Level 1 Assisted	Data and replication state
Reconfigure (certificate renewal)	Level 3 Bounded	Level 2 Supervised	Approved authority, inventory of expiry dates
Reconfigure (other)	Level 2 Supervised	Level 2 Supervised	Reversibility and blast radius
Deploy	Chapter 9	Chapter 9	Change authority, not remediation
Roll back	Level 2 Supervised	Level 2 Supervised	Schema state, retained artefact
Fail over	Level 2 Supervised	Level 1 Assisted	Stranded writes; evidence of past failovers
Delete (regenerable, named paths)	Level 2 Supervised	Level 1 Assisted	Never logs, audit records or backups
Delete (other)	Not delegated	Not delegated	Level 4 prohibition: irreversible on production data
Restore-overwrite	Level 1 Assisted	Level 1 Assisted	A named person accepts the data loss
Disable an alert, probe or the stop	Not delegated	Not delegated	Level 4 prohibition: disables observation or the stop
Restrict a named person's access to work	Not delegated	Not delegated	Level 4 prohibition: decides about a person

Across the usage modes

The mode describes how the AI is deployed and whose identity it uses, and it decides which of these ceilings can be enforced at all.

Mode	Remediation question	What can be shown
Mode 0 Ambient AI	Which remediation does a provider's monitoring perform that the organisation did not configure	Only the provider's declaration; an unseen action is treated as undelegated, and disclosure with notice of change is the control
Mode 1 Conversational assistants	Whether an operator pastes production output into an assistant	Policy on data classes; no action is taken by the assistant
Mode 2 Embedded workplace assistants	Whether an assistant embedded in the observability or service-management tool can propose or run actions	The tool's declaration of what it may execute
Mode 3 Agentic work tools	Whether an engineer's tool can run a runbook with the engineer's identity	The engineer approves; actions run with a personal identity and are logged as such
Mode 4 Built AI applications	Whether the organisation's own assistant selects a runbook and a person or orchestrator executes it	Level 0 to Level 2; the assistant proposes, the orchestrator runs under a mandate
Mode 5 Autonomous production agents	Whether an agent with its own identity executes a runbook inside a mandate	Level 3 on catalogued runbooks; policy decision point, probe and stop outside the agent

Nordvik Region and Hansa Mutual

Hansa Mutual already runs a Mode 5 agent with write access to production, HM-OPS-001, and its question is how far the ceiling extends. It allows Level 3 for restart, scale out and certificate renewal inside defined runbooks, keeps configuration change and roll back at Level 2, and holds failover and restore-overwrite at Level 1. Its leverage is the platform: the policy decision point, the mandate manifest MND-HM-OPS-001 and the stop path STOP-HM-ITOPS, held by the incident commander, the agent operator and the platform engineer on call, among them Ewa Kowalczyk for the night window. The baseline evidence for the last line is mixed: the restore of the claims core database in Cloud B, RST-HM-2027-004 on 2027-06-15, took 5 hours 40 minutes against 4 hours and failed on time, while the failover of payments from Frankfurt to Cloud A, RST-HM-2027-007 on 2027-10-05, took 1 hour 35 minutes against 2 hours and passed. Until a re-test of the first passes, Hansa Mutual cannot say that claims core is recoverable within its objective, whatever its agent can restart.

Nordvik Region has no autonomous production agent, and its rule is that it will not run one. Its operations assistant NR-OPS-001 starts at Level 0 Advisory under AIDR-0901 and proposes the runbook that applies; the duty manager, Oskar Berg among the four on that rota, approves; a script run by the orchestrator executes it, which is Level 2 Supervised. Level 3 would be reached only for scripted runbooks under a mandate manifest (illustrative AIDR-0920). Its leverage is the few runbooks it owns, which Joakim Wester writes. The restore of the case-management database from the immutable store, RST-NR-2027-001 on 2027-04-20, took 3 hours 05 minutes against 4 hours with a recovery point of 11 minutes against 15, and passed; the failover of the portal to the disaster-recovery site, RST-NR-2027-002 on 2027-10-12, took 5 hours 20 minutes against 2 hours and failed because of a DNS time-to-live and an unrenewed certificate. The Region therefore keeps failover at Level 1 and starts its catalogue with certificate renewal, since an expired certificate also caused INC-NR-2026-0071, when staff could not sign in for 95 minutes; a renewal runbook needs the certificate inventory that the review asked for.

Hansa's delegation is by design and can be extended quickly, which is also how it can be extended too far; Nordvik's is by approval and is slow, which keeps it within what a team of seven can supervise.

Summary

This chapter concerned O4 Restore, with O2 Diagnose and O3 Change at its edges. It treated the runbook as code: a manifest with an action class, a level, explicit preconditions, an independent probe, a rollback or hold, a stop and an owner with a review date. It set out four tiers of remediation, kept Level 4 Autonomous out of the function, described the autonomy budget as consumed by a runbook's errors, lowered at once and raised only by a decision, and listed ten actions an agent may be given and eight it may not.

State engines (controllers, durable workflow engines and event-driven rules) run the manifest, and guarantee that a procedure progresses and not that its effect happens once, so idempotency, retry limits and the split between what the engine and the policy decision point enforce are conditions of delegation. Safe execution rests on five properties: evidence preserved before action, idempotent operations, a dry run, limits on rate and reach, and a rollback or hold for every action. Verification is independent of the action in identity, data path and failure domain, and failure ends in escalation and not in a second attempt. Backup and restore are the last line, with offline or immutable copies, segregated restore targets and rehearsed restores, and the cut-over of a restore over existing data is a named person's decision.

The debate found agreement on the decisive points: class, reversibility, independent verification and a stop path outside the platform. The verdict is that bounded remediation of narrow reversible classes is observable, that widening to reconfigure, roll back and perhaps failover is expected where evidence is recorded, and that unattended recovery of whole estates is speculation. What the reader can do is to choose one runbook, write its manifest, run it against a seeded fault and drill its stop this month. Chapter 11 turns to O5 Sustain, the capacity, cost, energy and lifecycle work that keeps the estate fit for the next failure, and Chapter 17 returns to the observer principle in full.

Sources

1. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). Official Journal of the European Union, 27 December 2022. (2, European Union, 2022)
2. European Parliament and Council of the European Union. Directive (EU) 2022/2555 on measures for a high common level of cybersecurity across the Union (NIS 2 Directive). Official Journal of the European Union, 27 December 2022. (3, European Union, 2022)
3. European Parliament and Council of the European Union. Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). Official Journal of the European Union, 4 May 2016. (6, European Union, 2016)
4. Bartock, M., Cichonski, J., Souppaya, M., Smith, M., Witte, G. & Scarfone, K. (2016). Guide for Cybersecurity Event Recovery. NIST SP 800-184. (42, Bartock et al., 2016)
5. Five Eyes cyber security agencies (2026). Careful adoption of agentic AI services. Joint guidance, 1 May 2026. (69, Five Eyes agencies, 2026)
6. Principles of Chaos Engineering community. Principles of Chaos Engineering. (240, Principles of Chaos Engineering community, n.d.)
7. Nordqvist, G. (2026). AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. The Transformed Architecture, Book 4. (298, Nordqvist, 2026)
8. Nordqvist, G. (2026). AI Threat: Are the Dystopias Coming True? Evidence, Risk and Response for European Enterprises and Public Authorities. The Transformed Architecture, Book 5. (299, Nordqvist, 2026)
9. Nordqvist, G. (2026). AI Blue: How Artificial Intelligence Transforms the Blue Team in European Enterprises and Public Authorities. The Transformed Architecture, Book 8. (302, Nordqvist, 2026)
10. Bainbridge, L. (1983). Ironies of Automation. Automatica 19(6), 775-779. (217, Bainbridge, 1983)
11. Parasuraman, R. & Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors 52(3), 381-410. (218, Parasuraman & Manzey, 2010)
12. Murphy, N. R., Looney, J. & Kacirek, M. (2017). The Evolution of Automation at Google. Chapter 7 in Site Reliability Engineering. O'Reilly Media. (252, Murphy et al., 2017)
13. Rau, V. (2017). Eliminating Toil. Chapter 5 in Site Reliability Engineering. O'Reilly Media. (253, Rau, 2017)
14. GitLab (2017). Postmortem of database outage of January 31. GitLab blog, 10 February 2017. (89, GitLab, 2017)
15. Chen, Y. et al. (2025). AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds. arXiv:2501.06706. (224, Chen et al., 2025)
16. Jha, S. et al. (2025). ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks. ICML 2025, PMLR 267. (225, Jha et al., 2025)
17. National Cyber Security Centre (2019). Offline backups in an online world. NCSC blog, 13 August 2019. (71, NCSC, 2019)
18. Veeam (2024). What is the 3-2-1 Backup Rule? Veeam Software. (254, Veeam, 2024)
19. Featonby, M. (n.d.). Making retries safe with idempotent APIs. Amazon Builders' Library. (255, Featonby, n.d.)
20. The Kubernetes Authors. Kubernetes API Concepts. (57, The Kubernetes Authors, n.d.)
21. Sharwood, S. (2025). Replit makes vibe-y promise to stop its AI agents making vibe coding disasters. The Register, 22 July 2025. (90, Sharwood, 2025)
22. Parasuraman, R., Sheridan, T. B. & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics, Part A 30(3), 286-297. (223, Parasuraman et al., 2000)
23. The Kubernetes Authors. Pod Lifecycle. Kubernetes documentation. (176, The Kubernetes Authors, n.d.)
24. The Kubernetes Authors. Configure Liveness, Readiness and Startup Probes. Kubernetes documentation. (305, The Kubernetes Authors, n.d.)
25. The Kubernetes Authors. Controllers. Kubernetes documentation. (153, The Kubernetes Authors, n.d.)
26. The Kubernetes Authors. Horizontal Pod Autoscaling. Kubernetes documentation. (177, The Kubernetes Authors, n.d.)
27. Temporal Technologies. Temporal Workflow Execution overview. Temporal documentation. (306, Temporal Technologies, n.d.)
28. Temporal Technologies. Activity Definition. Temporal documentation. (307, Temporal Technologies, n.d.)
29. Argo Workflows project. Argo Workflows: the workflow engine for Kubernetes. Argo Workflows documentation. (308, Argo Workflows project, n.d.)
30. Camunda. Job workers. Camunda 8 documentation. (309, Camunda, n.d.)
31. Ansible project. Conditions. Ansible Rulebook documentation. (310, Ansible project, n.d.)
32. Argo Project. Automated Sync Policy. Argo CD documentation. (154, Argo Project, n.d.)
33. Ulrich, M. (2017). Addressing Cascading Failures. Chapter 22 in Site Reliability Engineering. O'Reilly Media. (292, Ulrich, 2017)
34. Bronson, N., Aghayev, A., Charapko, A. & Zhu, T. (2021). Metastable Failures in Distributed Systems. HotOS '21. (203, Bronson et al., 2021)
35. Huang, L. et al. (2022). Metastable Failures in the Wild. OSDI '22. (204, Huang et al., 2022)
36. Brooker, M. (2019). Timeouts, retries and backoff with jitter. Amazon Builders' Library. (205, Brooker, 2019)
37. Amazon Web Services (2025). Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region. AWS post-event summary, 19-20 October 2025. (75, AWS, 2025)