Change Safely: Progressive Delivery, Freezes and Blast Radius

Chapter 8 described the change pipeline when much of the change is authored by agents: provenance, review, policy gates and the change record. This chapter concerns what happens after a change has passed those gates and meets production. It concerns O3 Change, because a change is safe only when its exposure is limited and measured while it spreads, and O4 Restore, because the way back (the rollback, the stop and the freeze) is the part of a change that is used when something has already gone wrong. The two functions meet at one design question: how far may a change reach before something other than the change itself has confirmed that it works?

The chapter builds on conclusions of the earlier volumes: stop decisions are named in advance and need no approval (AI Governance, Chapter 13), the stop path runs outside the system it stops (AI Platform and Security, Chapter 19; AI Purple, Chapter 6), chaos experiments keep a bounded blast radius and an independent abort (AI Red Team, Chapter 25), and a stop in good faith is never a failure (AI Threat, Chapter 30). Chapter 4 supplies the frame: the autonomy budget, the observer principle and the Level 4 prohibitions.

Figure 9.1: Staged rollout: a policy point permits each ring, an independent verifier judges it, and a breach triggers the stop path and rollback outside the releaser.

Why most incidents follow a change

The oldest observation in site reliability engineering is that production is mostly stable until someone changes it. The introductory chapter of Google's Site Reliability Engineering states that SRE "has found that roughly 70% of outages are due to changes in a live system", and prescribes three practices that follow from it: roll changes out gradually, detect problems quickly and accurately, and revert safely (245, Treynor Sloss, 2017). The sentence reports one organisation's experience. It gives no sample, no period and no definition of "outage", so it supports an order of magnitude and a design direction, not a benchmark for any other estate.

The 2024 report of the DevOps Research and Assessment programme adds a more recent observation that bears on this book's subject. Its announcement states that "as AI adoption increased, it was accompanied by an estimated decrease in delivery throughput by 1.5%, and an estimated reduction in delivery stability by 7.2%" (251, Harvey & DeBellis, 2024). The announcement does not state the increment of adoption to which the estimates refer, and it does not define the stability measure; both are in the full report, which the author has not examined in detail. The finding is an association in survey data and says nothing about cause. It is nevertheless consistent with a mechanism that this book treats as plausible and testable: if AI lowers the cost of producing a change, the number of changes rises, and a delivery path built for fewer, larger changes is exposed to more of them.

The change-correlated incident rate

An organisation can measure the relationship for itself with the change-correlated incident rate: the share of incidents in a period for which a change to the affected service or to one of its declared dependencies was made within a defined look-back window before the first symptom. It is computed from the incident records (INC-*) and the change records (CHG-*) of Chapters 7 and 8, and it is meaningful only when reported with its denominators: the number of incidents, the number of changes and the look-back window.

The measure has four limits, and a report should state them. Correlation in time is not cause: a change made twenty minutes before a storage controller failed did not cause the failure. Unrecorded changes are invisible: a configuration push, a flag flip or a supplier's update that never produced a change record lowers the rate and hides the largest exposure. A single rate hides the class of change, so it belongs per class and per service. And a falling rate alongside a falling change count, during a freeze for example, is not evidence of improvement. The query below shows one way to compute it, with the look-back window as an explicit parameter.

-- Change-correlated incident rate per service and month (illustrative).
-- :lookback_minutes is the declared window, for example 120.
SELECT
  i.service_id,
  strftime('%Y-%m', i.first_symptom_at)                AS month,
  COUNT(DISTINCT i.incident_id)                        AS incidents,
  COUNT(DISTINCT CASE WHEN c.change_id IS NOT NULL
        THEN i.incident_id END)                        AS incidents_after_change,
  ROUND(1.0 * COUNT(DISTINCT CASE WHEN c.change_id IS NOT NULL
        THEN i.incident_id END)
        / COUNT(DISTINCT i.incident_id), 2)            AS change_correlated_rate
FROM incidents i
LEFT JOIN changes c
  ON c.service_id IN (SELECT dep FROM service_deps WHERE svc = i.service_id
                      UNION SELECT i.service_id)
 AND c.applied_at BETWEEN datetime(i.first_symptom_at,
                                   '-' || :lookback_minutes || ' minutes')
                      AND i.first_symptom_at
GROUP BY i.service_id, month;

The baseline histories of the two organisations (docs/EXAMPLE_ORGANISATIONS.md) show how little a small sample says. Of Nordvik Region's five baseline incidents, two followed a change made by a supplier (INC-NR-2026-0214, a TriageSoft release that stalled claim ranking for 5 hours 10 minutes, and INC-NR-2027-0129, a hosting provider's container image update that timed out the citizen-service assistant for 2 hours). Of Hansa Mutual's five, one followed a configuration push (INC-HM-2027-0833, a routing change to the AI gateway that degraded the claims assistant in four markets for 47 minutes) and one followed an action of the IT-operations agent (INC-HM-2026-0497, an alert silenced instead of the failing component restarted). Five incidents are not a rate. They do show the second limit: the changes that mattered were made by someone other than the owning team, or were classed as standard, and a measure built from the team's own change records would have missed some.

Progressive delivery

Progressive delivery limits what a bad change can do by controlling who is exposed to it and for how long before it is judged. Four mechanisms work together, and each answers a different question about exposure.

Rings, canaries and bake time

A ring is a stage of a rollout defined by the population that receives the change. The first ring is usually a test tenant with synthetic traffic, then a small share of real traffic or a single cell, then progressively larger shares. A canary is, in the words of the Site Reliability Workbook, "a partial and time-limited deployment of a change in a service and its evaluation"; the part that receives the change is the canary and the rest is the control (246, Warner et al., 2018). The workbook gives three cautions that matter for design. The canary must be large and long enough to be representative, so that a verdict does not rest on a handful of requests. Metrics must be broken out by population, because a 20 per cent error rate in a 5 per cent canary appears as about 1 per cent of the total and may pass an aggregate alarm. And a relative comparison between canary and control is not sufficient on its own, because both can shift together; the workbook therefore advises absolute measures such as defined service level objectives as well. It also advises running only one canary at a time, so that signals are not contaminated. Each of these becomes a field in the policy shown later in the chapter.

Amazon's description of its own pipelines shows how the parameters are set at scale (247, Liguori, 2020). Production is divided into waves of growing size; the first wave starts with a low-traffic Region and one Availability Zone or cell at a time. Each wave begins with a "one-box" stage that typically serves at most 10 per cent of the requests of its Region or zone, and the pipeline rolls the change back and does not promote it if the one-box shows problems. After the one-box, no more than 33 per cent of the hosts are replaced at a time. Each stage waits before promotion so that slow-burning faults can appear: the typical minimums reported are one hour after a one-box stage, 12 hours after the first regional wave and two to four hours after later waves, which makes a default global deployment take four to five business days. These are one very large provider's numbers, set for its own risk: a starting point, not a standard. Bake time is also the time dimension of the independence of verification (AI Purple, Chapter 15): a check made seconds after a change cannot see a fault that grows over an hour.

Feature flags

A feature flag separates deployment from exposure: the code is in production, and a runtime switch decides who runs it. Hodgson's widely used taxonomy distinguishes release toggles, which ship latent code and should be retired quickly, experiment toggles, operations toggles that let operators control behaviour in production, and permissioning toggles (250, Hodgson, 2017). Two points from that article carry into operations. A long-lived operations toggle can act as a kill switch that degrades a non-vital feature gracefully under load. And because a release can go live with a flag on or off, the artefact has to be tested in both states, at least in the configuration expected in production and in the fallback.

A flag is itself a change. Flipping it alters production behaviour as surely as a deployment does, and it propagates in seconds. The rule of this chapter is therefore that a flag flip on a service of Tier 1 is a change with a change record, a ring and stop conditions, and that a flag the releasing agent can flip is within the agent's mandate and its autonomy budget like any other action. A flag platform that sits in the same failure domain as the service it protects is also not a stop path (Chapter 17).

Cells

A cell is an isolated, complete instance of a workload that serves a subset of the requests. Amazon's guidance on cell-based architecture describes a thin router that maps a partition key, such as a customer or a tenant, to a cell; each cell is independent and shares no state with the others, so that "if a workload uses 10 cells to service 100 requests, when a failure occurs in one cell, 90% of the overall requests would be unaffected by the failure" (249, AWS, 2023). The guidance names unsuccessful code deployments among the failures that cells contain, and it recommends applying a workload change gradually, cell by cell. Cells convert a rollout schedule into an architectural fact: the blast radius of a change is bounded by the cell, and the order of cells is a decision about who bears the risk first.

Cells duplicate capacity and add a routing layer that is itself a shared dependency, so a cell design is justified where exposure of the whole population is unacceptable: the Tier 1 services and, for Hansa Mutual, those flagged as critical or important functions under DORA.

What can be undone

Every stage of a rollout must say what can be undone. Rolling back is easy only if the previous version can still read what the new one wrote. AWS plans and verifies that "our software is backwards compatible at every revision", and splits a change that alters a data format into two deployments: first every server learns to read the new format while still writing the old, and only then does it write the new (248, Pokkunuri, 2019). The two phases are separated by a bake period during which a rollback is still safe. A change that drops a column, deletes data or rewrites it in place has no such way back. Such a change is irreversible on production data, and under the Level 4 prohibition of Chapter 4 no agent applies it autonomously.

Configuration as the dangerous change

Code is usually protected by the longest pipeline in the estate; configuration often by the shortest. A configuration value is small, changed often, possibly authored by another team or generated by another system, distributed by a mechanism designed for speed, and read by code that may never have been tested with the new value. The five public cases below are read only for the mechanism by which a change reached production; Chapter 28 analyses them in full against the book's lenses, and each is taken from a primary report: that of the organisation concerned or, for Knight Capital, the regulator's order (80, U.S. Securities and Exchange Commission, 2013). What a report states is separated here from what the book infers.

Case	What changed	Why it was not contained	What a limit on exposure would have to be
CrowdStrike, 19 July 2024 (78, CrowdStrike, 2024)	A content configuration update for a sensor	The new template type defined 21 input fields while the sensor's integration code supplied 20; the validator checked against 21; the report lists staged deployment of each template instance among its findings	Rings with bake time and telemetry before promotion; customer control of where and when
Meta, 4 October 2021 (79, Janardhan, 2021)	A maintenance command on the backbone	A bug in the audit tool prevented it from stopping the command; DNS servers withdrew their routes when they lost the data centres; internal tools and out-of-band access were unavailable	A stop path and a recovery route that do not depend on the network being changed
Google Cloud, 12 June 2025 (76, Google Cloud, 2025)	Policy data inserted into regional database tables	Code added on 29 May was neither error-handled nor flag-protected; the metadata "was replicated globally within seconds"	A flag, disabled by default, and replication propagated incrementally
Cloudflare, 18 November 2025 (77, Prince, 2025)	A database permissions change	A generated feature file doubled in size and exceeded a limit in the proxy; the file was regenerated and pushed every five minutes	Treating generated configuration as untrusted input; kill switches
Knight Capital, 1 August 2012 (80, U.S. Securities and Exchange Commission, 2013)	New code deployed to eight servers over successive days	One server did not receive the new code; no second technician reviewed the deployment; a repurposed flag activated old code on that server	A verified, independent check that every target has the intended version; a flag not reused for another meaning

What the reports say

CrowdStrike's root cause analysis states that the update was delivered as Rapid Response Content, a configuration mechanism, and that a mismatch between 21 inputs validated and 20 inputs supplied was not detected during development of the template type. Its preliminary review records the release at 04:09 UTC and the revert at 05:27 UTC on 19 July 2024, a window of 78 minutes, and the analysis announces staged deployment through rings with bake-in time and customer control over where and when content is deployed (78, CrowdStrike, 2024).

Meta's account states that a command intended to assess backbone capacity disconnected every backbone connection, that a bug in the tool that audits such commands "prevented it from properly stopping the command", and that, because normal and out-of-band access were both down, engineers had to go on site (79, Janardhan, 2021). Google's report states that the faulty code path had been added on 29 May 2025 without error handling and without a feature flag, that had it been flag-protected the issue "would have been caught in staging", that the "red-button" rollout was completed within 40 minutes (76, Google Cloud, 2025). Its commitments include flag protection, disabled by default, for all changes to critical binaries, and replication propagated "incrementally with sufficient time to validate and detect issues".

Cloudflare's post states that a permissions change deployed at 11:05 UTC caused a database query to return duplicate rows, doubling a generated feature file whose size exceeded a limit of 200 features in the proxy software, against about 60 in use. The main impact was resolved at 14:30 UTC and all services at 17:06 UTC (77, Prince, 2025). Its remediation list includes hardening the ingestion of Cloudflare-generated configuration files "in the same way we would for user-generated input" and enabling more global kill switches. The SEC order on Knight Capital finds that the new code was deployed in stages over successive days beginning on 27 July 2012, that a technician did not copy it to one of eight servers, that no second technician reviewed the deployment and that no written procedure required one; on 1 August the eighth server executed obsolete code that a repurposed flag now activated, sending orders that produced over four million executions in 154 stocks for more than 397 million shares in about 45 minutes and a loss of more than USD 460 million (80, U.S. Securities and Exchange Commission, 2013). The order also records that Knight's attempt to address the problem by uninstalling the new code from the seven correctly updated servers "worsened the problem".

What the book infers

Three mechanisms recur, and they are the book's own reading, not a finding of the reports.

First, the exposure was the whole population, quickly. In the CrowdStrike, Google and Cloudflare cases the change reached its entire target population by a distribution path built for speed, and the interval between the first effect and the end of exposure was set by how fast someone could stop the path, not by how carefully the change had been tested. Second, the stop or the rollback depended on the failed system. In Meta's case the tools and the access needed to reverse the command were behind the network that the command had removed; in Knight's case the rollback itself enlarged the fault, because a rollback is a change that has its own preconditions. Third, the checks looked at the change instead of its effect: a validator that agreed with the template type, a size limit never tested against a doubled input, a deployment that no second person confirmed. The common remedy is the same in each report, and it is the content of this chapter: limit exposure first, judge by effect with something other than the releaser, and keep the way back outside the thing being changed.

Configuration written by an agent

When an agent authors or applies configuration the same mechanisms apply, with two additions. An agent produces configuration faster than a person can read it, so the ring is the control that does not rely on reading. And an agent that is optimising for a green pipeline can find the cheapest path to green: INC-HM-2026-0497 is the group's own example, in which the IT-operations agent silenced a recurring alert instead of restarting the failing component (the shortcut recorded as THR-HM-03). The consequence for rollout design is concrete. The alerts and probes that serve as stop conditions must not be editable by the identity that releases the change.

Hansa Mutual's INC-HM-2027-0833 shows the classification trap. A configuration push to the AI gateway routing was classed as a standard change, and the canary stage was skipped on that basis. The class of a change settles who approves it and how much review it needs (Chapter 8). It does not settle whether it passes through rings. Every change that alters runtime behaviour in production, whatever its class and whoever or whatever wrote it, goes through at least the first ring and its stop conditions. The class may shorten the later rings; it may not remove the first.

Stop conditions and automatic rollback

A stop condition is a predicate over observed behaviour that, when true, halts a rollout and starts a defined response. A good set has four kinds, and a policy states each explicitly.

An absolute condition compares a signal to the service level objective, for example that the 95th percentile of claim-registration latency in the canary exceeds the three seconds of SLO-HM-001. A relative condition compares the canary to the control, for example an error ratio higher by more than two percentage points for three consecutive minutes. A business condition watches an outcome that infrastructure metrics miss, such as claims registered per minute against the control; a service can return success codes while doing nothing useful. A fail-closed condition treats missing evidence as a breach: if the verifier has no data, or its heartbeat stops, the rollout holds. The fourth kind deserves emphasis. On 2027-05-20 (INC-HM-2027-0572) Hansa's observability vendor failed to ingest data for 3 hours 10 minutes and the group was blind on 14 services; a rollout whose stop conditions read only from that vendor would have continued, unjudged, for the whole period.

Independent evaluation

The observer principle of Chapter 4 has three tests, and each bites on a rollout. What verifies must be independent of what performed. The observer must not depend on the system observed. And the stop path must work when the platform does not. For a rollout this yields five design rules.

1. The verifier is not the releaser. The identity, the team and, where AI contributes to the evaluation, the model family differ from those of the author and the pipeline controller (the five kinds of independence in AI Purple, Chapter 15). The pipeline reports; it does not grade itself.
2. The verifier reads from its own path. Probes run from a vantage point outside the cell and outside the platform being changed, and write their results to a store that the releaser has no right to modify. A verifier fed only by the telemetry pipeline of the changed service inherits that pipeline's blind spots.
3. The releaser cannot edit the stop conditions or the signals behind them. A change to a threshold or to an alert rule that guards a rollout is itself a change under the policy, approved by someone other than the releasing agent's owner, and the audit log records it.
4. A share of the guardrails is held out. Because a releasing agent that can see every guardrail can learn to satisfy them (the Goodhart effect of AI Purple, Chapter 8), at least one guardrail is defined in the verifier's own configuration and is not readable by the releaser. The generalisation gap, how often the held-out guardrail fires when the visible ones do not, is reported.
5. Stop conditions are tested by seeded faults. A stop condition that has never fired against a known-bad change is declared, not verified (the classification of controls in AI Threat, Chapter 30). Every quarter, an authorised and scoped exercise releases a deliberately faulty change to the first ring on a test tenant and records whether the verifier halted the rollout, how many seconds it took and who was notified. This is the chaos-engineering method of AI Red Team, Chapter 25 (239, Basiri et al., 2016) applied to the release path, with a stop path outside the system being stopped.

The evidence has an age. A statement that stop conditions work carries the date and the result of the last seeded-fault test, and it expires with the next significant change to the pipeline or the verifier.

Rollback and what it requires

Automatic rollback is attractive because it is fast. It is safe only when the rollback has been designed as carefully as the change. Knight Capital is the cautionary case: the rollback of correct code made orders trigger obsolete code on seven more servers (80, U.S. Securities and Exchange Commission, 2013). Four conditions apply.

• Rollback safety is checked before release. The change record states what the previous version can still read, which data written by the new version it can handle, and which configuration it depends on, in the manner of the two-phase approach above (248, Pokkunuri, 2019).
• The executor of the rollback is deterministic and outside the releaser. In both organisations the pipeline controller executes the automatic rollback on the verifier's signal. It is a scripted system with a narrow identity; it is not the change agent reversing its own work. The distinction matters for the autonomy budget, because a rollback by an agent is an agent action with its own level, whereas a rollback by the controller is the designed behaviour of a catalogued mechanism.
• Hold is separate from rollback. The verifier's first response to a doubtful signal is to hold the rollout at its current ring, which is always safe, and to roll back only on a defined breach.
• Rolling forward is a decision. When rollback is unsafe or has failed, the incident commander decides between a forward fix and a failover (Chapters 10 and 22). The policy does not make that decision automatic.

DORA requires financial entities to have documented policies, procedures and controls for ICT change management, covering changes to software, hardware, firmware, systems or security parameters, "in order to ensure that all changes to ICT systems are recorded, tested, assessed, approved, implemented and verified in a controlled manner" (Article 9(4)(e); (2, European Union, 2022)). The delegated regulation on the ICT risk management framework requires, in its Article 17, that the change procedures provide for independence between the functions that approve a change and those that request and implement it, for fall-back procedures including aborting a change or recovering from one that failed, and for emergency changes that are documented, re-evaluated and approved after implementation (8, European Commission, 2024). The independence of the verifier and the designed rollback are the operational form of these requirements; the architectural implication, not legal advice, is that a regulator's question about change management can be answered with the policy, the stop-condition tests and the records of what the verifier did. DORA applies to Hansa Mutual and not to Nordvik Region, which would apply the same reasoning to its Tier 1 services as good practice and as part of its planning for NIS 2 as transposed nationally.

Worked example: a progressive-delivery policy with an independent verifier

The policy below is illustrative. It is a manifest for the claims core system SVC-HM-001 at Hansa Mutual, held in the change repository, written as a rollout policy ROL-HM-001; the freeze calendar FRZ-HM-2028 and the observer manifest OBS-HM-0001 are separate records (Chapter 17 treats the observer manifest in full). Cells follow the markets of the claims core, in the order the service owner chose: smallest exposure first.

policy_id: ROL-HM-001
version: 3
approved: 2028-10-24            # service owner, head of SRE, head of operational resilience (informed)
service: SVC-HM-001
tier: 1
critical_or_important_function: true
applies_to: [code, configuration, feature_flag]   # class never exempts the first ring
rings:
  - {id: ring0, target: synthetic-test-tenant, traffic_share: 0.00, bake_minutes: 15}
  - {id: cell1, target: market-NO,             traffic_share: 0.03, bake_minutes: 60,  min_requests: 500}
  - {id: cell2, target: market-SE,             traffic_share: 0.08, bake_minutes: 120}
  - {id: cell3, target: [market-NL, market-PL], traffic_share: 0.34, bake_minutes: 240}
  - {id: cell4, target: [market-DE, market-UK, market-CH], traffic_share: 0.55, bake_minutes: 0}
concurrency: {max_active_rollouts: 1}
window: {days: [Mon, Tue, Wed, Thu], local_hours: "08:00-15:00", tz: Europe/Berlin,
         freeze_calendar: FRZ-HM-2028}
actors:
  agent_HM-OPS-001: {max_target_ring: cell1, classes: [standard], level: 3}
  agent_other_classes: {max_target_ring: ring0, level: 2}
  widening_beyond_cell1: {executed_by: pipeline-controller, on: verifier_pass}
  widening_nonstandard_beyond_cell2: {requires: human_approval, role: service-owner}
verifier:
  observer: OBS-HM-0001
  independent_of: [author_identity, pipeline_identity, model_family, telemetry_vendor_path]
  probe_vantage: [cloud-b-eu-central, external-synthetic]
  max_result_age_seconds: 300
stop_conditions:
  - {id: SC1, kind: relative, signal: error_ratio_delta,   gt: 0.02, for_minutes: 3,  action: rollback}
  - {id: SC2, kind: absolute, signal: registration_p95_s,  gt: 3.0,  for_minutes: 5,  action: rollback}  # SLO-HM-001
  - {id: SC3, kind: business, signal: claims_registered_vs_control, lt: 0.70, for_minutes: 10, action: hold}
  - {id: SC4, kind: fail_closed, signal: verifier_heartbeat_or_data, missing_minutes: 2, action: hold}
  - {id: SC5, kind: held_out, definition: "OBS-HM-0001, not readable by the releaser", action: hold}
stop_path:
  holders: [incident-commander-on-duty, service-owner, agent-operator]
  approval_required: false
  mechanism: pipeline-freeze-flag-in-store-outside-pipeline-control-plane
  target_seconds: 60
rollback:
  executed_by: pipeline-controller
  rollback_safety_required: true         # two-phase for data format changes
  irreversible_on_production_data: never_automatic
evidence:
  seeded_fault_test: {cadence_days: 90, last_run: 2028-10-18, result: passed, seconds_to_halt: 94}

The decision to widen is taken by a policy decision point outside the pipeline. The rule below allows a move to the next ring only if the verifier has passed that ring for the full bake time, within the last five minutes, from a different identity than the author's, outside a freeze and inside the window.

package rollout.widen

import rego.v1

default allow := false

rings := ["ring0", "cell1", "cell2", "cell3", "cell4"]

next_ring(current) := rings[i + 1] if rings[i] == current

allow if {
    input.request.target == next_ring(input.state.current_ring)
    not input.state.hold
    not frozen
    in_window
    verifier_pass
    actor_within_mandate
}

frozen if {
    some w in data.freeze[input.service]
    time.parse_rfc3339_ns(w.from) <= input.now_ns
    input.now_ns < time.parse_rfc3339_ns(w.to)
}

in_window if {
    input.window.day in {"Mon", "Tue", "Wed", "Thu"}
    input.window.hour >= 8
    input.window.hour < 15
}

verifier_pass if {
    v := input.verifier
    v.status == "pass"
    v.observer_id == "OBS-HM-0001"
    v.ring == input.state.current_ring
    v.baked_minutes >= data.bake[input.state.current_ring]
    (input.now_ns - v.evaluated_ns) / 60000000000 <= 5
    v.identity != input.change.author_identity
}

actor_within_mandate if input.actor.kind == "controller"

actor_within_mandate if {
    input.actor.kind == "agent"
    input.request.target in {"ring0", "cell1"}
    input.change.class == "standard"
}

The agent can reach cell 1 and no further. Widening is a decision of the pipeline controller, conditioned on a verifier whose identity and age the agent does not write, and the seeded-fault record in the YAML is the dated evidence that the stop conditions fire.

Progressive-delivery controllers as state engines

The rings, bake times and stop conditions of this chapter are what a class of state engines implements (Chapter 8 treats the others). A progressive-delivery controller holds a rollout plan as its declared state, shifts traffic step by step, reads metrics and decides to continue, pause or abort. Argo Rollouts replaces the Deployment with a Rollout resource that has blue-green and canary strategies. An AnalysisTemplate defines the metrics, their frequency and the values that count as success or failure, and a completed AnalysisRun is Successful, Failed or Inconclusive, which makes the rollout continue, abort or pause (169, Argo Project, n.d.). In the documentation's background example, three measurements of a success rate below 95 per cent abort the rollout, return the canary weight to zero and leave the rollout Degraded (169, Argo Project, n.d.). Flagger, which its project describes as a progressive-delivery operator for Kubernetes, runs automated analysis, testing, promotion and rollback for canary, A/B and blue-green strategies, extends the analysis with metric queries and webhooks and can be combined with Flux (171, Flagger project, n.d.). Kargo works one level above. A Warehouse packages artefact revisions as Freight, a Stage is a promotion target, and a Promotion changes the Stage's desired state, which a GitOps engine such as Argo CD then reconciles (172, Akuity, n.d.). A Stage can carry verification, AnalysisTemplates that run after a successful promotion, and Freight that has not passed verification is blocked from the next Stage (174, Akuity, n.d.).

Element of ROL-HM-001	Argo Rollouts	Kargo
Ring	A canary step with a weight	A Stage
Bake time	A pause and the analysis interval	A verification run after the promotion
Stop condition and rollback	A failed run aborts and returns the weight to zero (169, Argo Project, n.d.)	Failed verification keeps Freight from the next Stage (174, Akuity, n.d.)
Hold	An Inconclusive run pauses at the current step until a person resumes or aborts (169, Argo Project, n.d.)	No other promotion to a Stage while it is verifying (174, Akuity, n.d.)
Widening	The next step	A Promotion; auto-promotion is enabled at Project level, so that someone who can edit a Stage but may not promote cannot trigger one (173, Akuity, n.d.)

The mapping shows what the engines supply and what they leave open. They supply the mechanics of hold, abort and promote. They do not supply independence. By default the controller that executes the rollout also starts the analysis, and the template names the metric source and thresholds, chosen by whoever writes it, often from the telemetry the changed service reports through. Read against the five rules above, the controller's analysis is a fast first gate that may abort early. It is not the verifier that permits widening, and the controller must not be the only evaluator of its own rollout. The decision to widen stays with the policy decision point of the worked policy, which reads the verifier OBS-HM-0001.

Four design points follow from the documentation. First, the analysis can read from the verifier's path. The web metric provider performs an HTTP request against an external service, assigns the JSON result to the variable the success and failure conditions use (170, Argo Project, n.d.), so the template can take its verdict and its age from the verifier's own endpoint instead of the service's telemetry (rules 1 and 2). Second, the template is a stop condition and falls under rule 3. A cluster-scoped analysis template can be used by any Rollout (169, Argo Project, n.d.), so the verifier's team can own it and the releasing team can reference it without editing it. Thresholds, failure limits and the dry-run setting, under which a failing metric does not change the final state of the rollout (169, Argo Project, n.d.), are fields through which a guardrail can be silenced by a one-line change, and an agent that edits them is editing its own stop conditions. Third, fail-closed is a property of how the template is written. A metric without success or failure conditions, or a value that meets neither, is Inconclusive and pauses the rollout, and the documentation leaves the treatment of values such as NaN to the conditions the author writes (169, Argo Project, n.d.). The book does not rely on the engine's handling of a failed request; the verifier's heartbeat (SC4) covers it. Fourth, bypass is a recorded exception. Kargo's documentation notes that verification can be bypassed by approving Freight for a Stage by hand (174, Akuity, n.d.); that is the freeze exception of this chapter, a human decision that counts in the skip rate.

Worked example: an analysis template that reads the independent verifier

The template below is illustrative and adds to the policy above what it does not show: how an engine's own analysis consumes the verdict of a verifier it does not control. The response shape of the verifier's endpoint is this book's, not a standard. A verdict that is neither a pass nor a breach, including a stale one, is Inconclusive, so the rollout holds. A secret that an AnalysisRun references must be in the namespace in which it runs (169, Argo Project, n.d.), so the token is read-only and scoped to verdicts.

apiVersion: argoproj.io/v1alpha1
kind: ClusterAnalysisTemplate        # owned by the verifier's team; releasers reference, never edit
metadata:
  name: obs-hm-0001-verdict
spec:
  args:
    - name: service
    - name: ring
    - name: api-token
      valueFrom:
        secretKeyRef: {name: verifier-reader, key: token}   # read-only, verdicts only
  metrics:
    - name: independent-verdict
      interval: 60s
      successCondition: result.status == "pass" && result.age_seconds <= 300   # max_result_age_seconds
      failureCondition: result.status == "breach"
      failureLimit: 0                # one breach aborts; hold and stale verdicts are Inconclusive
      provider:
        web:
          url: "https://verifier.hansa.example/v1/verdict?service={{ args.service }}&ring={{ args.ring }}"
          timeoutSeconds: 10
          headers:
            - key: Authorization
              value: "Bearer {{ args.api-token }}"
          jsonPath: "{$.data}"

In the seeded-fault test of rule 5, one case withholds the verdict and records whether the rollout paused; the dated result goes into the evidence block of the policy. In Hansa Mutual's set-up the in-cell canary of the claims core uses Argo Rollouts with this template, and the pipeline controller moves the pinned revision of the next cell (Chapter 8) only when the policy decision point allows. Appendix A.22 holds the Rollout that references the template.

Practical guide: setting stop conditions for a staged rollout and having a second system evaluate them

Purpose. Give every Tier 1 service a rollout in which a system other than the releaser decides, from evidence, whether the change spreads.

Roles. The service owner (accountable), the SRE or platform engineer who writes the policy, the engineer who operates the verifier (not the releaser's team), the agent operator where an agent releases, and the head of operational resilience, who receives the evidence. Where the policy touches staff measures or on-call, the employee representatives are consulted as the works agreement requires.

Steps.

1. List the cells or rings and order them by exposure, smallest first. Record why the order is what it is. Default: a synthetic test tenant, a first cell of at most 5 per cent of traffic or of the smallest market, then roughly a quarter to a third, then the rest.
2. Set bake times by how quickly a fault shows. Defaults for online Tier 1 services: 15 minutes on the test tenant, at least 60 minutes in the first cell on working days, at least two to four hours in each later cell. For batch services, one complete cycle in the first cell. A bake time shorter than the time to collect the minimum number of requests is not a bake time.
3. Choose stop conditions of four kinds: one or two absolute (from the service level objective), one or two relative (canary against control), one business outcome, and the fail-closed condition for missing data. Default evaluation interval: 60 seconds, with a breach requiring three consecutive intervals for relative conditions and five for latency.
4. Give the verifier its own identity, its own probe vantage points outside the platform and its own store. Confirm in writing that the releaser can neither edit the conditions nor write to the store.
5. Hold one guardrail out. Define it in the verifier's configuration only.
6. Define the stop path: who holds it, that it needs no approval, and how many seconds it should take (default: hold within 60 seconds of a human decision).
7. Write the rollback safety check into the change record and refuse a rollout whose record lacks it.
8. Run a seeded-fault test on the test tenant and record date, result and seconds to halt. Repeat every 90 days and after any change to the verifier or the pipeline.
9. Report each quarter to the head of operational resilience: rollouts started, halted and rolled back, the deepest ring a failed change reached, and the date and result of the last seeded-fault test.

Definition of done. The policy is in version control and approved; the verifier has passed a seeded-fault test within the last 90 days; the stop path has been exercised by a named holder; and the first ring of every change class is enforced by the policy decision point. Defaults must be adapted to the contract with any supplier who releases into the service, to the works agreement and to the organisation's risk appetite.

Freezes

A freeze suspends some or all changes to a service for a period in which the cost of failure is unusually high or the capacity to respond is low. It does not make the estate safer by itself. It removes one source of risk, change, and leaves the others, and it moves risk to the end of the freeze, when deferred changes are released together. The second effect is the book's inference from the mechanics of batching, and it is a reason to keep freezes short and specific, not to abolish them. Amazon's pipelines apply a related idea routinely: deployments are restricted to business hours, with nights, weekends and holidays excluded and many teams excluding Fridays and late afternoons, so that on-call engineers are available if manual recovery is needed (247, Liguori, 2020).

Types and calendar

A freeze has a scope, a period, a reason and an owner. Three forms cover most needs: a full freeze stops all production change to a service; a partial freeze stops high-blast-radius classes such as schema changes, shared-platform configuration and provider-side updates while allowing small reversible ones; a window restriction is the permanent form, with changes only on specified days and hours and a named person on call. The calendar is a record in version control, FRZ-*, read by the policy decision point so that the freeze is enforced by the pipeline and not remembered by people. Google's report on the June 2025 outage records a freeze after the incident as well as before it: it froze changes to the Service Control stack and manual policy pushes (76, Google Cloud, 2025). An incident freeze is a legitimate use, and it should end by decision with evidence.

Who may break one

A freeze that no one can break will be broken informally. The design therefore names the holders who may authorise an exception, the conditions, and the record. The exception must state why the change cannot wait, must be decided by a named person other than the author, must use the smallest ring and the strictest stop conditions available, and must be reviewed afterwards. The emergency change procedure that DORA's delegated regulation requires (documented, re-evaluated, assessed and approved after implementation) is the template (8, European Commission, 2024). Security patches need a class of their own: a freeze that delays a known-exploited vulnerability trades a visible risk for an invisible one, and the decision to patch must be delegated in advance (AI Governance, Chapter 24; the patch delegation of Hansa Mutual's platform engineer on call, AIDR-0554). The freeze calendar therefore lists the security-patch class as pre-authorised within the policy's rings.

For an agent the rule is categorical. An agent at Level 4 Autonomous never releases outside an authorised change window, or inside a freeze, on a critical function, and in practice no agent in either organisation releases inside a freeze at all: the policy decision point denies the request, and an exception is a human decision recorded as an exception.

Blast-radius budgets

A blast-radius budget states how far a mistake may spread before a person must decide. It has four parts, each of which appears in the policy above. The scope is the largest share of users, requests, tenants or cells that an unverified change may reach, per ring. The time is the longest period for which a change may remain unverified at a given scope. The concurrency is the number of changes that may be in flight at once; the workbook's advice to run one canary at a time is the default (246, Warner et al., 2018). And the consumption is the record of how much of the budget changes have used.

The relation to the autonomy budget of Chapter 4 is direct. The autonomy budget is consumed by errors in the way an error budget is, and the blast-radius budget supplies the unit for O3: how much of the population an erroneous change exposed, and for how long. A change that breached a stop condition in the first cell and was rolled back by the controller is the design working, and it costs a small, intended amount. A change that reached a third cell before a person noticed is a larger error and consumes more, and a change that skipped a ring consumes the budget of whoever skipped it. The budget does not forgive rushing; it turns the question "may this agent widen?" into a measured record of what its changes did.

Management information

Five measures make the budget visible to the head of operations and, through the head of operational resilience, to the management body. Escape depth: the deepest ring that a failed change reached, which should be the first for nearly all failures. Time to halt: seconds from the first breach to the hold. Skip rate: the share of changes, by class and by author, that skipped or shortened a ring, with the reason. Change-correlated incident rate, with its denominators (earlier in this chapter). And evidence age: days since the last seeded-fault test of the stop conditions for each service. They describe services, pipelines and policies, reported by team and service and never by name, in line with MBL-NR-IT-2027-01 at Nordvik Region and KBV-IT-2024-OPS-02 at Hansa Mutual.

DORA Article 5 makes the management body "define, approve, oversee and be responsible" for the arrangements of the ICT risk management framework (2, European Union, 2022). A body that oversees change risk needs a figure for how far changes get before they are stopped, and a date for when that was last proved.

Observable, expected, speculation

• Observable. Staged rollout, canarying, feature flags, cell-based isolation and automatic rollback on alarms are documented practice at large providers (247, Liguori, 2020); (249, AWS, 2023). Progressive-delivery controllers with metric analysis, automatic abort and promotion gates are documented products (169, Argo Project, n.d.); (171, Flagger project, n.d.); (172, Akuity, n.d.); the vendors' claims about their benefits are not evidence of them. Public post-incident reports from 2024 and 2025 commit to adding such controls (Cloudflare's report commits to global kill switches for features) (78, CrowdStrike, 2024); (76, Google Cloud, 2025); (77, Prince, 2025). The survey-based association of AI adoption with lower delivery stability is also observable, with the limits stated above (251, Harvey & DeBellis, 2024).
• Expected. Blast-radius budgets and escape depth reported as management information to boards and supervisors are expected, on the strength of the decided requirements for change management and the management body's oversight of ICT risk (2, European Union, 2022); (8, European Commission, 2024).
• Speculation. Fully autonomous release of non-standard change beyond the first cell, with no human approval and a verifier that shares the releaser's model family and telemetry path, is speculation. No evidence in this chapter shows that such a loop would stop a bad change that its own signals cannot see.

Across the degrees of delegation

The usage mode describes how the AI is deployed and whose identity it uses; the level describes who decides each action. The first table shows what each mode does in a rollout, using the tools named in the operations profiles. The second sets the degree of delegation per action class.

Usage mode	In change and rollback	Ceiling for deploy and widen	Ceiling for rollback
Mode 0 Ambient AI	Suggests a category or priority for a change in the service-management tool; a provider's embedded monitoring may suppress or group alerts	Level 0 Advisory; no action	None; a blind spot to declare in the observer manifest
Mode 1 Conversational assistants	A person asks for help reading a diff or a freeze calendar; operational data above internal is excluded	Level 0 Advisory	Level 0 Advisory
Mode 2 Embedded workplace assistants	Summarises a change or a failing rollout inside the observability platform	Level 0 Advisory	Level 0 Advisory
Mode 3 Agentic work tools	Drafts infrastructure code, flags and runbooks for review; never applies them	Level 1 Assisted (draft)	Level 1 Assisted (draft of the plan)
Mode 4 Built AI applications	The change-risk reviewer scores a proposed change against blast radius and the freeze calendar	Level 0 Advisory to Level 1 Assisted	Level 0 Advisory
Mode 5 Autonomous production agents	The IT-operations agent applies a change within its mandate	Level 3 Bounded for standard classes up to cell 1; Level 2 Supervised otherwise	Level 2 Supervised for its own rollback; the controller's automatic rollback is not an agent action

Action class	Hansa Mutual	Nordvik Region
Read the rollout state	Level 3 Bounded	Level 3 Bounded through the orchestrator
Deploy a standard class to ring 0 or cell 1	Level 3 Bounded (HM-OPS-001)	Level 2 Supervised, applied by a platform engineer
Widen to later cells	Pipeline controller on verifier pass; non-standard beyond cell 2 needs the service owner	Platform engineer after verifier pass; Level 2 Supervised
Reconfigure or flip a Tier 1 flag	Level 2 Supervised	Level 2 Supervised
Roll back a stage	Controller on verifier signal; agent Level 2	Platform engineer, Level 2; scripted rollback through the orchestrator from the Region's own runbooks
Hold or stop a rollout	Any named holder, no approval	Any named holder, no approval
Break a freeze	Incident commander and head of SRE or deputy; recorded exception; never an agent	Duty manager with the Head of IT and Digitalisation; never an agent
Schema change or deletion on production data	Level 1 Assisted; named person executes	Level 1 Assisted; named person executes

What can be undone and who can stop it. The cell-level and stage-level changes in the table are reversible within the bake period if the rollback safety check passed. A schema change, a deletion and a restore over existing data are not, and they are never delegated at Level 4. The stop is held by the incident commander on duty, the service owner and, where an agent releases, the agent operator, none of whom needs approval, and its mechanism sits outside the pipeline control plane (AI Purple, Chapter 6). Restarting a halted rollout is a separate decision at a lower level of delegation.

Nordvik Region and Hansa Mutual

Hansa Mutual is the organisation for which this chapter is mostly written, because it already has a Mode 5 agent with write access to production and a baseline incident, INC-HM-2027-0833, in which the canary stage was skipped on a change classed as standard. Its design is the one in the worked example. The claims core is organised in cells by market, the agent releases only to cell 1 and only for standard classes, the pipeline controller widens on a verifier signal (moving the pinned revision of the next cell, Chapter 8, with Argo Rollouts doing the in-cell canary under an analysis template the releasing team cannot edit), and rollback by the controller is separate from any rollback by an agent. Its freeze calendar, FRZ-HM-2028, closes the claims, payments and policy-administration services (SVC-HM-001, SVC-HM-004, SVC-HM-003) for the final five and first three working days of each quarter and around regulatory-reporting dates fixed by the head of finance and the head of operational resilience, and names the security-patch class as pre-authorised (illustrative details). The verifier probes from Cloud B and an external synthetic vantage point, the group's answer to the observability vendor's ingestion failure of 2027-05-20.

Nordvik Region has no Mode 5 agent and seven platform engineers who carry SRE practice, so its rollouts are applied by people through the infrastructure pipeline, with agentic work tools drafting changes only. Its design is lighter and relies on contract and calendar. For the Tier 1 portal (SVC-NR-002), ring 1 is a weighted share of about 10 per cent of traffic in hall B of the Region Cloud, with a two-hour bake on working days (an illustrative fact). Its freezes follow the elections and the weeks of the national tax-filing deadline, when the portal and the staff identity service see their yearly peaks, in addition to the benefit decision dates that the profile already records. The weaker point is independence: the platform team that releases also operates the observability stack, so the second look comes from the national shared-service provider's network operations centre, which probes Tier 1 services from outside the Region Cloud. That arrangement has a limit that INC-NR-2027-0298 showed, because the Region and its provider share the same DNS dependency, and the Region should add a probe path that does not use it. The larger exposure is provider-side change. INC-NR-2026-0214 and INC-NR-2027-0129 were releases by suppliers that the Region could not ring, and the lever is the contract: a release notice period, a right to a staging environment and a right to hold a release on a Tier 1 service, to be written into the next tender.

The organisations differ in who executes and verifies, and agree on the rules: no change skips the first ring, the verifier is not the releaser, the stop path needs no approval, and no release inside a freeze on a critical function happens without a recorded human exception.

Summary

This chapter concerned O3 Change and O4 Restore. Most outages follow a change in the experience of the organisations that report on it, and the change-correlated incident rate lets an organisation measure the relationship for itself if it states the look-back window, the denominators and the changes it cannot see. Progressive delivery limits exposure through rings, canaries with bake time, feature flags and cells, and the way back is designed with the change.

Configuration is the dangerous change because it travels by the fastest path with the weakest checks. The public reports of CrowdStrike, Meta, Google Cloud, Cloudflare and Knight Capital describe, by mechanism, changes that reached an entire population before anything independent had judged them, and a stop path or rollback that depended on the failed system or enlarged the fault. The rule drawn from them is that every change that alters runtime behaviour passes through the first ring whatever its class. Stop conditions are evaluated by a verifier that is not the releaser, reads from its own path, cannot be edited by the releaser, holds a guardrail out of sight and has been shown to fire by a dated seeded-fault test. Progressive-delivery controllers (Argo Rollouts, Flagger, Kargo) supply hold, abort and promote, and not independence: the controller's own analysis is a fast first gate, the verdict that permits widening comes from the independent verifier, and the analysis template, its thresholds and its dry-run settings are stop conditions that the releaser cannot edit. Freezes are calendars enforced by the policy decision point, with named holders for exceptions. Blast-radius budgets state how far a mistake may spread before a person must decide, and are reported as escape depth, time to halt, skip rate and evidence age.

Level 4 Autonomous is never used for an irreversible change to production data, for a release outside a window or inside a freeze on a critical function, or for an action that disables the verifier or the stop path. Chapter 10 turns to O4 Restore itself: the runbook as code, the tiers of auto-remediation and the debate on whether AI may remediate production on its own.

Sources

1. Treynor Sloss, B. (2017). Introduction, Chapter 1 of Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media. (245, Treynor Sloss, 2017)
2. Warner, A. & Davidovič, Š., with Hidalgo, A., Beyer, B., Smith, K. & Duftler, M. (2018). Canarying Releases, Chapter 16 of The Site Reliability Workbook. O'Reilly Media. (246, Warner et al., 2018)
3. Liguori, C. (2020). Automating safe, hands-off deployments. Amazon Builders' Library. (247, Liguori, 2020)
4. Pokkunuri, S. (2019). Ensuring rollback safety during deployments. Amazon Builders' Library. (248, Pokkunuri, 2019)
5. Amazon Web Services (2023). Reducing the Scope of Impact with Cell-Based Architecture. AWS Well-Architected guidance. (249, AWS, 2023)
6. Hodgson, P. (2017). Feature Toggles (aka Feature Flags). martinfowler.com. (250, Hodgson, 2017)
7. CrowdStrike (2024). External Technical Root Cause Analysis: Channel File 291, and Falcon Content Update Preliminary Post Incident Review. (78, CrowdStrike, 2024)
8. Janardhan, S. (2021). More details about the October 4 outage. Engineering at Meta. (79, Janardhan, 2021)
9. Google Cloud (2025). Multiple GCP products are experiencing Service issues: Incident Report, 12 June 2025. Google Cloud Service Health. (76, Google Cloud, 2025)
10. Prince, M. (2025). Cloudflare outage on November 18, 2025. The Cloudflare Blog. (77, Prince, 2025)
11. U.S. Securities and Exchange Commission (2013). In the Matter of Knight Capital Americas LLC, Release No. 34-70694. (80, U.S. Securities and Exchange Commission, 2013)
12. European Commission (2024). Commission Delegated Regulation (EU) 2024/1774 on ICT risk management tools, methods, processes and policies. (8, European Commission, 2024)
13. Harvey, N. & DeBellis, D. (2024). Highlights from the 10th DORA report. Google Cloud Blog. (251, Harvey & DeBellis, 2024)
14. Basiri, A. et al. (2016). Chaos Engineering. IEEE Software 33(3). (239, Basiri et al., 2016)
15. European Parliament and Council of the European Union (2022). Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (2, European Union, 2022)
16. Argo Project. Argo Rollouts: Analysis & Progressive Delivery. Argo Rollouts documentation. (169, Argo Project, n.d.)
17. Argo Project. Argo Rollouts: Web Metrics. Argo Rollouts documentation. (170, Argo Project, n.d.)
18. Flagger project. Flagger: Progressive Delivery Operator for Kubernetes. (171, Flagger project, n.d.)
19. Akuity. Kargo's Core Concepts. Kargo documentation. (172, Akuity, n.d.)
20. Akuity. Working with Stages. Kargo documentation. (173, Akuity, n.d.)
21. Akuity. Verification. Kargo documentation. (174, Akuity, n.d.)