Appendix B: Templates and Tools

This appendix is the directory of reusable templates and tools for AI operations, in the repository that accompanies AI Operations. Each artefact is designed to be copied into an organisation's own repositories, adapted to its contracts, works agreements and national law, and maintained under version control beside the services it describes. The chapter that introduces each artefact is listed so that readers can find the rationale behind the design. The field names below are those of the worked examples in the chapters; a template that departs from them should say why. None of the templates is legal advice, and none is tooling for attack: they describe services, objectives, authority, records and evidence.

The governance templates of AI Governance, Appendix C, the platform and security templates of AI Platform and Security, Appendix B, the detection, guardrail and chaos-experiment artefacts of AI Red Team, Appendix A, the role, control and mandate artefacts of The Transformed Security Architect, the detection, case, playbook and mandate artefacts of AI Blue, Appendix A, and the scenario, rules-of-engagement and coverage artefacts of AI Purple, Appendix A, remain the reference for those subjects and are not repeated here. The complete versions of the artefacts of this book are in Appendix A.

Templates by activity

No. Activity Artefact Where to find it Introduced in
B.1 Definition Service record with dependencies and blast radius (SVC-*) Chapter 3 Chapter 3
B.2 Definition Delegation profile (DLG-*) Chapter 4 Chapter 4
B.3 Observation Service level objective record (SLO-*) Chapter 5 Chapter 5
B.4 Observation Correlation policy and noise budget Chapter 6 Chapter 6
B.5 Diagnosis Incident record (INC-*) Chapter 7 Chapter 7
B.6 Change Change class and change record (CHG-*) Chapter 8 Chapter 8
B.7 Restoration Runbook manifest (RBK-*) Chapter 10 Chapter 10
B.8 Learning Post-incident review (PIR-*) and problem record (PRB-*) Chapter 12 Chapter 12
B.9 Agents Mandate manifest for an operations agent (MND-*) Chapter 16 Chapter 16
B.10 Agents Promotion and demotion policy (PRM-*) Chapter 16 Chapter 16
B.11 Independence Observer manifest (OBS-*) Chapter 17 Chapter 17
B.12 Independence Seeded-fault record (SDF-*) Chapter 17 Chapter 17
B.13 Security Tier-0 checklist for the operator's AI Chapter 18 Chapter 18
B.14 Resilience Restore-test record (RST-*) and resilience register row Chapter 19 Chapter 19
B.15 Roles Role charter for the operations conductor (ROLE-*) Chapter 21 Chapter 21
B.16 People On-call charter Chapter 25 Chapter 25
B.17 People Competence record (CMP-*) Chapter 26 Chapter 26
B.18 Review Autonomy-budget review checklist Chapter 4 Chapter 4
B.19 Review Observer review checklist Chapter 17 Chapter 17
B.20 Review Quarterly resilience register review checklist Chapter 19 Chapter 19
B.21 Change Checklist: reviewing a sync policy before it is switched on Chapter 8 Chapter 8

Template descriptions

B.1 Service record with dependencies and blast radius (SVC-*)

  • Purpose: States the service, its tier, owner, objectives, dependencies, blast radius, the evidence of recovery and the unknowns, in version control beside the service's infrastructure code.
  • When to use: For every Tier 1 service first, then Tier 2; re-draw at least every six months and after any incident that revealed a dependency not on the map.
  • Inputs: Traces, infrastructure code, network flow data, register of information, restore tests.
  • Outputs: services/SVC-*.yaml.
  • Chapter: Chapter 3
# services/SVC-XX-000.yaml (template; replace every value)
service: SVC-XX-000
name: <service name>
tier: 1                              # 1, 2 or 3
dora_critical_or_important_function: false
owner: <owner role>
map:
  drawn: 2029-01-01
  next_review: 2029-07-01            # six months at most
  drawn_by: [<role>, <role>]
objectives:
  availability: "99.9 % per month"
  rto: PT4H
  rpo: PT15M
blast_radius:
  markets: [<market>]
  processed_in: [<country>]
  degraded_mode: "<what works, how long it takes to enter>"
dependencies:
  - id: <dependency id>
    kind: hyperscaler            # internal-service, saas-observability, managed-service-provider
    role: <role of the dependency>
    shared_with: [<SVC ids>]
    register_of_information: <contract id, if a provider>
    seen_failure: "<incident id and effect, if any>"
    independent_probe: none      # or the observer probe id
    evidence:
      test: RST-XX-0000-000
      date: 2029-01-01
      result: "<measured time and point against the objective>"
      age_months: 0
      retest: none
actors_on_this_service:
  - <agent id>                   # with its level and action classes
unknowns:
  - "<what the team could not say in the first hour of an outage>"

B.2 Delegation profile (DLG-*)

  • Purpose: States for one agent and one service what the agent may do, per action class and level, how much of the allowance it has kept (the autonomy budget), what verifies it and who stops it.
  • When to use: Before an agent acts on a service, and at every review (every 90 days by default).
  • Inputs: Service record, mandate, action-class catalogue, observer manifest, works agreement.
  • Outputs: delegation/DLG-*.yaml and an AI Decision Record (AIDR). Lowering a ceiling is immediate; raising one is a named authority's decision.
  • Chapter: Chapter 4
delegation_profile: DLG-XX-OPS-000
status: proposed                 # proposed, active, suspended, retired
version: 0.1
date: 2029-01-01
decision_record: AIDR-0000
agent: {id: <agent id>, mode: "Mode 5", mandate: MND-XX-OPS-000}
service: {id: SVC-XX-000, tier: 1, cif: true, slo: SLO-XX-000, rto: PT4H, rpo: PT15M}
accountable: {agent_owner: "<name>", raise_authority: "<name>"}
policy_decision_point: {location: outside_agent_platform, deny_by_default: true}
window_rule: "outside an authorised window or inside a freeze, ceiling is Level 2 with named approver"
action_classes:
  read:               {ceiling: "Level 4", condition: "redacted personal data"}
  annotate:           {ceiling: "Level 4"}
  notify:             {internal: "Level 3", external: "Level 1"}
  scale:              {ceiling: "Level 3", condition: "within quota and monthly cap"}
  restart:            {stateless: "Level 3", stateful: "Level 2", runbook: RBK-XX-0000}
  certificate_renew:  {ceiling: "Level 3", runbook: RBK-XX-0000}
  reconfigure:        {ceiling: "Level 2", alerts_thresholds_monitors: "Level 0"}
  deploy:             {ceiling: "Level 2"}
  roll_back:          {ceiling: "Level 2"}
  fail_over:          {ceiling: "Level 1", approver: incident_commander}
  delete:             {temporary_artefacts: "Level 3", production_data: "Level 1"}
  restore_overwrite:  {ceiling: "Level 1", approver: "named person"}
  anything_unlisted:  "undelegated"
autonomy_budget:
  window_days: 28
  tolerance: {<class>: "<rate of actions or count of faulty actions>"}
  customer_visible_effect: "consumes 100 per cent at once"
  review_at_percent: 70
  counted_by: OBS-XX-0001
  on_full_consumption: "class falls one level; raising needs raise_authority"
verification: {observer: OBS-XX-0001, independent_of: [model, data, authorship, vendor, time]}
stop: {holders: [incident_commander, agent_operator, platform_engineer_on_call],
       path: "outside agent platform", target_minutes: 5, drill: quarterly}
employee_representation: "<works agreement id> agreed before command logging is used"
review: {every_days: 90, next: 2029-04-01}

B.3 Service level objective record (SLO-*)

  • Purpose: Records what users are promised and how it is measured, alerted on and evidenced, including the burn-rate alerts, the budget policy and the evidence that the alerts work.
  • When to use: For every service with a tier; review at the review date and when the service, gateway or collector changes.
  • Inputs: Service record, user journeys, measurement points, past incidents.
  • Outputs: slo/SLO-*.yaml.
  • Chapter: Chapter 5
record: SLO-XX-000
service: SVC-XX-000
version: 1.0.0
effective: 2029-01-01
review_due: 2029-04-01
owner_role: <service owner role>
operator_role: <operator role>
window: {objective: calendar month, alerting: rolling 30 days}
indicators:
  - id: availability
    spec: <share of requests answered within the stated time>
    good: <definition of a good event>
    measured_at: [<ingress metrics>, <synthetic probe outside the service's network>]
    objective: 99.9
    budget_minutes_per_30d: 43.2
    min_events_per_window: 5000     # below this, probe results only
  - id: settlement_deadline
    spec: <run complete by the stated time on run days>
    kind: deadline                  # binary per run day; not a ratio
    checkpoints: ["<time and share done>"]
alerts:
  burn_rate:
    - {severity: page,   long: 1h, short: 5m,  burn: 14.4, budget_used: 2%}
    - {severity: page,   long: 6h, short: 30m, burn: 6,    budget_used: 5%}
    - {severity: ticket, long: 3d, short: 6h,  burn: 1,    budget_used: 10%}
  deadline: {severity: page, rule: checkpoint missed on a run day}
budget_policy:
  exhausted: reliability work first; change freeze on the service except fixes
  quarter_end: freeze calendar applies regardless of budget state
dependencies: [<SVC ids or external services>]
recovery: {rto: PT4H, rpo: PT15M, evidence: RST-XX-0000-000, result: <passed or failed and date>}
telemetry:
  hot_retention_days: 30
  archive_months: 13
  personal_data: minimised at the collector
evidence:
  alert_test: {date: 2029-01-01, method: seeded fault, result: "<time to page>"}
  expires_on: [gateway change, alert rule change, collector change, "<date>"]

B.4 Correlation policy and noise budget

  • Purpose: Sets, per alert class, what an observability layer may group, suppress or page at which level, which alerts it must never suppress, the noise budget of the rota, sampling by a person, seeded alerts and the stop conditions.
  • When to use: Before an AI correlation layer is switched on, and after each evaluation.
  • Inputs: Golden set of replayed episodes, alert inventory, rota data (for the team, never per named engineer).
  • Outputs: observe/correlation-policy.yaml, evaluation record.
  • Chapter: Chapter 6
never_suppress:
  - <heartbeat observer id>
  - slo-burn-page
  - recovery-objective-breach
  - change-freeze-window
noise_budget:
  pages_per_shift_max: 20
  actionable_share_min: 0.5
  reported_for: rota               # never per named engineer
sampling:
  held_or_merged_share: 0.10
  minimum_items_per_week: 20
  reviewer_sees_raw_first: true
seeding:
  per_class_per_week_min: 1
  truth_store: sealed-outside-correlator
stop_conditions:
  seed_catch_rate_below_0_90_for_4_weeks: freeze_promotion
  seed_catch_rate_below_0_80_for_2_weeks: all_classes_to_level_1
  missed_page_on_customer_slo: all_classes_to_level_1_and_review

B.5 Incident record (INC-*)

  • Purpose: Keeps the clocks, the timeline, the hypotheses (with who raised each), the assistant's contributions, the decisions and their holders, and the impact, so that the review can replay what was known when.
  • When to use: For every incident, opened by the monitoring, the assistant (as a candidate) or a person.
  • Inputs: Alerts, change log, assistant output, commander's notes.
  • Outputs: incidents/INC-*.yaml; input to the post-incident review.
  • Chapter: Chapter 7
id: INC-XX-2029-0000
service: SVC-XX-000
slo: SLO-XX-000
commander: <name>
ops_lead: <rota or name>
assistant: {use_case: XX-OPS-002, aidr: AIDR-0000, level: 1,
            model: <model and version>, prompt: <prompt version>, tools: [read]}
clocks:                            # trusted time
  first_signal: "2029-01-01T00:00"
  assistant_first_flag: "2029-01-01T00:00"
  human_ack: "2029-01-01T00:00"
  declared: "2029-01-01T00:00"
timeline:
  - {t: "00:00", by: monitoring, e: "<event>"}
  - {t: "00:00", by: assistant, kind: drafted, e: "<event>"}
  - {t: "00:00", by: commander, e: "<hypothesis, decision or action>"}
hypotheses:
  - {id: H1, text: "<hypothesis>", by: assistant, rank: 1, score: 0.0,
     status: open, test: "<how it is tested>"}      # open, refuted, confirmed
assistant_contributions: {retrieval: <useful or not>, correlation: <note>,
                          ranking: <top1_right or top1_wrong>, draft_updates: 0}
decisions: [{what: <action>, holder: <role>, dlg: DLG-XX-OPS-000}]
impact: {sev: 2, start: "00:00", end: "00:00", minutes: 0}

B.6 Change class and change record (CHG-*)

  • Purpose: The change class states what may be released without a person in the path, with limits, window, delegation, independent verification, rollback, history and the rule that suspends it. The change record states who or what authored a change, with what inputs, what was tested and not tested, the policy results, approvers, rollback, independent verification and the claim the change does not make.
  • When to use: Class: when a pre-approved change type is proposed and at each expiry. Record: for every change.
  • Inputs: Change history, policy-as-code results, plan output, rollback test.
  • Outputs: change/classes/*.yaml, change/CHG-*.yaml, an AIDR for each class.
  • Chapter: Chapter 8
change_classes:
  - id: XX-STD-000
    name: <change type>
    approved_by: AIDR-0000
    eligible:
      service_tiers: [2, 3]
      dora_critical_or_important: false
    may_touch: [certificate, secret_reference]
    limits: {max_resources: 10, deletes: 0, identity_changes: 0, network_changes: 0}
    window: "daily 05:00-07:00 Europe/Berlin"
    delegation: {draft: "Level 2 Supervised", release: "Level 3 Bounded"}
    verification:
      probe: <probe not run by the pipeline>
      pass: "<criterion>"
    rollback: {automatic: true, tested_on: 2029-01-01, result: passed}
    history: {instances_12m: 0, correlated_incidents: 0}
    suspend_if: "two correlated incidents or one in Tier 1 within a quarter"
    sampling: {post_release_rereview: 0.10, seeded_errors: 0.05}
    expires: 2030-01-01

change_record:
  id: CHG-XX-2029-0000
  service: SVC-XX-000
  title: <what changes>
  motivated_by: [<RST, INC or PRB id>]
  requested_by: <person>
  author:
    kind: agent                      # agent or person
    tool: <tool>
    identity: <workload identity>
    model: <model and version>
    instructions_version: <skill or prompt version>
    inputs: [<ticket>, <repository and commit>, <runbook id>]
    tests_run: [<test>]
    not_tested: [<what was not tested>]
  provenance_stored: references_only
  declared_class: normal
  computed_class: normal             # computed from tier and catalogue, not declared
  plan: {add: 0, change: 0, destroy: 0, sensitive_types: []}
  policy_results: {gate: allow, supply_chain: "<note>", sbom_delta: 0, window_check: pass}
  approvers:
    - {id: <name>, kind: person, team: <team>}
  window: "2029-01-01 20:00-22:00"
  rollback:
    plan: "<how to undo>"
    reversibility: full              # full, partial, none
    tested_on: 2029-01-01
    result: passed
  stop_path: "<holders and mechanism>"
  verification:
    probe: <probe run by monitoring, not by the pipeline>
    result: "<result>"
    checked_at: 2029-01-01T00:00
  claim_not_made: "<what the change does not show>"

B.7 Runbook manifest (RBK-*)

  • Purpose: Makes a runbook executable under a bound: preconditions, a dry run, the action, an independent verification, what happens on failure, who may stop it and when it was last exercised.
  • When to use: For every runbook an agent or a pipeline may run; review at least every 90 days.
  • Inputs: Existing runbook text, delegation profile, mandate, probe inventory.
  • Outputs: runbooks/RBK-*.yaml.
  • Chapter: Chapter 10
runbook: RBK-XX-0000
title: <what the runbook does>
version: 1
service: SVC-XX-000
owner_unit: <unit>
accountable: <name>
decision_record: AIDR-0000
action_class: restart                # one of the classes of the delegation profile
delegation:
  profile: DLG-XX-OPS-000
  mandate: MND-XX-OPS-000
  level: Level 3 Bounded             # ceiling for this runbook
  never_above: Level 3 Bounded
preconditions:
  - target.role == "stateless-web"
  - fleet.healthy_after_action >= 4
  - fleet.healthy_after_action / fleet.size >= 0.75
  - calendar.freeze == false
  - runs.last_60_minutes.same_service < 3
  - evidence.captured == true        # logs and process state saved first
dry_run: required                    # plan and policy check, no effect
action:
  step: <step name>
  targets: 1
  max_runs_per_target_per_hour: 1
verification:
  probe: <probe outside the cluster and the agent's identity>
  location: <probe location>
  soak_minutes: 10
  pass_when: 5 consecutive passes at 60 second intervals
on_failure:
  rollback: <what is kept, what is added>
  then: escalate to incident commander, no second attempt
stop:
  held_by: [incident commander, agent operator, platform engineer on call]
  path: STOP-XX-ITOPS                # outside the agent platform, no approval needed
review:
  last_dry_run: 2029-01-01
  review_due: 2029-04-01
  approved_by: <name>

B.8 Post-incident review (PIR-*) and problem record (PRB-*)

  • Purpose: The review states who took part, how the timeline was drafted and verified, what independent accounts were written before the session, the contributing conditions with evidence and what AI did in the analysis. The problem record carries the known error, the workaround and the actions, each closed only by a verified re-test.
  • When to use: Review: for every incident that group policy or Article 13(2) of DORA, where it applies, requires one for. Problem record: whenever a condition is found that could produce further incidents.
  • Inputs: Incident record, change log, accounts, retrieved similar incidents.
  • Outputs: reviews/PIR-*.yaml, problems/PRB-*.yaml.
  • Chapter: Chapter 12
pir:
  id: PIR-XX-2029-0000
  incident: INC-XX-2029-0000
  held: 2029-01-01
  facilitator_role: <reviewer not on the incident>
  participants: [<roles>]
  timeline:
    drafted_by: <assistant id>
    drafting_service_version: <version>
    entries: 0
    verified_by_person: 0
    seeded_errors: {planted: 1, found: 0}
  independent_accounts_written_before_session: 0
  contributing_conditions:
    - id: C1
      text: "<condition>"
      evidence: [<record ids>]
  root_cause_for_reporting: C1       # reporting convention; the account is the whole set
  analysis_author_roles: [<roles>]
  ai_in_analysis: "<what AI did and did not do>"
  decision_record: AIDR-0000
  problem: PRB-XX-0000

prb:
  id: PRB-XX-0000
  opened: 2029-01-01
  owner_role: <role with authority to change the estate>
  service: SVC-XX-000
  linked_incidents: [INC-XX-2029-0000]
  known_error: true
  workaround: "<runbook section>"
  as_of: 2029-01-01
  actions:
    - id: A1
      text: "<action>"
      change: CHG-XX-2029-0000
      state: open                    # open, implemented, closed_verified
      retest: {type: game-day, id: GD-XX-2029-00, date: 2029-01-01, result: "<result>"}

B.9 Mandate manifest for an operations agent (MND-*)

  • Purpose: Bounds an agent's scope, action classes and levels, approvals, credentials, budgets, breakers, separation of duties, prohibitions, verification, memory and stop path, with a named owner, operators and level authority.
  • When to use: Before an agent is given an operations function; review at the date stated; expires on the date stated.
  • Inputs: Delegation profile, runbooks, containment profile, works agreement.
  • Outputs: mandates/MND-*.yaml and an AIDR.
  • Chapter: Chapter 16
mandate: MND-XX-OPS-000
version: 1.0.0
status: active
decision: AIDR-0000
effective: 2029-01-01
review_by: 2029-04-01
expires: 2029-07-01
agent:
  id: XX-OPS-000
  role: runbook-runner
  usage_mode: 4
  identity: <workload identity URI>
  model: <declared model family on file>
owner: <name>
operators: [<name>, <name>]          # the rota may stop without approval
level_authority: <name>              # approves any rise in level
scope:
  services: [SVC-XX-000]
  environments: [production, staging]
  blackout_calendar: calendars/<file>.yaml
actions:
  read:       {level: 3}
  annotate:   {level: 3}
  notify:     {level: 3}
  scale:      {level: 2, runbooks: [RBK-XX-0000]}
  restart:    {level: 2, runbooks: [RBK-XX-0000]}
  reconfigure: {level: 2, runbooks: [RBK-XX-0000]}
  roll_back:  {level: 1}
  fail_over:  {level: 1}
  deploy: undelegated
  delete: undelegated
  restore_overwrite: undelegated
approval:
  approvers: [duty-manager, platform-lead]
  max_age_minutes: 15
credentials:
  issuer: <token service>
  lifetime_minutes: 10
  audience: ops-policy-decision-point
budgets:
  actions_per_hour_per_service: 6
  actions_per_day: 20
  autonomy_budget: {window_days: 28, tolerance: "per Level 3 class, as in Chapter 4", review_at_percent: 70, on_full_consumption: demote_one_level}
breakers: [same_component_3x_in_30min, no_metric_change_after_action, scope_over_catalogue]
separation: [does_not_write_runbooks, does_not_verify_own_result, does_not_edit_observer_config]
never: [change_own_mandate_or_skills, silence_alert, disable_probe_or_logging,
        act_in_freeze_on_tier1, delete_data, restore_over_existing, decide_person_access]
verification: {probe: OBS-XX-000, independent_of_agent: true}
memory: {between_runs: none, run_log: append_only}
stop:
  authority: [duty-manager, platform-lead, service-desk-lead]
  path: out-of-band revocation at token service; verified by drill
  last_drill: 2029-01-01

B.10 Promotion and demotion policy (PRM-*)

  • Purpose: States the evidence required to raise an agent's level (shadow, offline, seeded, stop) and the events that lower it at once.
  • When to use: Before any rise in level, and when a model, skill or tool changes.
  • Inputs: Shadow-mode proposals, held-out cases, seeded-fault results, stop drills.
  • Outputs: mandates/PRM-*.yaml.
  • Chapter: Chapter 16
promotion_policy: PRM-XX-OPS-000
applies_to: MND-XX-OPS-000
decision: AIDR-0000
evidence_max_age_days: 90
steps:
  shadow:   {min_days: 30, min_proposals_per_runbook: 50, max_disagreement_rate: 0.05}
  offline:  {held_out_cases: 40, authored_by: not-agent-author, refusal_cases_min: 10}
  seeded:   {scope: written, window: designated, observer: OBS-XX-000}
  stop:     {max_seconds: 300, drill_age_days: 90}
promotion_authority: <body or role>
demotion:
  owner_or_commander: immediate
  autonomy_budget_exhausted: one_level_for_class
  independent_probe_near_miss: pending_review
  model_skill_or_tool_change: re-evaluate
  evidence_expired: level_0
manual_practice: {interval_days: 90, runbooks: [RBK-XX-0000], agent_off: true}

B.11 Observer manifest (OBS-*)

  • Purpose: Declares the probes, heartbeat and stop path that verify an agent's actions, and the independence of the observer from the performer in model, data, authorship, credential, failure domain and time.
  • When to use: Before an agent acts on a service, at every change of provider, and when the statement of independence expires.
  • Inputs: Mandate, service record, provider inventory, stop-path design.
  • Outputs: observers/OBS-*.yaml, an AIDR, drill records.
  • Chapter: Chapter 17
observer_manifest:
  id: OBS-XX-0001
  version: 1.0.0
  decision: AIDR-0000
  services: [SVC-XX-000]
  owner: <role>
  independence_reviewer: <role outside the owner's line>
  effective: 2029-01-01
  statement_of_independence_expires: 2029-04-01
  performer:
    agent: XX-OPS-000
    mandate: MND-XX-OPS-000
    runs_in: <location>
    service_runs_in: <location>
    telemetry_vendor: <vendor>
  probes:
    - id: OBS-XX-0001-P1
      claim: "<what the probe shows is true>"
      method: scripted-synthetic-transaction   # no model involved
      from: [<location outside the performer's failure domain>]
      interval_seconds: 60
      page_after_consecutive_failures: 2
      identity: <read-only observer identity>
  provider_status_feeds: [<provider>]          # telemetry, never a verdict
  heartbeat:
    receiver: <service outside the platform and telemetry vendor>
    interval_seconds: 60
    page_after_missed: 3
    page_channel: <channel not run by the platform or the vendor>
  stop_path:
    holders: [incident-commander, agent-operator, platform-engineer-on-call]
    approval: none
    mechanism: [revoke-workload-identity-at-idp-emergency-control,
                disable-orchestrator-trigger-at-network-edge]
    runs_outside: [<agent platform>, <service management>, <telemetry vendor>]
    target_seconds: 300
    last_drill: {id: STOP-XX-ITOPS-2029-Q1-01, seconds: 0, result: passed}
    next_drill: 2029-04-01
  delegation:
    restart: {level: "Level 3 Bounded", condition: "P1 and heartbeat green"}
    roll_back: {level: "Level 2 Supervised"}
    fail_over: {level: "Level 1 Assisted"}
  independence:
    model: "<how verification differs in model from the performer>"
    data: "<outcome probes, not the health endpoint>"
    authorship: "<who wrote the verification clauses>"
    credential: "<separate and read-only>"
    failure_domain: "<where the probes run>"
    time: "<when the result is re-verified>"

B.12 Seeded-fault record (SDF-*)

  • Purpose: Records a deliberately introduced fault, authorised and scoped, to test whether the agent's own verification and the independent probe notice it, and what followed.
  • When to use: Each quarter per autonomy-budget class, and when a mandate, runbook or probe changes. Pre-production or in an authorised window only.
  • Inputs: Hypothesis, authorisation, change record, undo plan.
  • Outputs: seeded/SDF-*.yaml, a review (PIR-*).
  • Chapter: Chapter 17
seeded_fault:
  id: SDF-XX-2029-000
  kind: silent-failure
  service: SVC-XX-000                # pre-production mirror only
  authorised_by: <role>
  change_record: CHG-XX-0000
  seeded_on: 2029-01-01
  author: <person outside the agent team>
  hypothesis: "<what the independent probe should show>"
  seed: "<the fault, described by effect>"
  stop: "<who can end the exercise and how>"
  undo: "<how it is reversed>"
  observed:
    agent_verification: <passed or failed>
    agent_report: "<what the agent reported>"
    independent_probe_first_failure_seconds: 0
    independent_probe_page_seconds: 0
    exercise_ended_by: <role>
  result: "<who caught the seed>"
  review: PIR-XX-2029-0000
  findings:
    - "<finding>"

B.13 Tier-0 checklist for the operator's AI

  • Purpose: Controls for the remediation agents, runbook libraries, observability pipelines and configuration maps that hold the estate's strongest credentials: classification, access, segregation, supply chain, content classes and the ceiling suite.
  • When to use: Before an operations agent goes live, and annually.
  • Inputs: Asset classification, access model, supply-chain inventory, ceiling suite results.
  • Outputs: Completed checklist and evidence.
  • Chapter: Chapter 18

B.14 Restore-test record (RST-*) and resilience register row

  • Purpose: Records a restore or failover test with its scenario, objective, measured time and point, independent verification, result and effect on the claim, and the conditions for the next pass. The register row holds one claim with its latest result and its expiry.
  • When to use: For every recoverability claim; a failed result withdraws the claim and opens a problem record.
  • Inputs: Runbook, objective, backup copy, independent probe, attestor.
  • Outputs: restore-tests/RST-*.yaml, register row.
  • Chapter: Chapter 19
restore_test: RST-XX-2029-000
status: planned                      # planned, passed, failed
service: SVC-XX-000
claim: <claim name, for example restore-from-immutable-backup>
scenario: >
  <the loss, the source copy and the target environment>
objective: {slo: SLO-XX-000, rto: PT4H, rpo: PT15M}
executed:
  date: 2029-01-01
  declared_by: <role>
  runbook: RBK-XX-0000@1.0.0
  runner: <roles>
  source_copy: <copy and failure domain>
measured:
  rto: PT0H
  rpo: PT0M
  phases: {}                         # one entry per phase, summing to the measured time
verification:
  probe: <probe independent of the runner's tooling and credentials>
  integrity: pass
  reconciliation: pass
  synthetic_transaction: pass
result:
  rto_met: true
  rpo_met: true
  outcome: passed
attested_by: {role: <role>, independent_of_runner: true}
claim_effect: verified               # verified or withdrawn
expiry: 2029-07-01                   # clock; ends earlier on listed changes
conditions_for_next_pass: []

B.15 Role charter for the operations conductor (ROLE-*)

  • Purpose: Defines the accountable person for a loop over a set of services and a window: mandate, decision rights, prohibitions, independence, competence and the protection of the holder.
  • When to use: When a rota or post takes on the conduct of agents, and at review.
  • Inputs: Operating model, delegation profiles, works agreement.
  • Outputs: roles/ROLE-*.yaml.
  • Chapter: Chapter 21
role: ROLE-XX-OPS-000
title: <operations conductor, window>
version: 1.0
effective: 2029-01-01
owner: <role>
reviewers:
  - <second-line role>
  - <employee representative, under the works agreement>
mandate:
  services: [SVC-XX-000]
  window: "22:00-06:00 CET"
  loop_functions: [O1, O2, O3, O4, O5]
  agents_conducted: [XX-OPS-000]
decision_rights:
  may: [stop_any_operations_agent, approve_level_2_action, declare_tier_1_incident]
  referred_to: "delegation profiles DLG-XX-*"
  stop_right: no_approval_needed
  stop_path: outside the agents' platform
never_delegated:
  - irreversible action on production data
  - action outside an authorised change window or freeze on a critical function
  - action that disables observation or the stop path
  - any decision about a named person's access to work
independence:
  verification_of_agent_action: independent probe, not the acting agent
  conductor_may_not_verify_own_approval: true
competence:
  evidenced_by: exercises (CMP-XX-* records), not certificates
  minimum_evidenced_holders_per_window: 2
  fallback_if_fewer: Level 3 classes fall to Level 2
  manual_practice: one agents-off exercise per holder per quarter
people:
  metrics_reported_for: [teams, services]
  per_named_person_reporting: false
  minimum_group_size: 5
  agreement: <works agreement id>
review: 2029-07-01

B.16 On-call charter

  • Purpose: Sets the terms of the rota that the operations conductor and the agents' operators serve: coverage, response targets, the noise budget, rest after a night call-out, escalation, handover, and the rule that measures are reported for the rota and never for a named engineer.
  • When to use: When a rota is set up or changed, agreed with employee representatives and reviewed monthly or as the agreement requires. This template is a skeleton made for this appendix; it follows the noise-budget figures of Chapter 6 and the reporting rule of the operations annexes, and the chapter that develops it is not yet in its final form.
  • Inputs: Works agreement, working-time rules, noise-budget record, service tiers.
  • Outputs: oncall/CHARTER-*.yaml and the signed agreement.
  • Chapter: Chapter 25
oncall_charter: OCC-XX-000
version: 1.0
effective: 2029-01-01
rota: <rota name>
services: [SVC-XX-000]
coverage: {hours: "24x7", tiers: [1], response_minutes_tier1: 15}
agreement: <works agreement id>        # co-determination or consultation as national law requires
noise_budget:
  pages_per_shift_max: 20
  actionable_share_min: 0.5
  non_tier1_pages_out_of_hours_max: <figure>
  reported_for: rota                   # never per named engineer
  review: monthly
rest: {after_night_call_out: <hours, per the working-time rules>, compensation: <per agreement>}
handover: {record: required, includes: [open incidents, active freezes, agent levels in force]}
escalation: [<role>, <role>, <role>]
agent_duties: {agent_operator_on_rota: true, stop_drill: quarterly}
measures:
  reported_for: [teams, services]
  per_named_person_reporting: false
  minimum_group_size: 5
review: 2029-07-01

B.17 Competence record (CMP-*)

  • Purpose: Records the competences an operator has shown, in the assisted and the unassisted form, with the exercise, the assessors, the purposes for which the record may and may not be used, who may read it, and a development plan.
  • When to use: For each holder of an operations conductor or agent-operator role, each year, and after an agents-off exercise. Records of judgement exercised are personal data: exercise evidence only, no ranking, the subject reads all of it, and use agreed with employee representatives.
  • Inputs: Role charter, exercise results, learning paths.
  • Outputs: competence/CMP-*.yaml.
  • Chapter: Chapter 26
# competence/records/CMP-XX-00.yaml (skeleton)
competence_record:
  id: CMP-XX-00
  person_ref: <pseudonymous reference>
  role: ROLE-XX-OPS-000
  framework: {id: <framework id>, version: "2029.1"}
  assessed_on: 2029-01-01
  exercise: <exercise id>              # for example an agents-off restore or a stop drill
  assessors: [<roles>]
  purposes: [development_plan, input_to_conductor_roster]
  not_used_for: [individual_performance_ranking, pay_setting, disciplinary_process]
  readers: [record_subject, <line owner>, <hr competence partner>]
  competences:                          # both forms; the lower counts for the roster
    - {id: <competence id>, level: practitioner, unassisted: <exercise/part>, assisted: <exercise/part>}
    - {id: <competence id>, level: foundation, evidence: <record id>}
  subject_comment: <comment of the person assessed>
  development_plan:
    learning_path: <path id>
    protected_time_share: 0.05
    items:
      - {competence: <id>, target: practitioner, by: 2029-06-30, means: "<how>"}
  valid_until: 2030-01-01

Checklists

B.18 Autonomy-budget review checklist

Run at the review date of each delegation profile (90 days by default) and whenever the budget is exhausted or a near miss is recorded.

  1. The delegation profile names the decision record, the accountable owner and the raise authority, and the version and date are current.
  2. Every action class the agent has used appears in the profile; any class not listed is undelegated, and the policy decision point denies it by default.
  3. No class is at Level 4 Autonomous that is irreversible, outside an authorised window or freeze on a critical function, disables observation or the stop path, or decides about a named person's access to work.
  4. The budget window, the tolerance per Level 3 class and the review and drop thresholds (70 and 100 per cent) are stated, and the observer that counts them is named and independent of the agent.
  5. The count for the window is recorded per class, with the consumption as a percentage of the tolerance: reversals, failed verifications, refused attempts and customer-visible effects.
  6. Any class that reached 70 per cent has a review on record, any class that reached 100 per cent has fallen one level, and any rise is signed by the raise authority with evidence no older than the stated maximum age.
  7. The stop path was drilled within 90 days, ran outside the agent platform, and the measured time is recorded against the target.
  8. The seeded-fault and held-out results still stand: no model, skill, tool or runbook change since they were produced, or they have been repeated.
  9. Employee representatives have agreed the logging of staff commands, and the figures are reported for the team and the service, never for a named person.
  10. The next review date is set, and a short record of the decision (AIDR) is filed.

B.19 Observer review checklist

Run at each change of provider, model, probe or failure domain, and when the statement of independence expires.

  1. The observer manifest lists each service verified and each claim a probe shows, and every Level 3 class on the service has a probe that checks its effect.
  2. Each probe measures an outcome (a transaction, a route, a result) and not an endpoint that the action resets or the agent can write.
  3. Independence is stated for model, data, authorship, credential, failure domain and time, and each statement was checked by a reviewer outside the owner's line.
  4. The probe identities are separate from the agent's and read-only, apart from what the probe needs to create a test record.
  5. The probes run outside the failure domain of the service and of the agent platform; a provider's status feed is treated as telemetry and never as a verdict.
  6. The heartbeat is received outside the platform and the telemetry vendor, and the paging channel does not depend on either; the heartbeat is on the never-suppress list.
  7. A seeded fault in the quarter showed whether the independent probe caught what the agent's verification missed, and any miss has a finding and a date.
  8. The stop path works when the platform does not, and the last drill time met the target.
  9. Verification clauses were not drafted by the model family that drives the agent, or a person reviewed each one.
  10. The statement of independence has a new expiry date, and the decision is recorded.

B.20 Quarterly resilience register review checklist

Run each quarter by the service owners, the head of operational resilience (or the equivalent role) and the incident commander rota.

  1. The register has one row per claim, including claims never tested; the denominator is every claim the organisation makes.
  2. The status of every row is derived by the rule in the repository and not edited by hand; the rule is unchanged or the change is recorded.
  3. Each verified claim has a measured time and point from a probe independent of the runner, an attestation by a person who did not run the test, a date and an expiry.
  4. Rows past their clock, or touched by a change that is listed as a condition, show expired; none is left verified by omission.
  5. Every claim for a system supporting a critical or important function has a test within the legal floor (yearly under Articles 11(6)(a) and 24(6) of DORA, where it applies) and within the book's shorter clock, if the organisation adopts one.
  6. At least one test in the year was run by someone who did not write the runbook, and at least one claim that depends on an agent has been tested with the agent off.
  7. Each failed test has withdrawn its claim, opened a problem record and set the conditions for the next pass; no finding is closed without a passing re-test.
  8. The restore tests are placed within the working-time rules and the works agreement.
  9. The report to the management body shows the denominator, the count in each status, the age of the verified claims and at most three decisions it is asked to take.
  10. The next quarter's tests are scheduled by earliest expiry among the highest tier.

B.21 Checklist: reviewing a sync policy before it is switched on

Run by the service owner, the platform owner and a reviewer outside the reconciler's line, before automated sync, self-heal or prune is enabled for an application or a cell, and at each change of the engine's major or minor line. Chapter 8 gives the reasoning and Chapter 18 the protection of the engine.

  1. The change record names the reconciler: the engine, push or pull, the applying identity, the sync policy and what the engine may do alone.
  2. Automated sync is limited to catalogued standard classes that passed the gate; for any other change a person triggers the sync.
  3. Self-heal is on only for catalogued drift classes outside a freeze, and never for the observer's own configuration; a person's reversal of a self-heal is counted against the budget of the drift class.
  4. Prune is off for stateful kinds, or a named person approves each prune; the guard against an empty target is on; no setting makes a prune of data possible at Level 4.
  5. Every application has an allow window, because an application with no matching window is open; the freeze calendar is held by the policy decision point, and the windows are a second line.
  6. Each ignore rule has an owner and an expiry, covers expected drift only, and is a protected-path change.
  7. The engine's credentials are narrowed to the namespaces and kinds it manages; the engine, its credentials, its repository and its project or Kustomization objects are protected paths; a token that can trigger a sync is held as the reconciler's own reach.
  8. Retry has a limit, and unlimited retry is refused.
  9. Verification after sync is run by a probe that the reconciler and the agent do not control; a hook or post-sync test the engine runs is the releaser's check.
  10. The stop path (suspend, disable automated sync, or revoke the credential) is held by named people without approval, works when the engine does not, was drilled within 90 days, and the way back is a Git revert.
  11. The engine's version is on a supported line, any beta feature in use has an owner and an exit, and the review date is set.

Repository tools

Tool Purpose
scripts/toc_definition.py Holds the table of contents and regenerates book_index.json, the MkDocs navigation and the chapter list in BOOK_REQUIREMENTS.md
scripts/generate_outlines.py Regenerates the chapter briefs in BOOK_BLUEPRINT.md and outline files for chapters not yet written
scripts/format_citations.py Derives citation labels from the bibliography
scripts/renumber_sources.py Keeps the bibliography numbered 1..N without gaps or uncited entries
scripts/check_mermaid_diagrams.py Renders and validates the diagrams

Open-source and standard tools referred to in the book

The tools below are those the chapters cite from the vendors' or projects' own documentation. Naming a tool is not a recommendation, and the organisations of the book use equivalents. Product capabilities change, so check the documentation of the version in use.

Tool or specification Use in the templates Documentation cited
OpenTelemetry (signals, sampling, handling of sensitive data) Collector processors that minimise personal data before telemetry leaves the service (telemetry in B.3); the signals and sampling concepts behind B.4 (49, OpenTelemetry, n.d.), (50, OpenTelemetry, n.d.), (51, OpenTelemetry, n.d.)
OpenTelemetry Semantic Conventions for Generative AI Common attribute names for the telemetry of AI clients and agents, useful in the run log of B.9 (61, OpenTelemetry, n.d.)
Open Policy Agent and the Rego policy language A policy decision point outside the agent platform (B.2, B.6, B.9), and policy checks on infrastructure plans such as Terraform (56, Open Policy Agent, n.d.), (65, Open Policy Agent, n.d.)
Prometheus Operator Watchdog alert The pattern of an alert that always fires, so that its absence shows a broken alerting path; the basis of the heartbeat in B.11 when the receiver sits outside the platform (263, Prometheus Operator, n.d.)
Argo CD A pull-based reconciler with sync policy, sync windows, projects and ApplicationSets (B.6; B.21; Appendix A.21) Chapter 8; protected as Tier 0 in Chapter 18
Flux A pull-based reconciler with Kustomization, suspend and tenant lockdown (B.6; B.21) Chapter 8; Chapter 18
Argo Rollouts, Flagger, Kargo Progressive delivery and promotion with analysis and abort (B.7 verification) Chapter 9
Kubernetes controllers Continuous restart, replacement and scaling declared in the service record (B.1) Chapters 8 and 10
Terraform and OpenTofu Planned and applied infrastructure change, drift detection by read-only plan (B.6) Chapter 8; state as a secret in Chapter 18
Crossplane Cloud resources reconciled through Kubernetes objects (B.6) Chapter 8
Temporal and Argo Workflows Durable runbook execution (B.7) Chapter 10; agent plans as workflows in Chapter 16

The service level objective method behind B.3 follows the chapter on service level objectives in Site Reliability Engineering (241, Jones et al., 2017). The regulatory instruments the templates answer to are DORA (2, European Union, 2022) and NIS 2 (3, European Union, 2022); Chapter 23 gives the mapping, and none of the templates is legal advice.

Sources

  1. OpenTelemetry. Signals; Sampling; Handling sensitive data. OpenTelemetry documentation. (49, OpenTelemetry, n.d.), (50, OpenTelemetry, n.d.), (51, OpenTelemetry, n.d.)
  2. OpenTelemetry. Semantic Conventions for Generative AI (GenAI). (61, OpenTelemetry, n.d.)
  3. Open Policy Agent. Terraform; Policy Language. (56, Open Policy Agent, n.d.), (65, Open Policy Agent, n.d.)
  4. Prometheus Operator. Watchdog. (263, Prometheus Operator, n.d.)
  5. Jones, C., Wilkes, J. & Murphy, N. (2017). Service Level Objectives. (241, Jones et al., 2017)
  6. European Parliament and Council of the European Union (2022). DORA; NIS 2. (2, European Union, 2022), (3, European Union, 2022)
  7. Argo Project; Flux project; The Kubernetes Authors; HashiCorp; OpenTofu; Crossplane Authors. Documentation of the state engines (Argo CD, Flux, Kubernetes controllers, Terraform, OpenTofu, Crossplane). (153, The Kubernetes Authors, n.d.), (154, Argo Project, n.d.), (155, Argo Project, n.d.), (160, Flux project, n.d.), (161, Flux project, n.d.), (162, HashiCorp, n.d.), (164, OpenTofu, n.d.), (165, Crossplane Authors, n.d.)
  8. Argo Project; Flagger project; Akuity. Argo Rollouts; Flagger; Kargo. (169, Argo Project, n.d.), (171, Flagger project, n.d.), (172, Akuity, n.d.)
  9. Temporal Technologies; Argo Project. Temporal documentation; Argo Workflows releases. (199, Temporal Technologies, n.d.), (201, Argo Project, 2026)