AI tools can be unavailable. Your production systems cannot be. The AI-Absent Drill is the exercise that closes the gap between those two facts, and the only way to find out which gap your team is carrying before a live incident reveals it.
Production operations teams have absorbed AI tools into their daily work faster than almost any other engineering function. AI assistants help on-call engineers diagnose slow queries, interpret unfamiliar log patterns, generate remediation commands, search runbook libraries, explain stack traces, and suggest configuration changes, in minutes rather than hours. The productivity gain in production support is real and significant.
The dependency that comes with it is equally real, and in most organisations, unmeasured. When an AI tool is unavailable, due to a vendor outage, API rate limits, network restrictions during a security incident, or simply a context window that has run out at the wrong moment, the on-call engineer faces their most stressful scenario without the tool they have come to rely on most.
The AI-Absent Drill does not question whether AI tools should be used in production operations. They should. It tests whether the team can operate without them when they have to, and produces a measured, auditable score that closes Gate 5 of the Navigator Framework.
Put differently: the drill is the question stated as a measurement. The agent drives. The human navigates. When the agent disappears, did the human navigators retain the capacity to operate the systems they were navigating, or were they only ever along for the ride?
Why this is a production operations concern, not a development one.
The AI-Absent Drill is Gate 5 of the Navigator Framework, the Run gate. It is not a development lifecycle control. It belongs in production operations because the risk it addresses is an operational risk: the possibility that on-call engineers cannot diagnose and recover production systems without AI tool assistance.
This distinction matters for how the drill is scoped, who runs it, and what scenarios it tests. The drill does not test whether engineers can write code without AI. It tests whether they can operate, diagnose, and recover the production systems they are responsible for, under the time pressure of a real incident, without the AI tools they typically reach for first.
The scenarios are production support scenarios. The skills being tested are diagnostic and recovery skills, not development skills. The tool restrictions are operational tool restrictions. The clock is the RTO the team has declared for these systems.
"The drill is narrow, measurable, and practical. Not: can engineers work without AI? But: can this team diagnose and recover these systems, within this RTO, without these tools?"
Production scenarios the drill should test.
The drill tests the top-N fault scenarios for the systems in scope, based on incident history and risk level. These are not hypothetical scenarios. They are the faults the team has actually encountered or knows to be likely based on the system's failure modes. Selecting scenarios from historical incidents ensures the drill tests real operational knowledge gaps, not abstract capabilities.
-
Database performance degradationTypically handled with AI: Query analysis, slow log interpretation, index recommendation, connection pool diagnosis. Drill tests: Can the team diagnose root cause and remediate from raw slow query logs and system metrics alone?
-
API cascade timeoutTypically handled with AI: Distributed trace interpretation, upstream dependency identification, timeout cascade analysis. Drill tests: Can the team trace the cascade from raw trace logs and identify the failing upstream dependency without AI-assisted trace analysis?
-
Memory leak and pod restart loopTypically handled with AI: Heap dump analysis, stack trace interpretation, remediation command generation. Drill tests: Can the team identify the leaking service from raw metrics, execute a controlled rollback, and restore service within RTO?
-
Error rate spike post-deploymentTypically handled with AI: Log pattern analysis, error classification, rollback procedure generation. Drill tests: Can the team confirm the deployment as root cause, execute rollback from the runbook, and verify recovery without AI assistance?
-
Cache invalidation causing load spikeTypically handled with AI: Cache miss rate analysis, origin server impact calculation, rate limiting recommendation. Drill tests: Can the team diagnose the cache failure from metrics alone and implement a mitigation without AI-generated commands?
-
Authentication failure, certificate or service accountTypically handled with AI: Certificate chain analysis, error message interpretation, renewal procedure lookup. Drill tests: Can the team identify the expired credential from raw error logs and execute renewal from existing runbooks within RTO?
Choosing the top-N scenarios. Select from your last 12 months of production incidents. Take the five to ten faults that most commonly required significant AI tool assistance to diagnose. These are the scenarios where your operational AI dependency is highest, and therefore where the drill will be most diagnostic.
How to set it up, three decisions.
-
Scope: systems and scenariosSelect the production systems in scope for this drill, typically the highest-risk or most AI-tool-dependent systems from the team's responsibility. Select the top-N fault scenarios from incident history. The drill should cover the scenarios most likely to occur and most dependent on AI assistance to resolve. Start with five scenarios for the first drill; expand as the process matures.
-
Duration: the AI-absent windowDeclare the period during which AI tools are restricted. Typically 2–4 hours. The window should be long enough to cover multiple diagnostic scenarios but short enough to be repeatable quarterly without significant operational disruption. The declared RTO for each system sets the time constraint within the window, engineers must diagnose and recover within the RTO, not just within the drill window.
-
Restriction approachThree practical approaches: block AI tool API endpoints at the network level for the drill window (most rigorous), rotate to non-AI-assisted tools for the period (operational), or run in a sandboxed environment without AI tool access (safest for live systems). Run the drill unannounced or minimally announced, advance notice allows preparation that removes the diagnostic value. The drill should replicate the conditions of an actual AI tool outage, not a training exercise.
How to score it, the Diagnosis Success Rate.
The drill produces one primary metric: the Diagnosis Success Rate, the percentage of the top-N fault scenarios that the on-call team correctly diagnoses and recovers within the declared RTO, without AI tool assistance.
| Risk class | Threshold | Gate 5 outcome | Interpretation |
|---|---|---|---|
| Standard | ≥70% | Gate 5 closes | Team can operate without AI tools for the majority of common fault scenarios. Gaps identified for targeted improvement. |
| Elevated | ≥80% | Gate 5 closes | Higher threshold reflects customer-facing and PII-handling systems where operational failure has direct client impact. |
| High / Critical | ≥90% | Gate 5 closes | Near-complete operational independence required. Any AI tool unavailability should have minimal impact on incident recovery time. |
Below threshold. Gate 5 does not close. The organisation has measured a real operational gap, not a team failure. The gap tells exactly which scenarios the team cannot handle without AI, which becomes the remediation roadmap.
The quarterly drill cycle, cadence by risk class.
A single drill score is a snapshot. The drill's real governance value comes from running it quarterly and tracking the trend, because AI usage patterns, system complexity, and team composition change constantly. A team that scored 85% in Q1 may score 72% in Q3 if AI tool dependency has grown faster than operational capability. The trend is as important as the individual score.
The cycle. Score the drill (Diagnosis Success Rate) → identify gaps (which scenarios failed) → remediate (runbooks, knowledge, tools) → re-test next quarter (same conditions) → track trend (score over time).
-
Standard risk · quarterly minimumRun once per quarter regardless of AI usage changes. Update scenarios annually based on incident history.
-
Elevated risk · quarterly + triggeredQuarterly mandatory. Triggered additionally by any material change in AI tool usage or dependency in operations workflows.
-
High / Critical risk · monthly or triggeredMonthly cadence or triggered by any significant model change, infrastructure change, or new AI tool adoption in production operations.
What the score tells you, and what to do with it.
A score at or above threshold is operational evidence. It closes Gate 5 of the Navigator Framework and provides the attestation that the team can operate this system when AI tools are unavailable. In a Navigator Audit, this evidence is the proof of operational resilience, not a policy that says the team can operate without AI, but a measured score that proves it.
Meets threshold. Gate 5 closes. Document the score, the scenarios tested, and the drill conditions. This becomes the operational resilience evidence in the Navigator evidence pack, and the baseline for the next quarter's drill.
A score below threshold is information, not failure. It tells exactly which production scenarios the team cannot handle without AI assistance, which is precisely the information needed to remediate effectively. A below-threshold score that leads to targeted runbook improvements, knowledge transfer sessions, and tool diversification produces a better-prepared team than a team that never discovered the gap.
The four remediation actions that follow a below-threshold score:
Update runbooks, for the specific scenarios that failed. Runbooks should allow a trained on-call engineer to diagnose and recover without AI assistance. If they do not, the runbook is incomplete.
Close knowledge gaps, identify which team members could not diagnose which scenarios, and structure knowledge transfer from engineers who can. System-specific knowledge should not live only in AI context.
Diversify tools, identify which AI tools have no non-AI equivalent in the current toolset. Build or adopt non-AI alternatives for the highest-dependency diagnostic functions.
Run targeted practice drills, for the specific failed scenarios, before the next quarterly drill. Targeted practice on the known gaps is more effective than another full drill before the gaps are closed.
What the drill is not.
Three clarifications that prevent the drill from being misapplied:
Not a general productivity test. The drill does not measure how productive engineers are without AI tools. It measures whether they can diagnose and recover specific production systems within a declared RTO. Productivity is not the variable being tested, operational capability is.
Not about speed for the sake of speed. The RTO threshold comes from the system's declared recovery time objective, not from an arbitrary speed target. Engineers who diagnose correctly but slowly are more valuable than engineers who recover quickly but incompletely.
Not a replacement for incident response drills. The AI-Absent Drill tests one specific capability: operating without AI tools. It runs alongside, not instead of, regular incident response rehearsals, chaos engineering exercises, and gameday drills.
Operational resilience is a capability. Test it. Measure it. Prove it.