Earlier study: approval decisions and human interventions

The original project was a compliance study: how do AI agents handle approval decisions in a multi-operator environment? We built Clide — a sandboxed terminal that captures every tool call, approval decision, and operator action as structured telemetry. 26+ operators ran 1,431 sessions. We collected 28,224 enriched decisions and found that 91.5% were auto-approved, with a deny rate of just 0.3%.

We built a classifier that reached 96.0% accuracy on approve/deny under 5-fold stratified cross-validation, with a Wilson 95% confidence interval of [95.7%, 96.3%]. XGBoost did marginally better (96.4% [96.1%, 96.7%]). Both models barely beat the 93.3% baseline of always predicting "approve" — a 2.7-point lift the model paid for almost entirely by learning class skew rather than command semantics. That's when we knew we were asking the wrong question.

We shifted focus from compliance classification to intent reconstruction — can we recover what the human actually meant from a natural language instruction, trace how operators interpreted it, and measure whether the executed action matched the original intent?

This is harder than it sounds. A single instruction like "update the metrics and push to the site" fans out across multiple operators, spawns subagent calls, touches several repos, and produces commits that may or may not reflect what was asked. The gap between instruction and action is where alignment lives.

Once we stopped trying to predict approval and started measuring interventions directly, five findings fell out — each from a different angle on the same question: when does Eric (the human in the loop) reach over and stop the train?

Counting Ctrl-C is harder than it looks. A naive transcript scan finds 190 events. A four-marker family scan with per-second dedup finds 311. A parallel mitmproxy intercept of the agent CLI's analytics endpoint finds six event types the transcripts don't record, and extrapolating from the intercept sample to the full corpus puts the upper bound near ~5,365 events (range [4,200, 7,000] under MCAR, ~2,800 [1,800, 4,400] under engagement-correlated reweighting). The honest answer is the band, not the point. The newer agent-CLI version deliberately deletes user prompts from transcripts after an interrupt, so the floor is the only thing we can audit; the ceiling rests on extrapolation.

87% of interrupts hit a bash command. Joining the 961-event interrupt set to each operator's pre-interrupt action puts the bash share at 87.0% — about 24 percentage points above the corpus baseline. When Eric stops an operator, it's overwhelmingly mid-bash. And 70.7% of the time the operator retries the same tool right after — interrupts are course-corrections, not kill-switches.

Interrupted hours have 3.7× the tool-call density of non-interrupted hours. But the tool mix is essentially identical (Bash 73.7% vs 76.2% in non-interrupted, Read +2.5pp, Edit nearly flat). Same shape of work, very different throughput. The interrupt rate isn't tracking weird tool-mix patterns — it's tracking engagement intensity. When an operator is firing one tool every 23 seconds, Eric is about three times more likely to interrupt. When they're firing one tool every minute and a half, he almost never does.

IRC chatter and Ctrl-C are the same engagement signal. Joining 6,623 messages Eric typed into IRC to the same per-(operator, hour) buckets: in hours where he sent zero messages, the interrupt rate sat at 6.8%. In hours where he sent six to twenty, it doubled to 14.4%. The hours Eric is most actively typing into the operator's IRC channel are also the hours he is most likely to reach over and stop them — and those are the hours the operator is firing tools fastest. The two channels aren't independent.

Eric steers, doesn't stop. Joining the 961 interrupts to the 2,985-commit git history by operator and timestamp: only 22.3% of interrupts are commit-adjacent (within ±6h of any commit by the same operator). Of those 214 commit-adjacent interrupts, 75.7% are followed by a commit within +6 hours. Only 5.4% are abandonments. The interrupt is, in the strongest empirical form, a steering signal — the operator commits a course-correction, usually within minutes. One concrete sequence: an operator commits to a repo at 22:25:07, gets stopped at 22:25:37 (30 seconds later), and ships a recovery commit at 22:27:36 — two minutes from Ctrl-C to course-corrected commit. Across all 39 mid-run interrupts in the corpus, the median is under ten minutes.

The last slice explains the apparent paradox of the four findings above: interrupt rate doesn't correlate with the labeling rubric's risk score because it's tracking something else. Per-operator commit-adjacency varies from 0% (a classification-only operator who never commits) to 100% (a heavy-engineering operator whose every interrupt lands during shipping work). Engineering-shaped operators get steered while shipping; coordination-shaped operators get steered while coordinating. Both are alignment shaping. They're shaping different things — which is why a one-size rubric can't predict either.

The CS522 course report covers these findings with tables, confidence intervals, and methodology. Its bibliographic entry appears under Course reports and working papers; a public report file is not linked here.

Feature engineering pipeline. 47 engineered features across eight groups — base extraction, structural syntax, domain tags, path and complexity, cluster and auto-tags, role-fit, python-call sub-features, and per-pass labels. The role-fit features operationalize alignment-as-domain-fit at the per-action level: every network call gets a destination service, every command gets a target-repo, every operator gets a role-match flag against their documented domain. 38.3% of records show a role-fit flag that isn't a clean match — most of those are legitimate cross-team work, not misalignment.

3-dimensional labeling. We moved beyond binary aligned/misaligned to a richer framing:

• Alignment — did the action match the instruction?
• Risk — what was the blast radius if it went wrong?
• Outcome — what actually happened?

This lets us distinguish between an operator who misunderstood but did something harmless vs. one who understood perfectly but took a risky shortcut.

Captured across continuous multi-agent operation — 26+ operators, 1,431 sessions, 103 repositories:

1. Tighter labeling rubric. The current rubric collapses half a dozen distinct intent categories under one "risky" bucket, which is where most of the per-archetype variance comes from. A revised rubric with granular sub-categories (risky-process, risky-mass, risky-cred) is in progress and the corpus will be re-labeled against it.
2. Multi-host capture coverage. About a third of the intervention activity in the current corpus is from the bare-metal hub; the other two-thirds come from a home-network server through conversation backups. Live capture on the off-host machines is the largest single coverage gap.
3. Lexical tokenization for the safety classifier. The paper shows command-text tokenization recovers labels at four to five times the lift of the boolean feature flags the production rule system uses. A tokenizing classifier would access most of that remaining headroom without changing model architecture.
4. Pattern library. A small set of named operational antipatterns — workspace-prep rm -rf + git clone, credential-inlining in fleet-management calls — turns out to be a tractable target for the permission system. A regex bank catches the genuine failure surface without the false-alarm rate of the broader syntax filters.

• We Were Asking the Wrong Question: From Safety Classification to Alignment Detection in Multi-Agent Permission Systems — CS 522 (Data Management, ODU, Spring 2026), 17 pages, IEEE format. Full empirical case for the safety-to-alignment reframe, including the five findings above with confidence intervals, methodology, and the multi-channel intervention decomposition. Course report; publication acceptance is not established here.

• MODSIM World 2020 — First-author research on data de-identification and synthetic data security, including analysis of re-identification attacks.