# From evaluation scores to enforceable delegation

Published: 2026-10-07 · Edition source review: 2026-10-07

A reviewer benchmark, an evaluation-to-control workflow, and a contextual security agenda sharpen the next decision about agent authority.

An enterprise can reduce routine human intervention without treating a model’s confidence as permission to act. The practical move is to define a limited task, test the behavior that matters, enforce its authority outside the acting model, and retain enough evidence to revisit the decision. Human-on-the-loop operation then becomes an operating arrangement with observable limits and an intervention path, rather than a promise that someone is watching.

This edition connects a new reviewer benchmark, an available evaluation-to-control workflow, and a contextual security research agenda. Their maturity differs. None establishes a self-governing enterprise factory. Together they sharpen what a delivery team should ask before giving an agent more responsibility. The briefs report their sources; the decision and exercise below are AISDLC proposals.

## GitHub opens a code-review benchmark with explicit limits on its scores

GitHub · Source published 2026-10-05 · Reviewed 2026-10-07

GitHub released ReviewBench as a research preview, evaluating 219 public pull requests across 187 repositories and 19 languages. Its analysis of 103.9 million pull requests informed population distributions; that was not the evaluation sample. The versioned methodology reports 96.6% agreement with human true/false-positive labels but 62.9% exact severity agreement. It warns that augmented recall uses an agent-dependent denominator. These results support reviewer comparison, with explicit judge and sampling limits; they do not establish release readiness for a particular application.

**Why it matters — AISDLC analysis:** Compare reviewers against a fixed set of known findings, then inspect novel findings separately. Calibrate severity judgments against your own consequences before using them to determine escalation or permitted authority.

[Primary source: ReviewBench: An open benchmark for AI code review](https://github.blog/ai-and-ml/github-copilot/reviewbench-an-open-benchmark-for-ai-code-review/)

[Supporting source: ReviewBench methodology, pinned to its October 5 revision](https://github.com/review-bench/ReviewBench/blob/8486c286e95e6d5a73167edd78921a0e2adf709d/docs/METHODOLOGY.md)

[Measure accepted outcomes and verification effort](https://aisdlc.ai/insights/ai-native-sdlc-measure-accepted-outcomes)

## Microsoft connects a measured agent failure to reviewed runtime controls

Microsoft · Source published 2026-09-24 · Reviewed 2026-10-07

Microsoft’s available run-assert-eval skill connects risk discovery, evaluation, and reviewed runtime policy. A billing-support demonstration reported cross-customer disclosure in 12/40 applicable baseline conversations and 2/34 governed conversations. Remaining violations matter. The comparison retained behavior definitions, cases, and judging approach while adding tool-boundary controls. Policy generation required review before activation. This small, model-judged example is not production assurance; a repeatable release gate is described as future work.

**Why it matters — AISDLC analysis:** Measure forbidden actions and legitimate work separately. A mitigation should reduce the measured failure without succeeding merely by refusing useful requests; preserve applicable denominators and every remaining failure.

[Primary source: Introducing run-assert-eval: Find the risk, fix it, prove it](https://commandline.microsoft.com/run-assert-eval-responsible-ai-agent-risk-discovery-at-runtime/)

[Design policy and evidence boundaries](https://aisdlc.ai/policies)

## Google Research proposes a contextual supervisor for agent actions

Google Research · Source published 2026-10-05 · Reviewed 2026-10-07

Google Research published a workshop-based research agenda for contextual agent privacy and security. It proposes a supervisory policy engine that evaluates data flows and actions against their context, including actors, information types, and transmission rules. Dynamic policy generation and multi-agent evaluation environments are research directions, rather than a deployed control or demonstrated security guarantee. The report draws on a workshop held in late 2025; the announcement is dated October 5, 2026.

**Why it matters — AISDLC analysis:** Make purpose, recipient, resource, and allowed action explicit in a delegation. A model may propose a context-sensitive rule; widening its own authority requires a separately authorized decision.

[Primary source: Open and Emergent Problems in Agentic Privacy and Security: A Contextual Angle](https://research.google/blog/open-and-emergent-problems-in-agentic-privacy-and-security-a-contextual-angle/)

[Explore governed autonomy operating modes](https://aisdlc.ai/insights/from-ai-assistance-to-governed-autonomy)

## A decision for the next delivery cycle

Start with one internal, reversible delivery task whose permitted effects can be enumerated. For example, an agent may investigate a failing synthetic regression, prepare an implementation change, and run the existing checks. It may not change acceptance criteria, widen credentials, release the change, or send data to a newly discovered destination. Those are distinct authorities, even if one model can explain how to perform all of them.

Use a model reviewer to discover issues and prioritize investigation. Keep its observations and uncertainty in the evidence record. A separately versioned acceptance process decides whether the required conditions were met. Where correctness is open to interpretation, that process may need a named reviewer or a bounded adjudication step; an automated test cannot make an underspecified requirement precise. Inside an approved scope, runtime authorization can permit ordinary actions without a human approving every tool call. Exceptions and changes of scope follow the organization’s separately authorized escalation path.

That architecture allows learning without letting the executing agent redefine what counts as success. Runtime failures can inform proposed tests, prompts, or policies. Activate a proposed change through a controlled revision with its own evaluation and approval rules, then monitor its effect. An organization may automate parts of that promotion process if the evidence and consequences justify it. Automatic promotion still needs a protected authority boundary and an independent way to stop or roll back the change.

## Make the tradeoff visible

More aggressive review may discover additional defects while increasing false alarms and verification work. Tighter runtime rules may block unsafe effects while interrupting valid tasks. Context-specific permissions make purpose and recipients more explicit, but they also create policy maintenance work as workflows change. Measure those costs alongside successful outcomes; do not collapse them into one score that hides why the workflow failed.

A useful trial records acceptable completed tasks, prohibited attempted and completed effects, missed seeded defects, unnecessary refusals, human intervention effort, and elapsed time including retries and recovery. Set the observation window and stop conditions before starting. If a consequential effect can occur faster than an operator can intervene, the boundary needs preventive enforcement before that effect. A dashboard alone cannot provide that protection.

## A bounded exercise: twenty synthetic cases, repeated three times

This is a proposed exercise, not an executed result or an assurance standard. Use an isolated environment with mock tools and synthetic data. Prepare five cases in each of four groups: valid in-scope actions, forbidden cross-resource actions, a changed recipient or purpose, and revoked or expired authority. Include more than one interaction pattern in each group. The numbers make a small rehearsal manageable; they are not a statistically sufficient sample for a consequential rollout.

Freeze the cases and expected outcomes before comparing a baseline with a candidate control. Run each case three times under both configurations and retain all attempts, timeouts, failures, and tool effects. Hold the agent version, model settings, judging method, and environment fixed except for the stated control change. A deterministic replay may be useful, but repeated agent runs can follow different paths; the result measures the configured test environment, not every possible interaction.

Check model judgments against the fixture’s expected tool effects, and separately audit disagreements and severity assignments. A correct explanation does not compensate for an unauthorized write. Report counts for each group, applicable denominators, and run-to-run variation. Identify cases marked inapplicable, failed, or unscorable and why; otherwise a better-looking percentage can conceal disappearing failures. Never count a refusal of legitimate work as an acceptable completion.

Treat any completed prohibited effect as a reason to withhold additional authority and investigate the boundary. Zero observed prohibited effects in this small sample supports further testing, not a general safety claim. Before a production trial, add representative conditions, dependency failures, permission changes, and the organization’s own operational review requirements. Keep release authority separate from this exercise.

## Carry an evidence record into the next revision

Record the task scope and sponsoring principal; agent, model, environment, tool, policy, and test revisions; expected effects; actual tool requests and results; authorization decisions; evaluator disagreements; retries; and unresolved failures. Minimize and protect the retained data according to its sensitivity. The artifact should let an independent reviewer reconstruct which boundary was tested and what changed, without relying on the agent’s summary.

This is a practical stage in an enterprise’s AI SDLC journey: delegate a bounded iteration, observe its consequences, and improve the system through controlled revisions. Progress is demonstrated by reliable accepted outcomes and recoverable failures. A factory model becomes a possible operating choice when those properties hold across the relevant work; organizations can also retain hybrid modes wherever consequences or ambiguity require closer judgment.

## Further reading

- [From AI assistance to governed autonomy](https://aisdlc.ai/insights/from-ai-assistance-to-governed-autonomy)
- [Measure accepted outcomes and verification effort](https://aisdlc.ai/insights/ai-native-sdlc-measure-accepted-outcomes)
- [Design policy and evidence boundaries](https://aisdlc.ai/policies)