How to evaluate enterprise AI delivery under real governance

An enterprise assessing an AI delivery configuration needs to understand how it behaves when a useful change conflicts with a control. A successful demonstration establishes that the agent can perform a task in that demonstration. It leaves open whether another team can reproduce the result, whether the agent can alter its acceptance criteria, and who can authorize the resulting release.

LockedIn Labs publishes AISDLC. The evaluation guide describes reference configurations under the same evidence requirements. This article proposes a method for assessing a deployed system. We have not run the configuration evaluations described here.

Our enterprise AI delivery evaluation provides nine reference configurations. I would use it to choose a bounded pilot around an actual delivery problem, then publish the tested configuration and the limits of the evidence. A reference configuration cannot substitute for testing the deployed system.

Compare deployments that have the same job

Separate developer assistants, delegated coding agents, delivery orchestration systems, and enterprise control services. They may be combined in one deployment, but they own different parts of the work. Record where execution happens, who runs the infrastructure, which repositories and tools are reachable, and which systems retain approval authority. Compare the complete configured path against the same task and control requirements.

Keep enforcement outside the agent’s editing authority

A governed harness needs controls wherever an action becomes possible. Start with organization identity and scoped workload identities. Route authorized model and tool requests through gateways that validate the caller, resource, tenant, action, and current policy. The destination service must enforce authorization server-side. A gateway is insufficient if the same credential can call the provider or deployment API directly through an unrestricted network path.

Keep policy, verifier configuration, and release credentials outside the generating identity’s write permissions. Protect the policy distribution mechanism and require a distinct authority for policy changes. Give the verifier its own identity and approved workflow, with results stored beyond the generator’s control. Running another model as a reviewer is useful feedback, but shared credentials and editable checks do not create independent verification.

GitHub’s rulesets documentation illustrates why administration matters: rules have enforcement states and can grant bypass permission to selected actors. The enterprise must inspect those actors and the status-reporting identities it trusts. Its artifact attestation guidance provides mechanisms for producing and verifying build provenance; release authorization still needs a separate decision about what that evidence permits.

Bind the release decision to the artifact digest, policy revision, verifier evidence, target environment, allowed action, and expiry. Recheck those relationships at deployment. If any material input changes, determine which checks and approval must be renewed. Retain the attempted action and rejection as well as successful releases so that an independent operator can reconstruct what happened.

Avoid claiming the system is impossible to circumvent. Define the threat model: ordinary developer and agent identities, compromised task input, and the privileged administrators whose actions require separate monitoring. Emergency access needs an independent approver, a narrow scope, expiry, and an audit record retained outside the requester’s control. Test unauthorized normal paths and the authorized exception process separately.

Ask the deployment to reject an unsafe path

Measure useful delivery and preserve the failures

Predefine a task pack covering maintenance, new behavior, dependency changes, and recovery. Freeze repository snapshots, task instructions, acceptance tests, model and runtime versions, tool permissions, budgets, and human assistance rules. Declare task counts and repeat counts before running, publish failures and exclusions, and record configuration changes. Keep a comparable baseline and separate functional success from control outcomes.

Report accepted-change success rate, total cost per accepted change including unsuccessful attempts, median and p95 elapsed time, and human review minutes. Measure escaped defects over a stated observation period. Provide uncertainty intervals for rates and timing estimates, explain their calculation, and acknowledge when the sample is too small to support tail claims. A required control failure should remain visible even when delivery speed improves.

For each candidate, publish the evidence category: reference criteria, observed pilot behavior, or a reproducible test result. Keep missing evidence explicit and corrections dated. Begin with the evaluation methodology and our release-evidence design, then choose a workflow whose acceptance and operating owner are clear enough to test.

AISDLC Insights publishes source-informed editorial synthesis and implementation positions. It is reference material, not a standard, certification, legal opinion, or authorization to deploy an agent.