View as Markdown

AISDLC-CQ: Code Quality and Verification Conformance#

Version 12.3 Status Published Editor Sam Sweilem, LockedIn Labs Cite as AISDLC-CQ 12.3, clause references in the form CQ-4.2

Publication note (informative). 12.3 adopts the updated edition numbering. Normative requirements are unchanged from the previously published 1.0 edition.

1. Scope#

This specification defines what must be true for software produced with AI assistance to be promoted to a production environment.

It defines delivery practice for LockedIn Labs and the SprintLoop platform.

It governs the engineering system around generation. It does not govern model selection, prompting technique, developer experience, or cost.

The specification exists because a single rule is doing most of the work in practice and is rarely stated:

A model cannot waive a deterministic failure.

Running multiple passes with a language model acting as judge satisfies no clause of this specification. Model review is a pre-filter, executed first because it is cheap. Under the conformance model in clause 3, a system whose strongest control is model judgment is Level 0.

2. Terminology#

Two terms are held apart throughout, because they are commonly used interchangeably and the distinction is the substance of this document.

Generation agent. The component that produces code. Interchangeable by design. This specification names no generation agent as a dependency.

Harness. The constraint system surrounding the generation agent: what it is given, what it is permitted to do, and what its output must pass. The harness is what this specification governs.

Gate. A check whose result is authoritative and which the generation agent cannot override.

Promotion. The transition of a change into a branch or environment from which it can reach production.

Waiver. A recorded, time-limited exception to a gate result, issued by a named person.

Evidence package. The retained record of a change sufficient to reconstruct the promotion decision without re-execution.

The keywords MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are to be interpreted as in RFC 2119.

3. Conformance levels#

Four levels. Each is a strict superset of the level below.

Level Name Defining property
0 Unverified generation Promotion rests on judgment, whether a model's or a human's reading of a diff. No check is authoritative.
1 Deterministic floor It builds, tests execute and pass, static analysis and secret scanning block. No model judgment overrides them.
2 Adversarial and architectural Quality measured on the diff, mutation or property-based testing above threshold, architecture rules enforced as failures, security analysis on the diff, verification independent of the generator, bounded repair.
3 Governed and attestable Evidence package per change, provenance recorded, named human release authority by risk tier, gate telemetry retained, and for runtime model decisions a protected-data gate, evaluation sets and shadow operation.

Level 2 is the floor for any codebase another organization has to maintain. Level 3 is required where a regulated obligation attaches to the software's behavior, including protected health information, financial reporting, and safety-relevant systems.

Most agent-assisted development in use today is Level 0. This is not a criticism of the practitioners. It is a statement about what has been built around the practice so far.

4. Normative requirements#

Each clause states the requirement, the level at which it becomes mandatory, and how conformance is verified.

CQ-1 Executable intent#

CQ-1.1 (L1) Before generation, the following MUST be explicit: functional requirements, acceptance criteria, architectural constraints, language and runtime versions, approved frameworks, interface and persistence contracts, prohibited dependencies and actions, and a definition of done.#

CQ-1.2 (L1) A generation agent MUST NOT introduce requirements that were not authorized.#

CQ-1.3 (L2) Every promoted change MUST be traceable to an identified requirement.#

Verification: the requirement record exists and is referenced by the change.

CQ-2 Repository legibility#

CQ-2.1 (L1) The generation agent MUST be supplied the architecture, module boundaries, conventions, dependency rules, existing interfaces and decision records relevant to the change.#

CQ-2.2 (L2) Context selection MUST be bounded and reproducible. Supplying an entire repository is not conformance; it is the absence of a decision.#

Verification: the context assembly is inspectable and repeatable for a given change.

Note. This is the largest single determinant of output quality and the clause most implementations fail. Systems that fail here compensate downstream by rejecting a large share of generations, which is visible in gate telemetry under CQ-10.4.

CQ-3 Constraint at generation#

CQ-3.1 (L1) Tool invocation available to the generation agent MUST be typed and argument-validated. An operation absent from the available contract cannot be invoked.#

CQ-3.2 (L2) Where an interface contract or schema exists for the work, generation SHOULD be contract-first.#

CQ-3.3 (L1) The generation agent MUST operate under an identity distinct from any human identity, with permissions scoped to its function.#

Verification: inspection of the tool schema and the agent's granted permissions.

CQ-4 Deterministic verification#

CQ-4.1 (L1) The change MUST compile or otherwise build against the real dependency tree.#

CQ-4.2 (L1) Tests MUST execute in the build and pass. Tests that exist but do not execute do not satisfy this clause.#

CQ-4.3 (L1) Static analysis MUST fail the build rather than record a warning.#

CQ-4.4 (L1) Secret scanning MUST run on the change and MUST block.#

CQ-4.5 (L1) These checks MUST execute in an environment the change author does not control. A check executed where its result can be suppressed by the author is not a gate.#

CQ-4.6 (L1) Model judgment MUST NOT override any result under this clause.#

Verification: the checks are configured as required status checks, and promotion is blocked on their failure.

CQ-5 Adversarial verification#

CQ-5.1 (L2) Coverage MUST be measured on the changed lines. Repository-wide coverage does not satisfy this clause, because it can be raised without testing the change.#

CQ-5.2 (L2) Mutation testing or property-based testing MUST run against the changed code, with a stated threshold that fails the build.#

CQ-5.3 (L2) The threshold MUST be recorded in configuration rather than applied by discretion.#

Verification: threshold present in configuration; a deliberately weakened test causes failure.

Note. CQ-5.2 is the clause that distinguishes a system measuring whether tests ran from one measuring whether tests would have caught the defect. It is the most common gap in otherwise well-built pipelines.

CQ-6 Architecture conformance#

CQ-6.1 (L2) Layering, package and module boundaries, forbidden dependencies, and naming rules MUST be enforced as executable tests that fail the build.#

CQ-6.2 (L2) These rules MUST be specific to the repository rather than generic quality heuristics.#

Verification: a deliberate boundary violation fails the build.

Note. This clause is what prevents generated code that is locally correct and architecturally wrong, which is the dominant failure mode at scale.

CQ-7 Security and supply chain#

CQ-7.1 (L2) Static application security testing MUST run on the change and MUST block above a defined severity.#

CQ-7.2 (L2) Dependency and license analysis MUST run, with a policy that blocks rather than files a notification.#

CQ-7.3 (L3) A software bill of materials MUST be produced per build artifact.#

Verification: policy configuration and a blocked build on a known-vulnerable dependency.

CQ-8 Independent verification#

CQ-8.1 (L2) The component that verifies a change MUST NOT be the component that generated it.#

CQ-8.2 (L2) A verifying agent MUST NOT hold write access to source, MUST NOT hold approval authority, and this separation MUST be enforced by permissions rather than by instruction.#

CQ-8.3 (L1) Deterministic results and probabilistic judgments MUST be distinguished wherever both are presented. A probabilistic judgment MUST NOT be recorded as a passing gate.#

Verification: inspection of the verifying identity's granted permissions.

CQ-9 Repair discipline#

CQ-9.1 (L2) A failed change MUST be returned to the generation agent with the failure output attached and classified. Retry without failure context does not satisfy this clause.#

CQ-9.2 (L2) After repair, affected gates and regression gates MUST re-run.#

CQ-9.3 (L2) A retry ceiling MUST be configured, after which the change escalates to a person.#

CQ-9.4 (L3) The repair history MUST form part of the evidence package.#

Verification: configuration of the ceiling; the escalation path is exercised.

CQ-10 Evidence#

CQ-10.1 (L1) A record MUST be retained per promoted change identifying the requirement, the change, and the gate results.#

CQ-10.2 (L3) The evidence package MUST be sufficient for a reviewer who was not present to reconstruct the promotion decision without re-executing anything.#

CQ-10.3 (L3) Provenance MUST be recorded: the generation agent and version, a context digest, tool invocations, and approvers.#

CQ-10.4 (L3) Gate rejection telemetry MUST be retained: how frequently each gate refuses a first attempt.#

Verification: the reconstruction test is performed against a retained package by someone not involved in the change.

CQ-11 Release authority#

CQ-11.1 (L1) Promotion MUST require approval by a human identity distinct from the generation agent.#

CQ-11.2 (L3) The approving authority MUST be assigned by risk tier rather than by availability.#

CQ-11.3 (L1) A system MUST NOT promote on its own confidence.#

CQ-11.4 (L2) A waiver MUST record a named person, a written reason, and an expiry. A waiver MUST reduce the recorded conformance level of the change it applies to. An expired waiver MUST fail closed. No agent may issue or approve a waiver.#

Verification: branch protection configuration; waiver records inspected for completeness and expiry behavior.

CQ-12 Generator independence#

CQ-12.1 (L1) Conformance MUST NOT depend on which generation agent produced the change. The same clauses, gates and thresholds apply regardless of model or tooling.#

CQ-12.2 (L2) The achieved level MUST be reproducible across generation agents.#

Verification: the same work item processed by a different generation agent reaches the same gates with comparable results.

Note. If output quality depends on which agent generated it, the organization has a preference rather than a pipeline. Provenance under CQ-10.3 still records which agent was used. It carries no authority.

5. Stack profiles#

Informative. A profile binds clauses to a toolchain. Named products are examples, not requirements.

Clause Java TypeScript Python SQL and migrations
CQ-4.1 build Maven, Gradle tsc, strict build or import check migration dry run
CQ-4.2 tests JUnit project runner pytest migration up and down tests
CQ-4.3 static SpotBugs, Checkstyle, Error Prone ESLint, failing on error Ruff, mypy linter for the dialect
CQ-5.1 diff coverage JaCoCo with diff filter runner coverage with diff filter coverage with diff filter not generally applicable
CQ-5.2 mutation PIT Stryker mutmut property tests on constraints
CQ-6 architecture ArchUnit dependency-cruiser, ESLint boundaries import-linter schema and naming rules
CQ-7.1 SAST Semgrep, CodeQL Semgrep, CodeQL Semgrep, Bandit Semgrep rules
CQ-7.2 dependencies OWASP Dependency-Check audit tooling, Dependabot pip-audit not generally applicable

Where a stack has no strong tool for a clause, the compensating control MUST be recorded and the clause MUST NOT be reported as satisfied.

6. Stating a conformance claim#

A claim identifies the specification version, the level, the scope, and the exceptions:

Repository X conforms to AISDLC-CQ 1.0 at Level 2, excluding CQ-7.3, assessed on [date].

Claims stated without exceptions are read as claiming none. An organization SHOULD publish the clauses it does not meet.

Annex A: Reference implementation#

A conformance record is maintained with the implementation it describes rather than inside this specification, since a standard should not carry one implementation's configuration in its normative text. The record states the level achieved clause by clause, including any clause not met, and is written from configuration and build history rather than from description.

LockedIn Labs' conformance record is available under non-disclosure.

Annex B: Worked trace#

A trace follows one change end to end through every gate, including at least one gate failure and its repair. A trace in which everything passes on the first attempt is not representative and should be treated as illustrative only. Traces are provided on request against the requesting party's own stack.

Annex C: Anti-patterns#

Named plainly, because each is common and each is disqualifying.

Model as judge, treated as a gate. A second model's approval is not a check. It is correlated with the first model's errors and cannot be reproduced.

Repository-wide coverage as a quality signal. Raised by testing anything, including code nobody changed.

Advisory static analysis. A warning nobody is required to clear is a warning nobody clears.

Retry without failure context. An agent retrying without the error output is sampling, not repairing.

Evidence assembled for the audit. Records reconstructed after the fact describe what someone remembers, not what happened.

Verification by the generating component. A system checking its own work reproduces its own blind spots.

Checks that run where the author controls them. A local test suite is feedback. It is not a gate.

A human approval with nothing beneath it. Approving a diff no deterministic check has examined is Level 0 with additional steps.

Annex D: Self-assessment#

Answer for the repository, not the intention. Each "no" identifies the next thing to fix.

Level 1

Level 2

Level 3

Annex E: Scope limits#

This specification does not address model selection, prompt construction, agent orchestration design, developer experience, cost, or throughput. It makes no claim that conformance produces good software. It claims that non-conformance makes the quality of software unknowable, which in a regulated environment is the same problem.

AISDLC-CQ is published by LockedIn Labs. Comments to sam.sweilem@lockedinlabs.ai.