Faster at the wrong thing

An AI program can make individual development tasks faster without improving the operation those tasks are supposed to serve. Consider a team that delivers more backlog items after adopting coding tools, while customers still wait just as long for their requests to be resolved. In this example, the productivity gain may be real. What we still need to know is whether the team was working on the cause of the delay.

A disappointing result can have several explanations, including limitations in the technology or a lack of support for the people using it. Before committing the next round of spending, I’d also look at how work reaches the team and what counts as completing it. If the organization continues to choose work in the same way and judge success by shipment, faster development may simply produce more of what it was already asking for.

In the feature-team system Cagan describes, leaders and stakeholders decide which features belong on a roadmap, then assign them to teams for delivery. A team can meet that commitment without checking whether the feature improved the business. AI can reduce the effort required to implement it, but that alone doesn’t test the decision to build it.

Cagan’s alternative gives the team responsibility for solving a problem that leadership has explained and prioritized. That responsibility continues after an initial solution ships. Leadership still decides which problems matter and sets the constraints, while the team has room to test its assumptions and change the solution when the evidence calls for it.

Having been a CIO, I understand why a company needs a roadmap when there is more work than its engineers can take on. But if AI gives the team more capacity, we should look again at what we are asking them to build and whether it will actually help.

Suppose, in the service example, a team discovers that an assistant won’t resolve the delay because two departments haven’t agreed who owns an exception. The team needs permission to work with those departments on the ownership problem, even if that means changing the planned software. If it must finish the assistant to meet its commitment, then the way the work is managed prevents it from acting on what it has learned.

Changing that arrangement across a large company would take considerable work. I find Cagan’s pilot-team approach useful because it lets the organization test the change with a small team before extending it. He recommends a meaningful problem and stakeholders willing to participate, with coaching to help the team. He describes a quarter or two as a typical learning window and cautions against choosing a problem dominated by dependencies the team can’t resolve.

The pilot also needs a clear account of what it was able to test. Finding that the workflow shouldn’t be automated is different from being unable to get the access needed to investigate it. In the first case, leadership may have good reason to reject the idea. In the second, it still needs to decide whether to resolve the access problem. A demonstration wouldn’t settle either question.

For regulated operations, I’d bring in the people who can approve data access and changes to the operation while the pilot is being planned. Cagan already addresses coaching; my concern here is whether the team has agreed how the work will meet the organization’s controls. Those decisions need attention before delivery begins, so the pilot can spend its time investigating the problem with the access and oversight it requires.

This is how I think about a forward-deployed pod at LockedIn Labs. It’s an embedded team that helps the organization try this way of working and prepares its engineers to continue the work. A lead and architect can work alongside the organization’s engineers, with the team chosen to suit the workflow. We need people who can build and independently review the work, and we need to know who can approve it for use. I’d agree on those responsibilities before deciding how many people the team needs.

For the service example, I’d start with approved historical cases to find where the delay occurs. If a model appears useful, a read-only shadow trial could compare its recommendations with the current process, if the data access has been approved. Its tools would need bounded permissions and checks on their inputs and effects. Typed interfaces help make those boundaries inspectable, but don’t enforce authorization by themselves. The trial would need separate approval before it could affect customer decisions.

Our release-evidence essay explains how to connect approval to the specific change being released. That work is still necessary when a team has authority to choose its solution. It should follow an investigation of whether the change addresses the operating problem, because technical release checks can’t tell us whether the organization chose the right problem to solve.

The receiving team should be involved while the pilot is running. Keeping the work in its repository and using the operating process it will inherit makes the handover easier to test. The people who will handle exceptions need to rehearse that responsibility, including when they would pause or reverse the workflow. This also exposes requirements that a demonstration outside the operating environment might miss.

I’d agree on how we’ll judge improvement before the team starts. Cagan recommends a quality backstop so that progress against one objective doesn’t come at the expense of another. I’d put someone outside the delivery team in charge of that counter-metric. For example, we need to look carefully at a reduction in handling time if it leads to more repeat contacts. Someone must be able to challenge the result on that basis, even when the delivery target has been met.

In Everyone has the models, I discuss why access to a capable model doesn’t settle the harder questions of implementation. The same reasoning applies to the work a team is asked to pursue. At LockedIn Labs, I want an AI implementation engagement to begin with the operating problem and the conditions needed to investigate it, so the choice of technology follows a clearer understanding of the work.

There is a related question about how people prepare for this responsibility. Show me your portfolio discusses evidence a learner can make available for review, while The field is the interview examines practitioner assessment. These related LockedIn Labs publications concern evidence of a person’s work. A course result or practice score on its own doesn’t tell us whether a team can take responsibility for an operating problem.

A pilot gives leadership something more useful to review when it records how the team investigated its assignment and what changed as a result. That record should include the constraints it encountered and enough evidence for the receiving team to check the proposed solution. Leadership can then decide whether the approach is worth extending, or whether it needs to address the constraints the pilot found before another team tries it.

AISDLC Insights publishes source-informed editorial synthesis and implementation positions. It is reference material, not a standard, certification, legal opinion, or authorization to deploy an agent.