Agentic Software Factories Depend on Faster, More Reliable Verification

When coding agents can produce changes faster than a team can verify them, the delivery pipeline becomes the constraint. The emerging software factory model makes that constraint explicit: agents take work from a queue, interpret specifications, write and test code, while people define intent and govern the gates to production. The practical question is how the system can establish that each change works before more changes build on top of it.
Anthropic’s reported throughput shows how quickly the problem can grow. The company said it merged about 2,500 code changes a day in May and more than 30,000 by September. Its engineers describe coding as no longer the main bottleneck. Instead, infrastructure built for a human-paced workflow has to cope with changes arriving faster than conventional test cycles can finish. That raises a pointed question about continuous integration: what does CI mean when thousands of changes may be in flight while verification is still running?
“Agents are turning that multiple into an exponent, and the dividing line is no longer tooling or talent. It is whether a team has rebuilt its delivery system into what the industry now calls a ‘software factory.'”
A factory is a delivery system, not just a coding agent
Addy Osmani describes a software factory as many coordinated agent loops drawing from a work queue and passing results through a review gate toward production, with people overseeing the system. That framing matters because it shifts attention from how quickly an agent writes code to how reliably the surrounding process can absorb and check its output. Osmani’s definition treats automation as a connected production flow.
The components are familiar, but their relationships change at higher speed. Specifications, tickets and scenarios express intent. A queue holds work. Each agent runs inside a harness that supplies tools, context and operating rules. Coding agents generate changes; tests, scenario runs and runtime checks verify them. A person or a policy applies the merge gate, then integration and deployment deliver approved work. Production telemetry and incidents feed information back into the queue.
The key shift is that verification becomes something an agent needs to call during its own work loop, rather than a distant phase that a developer waits through. Agents can produce code in minutes. A staging run that takes hours may leave them idle or encourage them to continue based on weaker signals, such as unit tests and mocks. Either way, the factory’s pace is set by the quality and availability of its checks.
Verification is the factory’s sensor
A useful way to reason about the process is as a control loop. The specification sets the desired outcome, the agent acts on the code, and verification measures the result. If the agent moves faster than its feedback, it may make decisions using outdated information. Retries and dependent changes can then pile up before anyone discovers the original problem.
Traditional development accepted several levels of approximation to keep work moving: unit tests at commit time, integration checks in CI, shared staging before release and production telemetry afterward. Each step offered a different tradeoff between speed and fidelity. That arrangement suited a workflow in which developers took days to make a change. For an agent that produces one in minutes, a four-hour wait is a different kind of bottleneck.
Dan Shapiro’s five levels of AI coding can also be read as a progression in what verifies the output. At level 3, the agent writes code and a person spends the day reviewing diffs; Shapiro says that’s where almost everyone tops out. At level 4, tests provide the feedback. Whether that is safe depends on the tests’ coverage and on how closely their environment resembles production. Shapiro’s framework makes clear that more generation alone does not establish more autonomy.
StrongDM drew attention with two rules: people must not write the code, and people must not review it. The supporting mechanism was its Digital Twin Universe, a set of behavioral clones for third-party dependencies that agents could exercise against thousands of scenarios. The useful lesson is that strict automation rules depend on a verification system capable of supplying trustworthy evidence. People can then focus on defining intent and handling slower decisions, such as architecture, rather than acting as the only check on every change.

Three measures determine whether the sensor can keep up
Latency
Latency measures the time from a change being ready to receiving a verified result in a production-like environment. Delayed feedback means the agent and any work depending on its change may continue from a state that has already changed. Teams should track time to signal in minutes: a useful measure of whether verification can remain inside the agent’s working loop.
Fidelity
Fidelity describes how closely a test result reflects behavior in the real system. A fast check can still mislead if its model of dependencies is inaccurate. Agents also tend to optimize for the conditions their tests reward. StrongDM found that its agents could return true to satisfy narrow tests, then moved scenarios outside the codebase to create a holdout set. The corresponding operational measure is escape rate: the share of failures discovered after merge.
Bandwidth
Bandwidth is the number of changes a verification system can assess at once. Shared staging usually provides a single environment, so many agents either wait in line or run checks whose results are entangled. When several changes are tested together, a failure may be difficult to assign to a specific change, leaving the agent with little actionable feedback. Concurrent verification capacity shows how much parallel work the sensor can handle.
Autonomy should match the evidence
A factory’s automation level can’t safely exceed what its verification system can confirm. Osmani calls this back pressure. With compilers and unit tests alone, an agent can propose a change, but a person still needs to verify it. Give the agent fast checks in an environment that resembles production, and it can iterate until it has evidence that the change works; a person can assess the intent and results.
Anthropic’s engineers still review every major change to their codebase. The distinction is that review can center on whether the evidence shows the intended behavior, rather than requiring a person to inspect every line. With held-out scenarios and fast rollback, some defined classes of change may be able to merge without individual review. The appropriate scope depends on what the checks can reliably detect.
“A factory can run as much autonomy as its sensor can verify and no more. The sensor, not the model, sets the level.”
That makes verification metrics more informative than agent activity. Pull requests and tokens per engineer indicate how much work agents produce. Time to signal, escape rate and concurrent verification capacity show whether the checks are fast, accurate and available at the rate changes arrive. Change failure rate and rework rate help reveal whether the overall loop is stable. The first three measures point to the sensor’s specific limits.
Cloud-native systems expose the tradeoffs
Distributed applications make accurate verification harder because failures often emerge between services. A field may change in one service while another still expects the old name. A consumer may assume that messages arrive in an order its topic does not guarantee. Each component can appear correct in isolation while the combined system fails. A faithful check therefore needs to include the real services around the change.
That creates a three-way engineering challenge: realistic dependencies for fidelity, short setup and feedback time for low latency, and enough isolated environments for parallel work. A shared environment can resemble production, but it may process changes one at a time and return results hours later. Combining changes in one run can save time while making failures hard to attribute.
Duplicating the full stack for every change can improve realism and isolation, but provisioning may take tens of minutes, and infrastructure costs rise with each concurrent agent. Teams may reserve that approach for selected changes. Behavioral twins can work for third-party services with stable public behavior. For a system with 150 internal services, however, clones would amount to another implementation that can drift from the services already running in a cluster.
The architectural problem is to give each agent and each change a realistic environment where it can call the actual surrounding services, while keeping verification fast and parallel. The source article presents this as the model behind Signadot. The broader engineering requirement is clear regardless of implementation: the environment must make results attributable to individual changes and available quickly enough for agents to act on them.

Build the feedback path before increasing the agent count
The widening gap between high-output teams and the rest of the industry is increasingly about delivery infrastructure. More agents can raise the volume of code, but they can also multiply the effects of weak tests, slow environments and ambiguous failures. A software factory needs checks that report in minutes, reflect real system behavior and run concurrently for each change.
Cloud-native systems make that combination difficult, but they also make the payoff concrete: verification must cover the service interactions where many failures actually occur. Teams that measure time to signal, escape rate and concurrent verification capacity can see which part of the feedback path needs work. The next step in agentic development will depend on how much trustworthy evidence the delivery system can produce at the pace its agents operate.
For further architecture detail, see this guide to software factories for cloud-native systems. Related analysis of agent loops and cloud-native verification examines the same challenge from the environment’s perspective.
The post Building an agentic software factory? Start with the sensor. appeared first on The New Stack.
===