How I built a recursive operating loop

I built a recursive operating loop around the orchestration layer because I want work to keep moving without me being the handoff between every step.

A problem should be able to enter the system, become a well-defined piece of work, get built, get challenged, and come back with evidence. If the work exposes a weakness in how the system operates, that weakness should become the next piece of work. The next run should inherit the improvement.

That last part is the recursion. The system builds the product, and it can build changes to the system that builds the product.

There is a fairly large amount of architecture hiding in that sentence.

A model writing code is one part. Someone still has to decide what is ready, prevent two workers from doing the same job, establish what an approval actually covers, recover interrupted work, and determine whether the result reached a customer. Those responsibilities need somewhere to live when the chat ends.

In the mid-sized SaaS company I’m building this for, I have split those responsibilities across three places. The orchestration layer holds operational state. The source-code host holds the code and the evidence required to merge it. The operating-rules repository holds the operating procedures, tools, review rules, and learning records that govern the development loop. Agents take bounded jobs across those systems.

The support loop connects to the same structure. A customer report can become an engineering story. That story goes through the development loop. The resulting release feeds back into the customer case. A failure anywhere along that path can also produce a change to the loop itself.

So there are two questions running together: did we solve the problem, and did we improve our ability to solve this class of problem again?

Where truth lives

The architecture starts with where truth lives.

The backend orchestration layer is a database-backed control plane. It records stories, work ownership, attempts, review decisions, release evidence, customer cases, and the questions that actually need a person. An agent's account of what happened is useful context. The durable records determine what happens next.

That follows the same principle we use in the SaaS product. The relational database owns the invariants. Clients operate through authenticated commands. A UI, an agent, and a scheduled worker should reach the same business rules.

This is why the MCP interface matters. MCP gives an agent a discoverable set of tools for reading and acting on the orchestration layer. Those tools need scoped authorization, typed inputs, audit records, and the same server-side commands the screens use. Adding an agent interface should not create a second version of the product's rules.

The practical consequence is that I can change the worker without moving the work.

A session can stop. A provider can hit a usage limit. A fresh session can take over. The story, the current attempt, the review, and the remaining obligations still exist.

Work ownership and leases

Earlier versions used files in a Git dispatch queue to hand work between scheduled planner, builder, and reviewer sessions. That was useful for getting the first loop running, and it exposed where coordination was too dependent on the runtime. The newer design moves the work contract into the orchestration layer and uses leases.

A lease is temporary authority to perform a particular unit of work. It identifies the story, operator, role, attempt, and expiry. For work on an existing pull request, it also identifies the commit the worker is expected to act on.

The worker asks for its next unit, renews the lease while working, and finishes with a defined outcome. It does not invent a job because something in the repository looks interesting. Its authenticated identity determines which operator and role it represents.

That difference becomes important as soon as two sessions overlap.

Imagine a builder stops responding. Its lease expires and a replacement takes the work. Then the original session wakes up. Both sessions may have a perfectly convincing explanation for why they should continue. The database has to settle that disagreement. A stale lease is refused, and the old worker's account of its progress does not restore its authority.

There is a second boundary beyond the lease: what can merge. Expiring database authority cannot magically prevent every delayed process from pushing to a Git branch. The merge gate must independently establish that the commit belongs to the admitted attempt and has the required review.

The work contract and merge contract therefore solve different parts of the same problem.

Before a builder gets a story, the story needs enough structure to make a bounded attempt possible. Grooming names the target repository, acceptance criteria, affected surfaces, relevant principles, and size. “Improve support” is an intention. “Preserve an unsent reply when this specific operation fails, with these observable outcomes” is work someone can take.

The surfaces are the files or areas the story is allowed to change. They make scope inspectable. The target repository matters because capacity, ownership, and the eventual pull request all depend on it. If a proposed change spans repositories, the planner needs to make those dependencies explicit.

A small example from the loop's own development made this concrete. Stories with no target repository could count against capacity across repositories. The symptom looked like a worker that would not pick anything up. The missing fact was the target. That is exactly the kind of problem I want the system to explain before anyone starts building another scheduler.

Capacity is also explicit. There are limits for repositories and operators. Work waiting for a long time must have a path out of the queue; new build work should not endlessly outrun fixes and reviews. These are scheduling policies that belong in the shared contract, where every operator can see them.

The current worker shape is an author routine and a reviewer routine. The author can take build work or planning work according to the available unit. The reviewer takes a specific review. They do not need a standing conversation with each other. The orchestration layer is the handoff.

That lets the runtime stay fairly small. A wake-up tells a worker to look for work. The durable lease tells it what it owns. Event triggers reduce the delay between a pull request becoming ready and the next action; scheduled runs provide a backstop when an event does not produce a useful execution.

I want the model spending its effort on understanding the problem, writing the change, and challenging the result. Claiming a row, checking expiry, or reconciling a finished pull request is deterministic work.

Evidence and independent review

The builder produces a pull request with an operating record attached. The ledger identifies the story, operator, and loop version. It records decisions and what was not verified. A regression fix carries attribution where the cause is known.

The loop version is particularly useful. A change in prompts or governing instructions changes the system producing the code. If I cannot connect an outcome to that configuration, I cannot tell whether an operating change helped.

The ledger is also a handoff that survives context loss. A reviewer should be able to see why an approach was chosen, which boundaries the author encountered, and where the evidence stops. “Tests pass” is too broad if only one test suite ran.

The review then binds to an exact commit.

If a reviewer approved commit A and the builder pushed commit B, B needs its own decision. The fact that the pull request has the same number does not transfer the approval. The orchestration layer stores the reviewed head with the decision, and the reviewer finishes its work against that head.

The review is deliberately wider than syntax or whether a test suite is green. The dimensions include safety, truth, usability, fault tolerance, refusals, integrity, scope, verification, and simplicity. A change can satisfy its happy path while dropping a customer's draft on failure. It can reject an unauthorized request while accidentally rejecting every authorized request too.

For a bug fix, I want the test to distinguish the broken behavior from the corrected behavior. A test that passes on the old implementation has not established the claimed repair. Where a check could not be executed, that remains visible.

The reviewer has its own orchestration-layer identity. The author cannot simply submit an approval through its author key. That is a server-side boundary. Runtime isolation is a separate question, and I want that distinction stated honestly.

In the agent-runtime setup, independent sessions can still share a broader account trust domain. Separate environment names alone did not prove a hard isolation boundary. The architecture records that ceiling. Independent review is valuable, but its strength has to match the actual access model.

The merge boundary

At merge time, the gate assembles these facts.

The orchestration layer publishes a named merge-admission check through its source-host integration. In enforce mode, repository protection pins the required check to that integration, so another identity posting a check with the same name does not satisfy the rule.

For a normal story change, the gate checks the story linkage, the applicable operator, the review on the exact head, and the changed files against the story's surfaces. It accounts for renamed and deleted paths. Sensitive areas such as workflow configuration need explicit scope. It also checks whether another open pull request is using the same head.

For leased work, admission has to match the actual repository, pull request, attempt, and commit. When a merge queue builds a group, every member has to satisfy the gate. A group is not allowed to inherit trust merely because one member was valid earlier.

There are deliberately narrow exceptions. The current implementation permits additions or edits consisting only of root-cause write-ups in the operating-rules repository under a defined rule. Changes to the RCA schema, renamed or deleted records, or executable changes take the full path. That exception is small enough to inspect and test.

The rollout is also per repository. Shadow mode produces diagnostic results without enforcing the merge decision. Enforce mode makes the result a requirement. A working shadow check and a protected merge are different milestones, and the operating rules distinguish them.

This is how I want autonomy to expand: through a demonstrated boundary, with a clear account of where that boundary applies.

Recovery when the outcome is unknown

The hardest part of that boundary is recovery after an ambiguous external action.

Suppose the orchestration layer publishes a passing check to the source-code host, but the response is lost. The caller cannot tell from the timeout whether the source-code host accepted it. Retrying as if nothing happened can create conflicting evidence. Reassigning the story can be worse, because the first attempt may already be eligible to merge.

The design records publication intent, execution, and receipt. It uses a fencing token to distinguish the executor currently authorized to publish. Before retrying an uncertain publication, it reads back the provider's state and checks the identity, commit, check, and payload against what was intended.

Once a passing gate could authorize a merge, a merging reservation holds the attempt. Builder lease expiry alone must not hand that story to someone else.

The reservation ends with confirmed merge or verified withdrawal. Withdrawal has to establish that the gate is no longer passing, the pull request has not merged and is out of the queue, and an older publisher cannot arrive late with still-valid authority. The contract accounts for the lifetime of previously issued installation tokens and requires another read.

That sounds fussy until you consider the alternative: two workers each behaving reasonably against different versions of reality.

Unknown is an operational state. The system needs to be able to say, “I cannot establish whether this happened, so this work is blocked with the evidence attached.” Silently treating unknown as failure and starting over is how retries become damage.

The basic structure looks like this:

How the loop improves itself

That return path is where the system becomes recursive.

The operating-rules repository is itself a target repository. A missing recovery rule, an inadequate review check, or a misleading operational read can become a story. The loop can build the correction through the same author, review, and evidence path it uses for product work.

It does not give the planner permission to patch its own machinery whenever something is inconvenient. If the planner finds a defect, it records a bounded change. The correction still has to pass through the controls appropriate to it.

The learning record is a root-cause analysis, or RCA. We record regressions and operating gaps in a consistent format: what went wrong, where it broke, why, what prevents it now, and the general pattern.

The structured fields matter. A vague paragraph saying “we should test more” is difficult to compare with the next incident. A record identifying an unverified assumption, a test double that hid reality, or a missing check can be counted across repositories and operators.

The record also names the review the fault escaped, where that is known. That keeps the learning loop connected to how work was accepted. If several failures pass through the same missing review dimension, there is a concrete proposal to make.

A recurring lesson should produce a useful mechanism. That might be a test, a constraint, a scope rule, or a change to a review procedure. A reminder in a prompt is appropriate for judgment; an enforceable invariant should live where the action occurs.

Corrections to the learning record preserve history. If the original analysis was wrong, the correction points back to it. Quietly rewriting yesterday's explanation makes the system look more certain than it was and removes information about how the misunderstanding formed.

The scorecard gives this learning process a denominator.

I care about cycle time, rework, escaped regressions, stuck attempts, review decisions, scope breaches, evidence coverage, and the human intervention required to get a story through. Those measures need to be broken out by operator and loop version. Otherwise a faster worker can hide a worse failure rate inside an aggregate.

The measurement code reads the durable records. Missing data is shown as missing, with a reason. A metric that could not be computed does not become zero. A recently merged cohort has had less time to reveal regressions than an older one, so comparing the two requires that qualification.

The auditor's role is to turn the evidence into proposals. The weekly procedure examines recurring failures, decisions, and questions. The monthly procedure proposes changes to principles through review. Those procedures and reporting tools are part of the design; they do not justify claiming a measured improvement before the runs and comparable observations exist.

I still own the decisions about what the system is allowed to do. Production and store releases retain their human gates. Changes to credentials, protection, or governing principles have their own authorization. A loop that can improve its implementation should not quietly expand its own authority.

Connecting the customer outcome

The support loop makes this whole arrangement more useful because it supplies outcomes beyond the repository.

A customer case can reveal that a merged change was never released to the relevant platform. It can reveal that the fix reached production but the customer was never told. It can reveal that the original reproduction missed the way the customer actually uses the product.

Each of those is a different failure, and each belongs at a different point in the architecture.

Consider an illustrative case where a customer loses a draft after reconnecting. Support captures the report and context. Engineering turns it into a sanitized story with a reproduction and acceptance criteria. The development loop builds and reviews the fix. The release path proves where the fix landed. Support communicates that result and keeps the case open to a reply.

If the root cause was that tests only exercised the successful save, the recursive output is larger than the repaired draft. We can add the missing failure case and propose a review check for that class of persistence change. The next story starts with a better contract.

The support loop carries the customer's obligation. The development loop carries the implementation. The learning loop carries the improvement to how both operate.

They chain because the output of one is structured input to the next. They collapse where two steps were really asking for the same fact. Support and engineering do not need separate competing explanations of whether a fix has merged. They need the same fact, attached to their different obligations.

I think that is the useful meaning of autonomous operations.

Work can advance inside defined authority. Interruptions have a recovery path. Evidence survives the worker. Decisions reach a person when they actually require one. And when the system discovers a repeatable weakness, it has a route to improve the machinery that allowed it.

The next run should require less rediscovery. That is what I am building the recursion for.