Architecture Amnesia

Solar Frontiers is a strategy game I have been building. Its simulations produce balance reports, which describe how the game economy behaved, and someone then has to read the report and repair whatever it exposes. That work is slow and repetitive, which made it look like a job for AI agents.

So I wrote a Bash loop. It started four agents, gave each one a job of its own, and waited. When an agent finished, the loop was supposed to inspect the branch, push the commits, move the task from development to testing, and free the slot for the next job.

The loop decided that an agent had finished by watching its Unix process, an OS PID rather than a BEAM pid(). Once kill -0 reported the PID gone, the completion handler ran.

Going back over one run, I found that all four agents had done their work. The loop had recorded one of them. For the other three, job state and process presence had drifted apart somewhere along the way, so nothing inspected their branches, nothing moved their tasks on, and the finished work sat where it fell.

kill -0 answers one question: whether a process exists at this moment. It cannot say whether a job left usable evidence behind. I had asked the operating system a workflow question, and it answered a process question, the only kind it has.

The bug was ordinary. The embarrassing part is that I have spent three decades building Erlang systems, and the way you build those is by keeping transient workers apart from the processes that own state. Agents felt new enough that I put that habit down at the moment I needed it.

Where authority lives in an agent system comes down to three placements: which component owns a task, which one decides that the work is finished, and where a rule has to sit so that the next session obeys it. The Bash script came after eighteen months of building orchestrators, and I had already got all three placements wrong, more than once.

Aurora

The first loop grew inside Aurora, my AI-assisted project organizer. Its build agents began as prompts wrapped around shell commands, and the first failure was the simple one: an agent produced a valid change and exited without staging or committing it. The code existed on disk. The task had no record that anything had happened.

I moved the loop into a long-running Aurora Daemon and gave every job persistent state. That fixed the record and exposed a deeper problem.

One result handler decided what should happen after an agent returned, and the project configuration separately described which transitions were legal. The function meant to connect the two was called dispatch_next. It dispatched nothing. Successful work could move into testing without a test worker being assigned, and the pipeline stalled at that boundary.

The missing function was easy to write. The flow still had no single owner, and most of what I thought of as the architecture lived in prompts.

A rule in a prompt

A conversation is a poor place to keep a rule. A session can run for hours, and an instruction from its first minute stops shaping decisions long before the session ends. A better model raises the odds that the rule is followed and does nothing about where the rule lives. The rule existed in the conversation, and nowhere in the repository.

Aurora was migrating its production code from Elixir to Erlang at the time, and the root instructions said so: no new Elixir modules. Three weeks later, an agent-assisted change added 276 lines of Elixir in two new modules. I cannot tell from the history whether the model forgot the rule, overlooked it, or followed the Elixir code around it. What the history does show is that the rule was in the prompt and the modules were written anyway.

I replaced the instruction with a CI ratchet: a new production .ex file fails the build unless it is explicitly allowed. A later agent added an Elixir redirect handler, CI rejected it, and the revision came back in Erlang. After that, no session had to remember the migration boundary, because the repository enforced it.

Gnomeforge

I extracted Aurora Daemon into Gnomeforge, an orchestrator for durable engineering tasks. Its task record and worktree outlive the agent session. The worker reports evidence, and the process that owns the task state decides whether the work advances.

Even there, one placement was wrong at first. Recovery treated the live process registry as the truth, so a missing registry entry could mark live work as abandoned and start a second execution of the same task. The correction persisted the worker-to-task assignment, used liveness only to trigger a bounded reconciliation, and moved anything unresolved into a blocked state. A missing process now triggers a check instead of rewriting the task.

Five roles, one script

In BEAM terms, the session had become a Worker and the task-state process its Resource Owner. That distinction is the repair, and I did not have to invent it. I had already written it down and published it, in the Gnome Village, where every process holds one role and only one.

A Worker does one job, reports what it did, and disappears. An agent session is a Worker. It receives an errand, produces evidence and exits, and if it crashes, one errand is lost and nothing more, because nothing durable is allowed to leave with it.

A Resource Owner owns state and the invariants that go with it. The task process is the Resource Owner. It handles one message at a time, so a transition either happens or it does not, and it decides on completion by looking at the evidence. It outlives every session that reports to it.

A Router decides where a job goes and owns nothing. Choosing the machine, the container or the model for an errand is routing. A Router that starts remembering which errands it has sent out is holding state it cannot protect.

A Gatekeeper limits what may start: how many sessions run at once, which stage a task may enter, how much budget is left. Admitting work is a decision of its own, separate from the work admitted.

An Observer watches, and owns neither state nor work. Supervisors restart what dies. Sentinels notice a task that has sat in one stage too long and trigger a reconciliation. An Observer can report that a session is gone. It cannot report that a task is finished.

Now look at the Bash loop again. One script chose the workers, admitted them, watched them, pushed their commits, and decided what their exits meant for the tasks. Five roles in one place, with the durable ones kept in shell variables. kill -0 was a weak Observer signal to begin with, and because the loop was also the owner, an observation turned into a verdict.

Across all of these systems, process presence kept standing in for workflow state and prose kept standing in for an invariant. Both put the authority somewhere a new decision was free to ignore.

I did not repair the Solar Frontiers loop. I threw it away. In its replacement an OS PID is a health signal and nothing else, completion is a state transition backed by durable evidence, and a worker that exits cannot take its task with it.

Symphony

Earlier this year OpenAI published Symphony, its orchestrator for Codex agents, as a specification rather than a product. The reference implementation is written in Elixir. It polls an issue tracker, creates a workspace per issue, runs a bounded number of agents, and when an issue reaches a terminal state it stops the agent and cleans up the workspace. The stated reason for the language is supervision: Erlang, the BEAM and OTP hold up under long-running concurrent processes, and the implementation can reload code without stopping active subagents.

Two placements in that design are the ones I reached the slow way. Session policy lives in a WORKFLOW.md inside the repository, versioned with the branch it governs. Work state lives in the supervised orchestrator, so an issue outlives every session that works on it.

Architecture amnesia

I call forgetting the basics architecture amnesia, which sounds better than forgetting. The pattern was on my own blog, in my head, but not in my new code.

If you run agents, three checks will find it. For every rule the agents must obey, find out whether it lives in a prompt, in a review, or in the build. For every task that can be finished, find the durable record that says it was. For every liveness check, name the workflow question it is standing in for.

A lesson is learned when the next agent does not have to remember it from an earlier conversation.

- Happi

Comments

Loading comments...

Leave a comment

Back to blog index. Tip: open Reader settings (the sliders icon) to adjust font size and reading direction.