I built an AI Software Factory, then stopped using it.

Who am I

I am a bootstrapped solopreneur building my second SaaS app. That matters as context, because I do not have VC money to burn on expensive pipelines that only offer promises. I need systems that deliver cost-effective outcomes.

Why I stopped

Three issues caused the factory to cost far more than I expected, while not delivering enough over running agents via an ADE (Agent Development Environment) or multiple Claude Code terminals.

• Extremely high token cost.
• You cannot easily hot-swap foundation models in lanes.
• A reviewed plan does not get you a well-built change.

What my Software Factory looked like

I did what all the blog posts say and built a team of AI agents to review PRDs (Product Requirements Documents), write the code, test in my development environment, fix issues, and ship it.

Lesson 1: extremely high token cost

These are medians drawn from 408 real software factory runs, where a PRD required 2 fix rounds before it was ready to ship and assuming a mid-90% cache-hit rate:

Notes:

• Stages do not share memory. Each one starts cold and re-reads whatever it needs to execute.
• The verify lane was so expensive because it ran full end-to-end browser-based testing via Claude's Chrome MCP. That meant the agent had to send screenshots to help it run the tests and to show evidence I could put in the PR. Sending screenshots == tons of tokens...
• If you tell your review agent to find issues with your PRD, it will find issues and you may approach an infinite loop of fix rounds. Instead, you need to tell the review lane agent to confirm that the plan will produce the documented outcomes.

Lesson 2: you cannot easily hot-swap models in lanes

The obvious way to control cost is to match the model to the lane. Run the expensive models where judgement matters, run something cheaper for the mechanical parts, treat the choice as a dial. That assumes the models are interchangeable. They are not.

Foundation models have different, often undocumented, constraints. Those constraints decide what a lane can physically do.

My verification lane signs in to a dev copy of the product using test credentials. On Sonnet 5, the agent refused to enter credentials. I had no choice but to use Opus 5. That forced me into a model that is 2.5x more expensive.

So, the per-lane model choice is not just a cost dial, it is a capability constraint, and you find out which one you are holding after you have built the lane.

Lesson 3: a reviewed plan does not get you a well-built change

Now the problem shifts from technical to operational.

The premise is that if you get the plan right, the build is mechanical. Scope it, review it against the real code, repair what the review finds, and what comes out should be roughly what a careful engineer would have written. That is not what happens.

Even if the code is functionally correct, it very plausibly does not meet my architectural hopes or coding practices. It is not trivial to just prompt engineer your way out of that.

So, the human gate was in the wrong place. I built the pipeline to put me at the end, signing off on it after it was built and tested. What I actually needed was to be in the middle: reading the diff while it is being written.

What I did instead

I still use the plan, review, and fix lanes. Those just happen locally, using the same prompts that I developed for the software factory.

But, I moved to an ADE, Orca, which is built around the assumption that I am there: several agents working at once, each in its own isolated copy of the repo, and me watching what they are doing rather than reading a report afterwards.

The practical difference is that I can catch the agent taking a suboptimal approach before it gets to the development environment. You likely cannot have a junior engineer do this as effectively because they don't have the "spidey sense" to feel when an agent is going down a bad path.

Verification became a choice rather than a lane. Some changes get automated end-to-end checks because they are worth the tokens. Most get checked by me, in the browser, in less time than it takes an agent to describe what it is about to click. That single decision removed the most expensive lane I had.

If you are about to build one

First, create prompts for each lane in isolation and tune them to see how well they work and how many tokens they use (pick whatever sets of models you think are best for you). The cost implications may surprise you and stop you from continuing any further.

Then, manually run through them end to end, in fresh agents per lane, as if it were being done through the software factory. This will show how long the entire process will take, where you might have review + fix loops, and how much a single PRD will cost you.

If the result is a software factory, good for you. You did it in a thoughtful, intentional, and deliberate way.

If you stopped short of a software factory, the prompts you created can become skills to help you in your software engineering efforts.

* Token counts and costs are medians across 408 real lane runs between 9 August and 3 September 2026, computed per run from its own usage record and priced at Opus 5 API rates as of 6 September 2026: $5 per million fresh input tokens, $25 per million output, $0.50 per million cache reads and $6.25 per million cache writes. Cache reads are 93-97% of billed input depending on the lane, which is why the totals are far below what the headline token counts suggest.