QA agent pipeline
Contract · Jan — Mar 2026
PRDs in, reviewed test plans out. 60% of the cases run without a human.
- fewer production bugs
- 40%fewer production bugs
- fewer regression bugs
- 30%fewer regression bugs
- cases run without a human
- 60%cases run without a human
- LLM calls in CI
- zeroLLM calls in CI
[01]How it fits together
[02]Step by step
- 01
Ingest the knowledge base
Confluence is the source — photos, renders, past PRDs and whatever else describes how the product actually behaves. Without that grounding a generator invents its own conventions, and QA has to rewrite everything it produces, which costs more than writing it by hand.
- 02
Take the PRD for the change
PRDs arrive incrementally, so each revision generates against what changed rather than the whole spec. Most of a document does not move between revisions, and paying a model to re-read it is the easiest waste to remove.
- 03
Generate cases — and pull regressions, in parallel
Two paths run at once. One writes new cases from the PRD against retrieval. The other pulls the regression cases already in the corpus that this change puts at risk. Regenerating cases that already exist is both expensive and how you end up with four slightly different versions of the same test.
- 04
Label every case with how it runs
Each case carries a title, steps, acceptance criteria and a run type: automated, AI-manual, or human. This is the judgement call the whole system rests on — everything downstream is just routing.
- 05
Assemble a test plan
New cases and pulled regressions come together into one plan for the change, rather than two lists someone has to reconcile by hand.
- 06
Human review, then finalise
A person adds, removes and updates before anything executes. This gate is deliberate: the pipeline proposes a plan, it does not decide one. It is also what kept QA using the tool — reviewing a draft takes seconds, and being able to overrule it is why the output gets trusted.
- 07
Execute in three lanes, in parallel
Automated cases are generated and run. AI-manual cases are executed and their results documented. Human cases go to people. All three run at once and land in the same result set, so a release has one place to look.
- 08
Promote approved tests into CI
Automated tests that pass review become part of the CI suite. No model in the loop at run time, so they run on every push at ordinary CI cost and never flake on a sampling temperature. The LLM spend is paid once, at authoring.
Context
A games studio where QA read every PRD by hand and wrote test cases from it. Slow enough that a full regression pass only happened near release — which is exactly when finding a bug is most expensive.
The part that was hard
The classification. Every case is labelled automated, AI-manual or human before anything runs, and being wrong in either direction kills the tool — over-automate and QA stops trusting the results, under-automate and you have built an expensive way to write documents. Draw that line honestly and the rest follows: tests cheap enough to run constantly, and QA spending their hours on the cases that needed a person.
Stack
- Python
- LangGraph
- LangSmith
- OpenRouter
- AWS
- pytest
Have something like this to build?
Thirty minutes is usually enough to tell whether it is worth either of our time.