Mikrowave is an internal Anghami project: the pipeline that encodes and packages our video and audio for streaming, from live channels and film catalogues to the music catalogue and Dolby Atmos® masters. Over the past year I led its development with coding agents writing the code. This post covers which agent did what, how the split changed as the models and tools changed, and which habits survived every change.
How It Started
Mikrowave started from need and curiosity. Anghami’s media work needed video encoding, multi-DRM packaging for HLS and DASH, an audio catalogue that never stops growing, Dolby Atmos® masters, and a fleet to run all of it. Last September I started building it, and the way I chose to do it was to let coding agents write the code. Not assist with it: write it. I wrote the specs, reviewed the plans and tested what came back.
Mikrowave is the result. It went from three bash scripts to a 44,000-line Python service with 70,000 lines of tests, 63 releases, and two production clients, both internal. The agents changed every few weeks: six Claude generations and six OpenAI models passed through this repository, along with a stream of new tool features.
This post is about the practice, not the product. Another post will focus on the product story and the broader production results.
Four Eras of Delegation
Era 1, September 2025: build, then rebuild from a spec
It started on the night of 23 September 2025 as a request to Codex CLI, running gpt-5-codex:
a program that launches an srt transmitter that takes an input file and transmits it as SPTS stream with an h264 video and a stereo english audio channel. the input file and destination IP and port are to be configured.
Codex proposed Python. I pushed back two minutes later. I was stubborn at the start and still wanted things done the way I would do them, maybe out of fear of losing control along the way:
Why python scripts? why not bash?
lets start with bash
Three nights of that produced a working live pipeline: an SRT receiver, an HEVC ladder, two packager instances fed from one mezzanine (“one cenc for Widevine/PlayReady and one cbcs for FairPlay, sharing the same mezzanine pipelines”), 6.08-second segments aligned to 25 fps so audio and video segment durations matched, and a near-real-time S3 sync. I debugged on EC2 with the shell output pasted straight into the chat. The delegation model was the obvious one: the agent types, I test and steer.
Four days later I did something different. Instead of asking an agent to “port this to Python”, I wrote a 333-line specification and handed it to Claude Code. It opened with a goal, a warning and an architecture:
Goal: Build a Python-based video ingestion, encoding, and packaging pipeline that replaces the old Bash scripts. It must: * Receive a live or file-based input. * Encode multiple renditions (H.264 and H.265). * Package the outputs for HLS and DASH. * Support encryption using CENC (Widevine + PlayReady) and CBCS (FairPlay). * Produce fully deterministic outputs and pass end-to-end validation tests.
A paragraph headed CRITICAL WARNING followed the goal. It described a crash the bash prototype had hit and told the agent to check for it first. The spec closed with an end-to-end test specification and acceptance criteria: “Deterministic output: two runs yield identical manifests”; “make tests passes all scenarios end to end”. The Python repository was born from that file on 30 September 2025. Twenty commits landed that day, all carrying a Claude co-author trailer. Two hours after the first one, I opened Codex on the same directory:
run the end to end tests and debug why they are failing
That is Era 1 in one afternoon. One agent implements from a written spec, the other is asked to verify. I trusted neither alone. I wrote the spec and the test plan, then reviewed the implementation and its test results. The warning paragraph about the crash was a lesson learned in bash on 24 September, written down so no agent would have to rediscover it.
Era 2, March to April 2026: one implements, the other reads and reviews
After a five-month pause (another post will explain it) the repository restarted on 23 March 2026. Two things had changed. Claude Opus 4.6 had shipped with a one-million-token context window, which made reading the whole repository in one conversation practical. And I had built a sandbox. I knew the horror stories, the agent that drops a database or wipes a laptop’s root, and I would not run one without a leash. I was also sick of being asked whether an agent may read a file because its command happened to contain a Unix pipe. A Debian container solved both problems. It has the entire media toolchain built from source, bash, and both agents installed. It is close enough to the fleet to trust the results, and safe enough to run everything with permission prompts switched off. Inside it, git cannot push, no AWS credentials are passed in, the root filesystem is read-only apart from size-capped scratch mounts, Linux capabilities are dropped and processes are capped. Agents can edit the working tree and create worktrees; they cannot deploy or reach production.
The first prompt of the restart went to Codex, and it was not “write code”:
I want you to scan this repo and understand the VOD parts of it. that’s the most important for our needs now
That session ran 121 turns. From April, Codex was routinely pointed at Claude’s plan files before any implementation:
review this plan ~/.claude/plans/purring-booping-goose.md
ok re-review this plan again ~/.claude/plans/purring-booping-goose.md
Twenty such sessions are in the record. Codex still wrote code in April; by June it had all but stopped, and its session logs show the drift away from implementation and towards review. The orchestrator, worker agent and dashboard arrived in one 11,000-line commit on 11 April. The manifest-driven audio pipeline that replaced our old PHP encoder arrived on 12 April. Both landed in Claude-trailed commits after their plans had been through Codex.
Model uptake in this era was quick but deliberate. New Claude and OpenAI models went into use a few days after release, rarely on day one. I waited a day or two on purpose, to give the early issues time to be fixed. I also hoped the previous model would run faster while everyone else was busy trying the new one.
Era 3, May to July 2026: reviewer and release manager
By May the division of labour had settled, and the reason was budget as much as ability. I had two subscriptions and one attention span, so I had to watch token usage and pick one tool to spend most of it on. I was more comfortable in Claude Code’s harness, so Claude became the main implementer with the larger budget. Codex became the fallback for the hardest, most surgical problems, at its highest reasoning effort, or in its fast mode when I was in a hurry. Day to day the two ran side by side. Claude implemented one feature in one terminal. In another, Codex reviewed the plan for the next feature or the branch of the previous one, and cut the release in between:
see what changed since last release tag, and create a new one
Sixty-three tags carry that routine. Codex’s role also included discipline I had to teach:
why are you doing make build? you need to review the plan logically not actually implement anything
this was a PLAN proposal, not a PR – no need to run make build since nothing is implemented yet
Model uptake sped up over the summer. When Claude Opus 5 shipped in late July it was in use within three days, and from that point every Claude generation was adopted within days.
Era 4, August to September 2026: review as a pipeline, work in parallel
The main change in the last era was in how the work was organised, not in the model. Two things happened.
First, most reviews moved out of conversation and into an invoked pipeline. From 1 August to 10 September, all 33 of the Mikrowave Codex sessions in the record have zero user messages: they were driven by Claude-side skills that hand a diff to Codex and read back the verdict. The most recent step is a skill that sends a diff or a draft to the OpenAI API directly, with the model tier chosen by the complexity of the change, and only when I invoke it.
Second, implementation began to split across parallel worktrees instead of always living in one long conversation. Claude Code’s dynamic workflows shipped at the end of May. Inside my sandbox, the first workflow runs in the surviving record are on 23 July, the day work restarted on SourceIntelligence, our per-title encoding analysis. Heavy use starts in the week of 19 August. Git worktrees appear, and my prompts change register:
attach all agents to this session
Implement using ultracode however make sure to not waste tokens using expensive models when the task is mechanical. but also do not use cheap models where a large and smart model produces better results.
(Ultracode is Claude Code’s mode that plans a multi-agent workflow and runs it.)
spin up another workflow in parallel to fix these bugs
The 24 August instruction about model cost became policy in my global CLAUDE.md, and not by choice: a workflow had eaten my entire usage limit in a few minutes by spinning up Fable agents to change a few lines of code, so I had to teach it how to act. The rule makes Opus 5 the default for development and review. Haiku takes mechanical work, Sonnet moderate tasks, and Opus everything else. Under that rule I no longer use Fable as a subagent. I keep it for the main session and let it hand the mechanical work to the cheaper models. In the week of 7 September my host session made 28 workflow calls, 17 of them on one day, while three stacked pull requests moved through plan, implementation, review and merge.
The counterweight arrived the same week:
why is it taking 13 minutes of wall time to do inference on the first workflow item?
A reproducer agent had spent three of its thirteen minutes on an unrequested full unit-suite run. The rule that came out of it has four parts. A bug fix is reproduce, fix, one reviewer. I run the test gates in parallel myself. Agents run only the named tests. The adversarial shape with several reviewers is reserved for large design changes.
In the first week of September I moved the main session to Claude Fable 5.1 and the Codex reviews to gpt-6 astra, each within days of its release.
What the eras do and do not show
Each generation changed a few specific things: how much of the repository an agent could hold at once, how its effort could be steered, and, from August, whether work could be split across worktrees and background agents. By the end, models were adopted within days; tool features took longer, and dynamic workflows took nearly two months to show up in my records. What the eras cannot show is that any release made the code better. Task mix, project maturity and my own skill changed at the same time. This is an observational history with no control group.
The Operating Loop: Five Steps and Three Controls
Strip away the model names and the development loop had five steps: specify, review the plan, implement, test, and review the branch before tagging. Three controls sat around it: versioned memory, the sandbox, and a deploy step that only I could run.
Step 1: specify. The first spec set the shape: architecture, config schema, end-to-end scenarios, acceptance criteria, and one paragraph of hard-won warning. Later large features got the same treatment in Markdown plan files. Claude drafted each plan, Codex reviewed it, and I made the final decisions before implementation.
Step 2: cross-review. One plan, the April 2026 design for scoring encode candidates against their source, went through five review rounds in two days. Each round had the same shape: Codex raised its findings, Claude revised the plan and summarised what had changed, and I decided which findings stood. To paraphrase Claude’s summary of the revision, the first round’s top finding was that the plan’s approach quietly changed what the metric was measuring. It was replaced with one that keeps both sides comparable. The next round caught an alignment error in the revised approach that would have “reintroduced the misalignment we’re trying to eliminate”; the round after moved that fix into the maths. Across five rounds, the second model caught three alignment mistakes in plans the first had just revised, and I verified each revision. None of them shipped. Months later a bug of the same family did ship, and that one is the second story in the bugs section below.
Step 3: implement. Claude implements from the reviewed plan, in the sandbox, on a branch or a worktree. From August, Claude often worked in several worktrees at once under a workflow script.
Step 4: test. Test code has outgrown source code since April 2026. As of September, a plain line count of Python files under tests/ gives 70,409 lines against 43,748 under src/, and 3,427 test functions, up from 37 at the first tagged release on 1 October 2025. Most of that is ordinary unit and integration coverage. The part that matters for agent work is a directory of golden tests. A golden test compares what the code produces against a stored, approved copy, byte for byte. Ours pin the exact encoder and packager command lines the pipeline generates. A refactor that changes an encoder flag, even by accident, fails a golden test. A pin is regenerated only as a deliberate decision, never to make a failing test pass.
Step 5: review the branch and tag. Before a merge, I ask Codex to review the branch as a change, not a plan. I go through its findings and send the ones I accept back through step 3. Once the tests pass and I approve the branch, Codex cuts the tag with a description of what changed since the last one.
Control 1: memory in a versioned file. CLAUDE.md, the file Claude Code reads at the start of every session, grew from nothing to about two hundred lines. AGENTS.md is a copy, so Codex reads the same rules. Three entries show what the file is for:
- bool(elem) is False when an ElementTree element has no children. Always use `elem is None`, never truthiness. - The worker heartbeat is a coroutine on the job's event loop. Any blocking call on the loop stalls it and the job gets reaped as WORKER_LOST. Offload with asyncio.to_thread. - The worker treats an HTTP 409 as "lease lost, give up". An orchestrator endpoint must never answer 409 for a job the worker should keep handling.
Each is a bug that cost real time once and is now cheaper to prevent than to re-find.
Control 2: the sandbox. The container builds pinned revisions of the whole media toolchain with the same recipe as the fleet image, which keeps sandbox encodes as close to production as I can make them. Its restrictions are listed above. They reduced the direct blast radius enough for relaxed command approvals: agents could alter the working tree and scratch space, but they had no push or deploy path, and review and tests still guarded anything that left the container.
Control 3: deploy from outside the sandbox. Tags are cut by Codex, but deploys run from my machine, not from the sandbox, through a script that verifies the release artefact, migrates, flips the version pointer and rolls workers without interrupting jobs. No agent has ever had the credentials to do that.
Long Sessions, Compaction, and Written Memory
The record shows two things about long sessions. The SourceIntelligence restart in July ran as one container session of 552 user turns, and it crossed context-compaction boundaries repeatedly. Across the 42 Claude Code sessions retained from July to September there were 44 compactions. Every compaction replaces the conversation so far with a summary, and every summary loses something.
Written material preserved state when the conversation could not. The agent forgets; the repository does not. This is how I steered every large feature. It starts as a plan file, written by Claude in the sandbox and reviewed by Codex until it approves; the SourceIntelligence marathon’s first prompt was literally “start from” the plan file of the previous session. As the work grows, the plan file gives way to a design document in the repository root: a proposal, an implementation plan, a policy-numbers file with the measured evidence, an open-questions file that is rewritten as a resolved-decisions record once nothing is open.
Thirteen of those documents sit in the repository today, about 13,000 lines, the largest ones for the source probe, an unpublished design, and the per-title ladder. Each carries a “decisions taken in review” or “the journey: decisions and reversals, in order” section, so a new session or a new agent can read one file and learn what was decided and what was tried and abandoned. Isolated reproduction scripts play the same role for bugs, and handoff notes do it for sessions that end mid-feature. The file preserves the state through a compaction, a change of agent, or a week away. The prompt is then one line: read this, continue from section 6.
And some failure modes never went away. Config-derived numbers leak into prose. Refactors touch golden pins. Fixes mask end-to-end failures; one commit from March 2026 is titled “stop masking e2e pipeline failures”. The tests and the second reviewer exist because these keep happening.
Three Bugs That Changed the Rules
The 149-second query (June 2026). A dashboard query took 149 seconds in production and 20 milliseconds when I ran the same query by hand. The problem was how the query was sent, not the query. The application sends it in a form that makes the database reuse one generic plan for every request, while a query typed by hand gets a plan made for the values you typed. So every by-hand test was fast, and production stayed slow. Once we sent the query the way the application does, the slow plan showed up on the first try, and the fix was a small index. The recipe for sending it that way went into the gotcha list beside the fix, so no agent has to rediscover it.
The 7.2 % frame misalignment (September 2026). Our per-title encoding analysis scores candidate encodes against a reference reel. The two came from different container formats that keep time at different precisions, and the scoring graph rounded the two clocks differently. In the isolated reproduction, 7.2 % of frames were compared with the wrong reference frame, at the same positions in every candidate. The quality score collapsed wherever the picture moved. Checking a reference against itself hid it, because both sides then rounded the same way. I traced all three canary titles that had missed their quality floor to this one cause. I first wrote an isolated reproduction script, then added a pinned end-to-end test. A one-line change made both inputs use the same clock. The rule that came out of it is now the first line of CLAUDE.md: always create an isolated test program when trying to figure out a complicated bug.
The libvmaf crash that Claude could not find (August 2026). On 18 August the per-title analyser started scoring candidates concurrently instead of one at a time, and the scorer began aborting in roughly one to two per cent of runs. Claude went in circles on it: hypotheses, instrumentation, more hypotheses, no root cause. So I opened a Codex session and briefed it with what the Claude session had already learned and ruled out. Codex found the cause inside libvmaf, the video-quality scoring library itself: a threading bug in the library’s 3.0.x line. Our new concurrency triggered it. Upstream had already fixed it in a later version. The fix was a library upgrade, pinned in every toolchain build and enforced by a version floor in the toolchain gate.
I then used Claude agents to run the validation across the fleet builds: one crash in 46 control runs and none in 150 patched runs. That looks convincing, but on its own it is inconclusive. One event in the control arm is too little to compare against. Confidence came from the confirmed upstream root cause and the enforced version floor; the A/B stayed on as a regression alarm, not as proof. We also learned to check which library the binary under test actually loads. A clean result may come from the unpatched code.
Why did Codex win this one? My assumption is not that it is the better debugger. I could guide it better the second time, and it started with a fresher context that already contained the first session’s dead ends. The lesson is about the process, not the model. When one agent circles, brief the other with what the first one learned.
Lessons Learned
- Spec before code. The best prompt I wrote all year was a 333-line text file.
- Make the agent’s memory a file you version. CLAUDE.md is a changelog of what hurt.
- Isolate before you debug. A reproduction script is worth more than a long conversation.
- Keep a second model in the loop, and brief it when the first one circles. Validate its findings yourself. A fresh context that starts from the first session’s dead ends is worth more than another hour in the same conversation.
- Pick one tool to spend on. Token budgets forced the division of labour more than any benchmark did; the fallback earns its keep on the hardest problems.
Conclusion
The tools changed monthly. By the end I was adopting new models within days of release. The practice that survived every change was the boring part: a spec, a plan, a reviewer, a test. Everything else, from the model to the number of agents running at once, was replaceable, and was replaced.
