Reviewing the robot’s homework.

The setup: a colleague in IT — sharp, not a developer, no repository access — used a third AI model (not one of the two in our enrichment pipeline) to build a purchasing module: generate purchase orders from the legacy ERP’s demand data, with approval workflows, configuration screens, the works. It arrived looking complete. Models, wizards, security groups, settings. If you judged code by its posture, it would have passed.

We have one rule that doesn’t care who wrote the code: nothing merges without review. Human, AI, me at 2 a.m. — same gate. So the module went through the same read-only standards review everything goes through.

Round one found three blockers. First, on every product it matched, the module silently overwrote our selling and cost prices with the vendor’s cost — exactly the class of damage our every-price-through-the-engine rule exists to prevent, written by a model that had never heard of that rule. Second, it stored vendor identifiers in a field our stack already uses for customer numbers, setting up quiet identity collisions between customers and vendors. Third, it shipped hardcoded user accounts whose passwords reset to known values on every upgrade — and granted an admin account both halves of a deliberately two-person purchasing approval, collapsing a fraud control into a single click. Behind the blockers, ten smaller findings: demo data shipping as real data, edits reaching already-submitted orders, the native workflow bypassed, dead code.

Here’s where it got interesting. The findings went back as a document written for two audiences at once — a plain-language summary my colleague could act on without reading code, with the full technical detail underneath for the model to work from. The module came off the deploy branch and onto a shelf in the meantime. Revision two came back with all three blockers fixed cleanly and most of the smaller findings resolved. Revision three closed nearly everything else — and notably, on the approval workflow it chose the harder-but-correct redesign instead of patching around the problem, and it flagged its own unknowns instead of guessing. After three rounds, the outstanding list is one unused field. One.

So the honest conclusion: round one wasn’t a capability failure. It was a context failure. The model had never seen our standards file. It didn’t know which identifier namespace was already taken, didn’t know every price flows through an engine, didn’t know that approval is two-person on purpose. Neither would a contractor on their first day — which is roughly what it was. Given the standards and the findings list, it performed like a competent contractor on day two.

One thing no revision could fix, because it isn’t a code problem: the module dials the legacy ERP’s database directly — and as the last entry explained, that ERP is deliberately unreachable from the cloud. The wall did its job again. Nobody told the model about the wall; it can’t know what it was never given. The recommendation on file is to read from synced tables like every other integration here does.

Status, plainly: the module remains shelved, and whether it gets adopted at all is a business decision, not a technical one. But the process proved something worth writing down. AI can do the homework — genuinely, impressively. Review decides what enters the codebase. Keep the same gate for humans and machines, feed the machine the same context you’d give a new hire, and saying yes gets a lot safer.

← All entries