Blog 3 of 3: the second engagement, in depth
This is the last post in a series about my experience as a PM on an AI data platform. Blog 2 was about an archive that never changes. This one is the opposite problem, told in the same six parts: the problem, what I did, what we built, what I learnt building an AI platform, how I used AI to do the job, and what came of it.
1. The problem
A leadership team making investment decisions under real pressure, with almost no joined-up data to make them on. The evidence they needed was public, but it sat across many sources that changed without warning, and nobody had pulled it into one picture of where the organisation stood against its peers or where it should invest next. The alternatives were expensive consultants, thin internal reporting, or deciding by anecdote.
The constraint that shaped everything was simple to state: these figures would be acted on. A number in a leadership briefing had to be traceable to its source, reproducible, and checked before anyone saw it. The client's own standard, set early, was that every material statement had to be independently checkable. So success wasn't a clever answer. It was an answer leadership could stand behind.
2. What I did
Discovery here was a cycle rather than an interview. The domain expert had spent a career on the method and knew far more than I did, so my usual approach, talk to people a few times and write it up, ran out within a fortnight. What worked instead was proposing a rule in plain language, letting them pull it apart, coming back with a better one, and going round again until we had something an engineer could build and the expert would sign off. About halfway through the build we stopped and went back to first principles: who are the customers, what are their problems, who else solves them. It felt like a detour at the time. It turned out to be the most useful week of the engagement, because it showed us that the problem we'd nearly set aside was the one nobody else was solving, and that our biggest competitor was, in effect, doing nothing.
When a new and genuinely exciting module turned up mid-build, I wrote down the five things that had to be true before the first deliverable could go out and said, in front of everyone, that nothing would jump the queue. Holding that line against my own enthusiasm was harder than holding it against anyone else's.
And I ran the stakeholders by lanes. Seven lines of authority sat in one head: product intent, method, architecture, the verification standard, commercial scope, institutional decisions, and everything else. Every question went to the right lane. Two versions of every document went out and were never mixed. And when sources changed, or a review raised hard questions six weeks before a big presentation, the answer was a plan grouped by owner with the same facts stated calmly. Calm, honest reporting of what wasn't working yet is what kept the sponsor investing.
3. What we built
Put plainly, it's a pipeline with a verification layer underneath and three views on top.
It reads public sources on a schedule. Agents pull from each source, keep every pull, and record exactly what was fetched and when, so any figure can be traced back to the pull it came from.
It resolves the sources into one picture. The same organisation turns up under different names in different sources; the platform reconciles them, scores them against a chosen set of peers, and rolls everything up into one view anchored to the organisation's own strategy.
It proves every figure before it is published. Nothing reaches a report until every number has been re-derived in code from its inputs. If the model and the data disagree, the data wins. If a claim doesn't have enough independent support, it gets dropped and flagged as a gap rather than published.
It shows three audiences three different things. An operations view where every status is visible, a review view where nothing reaches the client until a person has approved it, and a leadership view with plain-language answers and every statement checkable.

The verification layer in one picture: every catch became a rule, and every rule made the client more confident rather than less.
4. What I learnt building an AI platform
The product's model will, one day, produce a plausible number with nothing underneath it. Design for that day. I remember the afternoon. A report was nearly ready to go out and one of the totals read perfectly well. I traced it back, as I'd started doing out of habit, and there was nothing there. The model had filled a gap with a number that looked like all the others. It was sobering, partly because of the figure and partly because of how close it came to leaving the building with my name on the cover note. Nothing dramatic followed, because the check caught it. But from that afternoon verification stopped being a test at the end and became the feature I designed first, and the one the client ended up valuing most. The rule that came out of it, that the data is authoritative over anything the model says, is now the first guardrail I ask for on any build.
Design for trust before cleverness. The first question about any feature is how the person reading its output will know it's right. Every guardrail we named along the way, what the model could read, what it could reach, what happened when it was wrong, built the client's confidence rather than denting it, which surprised me at the time.
Sources move. A public source quietly renumbering its tables isn't a bug in your product. It will become one. Checking that the ground hasn't shifted under you is real engineering work and it has to be planned and paid for.
5. How I used AI to do the job
The other thread, kept separate on purpose: this is my working method, not the product.
I checked every build myself. I'm not an engineer. But every morning I went through the latest build with AI beside me, pulled the figures out of the reports, added them up again and compared them with the previous build. The first time I went to an engineer with "those two numbers should match and they don't" and turned out to be right, the way we talked to each other changed for good.
I learnt how to run an AI-assisted team. Three engineers, three stories, three AI coding assistants. Each story passed its own tests. Put together, the same fact showed different values in different places, because each assistant had a perfect view of its own story and no view of anyone else's. None of the figures was wrong on its own. Together, they were. That's not a failing of any engineer; it's what AI-assisted building does by default, and it took weeks of checking, mine included, before the pattern was obvious. Every catch became a rule in code so the same catch was never manual twice. What I took from it: treat generated code as a draft from a fast junior with no memory of anyone else's work, measure what's shippable rather than what's generated, and recompute rather than read.
I drafted twice and prototyped first. AI wrote the second version of every document every week, so I had the hours to get the first version right. Before a line of real code was written I'd built clickable views of what each audience would see, and I kept them current as the product took shape so the sponsor could watch it grow instead of waiting for it. Later I moved to an AI design tool and found I could go from "roughly like this" to "exactly like this" without a designer in the room. For someone who'd spent a decade handing requirements to other people to make visible, that was one of the quiet joys of the year.
6. The outcome
The first deliverable went out with every data point traceable to its source and every claim checkable, and it got used. The conversation in the room moved from whether the numbers could be trusted to what to do about them. The verification layer, which nobody had asked for on day one, became the part of the product the client talked about most and the reason it was worth extending. Trust didn't come from the AI being right. It came from being able to show, every single time, how we'd checked.
That closes the series. If you're a PM making the same move into AI, I'd love to know what stretched you, and what you had to unlearn. I'm learning this job again every week, and I'd rather do it out loud.
Drafted with AI assistance, as is my standing practice. The experience, the mistakes and the opinions are mine.
Comments
Join the conversation. You can edit your comments later using this browser. Clearing site data removes that edit access.