Everyone Is Learning to Grade AI. Who Owns the Answer Key?

AI output now arrives looking finished.

I run a small team of AI agents that help me build products and manage work. They work from rules I’ve been writing down for months, many of them from times I’ve sent something back.

Things still get through. Not long ago they ran overnight on one of those products and came back with ten pull requests. Each one had a preview. The run report said every preview built green. Two review passes had approved all ten.

Nobody had opened a preview in a browser, until I did. Every one of them was a blank page.

Nothing in that report was false. The builds were green. The reviews did pass. The standard was that the code builds and deploys without an error, and all ten met it. The work was checked against a standard, just not one that says a page has to render.

What My Search Days Taught Me

I’ve seen this problem before. I was the product lead on search at Yahoo!, and the editorial team built golden sets: their own judgment of what a good result was, written down. Every new version of the algorithm ran against those sets, side by side. The editors set the bar and they held it. If a release wasn’t good enough, it didn’t go out. Someone could overrule them. It rarely happened.

Years later, as CPO, I had the head of editorial reporting to me. By then her team had grown into the group that ran our evaluations, search ranking and side-by-side tests. I thought of them as the analysts of the product teams.

Three things from that have stayed with me. Someone wrote the standard down. Someone owned it. And it was checked before release, when saying “not yet” was still cheap.

Editors held the bar. Engineers ran the harness that checked it.

Testing My Own Answer Key

Last week I went a step further and tested the rules themselves. I had the system interview me to fill the gaps. Twenty questions on how I make product calls. What I’d approve, what I’d send back, and why. My answers went in as rules.

Then we ran twelve cases. Eight came from calls I had already made, like the ten green builds. Four were new, and I answered those cold.

With my rules, the agents matched my call on eight of the twelve. Without them, seven. One run, a small set, so I won’t lean on the number. But writing the standard down was clearly not the hard half.

What I learned was about me. One of my rules said a product doesn’t go in front of an outside user until its best moment lands every time. Then I was handed a case: a habit app whose weekly insight lands about half the time, but I use every day. Share it with a friend? I said yes. On two of the four new cases, my call broke a rule I had given earlier.

What I say good looks like, and what I decide when a real case is in front of me, turned out to be two different things. I rewrote three rules that afternoon, and I expect to keep rewriting them.

Two Sides, One Lesson

I’m not the only one doing this homework. Aakash Gupta recently put out a roadmap for people moving into AI product roles, with evals as a core skill. His best advice is to look at real outputs yourself before you let a model do the grading.

People who teach evals for a living, like Hamel Husain and Shreya Shankar, say much the same. At most small and mid-size companies, they’d make one domain expert the final word on quality. Shankar’s research also found that your criteria shift as you grade.

Engineers got there earlier. Their teams have spent years building the infrastructure that runs these tests at scale.

I think both are right. Product people are being taught evals as a skill. Engineering teams are building them as infrastructure.

What I don’t see in either is who owns the standard once forty people are each writing their own.

Forty PMs, Forty Answer Keys

Hand-labeling works for one person. You look at a hundred real outputs, you mark the good ones, and your marks become what the model gets graded against. Your judgment, written down.

Now put forty product managers in one company and have each of them do it. You get forty standards. Each one is defensible. None of them is the company’s. And I’m one person, and mine disagreed with itself inside a day.

One of the twelve test cases was a product of mine with thirteen users, all people I know. A review found its usage cap didn’t really hold. One person could use far more than their share. The fix was a day of work and a new vendor. The expected answer on the sheet said defer.

I hadn’t written that answer. The system had, from a call I had made earlier, in my own words. Reading the case fresh, I disagreed with it. Add the cheap cap, skip the vendor. That wasn’t even one of the two choices offered.

So the key was drafted by a machine, from my own record, and my record was out of date. Until I was asked, nobody had picked.

Everyone is learning to grade. Who owns the answer key?

The labels are the standard. A scoring tool only measures distance from whatever a person decided was right. So the open question is a name.

My answer: whoever runs product owns the standard, the CPO at a big company, the head of product at a small one. A group PM can own it for their group of products. Each product lead owns their own answer key under it, and keeps it current. Forty answer keys can live in one company. They can’t live without one standard above them.

Before the Build, and After It

Felix von Kunhardt and I wrote a guide this summer that says you discuss, agree and write down what good looks like before you build anything. I still think that’s the first move. It tells you what you’re aiming at while changing course is cheap.

The roadmap everyone is passing around puts evals fifth, after prototyping and agents. I’d move writing down what good looks like to the front. But no order fixes what happens after launch. Once the thing is built and running, the standard needs an owner. Models change, the product changes, and so does what your users will accept. Mine needed three edits on its first day.

It happens to things that already work, too. I have a daily digest that reads the newsletters I follow and saves what’s worth keeping. After a small fix to it, it saved nothing for twenty days. I didn’t notice, because a quiet day and a broken day left the same record behind. “Nothing worth keeping today” looked exactly like “nothing is getting through.”

Where nobody owns it, it’s easy to see how it happens. Product assumes engineering has it, because engineering runs the harness. Engineering assumes product has it, because product wrote the spec. Each is half right, so the output ships on whoever looked at it last.

Where to Start

Three places to start, smallest first.

In your own setup, have your AI interview you on what good looks like for one kind of call you make every week. Write the answers down. Then decide ten cases cold, before you look at what the model said, and check your calls against your own rules. That second step is where I found out.

In your own work, take the next AI-assisted thing you’re about to ship and write one paragraph on what good looks like for it. Then put a name next to who checked. If that feels uncomfortable, that’s the finding.

On your team, pick one kind of AI output you ship every week and ask, out loud, who owns the standard for it. If the answer is “all of us”, you have your forty answer keys.

Who wrote down what good looks like where you work?