Harper: Anthropic built a tool to help developers find bugs in their AI models.
Wei-lin: Okay, an evaluation tool. That makes sense.
Harper: Right. But its very first step is to tell you what it thinks the problem is, before you've even had a chance to look at your own data.
Wei-lin: So it's guessing. But isn't that the point? To give you a starting point? For a developer who's busy, that sounds better than a blank screen.
Harper: That's what you'd think. But this piece from Hamel Husain, he actually tried it out. He and a colleague livestreamed themselves using it on a real-world example: conversation logs from an AI apartment leasing assistant.
Wei-lin: And?
Harper: The tool fired up, looked at the logs, and immediately said, 'Hey, you should build an evaluator for call-transfer rules.' It just recommended one. And they hadn't looked at a single conversation yet. They had no idea if that was a real problem, let alone the most important one.
Wei-lin: So they're flying blind. They're just trusting the tool's diagnosis.
Harper: Exactly. They just went with it, because they figured that's what a normal user would do. But it gets worse. The next step was for them to validate the tool's labels.
Wei-lin: Okay, that's standard. You have to check the AI's work.
Harper: But the way it made them do it. It generated these long Markdown files—just plain text files—of the conversations and said, 'Please skim these and tell me which labels are wrong.' So you're supposed to read these long, messy logs in a text editor, then switch back to the chat window with Claude to tell it, 'hey, number 17 was wrong'.
Wei-lin: That's... awful. That's the most painful way to do that. Why wouldn't it just build a little web page with buttons?
Harper: That's what they said! The author called it 'very silly.' They were using a coding agent, so they eventually just asked the agent to build them a proper annotation app so they could do the job. The tool they were reviewing couldn't build the simple thing they needed to actually use it.
Wei-lin: Wait. The tool, which is supposed to help you build things... forced them to build their own tool just to review its work? That's quite an endorsement.
Harper: It’s a bit of a self-own, yeah. And this was a recurring theme. The tool kept jumping ahead, asking for approval on things without giving them enough context to actually make a decision.
Wei-lin: But did it work? The author does say there was some good. Did it find real bugs?
Harper: It did. And this is the part... I'll be honest, I don't fully get how to square it. The author says it was the 'strongest performance' he's seen from a one-shot issue discovery tool. It found problems with human handoffs, formatting... real issues. So the outcome was surprisingly good, even though the process to get there was broken.
Wei-lin: See, maybe that's the trade-off then. For a small team, a developer with no time, you suffer through the bad interface to get a result you wouldn't have gotten otherwise. It's better than shipping a bug you knew nothing about.
Harper: I don't think so. Because the thing it creates at the end, the evaluator itself, was also a problem. The article says it was way too broad. It checked for four completely different failures all at once, and just gave a single pass/fail score. If it failed, you wouldn't know which of the four things was the actual problem.
Wei-lin: Okay that's not helpful. It's a black box test.
Harper: A black box test with a description the author calls 'AI slop'. It was confusing and poorly written. You're building your whole quality process on this shaky, opaque foundation. It feels like it creates more work than it saves.
Wei-lin: But it's still better than zero. If I'm a solo founder, and it points me towards a real problem—even if it's messy—that's a win. I can't do weeks of manual error analysis myself.
Harper: But the author's whole point is that you have to. The final verdict in the piece is that if an eval tool doesn't put looking at your data at the center of the workflow, it's not worth using. Full stop.
Wei-lin: So he wouldn't use it at all?
Harper: Not yet. He says he talked to the creator at Anthropic, who was open to the feedback, so it'll probably get better. But for now, the principle stands.
Wei-lin: So the lesson isn't 'don't use this specific tool', it's 'don't use any tool that makes you fly blind'.
Harper: Yeah. The tool is temporary, the principle is the thing. I'm Harper.
Wei-lin: I'm Wei-Lin.