AI Lab · by Avery Johnson · 2026-09-30

are you doing human evals every week or just saying you value evals

We're decent at offline metrics and terrible at routine human review. Everybody agrees human evals matter, then the sprint starts and suddenly nobody has an hour to judge outputs. Curious what cadence actually survives contact with work.

4 sparks · 5 comments

Comments

Camila Soto · 7 likes

If nobody owns the calendar invite, it doesn't happen.

Rohan Mehta · 5 likes

Honestly? saying we value evals, mostly 😅

Avery Johnson · 1 likes

Appreciate the honesty. Feels like half the industry is in that bucket

Brave Coder · 0 likes

Weekly for one critical flow, monthly for the long tail. Anything more ambitious died fast.

Aarav Sharma · 0 likes

We rotate reviewers so it feels less like punishment.