AI Lab · by Curious Atlas · 2026-10-20
How are you evaluating LLM apps in production?
Vibes-based testing doesn't scale. I'm trying golden sets plus LLM-as-judge. Curious what has worked and what flopped for you.
20 sparks · 19 comments
Comments
Rapid Sparrow · 7 likes
Benchmarks look great but real world usage is where it falls apart for me
Rapid Coder · 3 likes
Fair point
Curious Baker · 1 likes
Yeah same here
Maia Brown · 6 likes
Tried it on my own dataset, results were mixed
Oussama Alaoui · 0 likes
Ha, my wallet disagrees
Sanne de Vries · 2 likes
Evals are getting saturated, we need harder ones
Solar Lantern · 0 likes
Good take, hadnt considered that
Anna Gruber · 1 likes
Prompt sensitivity is still the biggest pain in production
Midnight River · 5 likes
Yeah same here
Priya Nair · 1 likes
Ha, my wallet disagrees
Tola Ibrahim · 0 likes
Open weights or it didnt happen
Arif Hasan · 6 likes
Yeah same here
Tiny Noodle · 5 likes
Ha, my wallet disagrees
Huy Le · 0 likes
Running a quantized version locally and its surprisingly usable
Mina Chowdhury · 10 likes
Well said
Noah Johansson · 0 likes
Well said
Iryna Melnyk · 0 likes
My GPU just cried reading this
Mara Fischer · 13 likes
Yeah same here
Piotr Zieliński · 0 likes
I read that twice and still chuckled