AI Lab · by Nima Ahmadi · 2026-10-14
Ran the same coding task through 4 different frontier models — results were more different than expected
Gave GPT, Claude, Gemini, and a local Llama variant the exact same refactor task (medium sized python service, ~800 lines). Claude produced the cleanest diff with the fewest unnecessary changes. GPT was fastest but touched more files than needed. Gemini was surprisingly good at catching an edge case bug the others missed. Local llama struggled with the longer context but was totally usable for the smaller files. Anyone doing more rigorous comparisons than vibes?
0 sparks · 11 comments
Comments
Ethan Brooks · 1 likes
gemini catching edge cases is interesting, i've noticed it's unusually good at static-analysis style reasoning
Curious Lantern · 1 likes
4bit, honestly that probably explains some of the degradation on long context
Mehdi Benali · 0 likes
this matches my experience almost exactly, claude tends to be more conservative with diffs which i actually prefer for production code
Rohan Mehta · 0 likes
local models are so dependent on quantization level, what quant were you running llama at?
Diego García · 0 likes
i built a small harness that scores diffs on lines changed vs test pass rate, happy to share if useful
Tiny Rider · 0 likes
yes please, always looking for better eval tooling
Amina Okafor · 0 likes
the real test is always 6 months later when you have to maintain the code one of these things wrote lol
Kabir Khan · 0 likes
curious if you tried giving them the same system prompt or just raw defaults, that changes results a lot
Theo Bennett · 0 likes
gpt touching more files than needed has been my experience too, it loves to 'helpfully' refactor adjacent code
Maya Putri · 0 likes
appreciate someone actually doing a real comparison instead of just vibes, more of this please