New frontier model benchmarks just dropped — are we hitting diminishing returns on scale?

Narin Chai · 1 likes

honestly as someone building products on top of these models, the benchmark gap matters less than the latency and cost improvements lately

Paolo Garcia · 1 likes

the agentic/tool use benchmarks still have tons of room to improve though, thats where i see the real action next

Mateo Cruz · 0 likes

this tracks with what a lot of researchers have been saying privately for like a year now, scaling laws are bending

Ananya Rao · 0 likes

Diminishing returns on pretraining scale yes, but test time compute / inference scaling is where the real gains are coming from now imo.

Cian Kelly · 0 likes

i think synthetic data and better curation is doing more heavy lifting than people give credit for

Elif Yılmaz · 1 likes

agreed, garbage in garbage out still applies even at massive scale

Lucas Williams · 0 likes

can we talk about how saturated some of these benchmarks are too, models are probably partially trained on leaked eval sets at this point

Priya Nair · 0 likes

lol remember when gpt3 felt like magic, now we complain about marginal benchmark gains. wild how fast the bar moved

Omar Hassan · 0 likes

not an expert but genuine question, does anyone actually know how big these models are anymore or is it all guesswork now

Yasmine El Amrani · 0 likes

its basically all guesswork from the outside at this point, labs stopped publishing params a while ago