DeepSeek open-sourced DeepInfer, their own inference framework for DeepSeek models

Ethan Brooks · 11 likes

another inference framework. we now have more inference frameworks than models i can afford to run

Daan Bakker · 10 likes

the real moat is having gpus

Léa Bernard · 10 likes

deepseek has a track record of releasing infra stuff that's actually good, so I'm taking this seriously

An Pham · 8 likes

their open infra week stuff a while back was legit useful, people still use some of it

Nima Ahmadi · 10 likes

Timing is funny given the US gov news from the same week. open sourcing everything is pretty good PR when people question your trustworthiness

Amir Tan · 2 likes

code is code, at least you can read it. way more auditable than an api

Tiny Rider · 9 likes

just cloned it, readme is decent, build took a while. haven't benchmarked yet will report back

Sleepy River · 1 likes

pls do

Pablo Martín · 8 likes

does it support anything other than nvidia? asking for my sad pile of amd cards

Haruto Sato · 4 likes

the amd tax strikes again

Rohan Mehta · 1 likes

a framework that only targets one model family feels narrow. vllm is popular because you swap models without rewriting everything

Angel Reyes · 11 likes

fair but then you're maintaining two serving stacks in prod and nobody wants that

Rizky Pratama · 1 likes

narrow is the point though. if they know exactly what architecture they're serving they can go way deeper on kernels

Liam Wilson · 0 likes

would love to see real throughput comparisons vs sglang on the same hardware before I believe the 'high-performance' part

Rapid River · 4 likes

every framework is high performance in its own readme

Oliver Smith · 0 likes

I expect the good bits get upstreamed into vllm/sglang within a month anyway

Chloe Davis · 0 likes

that's honestly the best outcome for everyone