Throughput only counts after correctness
Open inference is attaching revenue claims to extreme throughput while runtime releases show how numerical, memory and transport bugs can invalidate the result.
Open inference is entering the stage where performance claims come with revenue models—and correctness bugs can erase them.
Open inference is entering the stage where performance claims come with revenue models—and correctness bugs can erase them. vLLM reports 954K total tokens/s/GPU on a GB300 NVL72 AgentX run, translating DeepSeek off-peak pricing into $38.96/GPU-hour. Meanwhile llama.cpp added native DeepSeek V4.1 Flash support, but its same-window releases also fixed a MiniMax scale-propagation bug that produced token soup, disabled an invalid multi-device MoE split, and added an RDMA bounds check. @vllm_project ↗ · @vllm_project ↗ · github-actions[bot] ↗ · github-actions[bot] ↗ · github-actions[bot] ↗ · github-actions[bot] ↗
Ollama’s stable 0.40.11 broadens local workloads with streaming speech and decision models. The caveat: vLLM’s numbers are project-reported on frontier hardware, and community results are single rigs. The implication is simple: capacity, revenue and features are downstream of numerical and operational integrity. Publish the complete serving recipe, validation cases and failure behavior—not only peak throughput—because a fast system that silently corrupts weights or streams partial failure is not production-ready. github-actions[bot] ↗ · Due-Sand6422 ↗ · GapNew4766 ↗
- The caveat: Project benchmarks can still be useful engineering targets, but they do not transfer automatically across hardware, traffic mixes, quantization paths or client behavior.
- What to watch: Treat numerical validation, transport safety and observable failure semantics as prerequisites for performance and cost claims.
OS Twitter / X Recap
Collection scope
The bounded logged-out sample returned 18 in-window posts from configured profiles, with one duplicate cross-profile observation. x-eliebakouch again exposed no supported payload; this is not a complete timeline or reply view.
vLLM turns throughput into revenue math
- vLLM says it matches DeepInfer with fewer ranks The project reports matching DeepInfer’s DeepSeek V4.1 Flash decode curves from batch sizes 8–256 while using DEP32 rather than EP64DP64, after integrating DeepSeek’s open kernels and optimizing prefill and decode. @vllm_project ↗
- A GB300 rack reaches 954K tokens/s/GPU With prefill/decode disaggregation and Wide-EP on GB300 NVL72, vLLM reports 954K total tokens/s/GPU and 196 tok/s P90 per user on AgentX—7.9× its single-node result. @vllm_project ↗
- The benchmark is translated into GPU-hour revenue Using DeepSeek off-peak pricing, vLLM calculates $38.96 per GPU-hour at the AgentX operating point. That is project-reported unit economics, not an audited provider margin or hardware-cost comparison. @vllm_project ↗
Reddit Recap
Collection scope
Twelve of 14 configured communities returned bounded archive samples after pacing and one permitted retry; r/llamacpp and r/Oobabooga were empty. The 221 records are community reports with provider lag and incomplete thread coverage.
A fast stack still has to finish the job
- Five agent harnesses get the same model A blind M5 Max test gave five agents the same Qwen3.8-27B quant and five scripted tasks. Four completed all tasks; one failed two after asking for an unanswered install approval. GapNew4766 ↗
- Passing agents varied from 10 to 24 minutes Among the four harnesses that passed every task, total time ranged from 10:29 to 24:07. The author withheld names pending a later reveal, limiting product-level conclusions. GapNew4766 ↗
- Long-context speed does not guarantee task success A 9060 XT user reports 40–55 tok/s through 130K–262K contexts in Strata, yet Qwen3.8-Flash-Next and Swift 1.5 both failed a Figma-plugin task that Argon completed after several prompts. Due-Sand6422 ↗
- Exact speculation still helps legacy K80s A community benchmark headline reports 1.3–1.4× speedups for Qwen3.6-35B-A3B through 14K context on NVIDIA K80s, with no slowdown beyond. The collected post contains no configuration table. Electrical_Ninja3805 ↗
Local tooling fills operational gaps
- DeepSeek Harness gets a source-control panel A community plugin reviews diffs, generates Conventional Commit messages, switches branches, previews changed media, pushes commits and can create GitHub repositories through the local `gh` CLI. Aggravating-Mix-8663 ↗
- Backlog.md tasks move into the harness Another plugin adds a filterable project-task panel, task details and “Send to Chat,” while syncing CLI and agent edits over WebSocket. It is an author announcement, not a core release. Aggravating-Mix-8663 ↗
- GPU unlock gains depend on precision and pipeline A comparison of three Linux reports shows MOSS BF16 on a 3090 Ti dropping from 71.98 to 45.40 seconds, while INT8 barely changes. The compiler did not run the hardware tests. Acceptable_Put9983 ↗
- MOSS native edit nodes add an HQ LoRA path The workflow author says updated defaults now cover the HQ edit LoRA and documents 14 edit types with prompts and known weak spots. Older strength-0.25 workflows need reopening. LatentSpacer ↗
- A four-minute local music video takes about a day A Longshot Studio user reports roughly 24 hours to assemble a four-minute H3 video and publishes the ComfyUI API studio plus a 1.4GB project containing unused takes. R34vspec ↗
Discord Recap
Collection scope
The four approved OSNews collector caches were still within the 14-hour reuse limit and retained true capture times, cloudBrowserRead:false, personalAccountUsed:false and freeOnly:true. However, all 194 returned messages predated this evening window, so no Discord message is presented as new.
Primary-Source Recap
Collection scope
Official GitHub release feeds establish shipped runtime changes. Performance and economic claims remain in the attributed X section; these release notes show the correctness, portability and safety work underneath.
DeepSeek V4.1 Flash reaches llama.cpp
- llama.cpp adds native DeepSeek V4.1 Flash support Build b11870 adds the deepseek41 architecture, converts Engram tables to BF16, computes compressed-KV read layers once, gathers Engram rows in-graph and adds the model to Metal fusion baselines. github-actions[bot] ↗
Correctness fixes behind the benchmark
- Missing MiniMax scales produced token soup Build b11861 forwards per-expert NVFP4 and dense/shared MXFP8 scales into the graph after the omission made routed-expert weights globally too large and swamped residuals. github-actions[bot] ↗
- An invalid MoE tensor split is disabled Build b11858 denies tensor split mode for architectures using row-wide amax reduction because each device otherwise computed only a partial maximum while reporting mirrored state. github-actions[bot] ↗
- RDMA receive gains a bounds check llama.cpp b11865 adds a bounds check in `rdma_recv`, a small release-note entry with direct importance for distributed inference transport safety. github-actions[bot] ↗
- Hexagon flash attention uses stranded cores Build b11868 adds KV-head team splitting so multicore HMX flash attention can keep extra cores active when models such as Gemma 4 expose fewer KV heads than cores. github-actions[bot] ↗
Ollama broadens local interaction
- Ollama 0.40.11 streams speech on macOS The stable release can play generated speech as it arrives and exposes an OpenAI-compatible `/v1/audio/speech` endpoint for applications that need streamed local audio. github-actions[bot] ↗
- Decision models become a first-class family Ollama adds vLLM Semantic Router’s d3 models in 2B, 4B, 9B and 27B sizes for classification, yes/no decisions and probability scoring over text, state or images. github-actions[bot] ↗