Filters
- SonicBoom, a Python-free C++/CUDA runtime, claims 52 tok/s decode on a laptop RTX 3050 versus 45 for llama.cpp
A developer tagged v0.0.1 of SonicBoom, a native tensor runtime in C++/CUDA with a C ABI and no Python, aimed at local LLM inference on consumer GPUs. On Gemma 4 E4B-it Q3_K_M on an 8 GB laptop RTX 3050, using the same GGUF and quantized kernels, the author reports 51.65 tok/s decode versus 45.20 for llama.cpp and credits a graph compiler for the gap. Prefill is slower in the author's numbers, about 1,051 tok/s versus roughly 1,944 for llama.cpp on a 4,100-token prompt.
- Nvidia's share of China's AI chip market fell to 34% in the first half as Huawei rose to 28%, IDC data shows, per Reuters
Nvidia shipped about 678,000 AI accelerators in China in the first half of 2026, down 44% and giving it a 34% share, while Huawei shipped about 572,000 for 28%, Reuters reports, citing IDC data. Alibaba's T-Head held 7%, Cambricon 5%, Kunlunxin 4% and Hygon 3%, out of roughly 2.008 million units in total. GPU shipments fell 26% to about 991,000, and non-GPU accelerators outnumbered GPUs for the first time.
- NVIDIA and Cisco introduce USFP pluggable optics with 400Gb/s per lane for AI factories
NVIDIA and Cisco introduced USFP pluggable optics, which NVIDIA says is the first module to support 400Gb/s per lane, with four times the bandwidth density of current optics. NVIDIA says builders can connect more compute to the same switch without extra network tiers and with less power per bit.
- Alibaba Cloud's Jason Wu says AI training, inference and agents are reshaping infrastructure needs
At Apsara Conference 2026, Jason Wu, Alibaba Cloud's head of Elastic Compute Service, discussed how AI is changing what infrastructure must deliver, from model training and inference to running AI agents at scale.
- Developer publishes a lecture and article on building a GPU compiler that lowers PyTorch graphs to CUDA
A developer released a GPU_MODE lecture and an article on building a GPU compiler from scratch, covering concepts shared by compilers such as TVM, Modular and PyTorch Inductor. The article shows how operations like RMSNorm, matmul and attention are lowered through multi-level IRs into CUDA kernels. The developer says the compiler works with vLLM for end-to-end model serving.
- Intel names Crescent Island the GPU D-Series, with D270 offering up to 480 GB LPDDR5X for inference
Intel published the GPU D-Series product page for its Crescent Island inference accelerator, headlined by the GPU D270. The air-cooled PCIe card is rated at 350 W with up to 480 GB LPDDR5X, 1.5 TB/s memory bandwidth, 32 Xe3P cores, 256 XMX engines and up to 1.3 PFLOPS of FP8 compute, aimed at agentic AI, long contexts and high-concurrency serving without HBM.
OfficialSource: r/LocalLLaMA - Microsoft will scale Azure Maia to multi-gigawatt deployments from 2027, making it the primary inference system for its products
Microsoft says Maia 200 now serves production workloads including Microsoft Copilot, with GitHub Copilot and Microsoft Foundry to follow by year end, running OpenAI, MAI and open-source models. Its next chip, Maia 300, is built for long-context reasoning, latency-sensitive chat and multi-step agentic tasks and begins production deployments in 2027. Microsoft expects 40% more tokens per megawatt than leading GPU-based systems and up to 7x Maia 200's rack-scale token throughput in a 96-accelerator configuration.
- NVIDIA contributes Vera Rubin rack and compute tray designs to OCP and open-sources new DSX AI factory software
At the Open Compute Project Global Summit, NVIDIA contributed Vera Rubin rack and compute tray designs through its open MGX architecture, including the MGX 1.2 spec with blind-mate liquid cooling and support for racks up to 250kW, plus a Vera CPU rack design for agent tool calling and sandboxed execution. It also released open-source DSX OS components for network configuration, cluster stress-testing, rack management and bare-metal provisioning. NVIDIA joined AMD, Google and Meta on the Open Data Center Specification v0.7, and Google, Meta, Microsoft and NVIDIA published an OCP Grid Resilience Specification.
- shad0w learns a local LLM's classification answers and serves the repeats in microseconds
shad0w is an open-source Apache-2.0 tool for Python, JS and C that sits in front of any OpenAI-compatible server, including Ollama, logs the model's answers and, after about 1,000 of them, compiles a small table that answers repeat classification queries on CPU in about 2.4 microseconds. It only answers where it matches the model at least 95% of the time on held-out answers and sends everything else to the LLM; on its benchmarks it handled 61 to 94% of messages.
- LocalLLaMA user ports Gemma 4 E4B EXL3 via exllamav3 as a Jetson Orin alternative to llama.cpp
A LocalLLaMA post describes little-gemma V1.0, a Jetson Orin–optimized engine that the author says substantially outperforms llama.cpp on Gemma E4B, yet still falls short of ideal speed on an Orin Nano Super 8GB after trying practical GGUF options. They forked exllamav3, ported key little-gemma V1.0 code into an EXL3 path, and present it as another way to run Gemma 4 E4B on Jetson hardware.
- Decapybara open-sources a C++20 and CUDA rebuild of the MegaCapybara inference server
GitHub user giveen published Decapybara, an open-source C++20 and CUDA reconstruction of the MegaCapybara inference server, arguing the community should not have to run closed “trust me” binaries that claim top speed without source. The repository is linked from a LocalLLaMA thread announcing the release.
OfficialSource: GitHub giveen/Decapybara - LocalLLaMA user posts a final zzpanic vllm-radiance build tuned for one R9700 running Qwen3.8-27B
A LocalLLaMA post shares a last revision of the zzpanic vllm-radiance personal build for a single R9700 GPU running Qwen3.8-27B. The author says newer radiance engines have largely superseded older vllm-radiance builds and is wrapping remaining single-digit percent speedups.
- Cheaper AI tokens are driving more demand, and that's Jensen Huang's best-case scenario, per The Decoder
The Decoder covers a16z-cited data showing a Jevons paradox in AI: token prices keep falling while H100 GPU rental prices hold or climb, so cheaper inference fuels demand faster than unit costs drop. If that demand flattens, chip makers through cloud providers take a hit.
- LocalLLaMA user reports Next Flash IQ3_XXS ~70 tok/s or dual Qwen 3.6 35B-A3B ~145 tok/s on $500 BC-250 boards
A LocalLLaMA user’s third BC-250 cluster update says four ex-mining BC-250 boards on 2.5GbE with llama.cpp Vulkan RPC run Next Flash IQ3_XXS at about 70 tok/s with 100k context, or two Qwen 3.6 35B-A3B IQ4 instances at about 145 tok/s with 256k context, for roughly $500 of boards.
- QbitAI says Silicon Valley AI competition is now gated by who controls GPU supply, per QbitAI
QbitAI reports that compute scarcity is reshaping Silicon Valley AI: Anthropic sought capacity from rivals, SpaceX resold Nvidia GPUs, and even returning Google researcher Noam Shazeer left again for OpenAI amid internal GPU rationing. Venture firms are described as stockpiling GPUs as the scarce resource.
- NVIDIA reportedly ends new RTX 5090 GB202 supply for GeForce, reserves chips for RTX PRO, per VideoCardz
VideoCardz reports leaker MEGAsizeGPU saying NVIDIA will no longer allocate new GB202 silicon to GeForce RTX 5090 production and will reserve future GB202 supply for RTX PRO workstation cards. A separate factory-notice claim says an RTX 5080 24GB would become the top GeForce card; NVIDIA has not confirmed either report.
- strata-mlx runs Qwen3.8-Flash-Next GGUF files on Apple Silicon via MLX
yibie open-sourced strata-mlx, an unofficial MIT-licensed MLX engine that loads Strata's GSQ-RCO GGUF files unchanged and speaks Strata's line protocol so Strata's server can drive it on Mac. The README reports that on one M4 Max 128 GB machine, answer generation was about 1.6–2.9× faster than llama.cpp/LM Studio for Q2_0 and IQ3_S quantizations, with draft-head verify optional. It needs Apple Silicon, Python 3.12+, and large GGUF shards (about 66 GB for Q2_0); image input and multi-conversation decode are not supported yet.
OfficialSource: GitHub @yibie/strata-mlx - Huawei pitches OceanStor M900 Context Memory Storage for long-context AI inference
Huawei says its OceanStor M900 Context Memory Storage offers large shared capacity, high-performance data access and optimized token costs for hyperscale AI data centers, as context windows grow and inference workloads scale.
- LlamAmpere update runs Qwen3.8 27B with 200K context and MTP on 12GB Ampere GPUs
A developer released a new version of LlamAmpere, a runtime tuned for RTX 3090-class Ampere cards, that runs Qwen3.8 27B with 200K context and multi-token prediction on 12GB GPUs. The developer says it is 3-4% faster with a runtime a few hundred MB smaller, and supports up to about 340K context with YaRN.
- Basalt inference engine runs Qwen3.8 Flash-Next at 665 tok/s on an RTX 5090 plus 5060 Ti
A developer released Basalt, a heavily tuned fork of the Strata inference engine that supports only Qwen3.8 Flash-Next on Nvidia Blackwell GPUs, including dual-GPU setups. On an RTX 5090 plus RTX 5060 Ti, the developer reports 665 tok/s for structured output and 354 tok/s for prose, up to 2.6x Strata on the same weights.