Chips & infra updates · Official

Official news, first rumor reports and substantive progress. Each entry keeps the information published at that time.

• llama.cpp v0.9.0 adds OCP FP8 E4M3 quant, multi-sequence KV flash-attn, and four new architectures

  ggml-org released llama.cpp v0.9.0 with OCP FP8 E4M3 quantization, a new flash_attn_ext_rows op aimed at efficient unified KV cache multi-sequence inference, and a generic batched tensor copy API. The notes add four architectures (Clover, Limite, Metis-1.6, and Mistral Large 4), expand audio generation and transcription with Chatterbox TTS, Nemotron 3.5 Transcribe, and predefined Pocket TTS voices, and cite major Gemma 4 E2B/E4B prefill optimizations alongside a ggml bump.

  OfficialOfficial updateNov 4Source: ggml-orgPart of llama.cpp
• llama.cpp v0.8.0 adds LiquidAI LFM3, streaming-audio mtmd APIs, and architectures including DeepSeek V4.1 Flash and Muse Spark

  ggml-org’s llama.cpp v0.8.0 adds the LiquidAI LFM3 family (text, vision, audio, omni) with an experimental streaming-audio API in mtmd and server realtime audio via /v1/realtime and /tts. New architectures include DeepSeek V4.1 Flash, SenseNova U1, Beam, Kolibri-1, Flint, Muse Spark, and Granite Switch Decision, plus sparse-attention and host-resident MoE cache work in the core.

  OfficialOfficial updateOct 28Source: ggml-orgPart of llama.cpp
• llama.cpp merges faster CUDA matrix-vector kernels for low-bit quantizations, speeding up Q2_K and Q3_K generation on older GPUs

  A pull request by am17an, merged into llama.cpp on October 17, speeds up the CUDA matrix-vector kernels used for low-bit quantization types. In the author's llama-bench runs, Qwen3.5-9B token generation on an RTX 3090 rose from 102.67 to 141.02 tokens per second at Q2_K and from 81.04 to 126.22 at Q3_K_S, and an A16 saw up to 1.73x, while the RTX 4090, RTX 5090 and RTX PRO 6000 gained at most 8 percent. The author says Q4_K and Q8_0 are unchanged and discloses that the change was written with help from Claude.

  OfficialOfficial updateOct 17Source: GitHub ggml-org/llama.cppPart of llama.cpp
• llama.cpp v0.7.0 moves the server's default port to 9931 and adds LiliCorr speculative decoding, K2 Horizon support and a web UI models manager

  ggml-org's llama.cpp v0.7.0 changes the server's default port from 8080 to 9931 and refactors its chat layer, adds a models manager to the web UI, lets RPC split tensors in parallel and moves ggml to v0.27.0. It adds LiliCorr speculative decoding and MTP for GLM5-Next, and new models including K2 Horizon (dense and MoVA), Prism Bonsai 2 27B, EmbeddingGemma-2, MiniCPM-V 4.7, Nemotron Super 3.5 VL and Nemotron Bolt, plus decision models such as LiquidAI d1-3B and pplx-decider. The release also ships the GPU cache for host-memory MoE experts (--moe-cache-mib) first reported from its pull request on October 7.

  OfficialOfficial updateOct 15Source: ggml-orgPart of llama.cpp
• llama.cpp adds --spec-draft-n-ubatch to set the draft context's batch size and save VRAM in speculative decoding

  A llama.cpp pull request from Georgi Gerganov, merged on October 14, adds a --spec-draft-n-ubatch option that sets the physical batch size (n_ubatch) of the draft context used for speculative decoding. Because drafters feed that context at most the drafted tokens per sequence, it does not need the target model's ubatch, and a smaller value shrinks the draft compute buffer and leaves more VRAM for the target's KV cache.

  OfficialOfficial updateOct 14Source: GitHub @ggml-orgPart of llama.cpp

RSS for this view →