All updates

Official news, first rumor reports and substantive progress. Each entry keeps the information published at that time.

• llama.cpp v0.9.0 adds OCP FP8 E4M3 quant, multi-sequence KV flash-attn, and four new architectures

  ggml-org released llama.cpp v0.9.0 with OCP FP8 E4M3 quantization, a new flash_attn_ext_rows op aimed at efficient unified KV cache multi-sequence inference, and a generic batched tensor copy API. The notes add four architectures (Clover, Limite, Metis-1.6, and Mistral Large 4), expand audio generation and transcription with Chatterbox TTS, Nemotron 3.5 Transcribe, and predefined Pocket TTS voices, and cite major Gemma 4 E2B/E4B prefill optimizations alongside a ggml bump.

  OfficialOfficial updateNov 4Source: ggml-orgPart of llama.cpp
• llama.cpp v0.8.0 adds LiquidAI LFM3, streaming-audio mtmd APIs, and architectures including DeepSeek V4.1 Flash and Muse Spark

  ggml-org’s llama.cpp v0.8.0 adds the LiquidAI LFM3 family (text, vision, audio, omni) with an experimental streaming-audio API in mtmd and server realtime audio via /v1/realtime and /tts. New architectures include DeepSeek V4.1 Flash, SenseNova U1, Beam, Kolibri-1, Flint, Muse Spark, and Granite Switch Decision, plus sparse-attention and host-resident MoE cache work in the core.

  OfficialOfficial updateOct 28Source: ggml-orgPart of llama.cpp
• llama.cpp merges faster CUDA matrix-vector kernels for low-bit quantizations, speeding up Q2_K and Q3_K generation on older GPUs

  A pull request by am17an, merged into llama.cpp on October 17, speeds up the CUDA matrix-vector kernels used for low-bit quantization types. In the author's llama-bench runs, Qwen3.5-9B token generation on an RTX 3090 rose from 102.67 to 141.02 tokens per second at Q2_K and from 81.04 to 126.22 at Q3_K_S, and an A16 saw up to 1.73x, while the RTX 4090, RTX 5090 and RTX PRO 6000 gained at most 8 percent. The author says Q4_K and Q8_0 are unchanged and discloses that the change was written with help from Claude.

  OfficialOfficial updateOct 17Source: GitHub ggml-org/llama.cppPart of llama.cpp
• llama.cpp v0.7.0 moves the server's default port to 9931 and adds LiliCorr speculative decoding, K2 Horizon support and a web UI models manager

  ggml-org's llama.cpp v0.7.0 changes the server's default port from 8080 to 9931 and refactors its chat layer, adds a models manager to the web UI, lets RPC split tensors in parallel and moves ggml to v0.27.0. It adds LiliCorr speculative decoding and MTP for GLM5-Next, and new models including K2 Horizon (dense and MoVA), Prism Bonsai 2 27B, EmbeddingGemma-2, MiniCPM-V 4.7, Nemotron Super 3.5 VL and Nemotron Bolt, plus decision models such as LiquidAI d1-3B and pplx-decider. The release also ships the GPU cache for host-memory MoE experts (--moe-cache-mib) first reported from its pull request on October 7.

  OfficialOfficial updateOct 15Source: ggml-orgPart of llama.cpp
• llama.cpp adds --spec-draft-n-ubatch to set the draft context's batch size and save VRAM in speculative decoding

  A llama.cpp pull request from Georgi Gerganov, merged on October 14, adds a --spec-draft-n-ubatch option that sets the physical batch size (n_ubatch) of the draft context used for speculative decoding. Because drafters feed that context at most the drafted tokens per sequence, it does not need the target model's ubatch, and a smaller value shrinks the draft compute buffer and leaves more VRAM for the target's KV cache.

  OfficialOfficial updateOct 14Source: GitHub @ggml-orgPart of llama.cpp
• strix-llama rdna35-perf branch speeds Strix Halo prompt processing vs Vulkan and stock ROCm

  A LocalLLaMA author published the rdna35-perf branch on GitHub asf0/strix-llama.cpp (a Strix Halo–focused llama.cpp fork, ROCm/HIP only). On a Radeon 8060S / 128 GB Strix Halo box they report prompt processing 21–54% faster than Vulkan and up to about 2.7× faster than the stock ROCm kernel at 128k context with unchanged output quality, with the ROCm gap widening as context grows.

  OfficialOfficial updateOct 12Source: GitHub asf0/strix-llama.cppPart of llama.cpp
• llama.cpp merges probabilistic MTP drafting with rejection-sampling verify for ~14% faster prose decode

  llama.cpp pull request #27694 is merged: model-based drafters keep probabilistic samples and the target verifies with rejection sampling for simple draft and MTP, instead of discarding sampler output for the top candidate. A LocalLLaMA post reports about +14% decode on prose and recommends updating llama.cpp.

  OfficialOfficial updateOct 11Source: GitHub ggml-org/llama.cppPart of llama.cpp
• Georgi Gerganov says llama.cpp appeared on stage at Microsoft's Windows event, per a Reddit link post

  A link post on r/LocalLLaMA shares an X post from llama.cpp creator Georgi Gerganov saying it was 'very cool' to see llama.cpp on the big stage at the day's Windows event. He wrote that the software and hardware stacks are finally coming together after years of community work and that he expects more users to embrace local AI.

  UnconfirmedPublished newsOct 7Source: r/LocalLLaMAPart of llama.cpp
• llama.cpp pull request adds a GPU cache for MoE experts kept in host memory, with up to 2.2x faster decode

  A llama.cpp pull request by am17an ports qvac-fabric's MoE expert cache, keeping an LRU cache of experts on the GPU and uploading only misses when experts live in host memory, enabled with --moe-cache-mib. Benchmarks on Qwen3.8-Flash-Next Q4_0 show 1.57-1.62x faster token generation on an RTX 4090 and up to 2.2x on an RTX 5090, per a post on r/LocalLLaMA. It currently supports single-GPU inference.

  UnconfirmedPublished newsOct 7Source: r/LocalLLaMAPart of llama.cpp
• llama.cpp pull request adds GLM5Next multi-token prediction for running GLM 5 Flash locally

  A llama.cpp pull request by pwilkin adds multi-token prediction (MTP) support and optimizations for GLM5Next, so users can run GLM 5 Flash with MTP locally, according to a post on r/LocalLLaMA.

  UnconfirmedPublished newsOct 7Source: r/LocalLLaMAPart of llama.cpp
• User finds mainline llama.cpp beats ik_llama.cpp on a hybrid multi-GPU rig, per a Reddit post

  A post on r/LocalLLaMA says mainline llama.cpp consistently outperforms the ik_llama.cpp fork on the author's hybrid multi-GPU setup, despite recommendations to the contrary.

  UnconfirmedPublished newsOct 6Source: r/LocalLLaMAPart of llama.cpp
• Qvac's Metal kernels for speculative decoding on Apple Silicon merge into llama.cpp

  Qvac's pull request adding new Metal kernels for speculative decoding on Apple Silicon was merged into llama.cpp, per the company. The PR lands alongside llama.cpp's recent run of Apple Silicon performance work.

  OfficialOfficial updateOct 5Source: X @qvacPart of llama.cpp
• llama.cpp v0.6.0 adds GLM-5.3-Flash 320B, Clef decision models and a /v1/systemone API

  llama.cpp released v0.6.0, headlined by support for the GLM-5.3-Flash (GLM5-Next) 320B text+vision hybrid model and the Clef decision model with both text and vision. The release ships a new /v1/systemone server API covering five decision models: laya, julia-1, lev, openjev (with vision) and kev. Qwen4Exp gets high-quality support with MTP speculative decoding, about a 1.5x decode speedup on DGX Spark. Other changes include a new llama_batch_ext API for mixed token/embedding inputs, an overhauled Web UI with a Hugging Face Hub data layer, Metal F16 KV flash attention and ggml v0.26.0.

  OfficialOfficial updateOct 5Source: ggml-orgPart of llama.cpp

RSS for this view →