Filters
- Ai2 replaces its priority-based GPU scheduler with time budgets and fair-share allocation
Ai2's AI infrastructure team says it replaced a priority-based scheduler for its clusters of thousands of NVIDIA H100, B200 and B300 GPUs with a system of GPU time budgets, hierarchical fair-share allocation and a time-slicing contract. With requests for two to three times more GPUs than it has, Ai2 says the change turned decisions about how much compute each research project gets into a transparent budgeting process while keeping clusters fully occupied.
- LocalLLaMA shares Vulkan llama.cpp GPU benchmarks sorted by price/performance
A r/LocalLLaMA post compiles mid-range GPU results from the llama.cpp Vulkan performance discussion (#10879) into a price-to-performance table aimed at budget data-center homelabs. The author says the source thread lists 76 GPU models and that the table focuses on value-oriented, mid-level, decent-speed cards.
- Builder runs Qwen3.8-Flash-Next at 60–100 t/s decode on eight Radeon Pro V620s with a custom vLLM fork
A post on r/LocalLLaMA describes a roughly $2,800-class rig with eight Radeon Pro V620 cards (256 GB total VRAM) and a custom vLLM fork reaching about 60–100 tokens/s decode and 3,000+ tokens/s prefill on Qwen3.8-Flash-Next. The cards are older RDNA2 enterprise GPUs bought cheaply as a gamble.
- TechCrunch Disrupt 2026 to feature Ambrosia Energy and Bloom Energy on AI infrastructure opportunity, per TechCrunch
TechCrunch previews a Smart Systems Stage session at Disrupt 2026 with Ambrosia Energy CEO Ben Longmier and Bloom Energy SVP Bill Thayer on where the AI infrastructure boom is creating opportunity.
- User clusters six mismatched devices to run a 120B model none could load alone
A post on r/LocalLLaMA shows a cluster of six machines — a 12 GB Windows laptop, an RTX 3060 mini PC, a 16 GB Mac mini, an M3 MacBook, a 2017 Intel MacBook CPU-only, and an Android phone — jointly loading a 120B model that none of them could run alone.
- LoRA-over-GGUF recipe trains Qwen3.8-Flash-Next in 40 GiB VRAM without CPU offload
An update to the transformers5-qwen3.5-recipe reports LoRA training over GGUF for Qwen3.8-Flash-Next (125B-A6B plus 51B engram) in 40 GiB VRAM with engram on disk and no CPU offloading. On Strix Halo it cites context chunk size 2048 at about 9.5 s/it (~200 token/s).
- RTX 5070 Ti user reports a multi-hour CUDA job ending in an NVRM GPU lock watchdog error
A post on r/LocalLLaMA from a user with prior stable 3090/4090/2080 Ti workloads says a new RTX 5070 Ti died near the end of a six-hour CUDA job with an NVRM RC watchdog "GPU is probably locked" journal message, asking whether Blackwell cards have PCIe 5 issues.
- audio.cpp update cuts Higgs Audio TTS peak VRAM by about 48% and speeds HTDemucs and PocketTTS
A post on r/LocalLLaMA highlights recent audio.cpp performance work: Higgs Audio TTS now peaks around 6 GB VRAM (about 48% less than before), HTDemucs runs about 2.2× faster on GPU, PocketTTS about 2.2× faster on CPU, plus a WebUI generation-history feature.
- TornadoVM and jitLLM claim about 90% of llama.cpp local NVIDIA performance by compiling Java to CUDA
A post on r/LocalLLaMA points to TornadoVM (Java-to-CUDA) and jitLLM, a Java vLLM-like inference engine, claiming about 90% of llama.cpp performance on local NVIDIA GPUs via CUDA and cuTile, with a linked deep-dive talk.
- Microsoft adds experimental llama.cpp support to Windows ML so apps can run GGUF models locally through new task-specific APIs
Microsoft says Windows ML now has an experimental llama.cpp integration that runs GGUF models from Hugging Face through a new Text Generation API, which also accepts ONNX models and exposes an OpenAI-compatible local endpoint, alongside a Speech Recognition API for ONNX Whisper models. A new experimental Windows-native Windows ML Runtime API is in preview for zero-copy data paths and multi-model pipelines across CPU, GPU and NPU. Microsoft also says it contributed CUDA and speculative decoding work to llama.cpp with NVIDIA, and that PyTorch now ships native Windows Arm64 CPU builds, with Triton available on Windows.
OfficialSource: Microsoft Foundry on Windows Blog - r/LocalLLaMA thread looks for a name for hardware-specific inference engines like Strata and ninfer
A post on r/LocalLLaMA asks which acronym fits engines such as Strata and ninfer, proposing HOMIE (Hardware-Optimized Model Inference Engine) or MADE (Model-And-hardware Dedicated Engine). The poster notes these engines run only on specific hardware and support one or a few models, trading generality for performance that beats general engines like llama.cpp and vLLM on that hardware.
- r/LocalLLaMA shares report that Micron's NVHBM moves the memory controller into the base die
A post on r/LocalLLaMA shares a report that Micron expects its NVHBM memory to improve profitability even with an outsourced base die. Per the quoted text, NVHBM moves the memory controller from the main compute die into the base die, cutting power use by 15%, raising memory bandwidth by up to 30% and shrinking the I/O PHY package area by up to 67%.
- Ollama 0.40.1 proxies cloud usage and balance APIs and fixes Windows reads past 2GiB
Ollama's v0.40.1 release lets the server proxy cloud usage and balance APIs, fixes reads past 2GiB on Windows, removes the account step from CLI onboarding and avoids symlinks in manifests on Windows. It also drops a Metal residency patch for MLX now that the fix is upstream, per the GitHub release notes.
- Reddit user's llama.cpp fork runs Qwen3.8-Flash-Next on six 3090s with 8-10x faster prefill
A post on r/LocalLLaMA shares a multi-GPU llama.cpp fork, with binaries and Docker images, built to run Qwen3.8-Flash-Next on six RTX 3090s over plain PCIe without NVLink, serving five 262k-context sessions. The author says stock llama.cpp slowed as context grew, falling to 2.3 tokens/s per session at 5x250k, and reports 8-10x faster prefill and 2-3x faster long-context decode after the fixes.
- Google launches its first Project Suncatcher test satellite with four TPUs in orbit
Google said Project Suncatcher, its moonshot exploring whether machine learning infrastructure could one day be hosted in space, launched its first test satellite last week carrying four TPUs. The aim is to learn whether its AI hardware can handle the physical stress of spaceflight and the radiation and thermal extremes of orbit, questions Google researchers have studied for the past few years.
- cmpunlocker v0.5 adds ECC support and unlocks 4 more SMs on the CMP 170HX, per a Reddit post
A post on r/LocalLLaMA says cmpunlocker v0.5 brings ECC support and four extra SMs to the CMP 170HX for free, and pitches the card's 1.5 TB/s of memory bandwidth, against 273 GB/s on a DGX Spark, at under half the price of the new 64GB DGX Spark. The author says a gen3 unlock is also on the way.
- Reddit user asks how to put a spare RTX 2060 to work inside a Framework Desktop Strix Halo build
A post on r/LocalLLaMA describes getting a spare RTX 2060 running with a Framework Desktop (Strix Halo) motherboard in a Phanteks Enthoo Pro case through two PCIe risers and a vertical GPU bracket, and asks what the card is useful for in a local AI setup.
- Reddit user with dual Radeon 7900 XTX cards weighs a PCIe 4.0 or 5.0 motherboard upgrade for tensor-parallel inference
A post on r/LocalLLaMA describes running a 27B model on two 7900 XTX cards on a Z390 board at PCIe 3.0 x8, and asks whether moving to an X570 (PCIe 4.0 x8) or X870 (PCIe 5.0 x8) board would speed up tensor parallelism while waiting for 256GB LPDDR6 machines.
- Reddit post weighs one H200 against several RTX PRO 6000 Blackwell cards for an inference server
A post on r/LocalLLaMA compares spending an inference budget on a single H200 versus multiple RTX PRO 6000 cards, noting the PRO 6000 has 96GB of GDDR7 at 1.792 TB/s (1.6 TB/s on the Server Edition) and native FP4 but no NVLink, and lays out which it would pick at each budget.
- Reddit thread argues the 2019 Mac Pro's 1.5TB RAM design would suit local LLMs today
A post on r/LocalLLaMA notes the 2019 Mac Pro supported up to 1.5TB of system RAM and four passively cooled GPUs with 128GB of VRAM, while its successor tops out at 192GB of shared memory, and argues an updated version would be the ideal local LLM machine.