𝗦𝘂𝗻𝗱𝗮𝘆 𝗖𝗼𝗳𝗳𝗲𝗲 & 𝗖𝗼𝗱𝗲 - 𝗗𝗶𝗳𝗳𝘂𝘀𝗶𝗼𝗻𝗚𝗲𝗺𝗺𝗮 𝗼𝗻 𝗮 𝟮𝟰𝗚𝗕 𝗚𝗣𝗨 (𝘆𝘂𝗽, 𝘀𝘁𝗶𝗹𝗹 𝗚𝗲𝗺𝗺𝗮)

Grab a big coffee - last week I said ”𝘪𝘵 𝘸𝘰𝘶𝘭𝘥 𝘣𝘦 𝘪𝘯𝘵𝘦𝘳𝘦𝘴𝘵𝘪𝘯𝘨 𝘵𝘰 𝘵𝘳𝘺 𝘵𝘩𝘦 26𝘉 𝘔𝘰𝘌 𝘮𝘰𝘥𝘦𝘭.” Google 𝗗𝗶𝗳𝗳𝘂𝘀𝗶𝗼𝗻𝗚𝗲𝗺𝗺𝗮 enters the room - a 26B MoE that doesn’t generate text one token at a time, it 𝘥𝘦𝘯𝘰𝘪𝘴𝘦𝘴 blocks of 256 tokens in parallel, like a photo coming into focus. Mission: get a new diffusion LLM running on my single L4 (24GB) via vLLM (𝘮𝘢𝘺𝘣𝘦 𝘵𝘪𝘮𝘦 𝘵𝘰 𝘶𝘱𝘨𝘳𝘢𝘥𝘦 𝘮𝘺 𝘈𝘞𝘚 𝘌𝘊2 𝘴𝘦𝘳𝘷𝘦𝘳).

Small problem: the weights are 49GB. My card is 24GB (Claude to the rescue).

𝗧𝗵𝗲 𝗼𝗯𝘀𝘁𝗮𝗰𝗹𝗲 𝗰𝗼𝘂𝗿𝘀𝗲:

• CUDA mismatch: PyTorch wheels built for CUDA 13, my driver stuck at 12.8. Fixed with vLLM’s cu129 wheel.
• 49GB of weights vs 24GB of VRAM: Red Hat published a 4-bit quant version - 17GB, fits.
• A vLLM bug where an unrelated model’s warmup kernel crashed startup. Patched locally, bug report heading upstream.
• Memory Tetris: 0.90 GPU utilization OOM’d, 0.80 starved the KV cache. 0.86 + 16k context: 𝘈𝘱𝘱𝘭𝘪𝘤𝘢𝘵𝘪𝘰𝘯 𝘴𝘵𝘢𝘳𝘵𝘶𝘱 𝘤𝘰𝘮𝘱𝘭𝘦𝘵𝘦.

𝗧𝗵𝗲 𝗽𝗮𝘆𝗼𝗳𝗳:

• Asked it about the Chamber of Secrets. Coherent, well-structured answer - you can literally read the model’s thinking block planning the outline before it writes.
• The metrics tell the diffusion story: 𝟭𝟯.𝟰𝟳 𝘁𝗼𝗸𝗲𝗻𝘀 𝗰𝗼𝗺𝗺𝗶𝘁𝘁𝗲𝗱 𝗽𝗲𝗿 𝗳𝗼𝗿𝘄𝗮𝗿𝗱 𝗽𝗮𝘀𝘀. An autoregressive model commits exactly 1. It even stopped denoising early when confident.
• ~51 tokens/s on a $0.80/hr GPU - and that’s the floor, not the ceiling.

𝗪𝗵𝗮𝘁 𝗜 𝗹𝗲𝗮𝗿𝗻𝗲𝗱:

• The memory squeeze is three levers: quantization, context window (hello again, num_ctx), and GPU memory utilization. vLLM makes you set all three; Ollama quietly decides for you.
• Model released → community quants → running on commodity hardware happens very quickly. The gap between “announced” and “test it yourself” has collapsed.

Anyone else poking at diffusion LLMs yet,?

A good Sunday, and the coffee went cold around the third OOM.

• Model: https://huggingface.co/google/diffusiongemma-26B-A4B-it
• The quant that made it fit: https://huggingface.co/RedHatAI/diffusiongemma-26B-A4B-it-NVFP4