@diy_smart_code: Your local LLM fits in RAM — the KV cache doesn't. The follow-up to my Ornith 1.0 video: why the 5.6 GB model needs about 10 GB in a real ollama agent session, and the two-variable q8_0 fix that halves the cache. ---- 🚀 DYNAMOUS AI COMMUNITY Want to learn agentic coding with live daily events and workshops? Check out Dynamous AI: https://dynamous.ai/?code=646a60 Get 10% off here 👉 https://shorturl.smartcode.diy/dynamous_ai_10_percent_discount ⚡ HOSTINGER — RELIABLE HOSTING FOR YOUR PROJECTS (10% OFF) Whether you're shipping a portfolio, a side project, n8n flows, or AI agents — I use Hostinger for fast, affordable VPS + web hosting. Get 10% off here 👉 https://hostinger.com/DIYSMARTCODE (Affiliate link — costs you nothing, supports the channel.) ---- What you'll see in this 110-second breakdown: → The comment that called it: the model fits, the context for your session likely won't → The memory bill: 5.6 GB model file vs about 10 GB for a real 128k agent session → The KV cache math, derived from config.json: ~32 KiB per token → 4 GB at 128k tokens → The honest part: only 8 of 32 layers grow the cache — a dense model would need 16 GB → The trapdoor: ollama's default context window is 4,096 tokens; the model supports 262,144 → The fix: OLLAMA_FLASH_ATTENTION=1 + OLLAMA_KV_CACHE_TYPE=q8_0 — 4 GB of cache becomes 2 The receipts: → Model weights: https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B → Run it: https://ollama.com/library/ornith → Ollama FAQ (default context + q8_0): https://docs.ollama.com/faq → Tech blog: https://deep-reinforce.com/ornith_1_0.html → Part 1 — Ornith 1.0 breakdown: https://youtube.com/shorts/8Cw1jX6o9w0 Quantize the cache, or cap the context — which one are you running? One word below. #ornith #localllm #ollama #kvcache #kvcacheoptimization #localai #localaicoding #localaiagent #runllmlocally #contextwindow #quantization #flashattention #deepreinforce #opensourceai #agenticcoding #openweights #aicoding #selfhostedai #ollamatutorial #diysmartcode

DIY Smart Code

Open In TikTok:

Region: DE

Friday 03 July 2026 18:59:50 GMT