@pro.glitch: How to build a private RAG on your Mac (Ollama + AnythingLLM, step by step, free models) How to build a private RAG on your Mac: an AI that answers from your own business files, shows you where it found each answer, and keeps every file on your Mac. Two free apps (Ollama and AnythingLLM Desktop), free open models, no Docker, no cloud account. By the end you'll know what every piece of a RAG system does, so when it gets something wrong you know exactly where to look. RAG in plain words: filling the shelf, then asking. A reader pulls the text out of your files, a chopper cuts it into small passages, an embedding model turns each passage into numbers that capture its meaning, and they all sit on a shelf called a vector database, on your disk. When you ask, your question gets turned into numbers by the same embedding model, the shelf hands back the closest passages, a second reader (a reranker) keeps the best few, and the chat model answers only from those and names the file. The build: 1. Check your Mac. Apple menu, About This Mac. You want an Apple M-series chip and at least 16 GB of memory. 8 GB is too tight. Install Ollama from ollama.com. 2. In Ollama's settings, switch Cloud off, so nothing you ask goes to Ollama's servers, and leave "Expose Ollama to the network" off. 3. Download your private models in Terminal. Pick by memory: 16 or 24 GB, ollama pull qwen3.5:9b (about 7 GB). 32 GB, gemma4:26b. 64 GB or more, qwen3.6:35b. Then the embedding model, ollama pull qwen3-embedding:0.6b. All Apache 2.0, free for business use. Run ollama run qwen3.5:9b --verbose to see your real eval rate. 4. Install AnythingLLM Desktop (Apple Silicon). Keep Ollama running, choose Ollama and your chat model, choose Just me, skip the survey. 5. Before you upload a single file: Embedder = Ollama with qwen3-embedding:0.6b (change it later and every document has to be embedded again), Vector Database = LanceDB, Text chunk size 1,500 with 150 overlap, and anonymous telemetry off. 6. Create a workspace, upload your files, Save and Embed. Scanned PDFs are read with OCR right on your Mac, but only when the whole PDF is pictures. 7. The rules: Chat mode Query, your own refusal message, a system prompt that says answer only from the context, name the file, copy prices exactly, never guess. Reasoning Effort Off for speed. Search Preference Accuracy Optimized, Max Context Snippets 6. The three traps: part codes like H-40 and H-50 look almost the same to a meaning search (pin small price lists), totals are a spreadsheet's job, and old versions keep answering until you remove them. 8. Prove it works with 10 questions you already know the answers to. 8 out of 10 right, and the one that isn't in your files has to refuse. Click Sources under every answer to see if the reader, the chopper, the search or the model broke. Last step: ask once online (it downloads the reranker the first time), then turn the Wi-Fi off and ask again. It still answers. And check FileVault is on. This is for small business owners, consultants, trades, agencies and anyone who wants AI on their own documents without uploading them. #aiautomation #aiagents #aiemployee #localai #privateai

Professor Glitch

Open In TikTok:

Region: US

Monday 12 October 2026 17:25:38 GMT