@bashifuirkashi: When an interviewer says “we have millions of customer support tickets…” they’re not testing AI. they’re testing systems thinking. Here’s what that actually looks like 👇 I’d break it into 5 parts: ingestion, embeddings, retrieval, generation, monitoring 1️⃣ Ingestion is offline, not request-time At this scale, everything is async • Stream tickets from DB / event logs • Normalize messy text (typos, threads, duplicates) • Thread conversations properly (not just single messages) • Chunk by conversation flow, not fixed tokens • Attach metadata (user, product, issue type, timestamp) This NEVER touches your request path 2️⃣ Embeddings + indexing built for scale You don’t embed on demand • Batch jobs for new tickets (queue-based) • Distributed vector DB (Qdrant, Milvus, etc.) • Hybrid search (BM25 + dense vectors) • Sharding by product / issue type • Store metadata alongside vectors Key insight: metadata filtering > vector search 3️⃣ Retrieval + generation (tight path) Keep this path fast and lean: query → metadata filter → cache → hybrid search → rerank → LLM • Most queries hit cache • Many never reach vector search • Rerank small candidate sets only • Send 5–10 HIGH quality chunks More data ≠ better answers Relevance wins every time 4️⃣ Caching is everything This is where scale actually happens • FAQ cache (repeat questions) • Retrieval cache (top ticket clusters) • Embedding cache (hot queries) Real flow: query → cache → (miss) retrieve + LLM → store result This is how you cut cost AND latency 5️⃣ Monitoring closes the loop Without this, your system slowly dies • Are we retrieving the right tickets? (recall@k) • Are answers actually helpful? (feedback) • Latency + cache hit rate • Auto re-embed as new tickets come in BOTTOM LINE: RAG at this scale is NOT an AI problem it’s a search + data + caching problem Most engineers build chatbots Real AI engineers build infrastructure #aiengineer #ai #embedding #rag #softwareengineer

Bashi | Software Engineer

Open In TikTok:

Region: US

Thursday 16 April 2026 19:00:00 GMT