@bashifuirkashi: When someone says “RAG over millions of PDFs” they’re not asking about AI… they’re asking about search + systems. Here’s what that actually looks like 👇 I’d break it into 5 parts: ingestion, embeddings, retrieval, generation, monitoring 1️⃣ Ingestion is offline, not request-time At scale, this must be async • Stream documents from storage (S3/GCS) • OCR only when needed • Clean + normalize text • Chunk intelligently (not randomly) • Attach rich metadata (doc, page, section, etc.) None of this should ever touch your user request path 2️⃣ Embeddings + indexing built for scale You don’t embed on demand • Batch embedding jobs (GPU or queued) • Distributed ANN indexes (Milvus, Qdrant, Vespa, Elastic) • Sharding + HNSW / IVF / PQ • Store metadata alongside vectors Key insight: metadata filtering is your first gate vector search is the fallback 3️⃣ Retrieval + generation (tight path) Your request path should stay minimal: query → metadata filter → cache → ANN search → rerank → LLM • Most queries never even hit vector search • Many don’t hit the DB at all (cache wins) • Rerank a small set only • Send 5–10 chunks max to the LLM More context ≠ better results More chunks usually hurt 4️⃣ Caching is everything This is what controls cost + latency • Query → answer cache (FAQs, repeats) • Query → retrieval cache (top chunks) • Data/index cache (hot vectors, parsed docs) Real path looks like: query → cache → (miss) retrieve + LLM → write back 5️⃣ Monitoring closes the loop Without this, your system silently degrades • Retrieval quality (recall@k) • Answer quality (feedback loops) • Latency + cache hit rates • Re-embed + re-shard as data evolves BOTTOM LINE: RAG at scale is NOT an LLM problem It’s a search + caching architecture problem Most people are building demos Real AI engineers are building systems Link in bio for the full breakdown + a group with live expert-led calls and systems for you to get hired ASAP

nadav.ai :

RAG is just not the right solution.

eruvierda :

I in need of ocr with a certain form, for identity doc. I try tesseract and it can't read the sequence properly, any idea?

sallymusic :

in theory everyone will sound smart in practice these are nasty problems only companies like google get to solve

thedataguy :

Currently working on a project.. There are somethings i didn't put in place due to microsoft azure foundry.. There is document intelligence, ai search service nd many more. But for non azure users nd azure users the first 3 is essential

Cheno :

If the structure of the of the pdf matters, then I will go with recursive chunking strategy with optional overlap , semantically embedding the chunks whiles employing indexing by pdf name, perform both sparse and dense retrieval then finally applying reranking. Optional but necessary I will use Qdrant graph to visualise the embedding point to see how different strategy affect the embeddings.

Dwigth Astacio :

Chunking strategies, genérate embedding of those chunks, store them in a vector database, process user query -> hybrid semantic searchwith lexical search and promting ?

illyabvd001 :

Does Microsoft Foundry do these capabilities as a Knowledge Source, in the b.End?

albertmagarire :

Karpathy’s LLM Wiki idea..

Kabo Kable Molefe :

m :

if anyone is looking for more advanced solution there is a thing called PageIndex its a miracle for complex documents but costs more in tokens

rat :

prefect & grafana+prometheus

Critique :

need to copy the caption

Cisco23 :

Where do I go to learn all this?😭😭

gotthatsnipe :

hold up let me ask claude for step by step instructions and ima tell claude to not make mistakes and double check thework

Yassin :

I use pageindex with a metadata db to do this for a chatbot I built. it asks the user questions to narrow down the meta data then uses the llm the search thru the nodes before it selects the relevant source and ingests that to provide the answer.

baldezo313🇫🇷🇸🇳🇬🇳 :

Metadata filter >>>>>>>>>

CaptEmma Dev :

For me convert all to markdown files using docling then get ur Ai read the md files

Nick :

Didn’t read the whole thing but I would probably go about it differently if it’s my choice. I would spend a lot of time upfront having Claude use the markitdown skill to turn the pdfs into markdown files. Once that is done, build indexes (wil be so much faster and use less tokens) and then set up the rag system however works best for the task.

JSiron :

Add offline evaluation to keep track of retrieval quality

Alan :

Am building simple use case using Onedrive + AI build GPT, however the user interacts prompts are exchanged via email with the agents.

Tommy Matin :

Are you think this is big data?

Vũ bình dân :

Why cache? how can you implement that? too many different queries and they will all hit the vector db

jambutterpeanut :

if you have millions of pdfs, then u r the problem, not the system.