Why We Built This — Specs, Scale and the Stakes
1.6 million HTML files. Sixty years of Indian language newspaper stories.
That was the archive we started with—decades of Indian language journalism stored as raw HTML. It was searchable by keywords, but not by questions. We wanted readers to ask things like: “How has coverage of agricultural policy changed over the last three decades?”
Answering that requires semantic search, not keyword matching. So we built a retrieval-augmented generation (RAG) system that answers questions directly from the archive. The system works across multiple Indian languages and grounds every response strictly in the source data—hallucinations were significantly reduced through grounded retrieval + reranking + filtering.
The constraints were real. The archive spans decades of Indian language reporting with variations in language and structure. At this scale, naïve indexing becomes expensive in both storage and query time. We needed high recall, controlled latency, and an indexing approach that could scale with the corpus.
And the numbers only grow from there: 1.6M HTML files → ~5M vectors after chunking and embedding.
Turning that scale of raw archive data into a system that can return one accurate, grounded answer is the story we want to tell.
When We Stopped to Understand the Data—Preprocessing, Chunking Strategy, and What We Chose to Embed
Before we touched an embedding model or a vector store, we had to understand the data—not in a vague way, but concretely: what we were cutting, what we were embedding, and what we were keeping as metadata. We started with a sample of around 300 articles and a clear set of requirements, plus open questions about language, model size, and indexing. Every choice we made in this phase became the foundation everything else sat on.
From raw HTML to structured content
The archive is HTML: 60 years of Indian language newspaper text with all the usual markup inconsistencies and encoding variations. We built an extraction pipeline that converts this into structured JSON.
During extraction, we parse out both the main text and associated metadata, then clean them—removing unnecessary special characters, normalising whitespace, and stripping residual HTML artifacts—to ensure consistency and reliability downstream.
For each field, we then decide: embedding content (the text we vectorise) or metadata (dates, sections, bylines). That split drives everything downstream.
Chunking
Chunk size directly affects retrieval. Too large → “lost in the middle”; the model underweights the middle of long passages and you get a poor query match. Too small → fragments with no context. We cared about where to split first: sentence and paragraph boundaries, then section breaks. Character count was a constraint, not the driver.
We use RecursiveCharacterTextSplitter: 512 characters, 50-character overlap. It respects natural breaks (sentences, paragraphs) and uses character count as a fallback. Chunks are coherent story segments, not arbitrary slices.
What to embed
We A/B’d it. Chunk text only: 74% accuracy. Chunk text + article title: 77.5%. The title added the contextual signal we needed. Adding the dateline added noise; we left it out. We ship chunk text + title. Clean and meaningful beats dumping everything in—get chunking and embedding content wrong, and indexing can’t fix it.
The First System That Worked—Until Scale Hit: Embedding Model Selection, Vector Store Choices and Why Recall Dropped
With preprocessing and chunking in place, we had to choose an embedding model and a vector store. On a small sample, everything worked. Then we scaled—and the same setup that had given us strong results started to fail. This is the part of the story where we had to dig into why.
Embedding model
We prioritized models that met multilingual quality + operational constraints (CPU throughput, memory, dependency stability). We tried IndicBERT, MuRIL, Cohere, BGE-M3, XLM-Roberta, OpenAI text-embedding-3-large and Paraphrase-multilingual-MiniLM-L12-v2.
Finally LaBSE (Language-agnostic BERT Sentence Embedding) was the fit: 768 dimensions, strong Indian language semantic search, good recall across Indian languages, and the full workload runs on CPU. We later moved to Apple Metal/GPU for speed; LaBSE held up.
Vector store
We evaluated FAISS, Chroma, Milvus, Weaviate. Chroma, Milvus, Weaviate had managed options but didn’t match our data shape, growth, or the control we wanted. We chose FAISS: we keep vectors in RAM and own the store and the algorithm. That overhead became an advantage when we had to change how we search.
First system: flat index + reranker
Stack: LaBSE for embeddings, flat index (IndexFlatL2), Euclidean search, vectors in RAM.
We put in place an interface and API for query handling and backend retrieval. When a request comes in, it first goes through query decomposition: an LLM analyses the user’s question and extracts structured intent – a distilled query context (core semantic content, stripped of filler), plus parameters such as timeline and place. We send this distilled context to vector search, because it retrieves better than the raw query string.
After retrieval, we added two components:
- a deduplication and consolidation step to remove redundant chunks from duplicate articles and merge coherent chunks from the same article (using SHA-256 hashing plus fuzzy dissimilarity checks), and
- Cohere’s reranker to re-score retrieved chunks against the query and pass only top-ranked documents to the LLM.
Result: fewer hallucinations and less junk in the final answer.
We indexed about a year of data. Index size ~1.5 GB. Then the recall collapsed.
What broke
Queries that had worked stopped returning the right docs. Relevance didn’t taper—it collapsed. ~85% relevant score dropped to ~10%. The right docs were in the store (we verified); they just didn’t show up in retrievals, even at k=100, 200, 300. The bug was in how we were searching.
FAISS similarity scores were nearly identical across large portions of the index—e.g. 0.543267997 vs 0.54326998. With that much tie-breaking, ranking was effectively random. Retrieval latency also blew up: brute-force over the full flat index at that scale wasn’t sustainable.
The cause wasn’t the model or chunking. It was dilution. More data diluted relevance; too many vectors sat too close in score, and flat search couldn’t surface the right ones. We had to change how the vector store worked. That’s when we went deeper on FAISS and switched the algorithm.
How Clusters Saved Our Recall and Why Reachability Beat Brute-Force Search
We moved off LangChain’s embedding helpers and used the FAISS library directly. We tried IndexFlatIP (cosine similarity with normalization—inner product) to see if the distance metric was the issue. It wasn’t. The fix was IVF—Inverted File index. IVF sped things up and brought recall back. The key idea: reachability as much as similarity.
IVF: Go to the right section first
Like a library: you go to the section that matches your question (History, Sports, Culture), then search within that section. IVF does the same for vectors.
It groups vectors into clusters. Each cluster has a centroid (a representative point) and a list of vector IDs that belong to it. During the search, we only look at the clusters most likely to contain our answer.
Training & Best Practices
FAISS learns centroids with a coarse quantizer; in practice that’s k-means clustering. That learning step is training. After training, we map each vector to its nearest centroid. Vectors are still stored in full (flat, uncompressed); we only change how we organize and search them.
Choosing nlist (Number of Clusters)
The nlist parameter is fixed at build time. Too few clusters means most vectors get crowded into a few huge buckets (barely better than flat search). Too many clusters can spread docs too thin, and we might miss the right ones unless we probe more. We tuned it to avoid both extremes: not overcrowded, not too sparse.
Here are some best practices:
- Standard formula: A common starting point is nlist = √N
- Best practice for $N < 3M$ vectors: Use nlist = √N + (100 – 200) as a small buffer.
- Best practice for large scales: Depending on your hardware and requirements, nlist can be scaled up to √N*4, * 8, or even *16.
The Training Set
To create accurate clusters, FAISS needs a representative sample of your data.
- The 40x rule: For effective training, your sample set should be at least 40*nlist.
- Small datasets: If your total dataset (N) is less than 1 million vectors, using the entire dataset for training is usually the best approach.
- Large datasets: For massive scales, the sample set should remain proportional, ranging from 40*nlist to 256*nlist.
Search: Centroids first, then the right clusters
At query time, we compare the query to centroids first, then choose how many clusters to search using nprobe. That creates a speed/recall trade-off. After that, we search only within those selected clusters.
We still do cosine similarity on the vectors in those clusters, but on a small subset of the index. Even with nprobe equal to nlist (search every cluster), we do far less work than brute force because the search space is cut by structure. In our setup the time difference between nprobe and nlist was tiny; the real lever was how many clusters we had.
Result: recall back to ~85%
Recall came back to around 85%. The documents that had been “left behind” became reachable again—not because we changed embeddings or chunking, but because we changed how we searched.
The algorithm wasn’t just about similarity; it was about reachability. With a flat index, the right docs were lost in a sea of near-identical scores. With IVF we narrow to the right neighbourhood (the clusters), then do fine-grained search there. Same vectors, different structure—a system that could scale with the data.
What’s Next
What you’ve read so far sets the foundation. In the upcoming parts, we’ll dive into how we processed decades of archive data with minimal supervision, reduced the index size by ~96% without losing recall, and moved metadata and lookups to disk—significantly lowering memory usage and overall infrastructure cost.
We’ll also cover how we evolved the system into a product, enabling users to search, expand, and generate from the archive. Building on the same retrieval backbone, we’ll go deeper into scale, compression, and production readiness.
End of Part 1
Uthra Karunakaran
Full-Stack Development, Automation, Image & File Processing, System Design & Architecture, RAG Systems, Cloud Deployments
