Ingestion
streaming
- feeds
- Encodertext
All case studies
NLPVector SearchProduction3 months
Semantic Deduplication at Ingestion
The same story kept arriving five times over. Now it arrives once.
Duplicate pair F1
0.94
Embedding ANN with calibrated rerank
Was
0.65
Exact hash + title match
In plain terms
What this actually was.
No jargon in this section. The technical write-up is further down.
What was going wrong
Syndicated articles came in as near-copies of each other — reworded, retitled, but the same story. Simple duplicate checks missed them entirely, and comparing every article against every other one was far too slow at the volume arriving.
What we built
We built a check that compares what two articles mean rather than how they are worded, fast enough to run on every item as it arrives instead of afterwards.
What changed
Complaints about a repetitive feed fell to almost nothing, storage and processing costs came down by about a fifth, and the editorial team stopped removing duplicates by hand.
Architecture
How it fits together.
Scroll through the stages. Anything marked as added is a component that did not exist before this project.
Encoder
sbert
- receives from
- Ingestiontext
- feeds
- ANN indexvector
ANN index
hnsw
- receives from
- Encodervector
- feeds
- Reranktop-k
Rerank
cross-encoder
- receives from
- ANN indextop-k
- feeds
- Decision
Decision
calibrated
- receives from
- Rerank
Ingestion
streaming
- feeds
- Encodertext
Encoder
sbert
We added this- receives from
- Ingestiontext
- feeds
- ANN indexvector
ANN index
hnsw
We added this- receives from
- Encodervector
- feeds
- Reranktop-k
Rerank
cross-encoder
We added this- receives from
- ANN indextop-k
- feeds
- Decision
Decision
calibrated
- receives from
- Rerank
The write-up
How it was actually built.
Dataset, approach, measured results and the stack. Written for whoever has to review it.
NLPVector SearchProduction3 months
Semantic Deduplication at Ingestion
Hybrid retrieval and semantic scoring to remove duplicates before downstream pipelines.
Problem
Near-duplicate articles from syndication flooded the feed. Exact hashing missed rewrites, and naive pairwise comparison at ingestion volume was computationally impossible.
Dataset
- rows
- 94M documents
- features
- 768-dim embeddings
- sources
- RSS, partner APIs, crawlers
- window
- 12 months
Approach
- 01Encoded documents with a sentence transformer tuned on labelled pairs
- 02Indexed embeddings in an ANN structure for sublinear candidate lookup
- 03Applied a calibrated similarity threshold plus a lightweight rerank
- 04Streamed decisions back to ingestion under a strict latency budget
Metrics
- 0.94
- pair F1 +0.29
- 23%
- duplicates removed of intake
- 38ms
- decision p95 at ingest
- 0.98
- index recall@10 vs exact
Business impact
- Feed quality complaints dropped to near zero
- Storage and downstream processing costs cut by a fifth
- Editorial team stopped manually de-duplicating
Tech stack
- Python
- Sentence-Transformers
- Apache Kafka
- Milvus
- Redis
- FastAPI
- Kubernetes
Next
Need a system like this?
Answer six questions and we will tell you whether this shape fits your problem — including when it does not.