Source APIs
market + chain
- feeds
- Kafkaingest
All case studies
Data PipelinesMetricsBI3 months
USDDub
Dashboards were six hours behind reality. Now they are five minutes behind.
Data freshness
5 minutes
Streaming with replayable stages
Was
6 hours
Scheduled batch extracts
In plain terms
What this actually was.
No jargon in this section. The technical write-up is further down.
What was going wrong
Data arrived faster than the overnight jobs could process it, so anyone opening a dashboard was looking at this morning at best. When something needed reprocessing, an engineer did it by hand — often at the weekend.
What we built
We replaced the scheduled overnight jobs with a continuous flow that can be replayed from any point, and put every number behind one shared definition.
What changed
Decisions are made on data minutes old rather than hours old, reprocessing no longer costs anyone a weekend, and every team is reading the same figures.
Architecture
How it fits together.
Scroll through the stages. Anything marked as added is a component that did not exist before this project.
Kafka
partitioned
- receives from
- Source APIsingest
- feeds
- ClickHouse
ClickHouse
columnar
- receives from
- Kafka
- feeds
- dbt modelstransform
dbt models
incremental
- receives from
- ClickHousetransform
- feeds
- BI layer
BI layer
governed
- receives from
- dbt models
Source APIs
market + chain
- feeds
- Kafkaingest
Kafka
partitioned
We added this- receives from
- Source APIsingest
- feeds
- ClickHouse
ClickHouse
columnar
We added this- receives from
- Kafka
- feeds
- dbt modelstransform
dbt models
incremental
We added this- receives from
- ClickHousetransform
- feeds
- BI layer
BI layer
governed
- receives from
- dbt models
The write-up
How it was actually built.
Dataset, approach, measured results and the stack. Written for whoever has to review it.
Data PipelinesMetricsBI3 months
USDDub
Canonical market data and signals that stay consistent across sources and time.
Problem
Market and event data arrived faster than the existing batch jobs could process it, so dashboards were hours stale and backfills required manual reruns.
Dataset
- rows
- 14M events / day
- features
- 203
- sources
- Market APIs, on-chain, internal services
- window
- Rolling 90 days
Approach
- 01Replaced batch extracts with a streaming ingestion layer
- 02Modelled a columnar warehouse tuned for the actual query patterns
- 03Made every stage idempotent so backfills are a replay, not a rescue
- 04Published a governed metric set behind a BI layer
Metrics
- 14M+
- events / day sustained
- 5 min
- data freshness from 6 hours
- 410ms
- query p95 -8.4s
- automated
- backfill effort from manual
Business impact
- Decisions made on data minutes old instead of hours old
- Backfills stopped consuming engineering weekends
- One metric layer shared by every internal consumer
Tech stack
- Python
- TimescaleDB
- Kafka
- Prefect
- Grafana
- dbt
- Web3.py
Next
Need a system like this?
Answer six questions and we will tell you whether this shape fits your problem — including when it does not.