Client SDKs
4 platforms
- feeds
- Kafkavalidate
All case studies
Product AnalyticsExperimentationPersonalization6 months
Masters
Four analytics tools disagreed about the same numbers. Now there is one set everyone trusts.
Offer CTR
6.4%
Ranked offers with experiment controls
Was
4.5%
Static rules, same offer for everyone
In plain terms
What this actually was.
No jargon in this section. The technical write-up is further down.
What was going wrong
The company ran on four different platforms that each counted users differently, so two dashboards could disagree by a fifth and nobody could say which one was right. Product decisions were being argued rather than measured.
What we built
We agreed a single definition for each important number with product, growth and finance together, rebuilt the reporting on top of those definitions, and added the controls needed to test changes properly.
What changed
One shared set of numbers, nine tests running at once instead of two, and arguments about attribution off the meeting agenda. The share of people who click an offer went from 4.5% to 6.4%.
Architecture
How it fits together.
Scroll through the stages. Anything marked as added is a component that did not exist before this project.
Kafka
contracts
- receives from
- Client SDKsvalidate
- feeds
- ClickHouse
ClickHouse
raw + marts
- receives from
- Kafka
- feeds
- dbt metrics
dbt metrics
governed
- receives from
- ClickHouse
- feeds
- LightGBMfeatures
LightGBM
offer ranking
- receives from
- dbt metricsfeatures
- feeds
- Experimentsserve
Experiments
lift + bias
- receives from
- LightGBMserve
Client SDKs
4 platforms
- feeds
- Kafkavalidate
Kafka
contracts
- receives from
- Client SDKsvalidate
- feeds
- ClickHouse
ClickHouse
raw + marts
- receives from
- Kafka
- feeds
- dbt metrics
dbt metrics
governed
We added this- receives from
- ClickHouse
- feeds
- LightGBMfeatures
LightGBM
offer ranking
We added this- receives from
- dbt metricsfeatures
- feeds
- Experimentsserve
Experiments
lift + bias
We added this- receives from
- LightGBMserve
The write-up
How it was actually built.
Dataset, approach, measured results and the stack. Written for whoever has to review it.
Product AnalyticsExperimentationPersonalization6 months
Masters
Unified analytics and personalization so teams run trusted experiments and move faster.
Problem
Four platforms emitted incompatible event schemas, so no one could agree on what a retained user was. Product decisions were being made on dashboards that disagreed with each other by up to 20%.
Dataset
- rows
- 412M events
- features
- 86
- sources
- iOS, Android, Web, CRM
- window
- 24 months
Approach
- 01Defined a single event taxonomy with tracking contracts enforced in CI
- 02Streamed all sources into one warehouse with identity stitching
- 03Built cohort and retention views on a governed metrics layer
- 04Deployed an offer ranking model behind an experiment framework
Metrics
- 0.87
- ranking AUC +0.11
- 6.4%
- offer CTR +1.9pp
- <1%
- metric drift across teams
- 340ms
- query p95 -2.1s
Business impact
- One shared KPI definition across product, growth and finance
- Experiment velocity up from 2 to 9 concurrent tests
- Attribution disputes eliminated as a recurring meeting topic
Tech stack
- Python
- ClickHouse
- Apache Kafka
- dbt
- Airflow
- FastAPI
- LightGBM
- Metabase
Next
Need a system like this?
Answer six questions and we will tell you whether this shape fits your problem — including when it does not.