Learning Insights Pipeline
Realtime learning analytics for education
Backend Developer2026 — Present3 engineers
0M+
events ingested / day
<0s
dashboard lag
0k
students served
OVERVIEW
A realtime analytics pipeline that ingests learner behavior events — lesson views, quiz attempts, agent conversations — and turns them into live dashboards for teachers and curriculum teams: funnels, retention, struggle detection.
CHALLENGES
- Ingesting spiky event traffic (evening study peaks) without dropping data.
- Dashboards had to reflect activity within seconds, not nightly batches.
- Aggregations across 20k students strained naive queries.
SOLUTIONS
- Buffered ingestion through RabbitMQ with idempotent consumers and dead-letter recovery.
- Maintained incremental aggregates in PostgreSQL updated per event batch, with Redis serving hot dashboard reads.
- Partitioned event tables by week and pre-computed funnel/retention rollups.
RESULTS
- Sustained 1M+ events/day with zero-loss ingestion during peak spikes.
- Dashboard freshness under 2 seconds end-to-end.
- Struggle-detection alerts adopted by teachers across 20k students.
TECH STACK
- Node.js
- RabbitMQ
- PostgreSQL
- Redis
- Docker
- AWS
ARCHITECTURE
PROCESS
- 011 week
Event schema design
Defined event taxonomy and aggregate models.
- 023 months
Pipeline build
Ingestion, consumers, rollups, dashboard APIs.
- 03Continuous
Scale tuning
Partitioning, batching, cache strategy.