●[PROJECT NAME: Real-Time Fleet Telemetry & Anomaly Analytics]
Architecting a fault-tolerant stream processing pipeline ingesting 18,000 sensor events/sec with real-time drift detection.
Thousands of vehicle sensors were sending temperature, vibration, and GPS readings every second. We built an analytics engine that compresses this torrent of data, flags abnormal mechanical vibrations in milliseconds, and lets dispatchers view fleet health without waiting for slow database queries.
The telemetry team was storing sensor events in an unpartitioned PostgreSQL instance.
As connected test vehicles scaled up, table locks began causing write timeouts. Historic queries over weeks of vibration data took up to 35 seconds to render, blocking operations staff from reacting to sensor alarms in real time.
Furthermore, transient network drops in rural test corridors led to duplicated sensor batches upon reconnection, skewing anomaly calculations.
“Sensors don't wait for your database to catch its breath. A telemetry architecture must ingest writes under pressure without complaining.”
Sustained throughput under simulated fleet peak
95th percentile aggregate query over 30 days of sensor points
Across the entire 90-day testbed window
Via columnar delta compression in ClickHouse
Architectural choices and what was explicitly rejected.
Partitioned ClickHouse Columnar Store for Time-Series Analytics
Traditional row-oriented databases choke on aggregate time-series math. By streaming pre-validated Kafka records into ClickHouse with delta-of-delta encoding, query times plummeted from 35s to under 50ms while cutting disk usage in half.
Rejected scaling up PostgreSQL memory and compute (prohibitively expensive and still fundamentally row-oriented).
Idempotency Keys with Redis Deduplication Buffer
To eliminate duplicate bursts when vehicles regain cellular signals, each sensor frame is stamped with device-generated monotonic UUIDs checked against a 15-minute sliding Redis bloom filter.
Rejected relying on client-side retry suppression (vehicle firmware was fixed).
Vehicle telemetry is received via MQTT broker, normalized, queued in Kafka, and fanned out to real-time alerting workers and columnar ClickHouse cold storage.
Edge Ingestion & Deduplication
Incoming telemetry streams through an edge gateway that drops duplicates using sliding bloom filters.
Kafka Event Distribution
Partitioned topics stream high-frequency vibration signals and lower-frequency GPS packets independently.
Columnar Storage & Anomaly Scoring
Batch inserts commit directly to ClickHouse; streaming anomaly worker flags Z-score outliers.
The revised engine handled [BENCHMARK: 18,500 EVENTS/SEC] with zero event loss and gave operators instantaneous dashboard feedback.
- 01.Pick the right storage engine for the shape of the data: columnar engines dominate time-series.
- 02.Idempotency is not an afterthought; it must be built into the ingestion edge.
- 03.Real-time streaming pipelines should fail gracefully by preserving raw append-only logs.
“Building resilient distributed systems requires designing for connection failure as a normal daily event.”