Vehicle data pipelines · Bhargava Mandapati

"Vehicles generate terabytes of data" appears in every SDV presentation. It is true and almost entirely misleading, because the interesting question is not how much is generated but how little can usefully leave.

The numbers that shape everything#

The pipeline#

Volume falls by orders of magnitude at every stage — Each stage exists to discard data. The design work is deciding what to discard, and that decision has to happen on the vehicle.

Collection#

Signals come off the buses, out of the data broker, and from application telemetry. At this point everything exists and nothing has been decided.

On-vehicle filtering#

One signal, four designs

The question: does a new suspension setting reduce harsh events?

Naive. Stream vertical acceleration at 100 Hz. That is roughly 400 bytes/s, about 1 GB per vehicle per month. Across a fleet of 100,000 vehicles it is 100 petabytes per month and a cellular bill nobody will sign.

Event-triggered. Send a 5-second window around each event above a threshold. Perhaps 30 events per month × 2 KB = 60 KB per vehicle. A factor of ~17,000 saved, and the answer is just as good.

Aggregated. Send a daily histogram of event magnitudes. About 200 bytes per vehicle per day. Loses the waveform, keeps the distribution — sufficient if you only need the trend.

Computed locally. Send one number per day: harsh-event count. 8 bytes. Sufficient if the question really is only "more or fewer".

All four answer a version of the question. The engineering skill is knowing which version you actually need — and the default mistake is choosing a design that answers questions nobody has asked yet, at a cost somebody will notice.

Buffering#

Transport and ingestion#

Batched, compressed, authenticated, usually MQTT or HTTPS. The cloud side is ordinary data engineering: ingest, validate, store, and make queryable.

Gotcha

Schema evolution is the part that bites. Vehicles in the field run software from several model years, all emitting slightly different payloads for years.

Version every message, never reuse a field name with a new meaning, and make the ingestion side tolerant of both older and newer shapes than it expects. A schema change that assumes the whole fleet updates together is a schema change that will lose data.

The categories of data, and their different rules#

Not all vehicle data is the same kind of thing, and treating it uniformly is a mistake.

Category	Example	Volume	Sensitivity	Typically
Diagnostic	Fault codes, ECU health	Low	Low	Always collected
Operational	Speed, energy use, temperatures	Medium	Medium	Aggregated
Behavioural	Braking style, feature usage	Medium	High	Consent required
Location	GNSS trace	Medium	Very high	Consent, minimised
Media	Camera, microphone	Enormous	Very high	Almost never leaves
Application	Crash logs, UI telemetry	Low	Medium	Standard app rules

What this means for cockpit engineering#

Your app telemetry is part of a shared budget. An analytics SDK sending events per interaction is trivial on a phone and can be significant across a fleet on metered cellular. Batch, aggregate, and consider whether each event earns its bytes.

Aggregate before sending, not after. The instinct from server work — collect everything, decide later — inverts here. Deciding later means paying to transmit data you will discard.

Design for offline as the normal case. Not an error path. Queue, persist across reboot, and define what gets dropped when the queue is full.

Never log location or personal data casually. A debug log with GPS coordinates that syncs to a backend is a compliance incident, and the investigation will find it.

Know which consent your feature depends on. If it needs behavioural data, what happens for a customer who declined? "It silently does nothing" is not an acceptable answer.

A well-behaved feature telemetry design

A team wants to know whether a redesigned media browser is easier to use.

Wrong: log every tap with a timestamp and screen name; upload continuously. Roughly 2 KB per drive per vehicle, always on, and it is behavioural data requiring consent for a question that does not need per-tap detail.

Right: compute the metric on the vehicle — median taps to start playback, per session. Accumulate a daily summary. Upload once, batched with everything else, about 40 bytes. No individual interactions leave. No consent-gated category is touched, because a distribution is not behaviour tied to a person.

The team gets a better answer — a distribution across the fleet rather than a pile of events to process — for a thousandth of the data and none of the privacy exposure.

Next#

Where the processing should happen — vehicle or cloud.