Continuous integration and delivery for vehicles

Continuous integration in a vehicle programme borrows the vocabulary of web development and then diverges sharply. The first half of the pipeline is familiar. The second half is not, and the differences are the interesting part.

What is the same#

What is different, and why#

	Typical web service	Vehicle software
Deploy target	Servers you control	Millions of devices you do not
Rollback	Redeploy, seconds	Another full update cycle, days
Failure blast radius	Users see an error page	Vehicle may be undrivable
Test environment	Identical to production	Approximation of a physical system
Release approval	Team lead	Safety case, homologation, regulator
Hardware variants	One	Dozens of trim, market and model-year combinations
Update bandwidth	Free	Metered cellular, per vehicle
Update opportunity	Any time	Parked, charged, and consented
Regulatory record	None	R156 requires an auditable trail

The pipeline#

Confidence increasing, execution frequency decreasing — Every stage is slower and scarcer than the one before it. The pipeline's job is to fail as early and as cheaply as possible.

Stage 1 — Commit (seconds to minutes, every commit)#

Static analysis, unit tests, licence and checks, secret scanning. On safety-relevant C, compliance is checked here.

Stage 2 — Component build (minutes)#

Build for every target variant. This is where automotive gets expensive quickly: a matrix of SoC variants, market configurations, and model years, each producing its own artefact.

# The variant matrix is the thing that grows without anyone deciding to grow it
matrix:
  soc:     [gen3, gen4]
  market:  [eu, us, cn]
  cluster: [standard, hud]
# 2 × 3 × 2 = 12 images, every time

Gotcha

Variant explosion is the most common reason automotive CI becomes unusably slow. The fix is not more build machines — it is questioning whether each dimension must be a separate image, or could be runtime configuration or an instead.

An overlay you can flip at runtime costs one build. A build flag costs a full matrix row forever.

Stage 3 — SIL / virtual ECU (minutes, every merge)#

The stage from the previous topic. Scenario tests against a data broker, replayed real traces, fault injection, API contract checks.

Stage 4 — HIL (hours, nightly)#

Real ECUs on a bench, driven by a real-time vehicle simulation. Catches timing, bus behaviour under load, boot sequencing, power transitions.

rigs are expensive and few. Treat rig time as the scarce resource it is: run the tests that genuinely need hardware, and be ruthless about moving everything else down to SIL.

Stage 5 — Vehicle fleet (days, per release candidate)#

A test fleet running the candidate in real conditions. Finds thermal problems, real network behaviour, GNSS in tunnels and multipath, and the long tail of things nobody modelled.

Stage 6 — Release (weeks)#

Safety case updated, homologation evidence assembled, /R156 documentation produced, staged rollout planned.

Testing on hardware that barely exists#

Early in a programme, silicon is scarce and late. The standard answers:

Emulate first. AAOS builds run on an emulator with a simulated VHAL long before boards exist. Most application and framework work needs nothing else.

Previous-generation boards. Slower and differently shaped, but they boot and they run integration tests.

Shared rigs with scheduling. A booking system, and jobs that queue rather than fail when a rig is busy.

Record and replay. Capture bus traces from the vehicles that do exist, replay them into SIL forever after. One drive becomes a permanent test asset.

A cluster feature, all the way through

A new charging-status view for the cluster.

Day 1. Written against a broker on a laptop, using a recorded charging trace. Commit stage green in four minutes. UI reviewed from a screenshot in the PR.

Day 1, on merge. SIL stage runs 40 scenarios: charging start, stop, interrupted, fault, unplugged mid-session, negative temperature. Two failures on edge cases nobody thought of at design time. Fixed the same afternoon.

Night 1. HIL run. Catches that the view takes 900 ms to appear after wake because it waits on a property the vehicle publishes late. Fixed with a cached last-known value.

Week 2. Test fleet. A driver reports the view flickers on a specific charger brand that reports state of charge at 20 Hz. Debounced.

Week 6. Ships in the quarterly release.

None of these four bugs would have been caught by the stage before it. That is the argument for having all of them — each stage exists because it finds a class of problem the cheaper stage structurally cannot.

Continuous delivery, honestly#

Gotcha

Beware the "we deploy to cars like we deploy to servers" claim. Ask three questions: what is the rollback path, what is the staged rollout policy, and what happens to a vehicle that is offline for two months.

Programmes with good answers exist. Programmes that have not thought about the third question are the ones that end up with a fragmented fleet nobody can reason about.

What good looks like#

Practical markers you can check on a real programme:

Commit stage under ten minutes, and engineers actually wait for it.
SIL scenarios in the repository, versioned with the code they test.
Nightly HIL, with failures triaged the next morning rather than accumulating.
A build produces its SBOM and licence manifest automatically.
Any artefact can be traced to its commit, its tests and its release.
Test data — recorded traces — treated as an asset with an owner.
Rig time trending down per feature as SIL coverage grows.

What to remember

The early pipeline is ordinary CI; the difference is that rollback is slow and the blast radius is a vehicle, so confidence must be front-loaded.
Stages run commit → build → SIL → HIL → fleet → release, each slower, scarcer and more expensive than the last.
Variant explosion is the usual cause of unusable automotive CI — prefer runtime configuration and overlays over build flags.
Push coverage down to SIL, because it is the last stage that is fast and cheap; reserve HIL for what genuinely needs hardware.
R156 requires an auditable update trail — emit that evidence from the pipeline rather than assembling it later.
Continuous integration is achievable; continuous deployment to customers is neither practical nor desirable. Real cadence is 4–12 weeks, staged.
A good sign of health: rig hours per feature going down over time.

Next#

The simulation asset that ties all of this together — the digital twin.