Follow one report through the system

This page is about lineage. It answers what happens to one upstream vehicle report after collection, which records preserve it, and where the system stops trusting it. The architecture page explains why the branches are separate. This page names the stored evidence.

The collector polls the upstream service every 20 seconds. A successful location response becomes a normalized vehicle_observations row with both reported_at, the time reported by the vehicle, and collected_at, the time this collector received it. A fetch attempt also becomes a raw_source_events row when that journal write succeeds. When the journal write succeeds, a failed request has an operational record even though it cannot produce a vehicle observation.

The diagram has one purpose: show that road samples do not come directly from the latest coordinate. They come from good route positions, a bounded trace, an accepted Valhalla match, and eligible moving intervals. The ETA path reads transit segment runs. The slowdown path reads road segment samples. Neither path needs to rewrite the source observation.

The stored records have different jobs

The source event and the normalized observation are related by the collection cycle, not by a foreign key. The road tables retain their own match-run identity. This matters when an upstream fetch failed, when a point was off route, or when a Valhalla match was rejected: the absence of a downstream row is then a decision to explain, not an unexplained data loss.

Raw dedupe happens before intelligence

The raw observation insert is intentionally idempotent for the vehicle's event time:

That is the actual conflict clause in db.ts. A repeated packet returns as a duplicate count. The intelligence processor has the same key on vehicle_route_positions, so reprocessing the same observation does not create a second position, passage, or segment run. The collector only enqueues the road processor when the raw batch succeeded and inserted at least one new observation.

Where a report stops

Projection is a classification, not a destructive filter. A point within 200 meters of the canonical route is good; a farther point is stored as off_route. A good point that implies more than 100 km/h of route progress over a positive same-route, same-direction gap of no more than ten minutes is stored as gps_jump, except for a recognized route wrap. Both rejected qualities remain available to data-health and profile queries, but neither advances the transit event stream.

The transit processor catches errors per observation and continues through the ordered batch. A good position without an open trip starts one. Forward movement across a stop inserts a passage with an interpolated crossing time. A passage only creates a segment run when the immediately preceding stop passage exists for that trip. A stop candidate is more cautious: it starts within 75 meters of a stop at no more than 5 km/h, and it becomes a visit only after two samples. The current uncertainty value is 40 seconds, derived from two 20-second collection intervals.

The road branch has its own stopping points. It reads good positions joined to their source observations, takes at most 12 points, and requires a continuous prefix of at least four. A route or direction change, a gap over 60 seconds, or backtracking over 250 meters splits the prefix. An accepted match needs matched or interpolated points, edge positions, no route discontinuity, and a maximum snap distance of 50 meters. Interval samples then exclude stop endpoints within 100 meters, route-segment changes, intervals outside 7 to 100 km/h, implausible path-to-route ratios, and edge pieces under 5 meters.

One test run, with the counts exposed

The R603 PM intelligence fixture supplies three ordered observations for one vehicle. The processor writes three route positions, opens one trip, crosses two stops, and creates one segment run. The counts matter because the segment is not inferred from the number of GPS points. It is produced by the pair of adjacent stop passages.

The dwell fixture uses two stationary reports at 00:00:00 and 00:00:20, followed by a moving report. It writes one visit with 20 seconds of observed dwell, 40 seconds of uncertainty, and a sample count of 2. If the candidate had only one sample, it would be cleared without a stop_visits row. These examples are from intelligence.test.ts, not claims about a particular live bus.

The road fixture makes the boundary visible in a different way. Four points are matched, but nearest-stop distances of [200, 50, 200, 200] cause the two intervals touching the 100-meter stop exclusion to be skipped. The match run remains, one road sample is stored, and two intervals are counted as skipped. A successful map match therefore does not imply that every input interval becomes traffic evidence.

Retry state is data lineage too

The road processor advances a vehicle's cursor with the accepted run in one D1 batch. If Valhalla returns a transient failure such as HTTP 503, it stores no match or sample and leaves the cursor unchanged. The Durable Object records retryNeeded and schedules another attempt after 60 seconds. A non-retryable structural rejection, such as an unacceptable match or a route discontinuity, advances past the rejected prefix without storing samples. This avoids repeatedly presenting the same bad trace to the matcher.

No VALHALLA_URL means no road branch. The cursor does not move, the processor reports disabled, and the transit tables continue to receive data. ETA and service queries are read-time analytics over transit_segment_runs, stop_visits, and current good positions; they do not require a road match run.

For the component boundaries, see Telemetry and Transit Intelligence. For operator actions when one lineage stops, see the transit intelligence runbook.

Current source anchors: collector.ts, intelligence.ts, road-intelligence.ts, and the collector migrations.