Use this when a result disappears

Use this runbook when the tracker is empty or delayed, an ETA is unavailable, stop visits stop accumulating, or road slowdowns stop appearing. Start with collection and raw observations. A missing derived row is often a quality decision; a missing raw row is an ingestion problem.

The decision is intentionally ordered from the durable source outward. Do not start by changing ETA thresholds or deleting a derived table. That removes evidence before the failure is understood.

Check the collection boundary

The dashboard exposes private, no-store endpoints for the collector:

The admin routes require the dashboard's existing access control. The command or request that reaches them is operationally significant, so record the response from /api/status before and after a manual collection.

A healthy cycle reports the number of active vehicles, valid observations, newly stored observations, duplicates, derived counts, failures, and duration. A repeated alarm delivery can be healthy: the D1 unique key on (vehicle_id, reported_at) should count the repeated packets as duplicates rather than create a second raw row.

If one location request returns HTTP 503, successful vehicles should still be stored. The collector test expects two active vehicles, one valid observation, one stored observation, and one failure for that partial case. If the schedule provider fails, a new snapshot is not written; the previous snapshot remains available and its age must be checked before it is shown as current.

Check freshness before interpreting quality

There are three clocks in the current code:

Do not use the 60-second public snapshot state as an ETA confidence label. A snapshot can be delayed while an individual estimate still meets its query age, or an estimate can be low confidence while the snapshot is still being served.

In /api/data-health, compare the source event window with the position window. Source requests with failures but no new observations point toward upstream response or validation. New observations with mostly off_route positions point toward route data or assignment. Many gps_jump positions point toward a timestamp, coordinate, or progress discontinuity. Preserve the raw rows in all three cases.

Check transit derivation

GET /api/intelligence/readiness reports raw observations, route-position qualities, trip states, segment-run count, and stop-visit count. For a vehicle profile, use /api/intelligence/vehicles/{vehicleId}. For an ETA, use /api/intelligence/vehicles/{vehicleId}/eta?stopId=...; for route service and slowdowns, use the corresponding route endpoints.

A good route position is no more than 200 meters from the canonical route. A same-route, same-direction progress change over a positive gap of no more than ten minutes is marked gps_jump when it implies more than 100 km/h, unless it is a route wrap. Neither off_route nor gps_jump advances trip passages or segment runs.

If positions are good but segment runs are not growing, inspect trip continuity and ordered stop crossings. A segment run requires adjacent passages in one trip_run; it cannot be created from a pair of coordinates alone. A route or direction change, a gap over ten minutes, or a route wrap closes the open trip and starts another.

For stop visits, the current detector requires the nearest stop to be within 75 meters and speed to be at most 5 km/h. A missing upstream speed is accepted only when the previous same-route, same-direction position is no more than 60 seconds away and its progress implies at most 5 km/h. A candidate needs two samples. When it is flushed, the visit carries 40 seconds of uncertainty. A one-sample candidate disappearing is expected behavior, not necessarily a failed insert.

Check ETA evidence

The current ETA path uses a segment baseline only after 30 samples. It chooses, in order, the same weekday and hour, the same hour, or all samples for the segment. If none has 30 samples, the segment uses a fixed 18 km/h fallback and the response reports a fallback segment.

Recent segment durations can adjust a historical baseline only with three independent vehicles in the last 30 minutes and a current p50 at least 1.3 times the historical p50. Five recent vehicles can preserve high confidence; three or four cap an adjusted high-confidence estimate at moderate. Two vehicles do not adjust the estimate. Recent rows are excluded from the historical baseline.

When an ETA is unavailable, distinguish no_position from stop_not_on_route. When it is available but low confidence, inspect positionAgeSeconds, historicalSegments, fallbackSegments, and minimumHistoricalSamples in the response. These fields say which evidence is missing; changing a displayed label does not create it.

Check the optional road branch

GET /api/road-status shows the road processor state. GET /api/road-readiness shows persisted match runs and road samples. The branch is disabled when VALHALLA_URL is absent. In that state, the collector should continue storing observations and deriving transit intelligence.

When enabled, the processor reads good positions in traces of four to twelve points and considers at most three vehicles per run. It splits a trace at a route or direction change, a time gap over 60 seconds, or backtracking over 250 meters. A match must have no unmatched or discontinuous points and must stay within 50 meters of the input trace. Road intervals also require the same route segment, endpoints more than 100 meters from stops, and a route-derived speed between 7 and 100 km/h.

The cursor behavior shows whether a failure is safe to retry. A successful match writes the match run, samples, and cursor together. A transient matcher failure such as HTTP 503 leaves the cursor unchanged and schedules a retry after 60 seconds. A structurally unacceptable match advances past the rejected prefix without writing samples. A route-assignment change is also split rather than sent as one trace. A processor run with retryNeeded: false, rejectedRuns: 1, and no samples is not waiting for an upstream retry; inspect the input and match response.

Migrations and deployment

Apply the collector migrations through the package scripts, in the same environment as the deployment:

The migrations create raw observations and indexes, transit intelligence tables, road cursors and match tables, and the source-event journal. Confirm the target database has the expected migration state before starting a worker that writes a newer table. A schema error is not fixed by restarting the alarm.

For a collector deployment, check /api/health, /api/status, /api/data-health, and /api/intelligence/readiness after the first collection boundary. If road matching is enabled, check /api/road-status and /api/road-readiness separately. A green collector does not imply a green Valhalla branch.

Roll back without losing the source

If the web or API response is wrong, roll back that consumer while leaving raw collection running. If Valhalla is unavailable, remove or unset VALHALLA_URL in the collector environment and deploy the configuration change; this disables only road processing. Do not delete vehicle_observations or reset a road cursor to make the dashboard look healthy. The raw rows are the evidence needed to replay or diagnose the derived branches.

Source anchors: collector.ts, data-health.ts, analytics.ts, and road-intelligence.ts.