The model-selection question is narrower than “which model is smartest?” Davao Transit needs a model that can extract intent and copy passenger wording across English, Filipino, Cebuano, fragments, abbreviations, and follow-ups. It must do that without adding a route, mode, or place that the rider did not mention. A fluent answer is not a passing result because the model is not supposed to answer the rider.

What the benchmark measures

The benchmark loads versioned cases from apps/api/src/ai/evaluation/data, sends each message to a provider, parses the structured response, and compares the result with the case's expected intent, language, outcome, and slots. The comparison is deterministic. There is no second language model judging whether the first model sounded helpful.

Schema validity answers whether the response has the allowed shape. Intent accuracy answers whether the top-level intent is right. Outcome accuracy answers whether the deterministic side can take the expected next step, such as resolving endpoints or asking for location. Slot metrics check exact copied text, omission, mismatch, and invention. Latency and estimated cost describe the run; they do not compensate for a wrong action.

The dataset separates development, regression, and holdout cases. The v2 manifest contains 60 development cases, 21 regression cases, and 20 holdout cases, with repetitions specified per case. Public discussion is used as provenance for many examples, while adversarial and synthetic cases cover prompt injection, incomplete requests, invented slots, and follow-ups.

The current release thresholds

The checked-in gate file is dated 2026-08-23 and names baseline run historical-v1-llama-3-3-model-eval-20260823. It requires:

These are release thresholds, not a claim that every checked-in artifact has passed them. The gate is evaluated against the summary file supplied to check-release.ts. A historical baseline ID also does not turn a later prompt version into a baseline without a new run.

A regression run that did not pass

The checked-in regression-v5 summary contains 21 cases and 42 runs. Its metadata records prompt version ai-intent-v5, schema version assistant-intent-v1, model @cf/meta/llama-3.3-70b-instruct-fp8-fast, temperature 0, and one-at-a-time execution.

The result is instructive because schema validity and language accuracy are perfect while the action boundary is not. A model can return valid JSON in the right language and still select faq for “Pila?” when the previous result makes it a fare follow-up, or add modeHint: jeepney to “Asa na ang R603?” without the rider naming a mode. Those are product failures even though the JSON parser is satisfied.

The language slice makes the imbalance visible. English reached intent accuracy 1.0 and outcome accuracy 0.7 over 20 runs. Cebuano reached intent accuracy 0.7 and outcome accuracy 0.45 over 20 runs. Filipino had two runs, both correct, which is too small a sample to carry the language conclusion by itself.

The artifact's prompt version is older than the current source version ai-intent-v11. It remains useful as a preserved regression result, not as a measurement of the current prompt. A new run is required before claiming that a prompt change improved these numbers.

Running the comparison

The evaluator reads its endpoint and token from the environment. A one-model development run looks like this:

For a model comparison, set AI_EVAL_MODELS to a comma-separated list. Pricing is supplied through the evaluation environment, not embedded in the product. The stored cost is therefore a property of a specific run and price input.

To evaluate a saved summary, use the API package script with explicit paths from the package directory:

The command exits non-zero when a configured minimum or maximum is missed. That is the intended release decision. Model size, lower token cost, or a good schema score cannot override a failed intent or outcome gate.