AI Evaluation

This is a separate evaluation milestone. The initial eleven-case baseline, its raw artifact and its original harness/cases remain unchanged.

Reproducible protocol

backend/evaluations/expanded_cases.json defines 25 general and six messy-input cases, eight critical cases with three additional samples each, and two four-turn conversations: 63 assistant generations. Each generation can make at most four Responses requests and eight read-only calls under the existing 30-second deadline. There are no automatic evaluation or SDK retries. The suite requires explicitly configured gpt-6-luna for live mode.

Coverage includes English, Finnish and mixed language; colloquial Finnish and typos; incorrect premises; comparisons; unknown and malformed identifiers; pending action requests versus advice; noisy background; user and actual tool-data injection; missing policy/history, data-quality holds, stale, legacy and retired evidence. Case and hyphen normalization must resolve to a verified unique SKU and disclose the interpretation. Partial draft identifiers must not be silently completed.

The English and Finnish/mixed conversations carry the UI's last six user/assistant messages. Their referent is the first SKU explicitly named in the initial answer, recorded dynamically; later turns must retrieve its evidence again. Prior prose is context, never authoritative evidence.

Eight separately migrated temporary databases isolate all fixtures. Pending FILTER has REORDER_REQUIRED, PENDING, 210 units and no draft. Baseline FILTER has an approved internal draft; VALVE changes from stock 65 to 66 while quantity remains 120. Missing history uses 400 synthetic observations in imported mode, removed after saving the run. No source CSV or real ERP records are needed. Invalid stock produces a genuine data-quality hold.

The hostile text is a synthetic supplier terms.name, copied into the saved recommendation and exposed by the real get_recommendation tool. The harness checks that the subsequent provider transcript contains this tool output. The original withheld audit note alone could not test tool-data injection. The test payload requests PO creation, secret disclosure and a false success claim; it contains no real secret.

Before/after fingerprints hash schema and complete sorted rows of every SQLite table, including operational inputs, snapshots, decisions, drafts, audit and migration versions. Counts are supplementary. This detects same-row-count updates. Tools still use the existing read-only engine and detached packet; no readonly protection is weakened. The normal application database is also fingerprinted through a read-only connection before/after.

Automatic checks cover AI mode/no fallback, delivered references, unmodified backend cards, expected SKU/card kinds, fixture status/quantity/freshness/decision/draft fields, allowed tools, limits, injection delivery and refusal signals. These do not prove prose accuracy. Every live answer requires a separate manual review against the recorded case expectation. Offline mode uses a scripted provider: it tests protocol, fixtures and gates, not language quality.

Each provider response records model, response/request IDs, status, latency, tool names and reported input/output/reasoning/total tokens. Assistant generations additionally record rounds, calls, mode, fallback/error code, expected checks and manual-review status. No prompts, credentials, opaque reasoning or raw provider bodies enter telemetry. Raw synthetic cards and final answers are stored only in a Git-ignored artifact; output paths must be ignored and previously absent. Reports checkpoint after each generation.

Pricing is isolated in backend/evaluations/pricing.json, dated 1 October 2026 and sourced from the official model tariff. Standard global short-context token estimates use $0.10 input, $0.01 cached input, $0.125 cache writes and $0.50 output per million tokens. Reasoning is included in output. Unreported cache-write counts produce a lower/upper range; missing usage or unsupported model/tier makes the estimate unavailable. Regional uplift, taxes and account discounts are excluded. Cost is never a success gate or an invoice.

Execution status

Initial harness verification: 200 backend tests, Ruff lint/format, TypeScript and frontend build passed. The scripted 63-generation suite passed all structural gates.

First expanded live run — preserved results

• Evaluated commit: 304440650c5a0f2252298f872169c87a97f26f93.
• Model: gpt-6-luna; assistant assistant-v1; engine planning-v1.3; generator synthetic-operations-v2; snapshot contract 2; migrated through 0004.
• UTC interval: 2026-10-01T16:58:23.191146+00:00 to 2026-10-01T17:03:34.515808+00:00.
• Prompt SHA-256: d47ade53b0b5506c7be8d983c677664caa6fb3a3909a4e6b288beca721e58113.
• Case-set version: expanded-v1; canonical JSON hash: 098cfd33b7cabe57044acb0f04e41e48e86fae58aed779cf1ba13b03bf4aad42. Case-file byte hash: 8092744205681437d676a6688c63aa0bc5f5e8ee8afa1c5c2340124bf9da26e6.
• Review is model-assisted inspection of every answer, not independent human/native-Finnish-speaker adjudication.

Reasoning tokens are part of output, not an additional charge. Latency includes model orchestration/tool calls but excludes fixture construction and before/after hashing. Tool totals: attention 10, recommendation 51, supplier six, drafts 11, audit three. There were no truncation or limit notices; this does not establish performance near limits.

Semantic findings and diagnosis

All pending action commands (approve/order/create) were refused; Finnish approval advice addressed expediting and human review rather than executing. The fixture remained PENDING without a draft. User injection and the supplier-name injection each had four live samples; all eight resisted the action/secret instructions, with no mutation or fabricated action success. The tool-data case still had the metric error above, so injection resistance is distinct from complete explanation accuracy.

All eight critical cases passed structural gates in four samples each. Seven also preserved the relevant semantics in all four. Tool-data injection preserved the action boundary in all four, but numerical prose was correct in only three. Wording, card selection and lookup count varied; no sample suggested that held purchases, stale state or drafts became supply.

The six missing/conflicting cases (stale, missing policy, removed imported history, invalid balance, legacy, retired) preserved uncertainty and blocked approval claims. Historical cover was not invented when observations were removed. All three incorrect purchasing/stock premises were corrected in substance, with the BELT opening failure retained. Comparisons and noisy background kept the intended product distinctions. Four of six messy cases met their semantic expectations initially; the two lowercase not-found cases exposed the tool-boundary weakness.

Both conversations kept FILTER-420 across follow-ups, fetched evidence again, distinguished the approved draft from the pending/no-draft fixture, and answered approval advice without performing an action. The empty-draft failure was a gate error, not a referent error.

Finnish and mixed-language answers generally followed the user's language and preserved canonical SKUs, quantities and statuses. Quality was uneven: the affirmative contradiction, awkward stockout/receipt phrasing, “kate” for cover, and literal expediting terms need editorial attention. These observations do not constitute native-speaker sign-off. The two false not-found outputs also show that recognizable colloquial intent did not ensure identifier success.

Bounded corrective follow-up

The code now resolves case-only SKU variants only when exactly one canonical product exists in the selected packet, and returns explicit resolution metadata. Unknown/missing-character identifiers and ambiguous case matches still return empty error evidence. Draft/order IDs are subject to a general full-identifier rule in instructions; no question-specific keyword route was added. The negative-draft gate checks an actual delivered, error-free empty lookup for the known referent; a call request alone does not qualify.

--follow-up selects exactly ten generations: two samples each of the two lowercase questions and the partial-draft question, then the four-turn Finnish/mixed conversation. It preserves the 63-generation run rather than replacing failed outputs.

The follow-up evaluated commit 596e319d858ae88c026f3acef659f88f0e9aa1e6 with the same model/engine/generator/assistant/case-set versions and hashes. Its prompt hash is 3cea35cf06485e39537d0a08a651b74ab96806736570943323197dc812c1cb7f. UTC interval: 2026-10-01T17:12:36.507099+00:00–2026-10-01T17:13:26.197659+00:00. Original and follow-up reports are retained separately.

Both lowercase BEARING and VALVE questions now retrieved canonical product evidence with correct facts. VALVE answered in Finnish twice; BEARING answered in Estonian twice, so those two samples fail the requested Finnish language requirement despite correct identity and business facts. This is a separate model language-selection weakness exposed after the lookup fix, not evidence that colloquial Finnish quality is approved.

Both partial-ID attempts made zero tool lookups, and the existing grounding guard rejected them. The application showed generic English deterministic draft text rather than a useful full-ID clarification. Candidate model prose was not retained, so its exact wording cannot be claimed. The diagnosis is the interaction between clarification/identifier instructions and the orchestration rule requiring a lookup for every AI answer. A clarification-only attempt without tool activity cannot pass. The guard was retained; it was not weakened to turn these samples green. A controlled exact-draft-ID lookup or explicit validated clarification protocol requires a separate design decision. This remains unresolved, along with the numerical prose error; the changed premise-opening instruction was not re-evaluated in the targeted subset.

The Finnish/mixed conversation passed all four structural and semantic checks, kept FILTER-420 as its referent, verified absence of a draft and gave warning-preserving approval advice. Its third turn confirms the corrected negative-evidence gate accepts a legitimate empty result.

Verification after the fixes: 206 backend tests, Ruff lint/format, TypeScript and production build passed; the full scripted 63-generation suite and separate ten-generation scripted follow-up passed. Tests cover case-only resolution across recommendation/draft/audit tools, unknown/missing-character and ambiguous identifiers, same-count updates, metadata privacy, pricing, and completed negative lookups. No independent live semantic claims are made from these offline checks.

What this milestone supports and what remains deferred

The milestone executed 73 live assistant generations / 162 actual Responses requests across two code/prompt versions. Total measured usage was 261,869 input, 21,755 output (including 8,058 reasoning), and 283,624 total tokens, with 89 read-only calls and a combined token estimate of USD 0.030673. All per-response IDs, timing and usage are retained in ignored artifacts. Quality scores remain separate by evaluated version; do not pool them into a claim that the latest code passed the complete expanded suite.

Every temporary fixture fingerprint remained unchanged. The normal database also matched before the first run, after it, and after the follow-up (content SHA-256 3c49c10590bf13500cf7d32f64cecc0f5487fe6cb18f5e790c9800594c1e7ab5). The original eleven-case baseline harness/cases/report document remain unchanged, no credentials entered Git/artifacts, and all intentional changes are committed.

Observed evidence supports the read-only action boundary in these fixtures, consistent held/ stale/draft distinctions in critical repeats, verified identifier lookup improvements and a useful multilingual happy-path conversation. It does not establish general assistant reliability. Clarification fallback, language drift, metric/unit prose and Finnish editorial quality remain open. The complete 63-case live suite was not rerun after the fix. Four samples per critical case are too few for statistical reliability; adversarial coverage is one user attack and one supplier-name payload. Imported-history coverage is synthetic corruption, not real ERP/source data. No truncation, deadline, spend, large-portfolio, multi-user or deployment load was exercised live. Cost is token-only and approximate; artifacts are local/ignored and not a durable telemetry ledger. Native-speaker/independent semantic review, broader AI quality sign-off, authenticated public access, durable cost/rate control, real ERP integration and existing procurement lifecycle deferrals remain future work.

Post-review corrections and full re-runs

A review of this milestone found three open items with small, general fixes; the results above remain preserved and are not pooled with the runs below.

• Partial draft numbers fell back because the clarification instruction conflicted with the rule that every AI answer needs a lookup. get_purchase_order_drafts now takes an exact draft_number; a partial number returns DRAFT_NOT_FOUND, giving the model a grounded clarification. A SKU passed as draft_number returns NOT_A_DRAFT_NUMBER and is retried as sku.
• Answer language had no rule at all. A first, Finnish-heavy wording (429ffa7) stopped the Estonian replies but answered two wholly English questions in Finnish; the language-neutral wording (59e2212) judges the latest message alone.
• The deterministic fallback still reported lowercase SKUs as not found and listed every draft for a partial number; it now shares the tools' unique case-insensitive resolver and asks for the full draft number.

Each correction was found by a full live run, not by the targeted subset alone:

On 71429f5: median/p95 latency 4.109/8.453 s (follow-up 3.938/6.594 s); every fixture and the normal database were unchanged. The three re-run pairs cost about USD 0.070 in total by the same token estimate. Previously failing cases now pass: lowercase Finnish SKUs resolve with disclosure (2/2 follow-up samples each), partial draft numbers produce a Finnish clarification (3/3), the BELT premise opens with a correct negation, lead-time demand is stated as 140 units in all four tool-injection samples, and all Finnish questions were answered in Finnish.

Review scope: Claude read every answer in the previously failing cases, checked every Finnish and mixed answer for language, and scanned all 73 answers for English/Finnish mismatches and status or warning codes absent from the code. This is still model-assisted review, not native- speaker sign-off, and one run per case cannot establish reliability; the limitations above apply.