Evals for grocery-shopping agents. A frozen catalog, a deterministic scorer, a fake store, and receipts an evaluator can check.
An agent handed a weekly menu should buy what is on the list, nothing it was not asked for, and nothing that breaks the household's stated budget. Most agent failures are quiet: an item drops with no note, a substitute swaps with no reason, a total creeps over budget. Carteval scores the cart against a frozen catalog and a synthetic household week, so the failure is visible instead of quiet.
Scoring is a profile, not a single number: completeness, stated-constraint compliance (over budget or a broken constraint is a hard fail), taste fit on discretionary lines only, waste and leftover handling, out-of-stock robustness, and efficiency reported, not scored. No composite score, no single winner claim.
Shop the demo store — search and aisles over the frozen catalog, a cart, a review step, and a submit that saves the cart plus its full shopping event log under a receipt id. Anyone with the id can pull the cart up and check it.
Score a cart — POST a cart and a task id, get the profile back. Iteration only, not ranked.
Submit a run — POST a cart plus an optional transcript. Entries with a transcript are eligible for the published leaderboard; entries without one are stored but not ranked.
Leaderboard — ranked transcript-eligible submissions per task, profiles only, hard fails flagged.
Discovery file — the whole public surface in one JSON document, for agents.
No account needed to run the public eval: shop the demo store, score a cart, look any record up by its id. Sign in and you can build your own eval — title, budget, the cart as you define good, pantry owned, leftovers — shop it in the demo store, and keep a history of every run against it. Free while validating: no billing, no paywall.
Not a real store. Nothing here orders, charges, or checks out anything, and no payment, checkout, or grocery-login field is ever accepted. Not a medical or allergy-safety certification: eval constraints are stated dietary and budget constraints, nothing more. Built from the bit.recipes bake-off; the scorer, catalog, and example carts are published in the public eval repo at github.com/espetey/bitrecipes-eval.