Sample data. These figures are not a customer bill.Learn more

Northwind AI

Scale · Sample · cloud

Home

Get startedSoon
Audits
Overview

Analyse

Spend
Cost per task
Agents
Model comparisonSoon
Budgets
AlertsSoon
Reports
PR cost checksSoon
AdvisorsSoon
Spend analysis

Optimise

RecommendationsSoon
Routing policiesSoon
Shadow testsSoon
CachingSoon
TrimmingSoon
BatchSoon
Waste detectorSoon

Gateway

SetupSoon
ProvidersSoon
ModelsSoon
API keysSoon
DeploymentsSoon
PlaygroundSoon
LogsSoon

Savings & billing

Savings history
Savings ledger
Bill match
Billing
Invoices

Settings

Data & privacySoon
OrganisationSoon
ProjectsSoon
TeamSoon
RolesSoon
SSOSoon
Audit logSoon

Plan: Scale · Sample ↗

Sample data. Not a customer bill.

Northwind AI/Shadow tests/New shadow test

New shadow test

Planned

Route, candidate models, sample size, threshold and scorer.

Route

Candidate models

GPT-4.1GPT-4.1 miniClaude Sonnet 4.5Claude Haiku 4.5Gemini 2.5 ProGemini 2.5 FlashGrok 3Mistral LargeDeepSeek V3Llama 3.3 70B

Sample %Share of sample traffic a replay would use. Capped at 20%. Nothing is sent.Pass thresholdThe customer's quality bar, from 0.50 to 1.

ScorerTask successScores the candidate against the route's task tag.Judge modelA stronger model grades the candidate. The judge's tokens are counted as cost.Exact matchFor structured routes, the candidate must match the schema and the expected fields.

I confirm the estimated replay cost for this preview. No provider call is made, and a cap would stop a later test.

Often opened from model comparison with the route and candidate already filled in.