Evaluate
Evals — quality checks outside the agent
Evals in Norcaster are not a separate scoring tool. They are quality checks inside the AI Runtime Control Plane: run by a separate service against a versioned reference dataset the agent does not read, tied to deployments, policies, and traces, and recorded on the same run record as the runtime policy decision.
What an eval is
An eval is a saved set of test inputs run against a workflow or deployment, with one or more scoring dimensions on the output. Suites group evals so you can compare versions side by side and treat regressions like failed tests.
Evals are part of the Evaluate pillar in the runtime control plane. They live next to your deployments and policies, not in a separate platform.
Verdicts on the record, not just scores
A score on its own is a report. A verdict on the run's record is something a reviewer can act on. Today, Norcaster evals:
- Run suites against a workflow or deployment and show failing items linked to their runs and traces.
- Compare two versions on the same suite and show the regression per metric.
- Score the stored output of an existing production run against the reference without re-running it.
- Show the eval verdict next to the runtime policy decision and any human approval on the run detail page.
Every completed run carries a digest of its item verdicts and, where the trace chain is enabled for the organisation, a signed chain entry. A run with items below target raises an alert; a run that regresses against its baseline raises a drift alert. With the promotion gate enabled, a canary cannot be promoted without a passed verdict scored on that deployment's own runs, and a regressed candidate must go through a config proposal. A workflow suite can also re-run on a schedule (hourly, daily or weekly), once per period, re-executing the suite against its target. Online scoring of production traffic is on the roadmap; scheduled runs do not score live traffic.
When to use evals
Before a workflow change ships
Run the regression suite against the new version and compare it with the current one. If pass rate drops or specific examples fail, you see it before you promote.
After a model upgrade
Compare the old and new model on the same suite. Surface where the new model regresses, not just where it improves.
When a customer reports a bad answer
Add the failing case to the suite. Future changes have to keep passing it before they reach production.
When a production answer needs a second opinion
Score the run's stored output against the reference without re-running it. The verdict lands on that run's record, next to the policy decision.
Scoring dimensions
Quality is not one number. Norcaster surfaces several axes per run so a regression in one dimension is not hidden by an improvement in another.
| Dimension | Question it answers |
|---|---|
| Groundedness | Did the answer cite the supporting source? |
| Citation precision | Are the cited sources actually relevant? |
| Safety | Did the response trip a safety check or policy? |
| Schema | Did the output match the expected shape? |
| Latency | Did the run finish under the budget? |
| Cost | Did the run stay under the per-request limit? |
Evals measure quality. Policies enforce what AI is allowed to do. The two are separate but related — see runtime policies for the enforcement side.
Where to start
- Open the evals workspace to create your first suite.
- Read the observability guide — eval results show up alongside traces.
- See the AI Runtime Control Plane explainer for how evals fit with deployments, policies, and providers.