Vizra Evals is a Composer package built for the official Laravel AI SDK. It runs inside your app, turns agent evals into Pest tests, and records every run — sampled scores, judge reasoning, tool calls, cost — so baselines catch regressions in CI instead of customers catching them in production.
it('answers only from documented policy', function () { expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->dataset(base_path('evals/support.jsonl')) ->samples(3) ->assert(fn ($a, $row) => $a ->notEmpty()->gate() ->contains($row->expected()) ->costBelow(0.02)) ->judge('Answers only from documented policy.', min: 7) ->gate(minScore: 0.8, maxRegressions: 0) ); });
FAIL Tests\Evals\SupportBotTest ⨯ answers only from documented policy Eval — score 61.7%, pass rate 33.3% across 9 samples Gate failed: 2 rows regressed against the baseline (allowed: 0) ↓ regressed: "What is your refund policy?" 96.7% → 51.7% ↓ regressed: "Can I still return it?" 93.3% → 55.0%
Everything you need when "it seems fine" stops being good enough to ship on.
Or whether it just rolled badly. Every row runs N times, so you get a distribution with spread — not a coin flip dressed up as a verdict.
The judge writes down its reasoning and we keep it. Structured output, no regex parsing, so the score and the explanation never drift apart.
Assert on the real response object — which tools ran, with which arguments, in what order, at what cost.
Build datasets from real production traffic and evaluate mid-conversation with the context replayed exactly as it happened.
Rows are joined across runs by content hash, so a row that drops below its baseline stops the build — by name.
Score history per suite, any sample's assertions and judge reasoning one click deep, and two runs side by side.
If you can write a Pest test, you already know how to eval an agent.
Point toPassEval() at any Laravel AI agent, give it a dataset — JSONL, CSV, or your real production conversations.
composer require vizra/evals --dev php artisan migrate
Every row runs N times — agents are nondeterministic, one sample proves nothing. Normal test runs skip evals entirely.
./vendor/bin/pest # evals skipped ./vendor/bin/pest --evals # against the real model
Scores, judge reasoning, tool calls and cost land in your database. Your first passing run becomes the baseline.
composer require vizra/evals-ui # then visit /evals
A second package that mounts a route in your own app — suites, trends, drill-downs, and the judge's reasoning one click deep. Reads the same database your evals already write to, so there is nothing to sign up for and nothing leaves your network.
Everything above runs against your own database. Which means your baseline is whatever the last person to run it happened to have, CI throws its history away with the container, and nobody else on the team can see the trend. That is fine for one developer. It stops being fine the moment it matters to anyone else.
The engine. Evals as Pest tests, judges, baselines, gates and a CI exit code. Runs entirely in your app.
A dashboard for the runs it records, served from your own app against your own database.
The same runs, kept where the whole team can see them — and a Run button for people who never open a terminal.
The first two are the whole product for one developer, and always will be. The third is for when the answer has to outlive your laptop.
Same engine, same evals, same commands. Runs report themselves to a shared dashboard, CI keeps its history, and someone who has never opened a terminal can press Run — while the eval still executes in your app, on your keys.
One reference run, not one per machine.
Branch runs stop vanishing with the container.
For the people who judge the answers.
No model keys, no code, no access.
Five minutes from install to your first recorded, baseline-gated eval.
Looking for Vizra ADK? It lives here now →