Gate responses that fail quality thresholds
LLM-as-judge scores every response server-side. Agentic evaluation judges whole runs: tool choice, trajectory, and multi-agent coordination. Datasets and prompt management are part of it.
Fluiq/ testsTotal Evals
847
across 312 traces
Avg Score
0.91
threshold ≥ 0.7
Pass Rate
88.4%
749 / 847 passed
By Metric
What you get
Metrics
Six judge metrics
Hallucination, faithfulness, relevance, toxicity, coherence, and completeness, scored per response.
Thresholds
Per-metric gates
Set a threshold for each metric. Warn mode logs the score; block mode stops the response.
Agentic
Whole-run evaluation
Layered judging of a full agent run: deterministic checks, tool-selection quality, trajectory against the goal, and multi-agent coordination across fan-outs and joins.
Jury
Multi-model judge panel
Borderline verdicts convene a jury of different judge models and aggregate their votes, with every member's score and reasoning kept for audit.
Your call
Pick the judge, or the whole jury
Choose which model scores a run, and which models sit on the panel, per evaluation or per dataset batch.
Your keys
Bring your own provider key
Save an OpenAI, Anthropic, or Google key and judge tokens bill to your account at the rate you already negotiated.
Auditable
Every score shows its prompt
Each result carries the exact judge prompt and version that produced it, so a shifting metric is traceable to a prompt change.
Datasets
Golden sets from real traffic
Curate datasets out of production runs and batch-evaluate every example. The regression gate for a prompt or model change.
Prompts
Versioned prompts and judges
Version and deploy prompt templates next to the evals that score them, and promote any saved prompt to a custom judge by slug.
The judge runs server-side. Opt in with one call.
- instrument() only traces; scoring starts when you call fluiq.eval()
- Warn mode logs scores; block mode raises FluiqEvalError below threshold
- Run Agentic Eval on any root trace to judge the whole run: tools, trajectory, coordination
- Batch the same judges over a golden dataset to catch regressions before release
- CI gates run the same checks in GitHub Actions
import fluiq
fluiq.instrument(api_key="fl_...")
fluiq.eval(
metrics=["hallucination", "relevance", "toxicity"],
thresholds={"hallucination": 0.8, "relevance": 0.75},
mode="warn", # "block" raises FluiqEvalError
)Part of the Fluiq platform
Compare plansUnlimited traces, always free.
Tracing, security, evaluation, datasets, and prompts all run on the free tier. Slack alerts unlock on a paid plan.