We benchmarked the guardrails. We came second.
Nine guardrails, 949 cases, four corpora. Two of those corpora are public datasets nobody here curated. Everything is published: the harness, the attack corpus, the adapters, and every case each product missed.
We publish this and we compete in it. No methodology removes that conflict. What we did instead: scoring was fixed before anything ran, competitors run at stock settings with nothing tuned to these cases, every miss is listed by case ID, and a thirty-line regex control is included so you can see when a sophisticated product barely beats it. Fluiq does not place first on either output-side corpus. Re-run it and disagree.
Output leakage: adversarial suite
Leak shapes a PII corpus does not contain: obfuscated identifiers, credential formats, injected instructions echoed into output, and refusals that leak while refusing.
49 cases · 32 should block, 17 should pass · source Written for this benchmark · licence MIT
| Guardrail | Category | Recall | False alarm | F1 |
|---|---|---|---|---|
llm-guard | Open source | 87.5% | 17.6% | 88.9% |
fluiq | Fluiq | 81.2% | 11.8% | 86.7% |
aws-comprehend | Commercial | 84.4% | 23.5% | 85.7% |
nightfall | Commercial | 71.9% | 5.9% | 82.1% |
fluiq (lite) | Fluiq | 68.8% | 5.9% | 80.0% |
regex-baseline | Control | 62.5% | 11.8% | 74.1% |
lakera-guard | Commercial | 46.9% | 5.9% | 62.5% |
presidio | Open source | 43.8% | 5.9% | 59.6% |
nemo-guardrails | Open source | 43.8% | 17.6% | 57.1% |
Recall is what it catches. False alarm is how often it blocks text that should have gone out. Neither means anything on its own, because a guardrail that blocks everything scores 100% recall and gets switched off in week two.
How it works
Every guardrail receives the same string and answers one question: block, or allow. Scoring was fixed before any contestant ran.
- Roughly a third of each output corpus is benign text built to bait false positives: order numbers shaped like card numbers, git SHAs shaped like secrets, a sentence that merely describes an injection attempt.
- A contestant whose library or credentials are missing is skipped and reported, never scored zero. Nightfall's first run errored on 190 of 300 cases through rate limiting; those results were discarded rather than published.
- Scope differs between products. Presidio detects PII and never claimed to detect API keys, so it scores zero on secrets. That is a scope difference rather than a defect, which is why per-corpus results exist and no single number should be read alone.
- One labelling disagreement is material. On the jailbreak corpus, persona prompts (“you are Black Panther”) are labelled benign; Lakera treats persona adoption as hostile. That single difference of opinion drives both its high recall and its high false-alarm rate there.
Think we got it wrong?
Adding a guardrail means implementing one method. If you think a product is misconfigured here, the fastest rebuttal is a pull request.
def scan(self, text: str) -> Verdict:
return Verdict(blocked=..., findings=[...])