Evaluation · BELLS-O
Specialized guardrails or frontier LLMs: what BELLS-O reveals
We evaluated 28 supervision systems on what matters in production: detection, but also false positives, latency and cost. None of them wins everywhere.
Every AI system deployed at scale now sits behind supervisors: systems tasked with flagging malicious requests, dangerous outputs and attempts to get around safeguards. Two families compete for the job. On one side, specialized guardrails trained for this single task. On the other, general-purpose frontier LLMs used as judges.
Existing benchmarks mostly compare their accuracy. In production, that is only part of the question: a supervisor that is too slow degrades the experience, one that is too expensive does not scale, and one that blocks legitimate requests ends up switched off. BELLS-O is the first independent operational benchmark to measure all four dimensions together.
What we measured
We evaluated 28 systems from 17 providers on two tasks. Content moderation covers 11 harm categories, on both model inputs and outputs. Jailbreak detection covers 13 attack techniques. For each system, we measure detection rate, false-positive rate, latency and monetary cost.
The datasets combine handcrafted prompts, expert-curated samples and synthetic data paraphrased to remove the fingerprints of the model that generated it: 1,400 prompts and 1,300 input-output pairs for moderation, and 6,406 jailbreaks built from 720 base prompts, complemented by public datasets.
Moderation: specialists lead
In content moderation, the best specialized guardrails reach around 95% detection, versus 94% for frontier LLMs, with false-positive rates of 2% or lower in both cases. For equal performance, specialized systems run 5 to 10 times faster and cost about 10 times less.
Jailbreaks: generalists win, at a price
Jailbreak detection flips the ranking. Frontier LLMs catch more attacks and raise fewer false alarms, but they cost 10 to 50 times more and their latency is 5 to 10 times higher.
The gap is especially stark on benign prompts that look like jailbreaks: the median false-positive rate reaches 30.5% for specialized supervisors, versus 10.2% for generalists.
| Task | Best detection | Cost | Latency |
|---|---|---|---|
| Content moderation | Specialized, ≈ 95% (generalists ≈ 94%) | Specialized ≈ 10× cheaper | Specialized 5–10× faster |
| Jailbreak detection | Generalists, with fewer false positives | Generalists 10–50× more expensive | Generalists 5–10× slower |
More capable does not mean more vigilant
Within a model family, the more capable version cuts false positives without improving detection. Claude Sonnet 4.6 drops to an 8.6% false-positive rate versus 15.5% for Sonnet 4.5, with near-identical detection (66.0% versus 65.8%). Ministral 14B shows 6.4% false positives, versus 22.7% for its 3B sibling.
Shared blind spots
Some attacks slip past both families. ASCII-art jailbreaks are detected at a median of only 43.1% by specialized supervisors and 10.1% by generalists. Prompt injections also remain poorly covered.
There is no best supervisor in the absolute, only a best supervisor for a given use.
The takeaway
Picking a supervisor off a fixed ranking is a mistake. The right choice depends on the task, request volume, latency budget and the acceptable cost of a false positive. These trade-offs should be revisited with each deployment, based on measurements taken under real conditions of use.
That is exactly RabelAI's approach: measuring what matters in your context, with open and reproducible protocols.
BELLS-O was led by Leonhard Waibl, Felix Michalak and Hadrien Mariaccia, in collaboration with the UK AI Security Institute, the French Center for AI Safety (CeSIA) and the Supervised Program for Alignment Research (SPAR).
Which supervisor fits your system?
We evaluate the options under your real conditions of use.
Let's talk