A fixed cut measures the calibration as well as the detector
Most published comparisons score every detector at a fixed threshold, usually 0.50. That number is a point on a scale the vendor chose. Calibrate the scale differently and the same detector, reading the same rows, produces a different F1.
The comparison then says as much about the calibration curve as about detection quality. A refit that shifts the curve moves every fixed-cut figure, even when the detector still ranks the same attacks above the same benign inputs.
Matching on false positives instead
For each dataset and each detector, we take the benign rows and sort them by score. We set the threshold at the lowest value that flags no more than 1% of them. Then we count the attacks that score above it.
Every detector now pays the same false-positive cost, so the recall that remains is directly comparable. The threshold comes from the score ordering alone, which means no rescaling of the scores can move the result.
We average across the six datasets that contain both attacks and benign rows. A dataset with a single class has no operating point to derive, so we report those separately and leave them out of the average.
How the two bases differ
| Fixed cut | Matched 1% false positives | |
|---|---|---|
| Where the threshold comes from | A constant chosen in advance | Each detector's own benign scores |
| Moves when scores are rescaled | Yes | No |
| Comparable between releases | No | Yes |
| What it measures | Detector and calibration together | Ranking quality |
Why one percent
An inline guard has to run at a strict false-positive budget, because every false flag lands on a legitimate request that someone then has to review. One percent is a demanding setting for a test, and it is the one we hold ourselves to.
Reading our figures
On benchmark v13 the default engine catches 72% of attacks at this budget, averaged over six datasets. We still publish fixed-cut figures where they answer a different question, such as how often a benign prompt containing trigger words is flagged. We label each one as fixed-cut, so you can tell which basis you are reading.