Prompt-injection vendors publish a recall figure, a latency figure and sometimes a false-positive figure. Each depends on choices the page may leave out. The questions below fit any vendor, including us. Each group gives the prompts to put to the vendor, the reason they matter and the shape of a good answer.
Operating point
- Ask whether the headline number uses a fixed threshold or a matched false-positive budget.
- Ask which false-positive rate the headline recall was read at.
- Ask for recall at two or three false-positive rates, or for the full curve.
Why it matters. Recall without a false-positive rate is half a measurement. A detector that flags 30% of benign input catches more attacks than one that flags 1%. A fixed threshold such as 0.50 also sits on a score scale the vendor chose and refits at each retrain, so two fixed-cut figures compare calibration curves as much as detection quality.
A good answer names the false-positive rate, says whether the threshold was set per dataset from benign scores, and gives any fixed-cut figure as a labelled extra. A gate in front of a model needs a strict budget, because each false flag lands on a legitimate request.
The benign side
- Ask how many benign rows the benchmark holds and what kinds they are.
- Ask whether short everyday messages such as greetings and thanks are included.
- Ask whether benign prompts that contain words an attacker might also use are included.
- Ask whether any benign rows were used to choose the threshold, and whether those rows are also in the scored set.
Why it matters. The false-positive rate comes entirely from the benign rows. A benchmark with few or easy benign rows reports a flattering rate. Short messages and trigger-word prompts are where over-flagging shows up first. A matched-budget figure reads its threshold from the same benign rows it scores, which suits a comparison between detectors but does not forecast a deployment. In a deployment you set the threshold on your own traffic.
A good answer gives row counts, describes each benign set, reports separate false-positive figures for short messages and for trigger-word prompts, and says where the threshold came from.
Data hygiene
- Ask whether any test row appeared in the training data, and how the vendor checked.
- Ask which datasets are public and which are in-house.
- Ask for the row count and a confidence interval beside every rate.
Why it matters. A detector scores well on text it has already seen. An exact-match check misses rows that embed a memorised span, so ask about partial overlap as well. In-house test sets drawn from the vendor's own training distribution measure fit, and public sets the vendor did not train on give the independent test. A rate measured on 40 rows has a wide interval. Wilson intervals suit the rates near 0% and 100% where detector figures tend to sit.
A good answer states the overlap method and its result, separates public from in-house sets in any average, and prints the row count and interval next to each rate.
Conditions
- Ask what thread budget the benchmark used, and on what machine.
- Ask the batch size, and whether the latency includes a network hop.
- Ask which percentile the headline latency is.
- Ask how long the prompts were.
Why it matters. Latency moves with thread count, batch size, input length and network. A median measured in-process on short prompts differs from a 99th percentile measured over a network on long documents. A median hides the slow tail that a request path feels.
A good answer gives the hardware and thread budget, the batch size, the input length and where the clock started and stopped, and reports the median beside a tail percentile.
Who ran it
- Ask whether the vendor or an independent party ran the benchmark.
- Ask whether the vendor ran the other detectors itself, on the same machine, with the same thread budget and the same rows.
- Ask how the vendor handled each detector's own defaults, such as input truncation.
Why it matters. A vendor-run benchmark can be sound, and it gets the scrutiny any self-reported result gets. A comparison carries more weight when every detector received the same rows, hardware and thread budget, and when the page says what was done wherever a default changes a score.
A good answer says who ran what, shares the code or logs so you can inspect the run, and welcomes a repeat by a third party.
Versions and drift
- Ask whether the figure comes from the version you will call.
- Ask for the date of the measurement.
- Ask how a retrain changes the number, and how the vendor tells customers when it does.
Why it matters. Detectors get retrained. A fixed-threshold figure from last quarter's model may not describe today's. A matched-budget figure holds steadier when scores are rescaled, and it still moves when the model changes.
A good answer names the version and the date, and points to a changelog that records benchmark movement between versions, including drops.
What the output is
- Ask whether the API returns a decision your code acts on, or whether the vendor enforces something for you.
- Ask what the detector returns when it is unsure.
- Ask what evidence comes back with a decision.
- Ask what the terms say about detection coverage.
Why it matters. A returned decision leaves the response to you: you block, hold for review or log. Vendor enforcement moves that choice to the vendor, and you need to know what happens on a timeout, an error and a low-confidence score. No detector catches every injection, so the terms should say what the vendor does and does not warrant.
A good answer documents the decision states, documents the unsure state, returns evidence such as a score, the signals behind it and where in the text they fired, and makes no promise of complete detection.
Contract and data
- Ask how long the vendor keeps submitted text, and whether you can shorten that period.
- Ask where the text is processed and stored.
- Ask who can read it, including staff and sub-processors.
- Ask whether the vendor uses submitted text to train or tune models.
- Ask for the answers in the contract or data processing agreement, in addition to the website.
Why it matters. Scanned text belongs to your users and your documents. Retention, location and access are the first questions your own security review will ask.
A good answer is in writing, with a retention setting you control and a list of sub-processors.
How Tawa Guard answers the first four groups
Every figure below is a benchmark figure from Tawa Guard benchmark v13, run on 17 September 2026. None is a production figure.
Operating point. The default engine catches 72% of attacks at a matched 1% false-positive budget, averaged over six datasets: three public and three in-house. Each dataset's threshold comes from its own benign rows, so the figure measures ranking quality and does not describe a deployed setting. Two of the three in-house sets are held-out splits of our own data distribution, which makes them an upper-bound check and weaker evidence than the public sets. Ask any vendor, us included, for per-dataset figures.
The benign side. The benchmark includes a set of short everyday messages and a set of benign prompts that contain trigger words. Both are scored separately at a fixed threshold and sit outside the 72% average, because a set with one class has no matched operating point.
Data hygiene. The benchmark scores 14,865 rows per engine across 9 datasets, and every per-dataset rate carries a Wilson 95% confidence interval. We check each row of the independent sets against our training corpus. Exact matches are dropped from the scores, and partial overlaps are flagged for review and still scored.
Conditions. Every detector in the comparison is pinned to the same thread budget on one machine, with no network. The default engine's median detection time is 23 ms in-process, measured on short prompts, one at a time. We ran the whole benchmark ourselves, other detectors included, so the vendor-run caveat in the "Who ran it" group applies to us.
You can repeat these checks on your own traffic. The free allowance is 5 million Tawa Units a month, and one Tawa Unit is one character scanned on the default engine.