A latency number carries a hidden parameter
Libraries default to using as many cores as they can find. A numerical library, a deep learning framework and an inference runtime each read the machine's core count at start-up and size a thread pool from it. Each applies its own rule, and some runtimes treat the default as "the whole machine".
Time two detectors on one machine, start both with defaults, and each gets whatever its runtime picked. The two latencies then reflect the model and the share of the machine it was handed. Nothing in the printed number tells you how large that share was.
A bare millisecond figure is a measurement of one detector under one thread budget. Quoted without the budget, it invites a comparison the measurement can't support.
What an unpinned figure hides
The allocation. A comparison on unpinned defaults measures the machine's allocation as much as the model. A detector that happens to claim more threads looks faster, and one that claims fewer looks slower, whatever the models do per unit of compute.
The tail. A median describes the typical prompt. The p95 and p99 describe the slow ones, and those depend on contention between threads, on input length and on how a pool schedules work. A change in thread budget can move the tail by a different amount than the median, so a median that looks fine tells you little about p99.
The difference between latency and throughput. More cores raise the number of requests a machine finishes per second when requests arrive at the same time. They shorten a single request only if the detector splits that request's work across threads. Core count and concurrency therefore move throughput and latency by different amounts, and a figure for one doesn't predict the other.
Batch size and the network. Scoring prompts in batches changes the cost per prompt and adds waiting time. A figure taken over the network includes the round trip and the client's own overhead. Both are legitimate things to measure, and they answer different questions from an in-process figure for a single prompt.
The protocol we use
Every detector in our benchmark runs under the same rules.
- Thread budget. The same thread budget for every detector. We set the limit for the numerical libraries, the deep learning framework and the inference runtime before any of them initialises. Each run records the budget it resolved and whether each limit took effect.
- Batch size. One prompt per call.
- Placement. In-process, with no network between the harness and the detector.
- What is timed. Wall-clock time per call, after warmup, with the first five calls discarded.
- Isolation. Each detector runs in its own process.
- Prompts. Pooled over the benchmark's short-prompt datasets. The pool leaves out the document-scale material, which would change the result through prompt length alone.
- Percentiles. p50, p95 and p99.
Quality metrics sit outside this control. A score is deterministic for a given model and input, so F1 and recall don't change with thread count, and a quality figure needs no thread budget beside it.
Benchmark v13 as a worked example
The table shows per-prompt latency from benchmark v13, measured on 17 September 2026. The conditions are a single machine, the same thread budget for every detector, no network, one prompt per call, pooled over the short-prompt datasets. Latency has no score threshold, so no operating point applies.
| Detector | p50 | p95 | p99 |
|---|---|---|---|
| Tawa Guard default engine | 22.8 ms | 360.7 ms | 589.1 ms |
| Detector A, the fastest open detector we measured | 12.6 ms | 80.4 ms | 140.8 ms |
| Detector B, the open detector nearest ours on recall | 137.8 ms | 929.6 ms | 1,596.2 ms |
| Detector C, an open detector | 119.9 ms | 823.3 ms | 1,759.9 ms |
| Detector D, an open detector | 111.5 ms | 1,020.6 ms | 2,044.8 ms |
Three readings follow from the table.
Detector A has the lowest latency of the five at every percentile, including against the Tawa Guard default engine. Each detector had the same thread budget, so the gap reflects the detectors themselves.
The ordering at the median differs from the ordering at the tail. Detector D has a lower p50 than Detectors B and C, and the highest p95 and p99 of the three. A reader who sees only the median would rank them differently from a reader who sees p99.
Every detector's p99 is more than ten times its p50. The median describes what most prompts experience, and a slow minority sits far above it. A latency budget for a request path needs the percentile that matches the requests you can't afford to delay.
A different thread budget or a different machine gives different numbers. The table compares these detectors when each has the same thread budget and scores one prompt at a time, and it makes no claim beyond that.
A checklist for reading any latency figure
When you read a latency figure from any vendor, including us, look for these:
- The thread budget. State the number of threads the detector used, and whether the limit was set or left at the default.
- The same budget for every detector. A comparison needs one budget across all of them.
- The batch size. One prompt per call, or a batch, and how large.
- Where the clock ran. In-process, or over the network, and what the client and the route added.
- The percentile. Median, p95, p99 or a mean. If only one is quoted, ask for the others.
- The prompts. Which dataset, how many prompts, how long they were, and whether long documents were in the pool.
- Warmup and isolation. Whether the clock started after warmup, and whether detectors shared a process.
- The hardware class. Quote nothing across machines unless the machine is stated.
A figure that answers all eight can be compared with another that does the same. A figure that leaves some unanswered describes conditions you don't know, and you can ask the vendor to fill them in.
What these numbers are
The figures in this article are benchmark figures from a controlled harness. Latency in your deployment depends on your hardware, network, load and prompt lengths, and the way to find it is to time the detector on your own traffic under a budget you state.