Guardrails have a latency budget

Every team that ships an LLM feature eventually has the same meeting: the guardrail is slowing everything down, can we turn it off for this path? The guardrail that gets switched off protects nothing. Speed is a safety property.

The guardrail that gets turned off

Guardrails sit in the critical path twice: on the way in, before the prompt reaches the model, and on the way out, before the answer reaches the user. Whatever they cost is paid on every request, and users feel it most in time to first token. A check that adds a noticeable pause gets an exception for the “low-risk” endpoint, then for the batch job, then for the internal tool, and eventually the protected path is the minority of traffic.

The fix is not to choose between safe and fast. It is to design guardrails in layers, so that most requests only ever pay for the cheapest checks.

Cheap checks first

• Deterministic checks, in microseconds to a millisecond: validated patterns for identifiers and secrets, allow-lists, argument schemas, policy rules. They run on CPU, on every request, inline.
• Small models, in milliseconds: compact classifiers for things patterns cannot catch, such as names in free text or likely prompt injection. They run inline when a cheap signal says they are needed.
• LLM judges, slowest: a second model reviewing the first adds at least one more model call to the request path. Use them asynchronously, in shadow mode, or for the small set of high-risk actions where a human-scale delay is acceptable anyway.

Detection also has a quality cost. Meta's LlamaFirewall cut attack success on AgentDojo from 17.6% to 1.75%, a big win, but task success fell from 47.7% to 42.7% at the same time. Every layer should be measured for what it blocks, what it breaks and what it costs in time.

What a fast check costs

Here is what the deterministic layer costs in practice, measured on our open benchmark with Saaf and Microsoft Presidio on the same machine. Presidio was configured with its spaCy model and all its recognizers on.

A cold start of over a second matters more than it looks: it is paid by every new serverless instance and every autoscaled pod, which is exactly when traffic is spiking. Small, dependency-free checks also run in places a 691 MB stack cannot, such as the browser, an edge function or a pre-commit hook.

The accuracy side of the same benchmark comes with caveats we publish alongside it: most of the corpus is synthetic and our rules were tuned on similar data. The speed and size numbers are measured, not modelled. Full method on the research page.

Don't break the stream

Output guardrails have a second latency trap: streaming. If a guardrail waits for the whole answer before checking it, the user waits for the whole answer too, and the product's main UX feature is gone.

Saaf restores masked values inside the stream. It only holds back the few characters that could be the start of a placeholder like [PAN_1]; everything else passes straight through as it arrives. Property tests split the same answer into randomly sized chunks and check the restored result is always identical. The same principle applies to any streaming check: buffer the minimum that could change the decision, never the whole response.

Write the budget down

Treat guardrail latency like any other service-level objective:

• set a budget per layer, for example under a millisecond for deterministic checks and a few milliseconds for inline models;
• measure p50 and p99 on real traffic, plus cold start, not just a warm benchmark;
• fail safely and explicitly when a check times out, per endpoint, instead of silently skipping it;
• publish the numbers, so the next “can we turn it off?” meeting starts from data.

Fast guardrails stay on. That is the whole design goal.

Sources

1. Safearly, IndicPII-Bench v1 results: latency per document, per KB, cold start and footprint for Saaf and Presidio 2.2.364 with spaCy en_core_web_lg.
2. Meta, LlamaFirewall: AgentDojo attack success and utility.
3. Safearly, Saaf source: the streaming restorer and its chunking property tests.

Latency figures are from our own benchmark run on one machine; your hardware will differ, and the scripts to rerun them are public. Checked on 27 October 2026.