Semantic Analyzer: Scoring Prompt Injections Rules Miss
Deterministic rules catch known attacks. ekmire's semantic analyzer scores what passes those rules but still looks wrong — here's how the thresholds work.
A deterministic rule catches what it was written to catch. That is precisely the problem with prompt injection: the payloads worth worrying about are the ones nobody has written a rule for yet — a novel jailbreak phrasing, an instruction buried inside a document an agent is asked to summarize, an indirect injection routed through a tool response instead of the user's own message.
ekmire's semantic analyzer exists for exactly that gap. It runs after the deterministic rule engine, only on payloads that passed rule checks but still carry signals worth a second look, and scores them 0.0 to 1.0 using an LLM. This post is about how that scoring model is built, why it runs second instead of first, and how to tune it without turning it into a system that blocks your own users.
Rules first, semantic second — the performance rationale
The deterministic engine matches an Ed25519-verified rule bundle against the request in low single-digit milliseconds. It is exhaustive for known patterns and costs almost nothing. The semantic path calls an LLM — Groq by default, or your own OpenAI, Anthropic, or Ollama endpoint on BYOK — and that costs real latency, typically in the hundreds of milliseconds to low seconds.
Running the expensive check on every request would be wasteful and slow. So the pipeline is cheap-first: rules run on everything, and only the requests that clear the rule engine but still look ambiguous move to semantic scoring. Most legitimate traffic never touches the LLM path at all.
The scoring model: what 0.85 and 0.60 actually mean
ekmire's defaults are a block threshold of 0.85 and a log threshold of 0.60. Above 0.85, the request is blocked and a semantic.threat_detected event fires. Between 0.60 and 0.85, the request passes but gets logged as semantic.logged for review. Below 0.60, nothing happens — it's treated as noise.
These numbers aren't arbitrary, and they land close to what published research is converging on. PromptGuard, a structured injection-resilience framework published in Scientific Reports, found its semantic validation layer hit the best precision-recall tradeoff at a 0.78 cosine similarity threshold. A separate visual prompt-injection study used 0.85 to define a successful targeted injection at the action-generation stage — the same number ekmire uses as a block threshold, arrived at independently. Other 2026 LLM firewall architectures use point-based systems instead of a 0–1 score, with a default trigger around 40 out of 100 — a different scale, the same underlying idea of an accumulated-evidence threshold rather than a single binary rule match.
The takeaway isn't that any of these numbers are "correct" in some universal sense — it's that the industry has converged on the same shape of solution: score payloads on a continuous scale, and pick a threshold based on your own false-positive tolerance rather than a fixed cutoff borrowed from someone else's traffic.
What the semantic path catches that rules miss
Three categories show up most often in ekmire's logged-but-not-blocked tier, which is exactly where you'd expect novel attacks to surface first:
- Novel jailbreak phrasing. Variations on "ignore previous instructions" that reword the instruction-override intent well enough to dodge a literal string or regex match, but not well enough to fool a model reading it for intent.
- Indirect injection. Instructions embedded in a document, a webpage, or a tool's response that an agent is asked to process — not the user's own input at all. Rules matching the user-facing request miss this entirely because the malicious content never appears there.
- Obfuscated payloads. Unicode homoglyphs, unusual whitespace, or split-token phrasing designed to break a literal pattern match while still reading clearly to an LLM.
This last category is getting more attention for a reason: research published in early 2026 found that as few as five carefully crafted documents can manipulate a RAG pipeline's responses roughly 90% of the time — a reminder that indirect, document-borne injection isn't a theoretical edge case anymore.
Fallback design: what happens when the LLM is slow
A security check that can hang a request indefinitely is worse than no check at all. The semantic path has a 1,500ms timeout with a configurable fallback — pass by default. If the LLM call doesn't return in time, the request goes through rather than stalling. You can flip this to block per project if failing closed matters more to your application than availability, but the default favors not breaking real traffic over a provider hiccup.
# project-level overrides (dashboard, or proxy env for self-hosted)
EKMIRE_SEMANTIC_BLOCK_THRESHOLD=0.90
EKMIRE_SEMANTIC_LOG_THRESHOLD=0.55
EKMIRE_SEMANTIC_TIMEOUT_MS=1500
EKMIRE_SEMANTIC_TIMEOUT_FALLBACK=passTuning thresholds for your own traffic
The default 0.85 / 0.60 split is a reasonable starting point, not a fixed answer. If your users are developers writing about security research, code containing attack strings, or prompts that legitimately reference "ignore" or "instructions" in a non-adversarial way, the default block threshold will produce false positives. Two adjustments matter more than the raw numbers:
- Start wider, then tighten. Set the block threshold higher than default (0.90–0.95) for the first week in a new deployment, watch what lands in the logged tier, and lower it once you've confirmed the logged events are actually attacks and not just unusual phrasing from real users.
- Use the false-positive button. Every logged or blocked event has a false-positive marker in the dashboard. Marked events feed the crowd-sourced threat candidate pipeline, and a rule's own suppression rate is tracked automatically — a threshold that's clearly too sensitive shows up in that data rather than staying a guess.
Three things to take away
- Semantic scoring is a second pass, not a replacement. Rules still catch everything they were built to catch, at effectively zero cost. Semantic analysis exists for the traffic that gets past them.
- Your thresholds don't need to match anyone else's. Published research clusters around 0.75–0.85 for good reason, but the right number for your app depends on what your real traffic looks like — tune from your own logged events, not a blog post.
- A slow LLM should never be a denial-of-service vector against your own users. The 1,500ms timeout with a pass fallback is what keeps a provider outage from becoming your outage.
The detection pipeline docs cover the full seven-step request flow and where semantic scoring sits in it. The false-positive workflow docs cover how marked events feed back into threshold tuning and the wider rule bundle. For background on the write-time side of prompt injection — the same attack class showing up inside AI coding IDEs — see our post on prompt injection in Cursor, Copilot, and Claude Code.