# PromptGuard benchmark — measured 11 September 2026

CodeIntegrity PromptGuard was downloaded from its official Hugging Face repository and evaluated locally against four established open-weight classifiers. The run contains **30,270 classifications: five models × 6,054 identical inputs**, plus separate latency measurements. All downloaded file hashes match the publisher metadata.

**Result:** CodeIntegrity records **72.72% balanced accuracy**, **53.26% attack recall** and **7.82% false positives** on the external SafeGuard clean v2 test after removing known corpus overlap. Protect AI v2 scores highest on this test at **90.34% balanced accuracy**. CodeIntegrity has the lowest measured median latency on this laptop sample, **9.88 ms**, but this does not compensate for missed attacks.

![Measured comparison](promptguard-benchmark.svg)

## Primary external test

5,140 retained inputs: 1,072 attacks and 4,068 benign. Balanced accuracy averages attack recall and benign specificity; an always-benign classifier scores 50%. All thresholds are 0.5.

| Model | Balanced accuracy (95% interval) | Attack recall | False positives | Precision | F1 | ROC-AUC |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Protect AI DeBERTa v2 | 90.34% (89.22–91.49) | 81.81% | 1.13% | 95.02% | 87.92% | 97.57% |
| Meta Prompt Guard 2 · 86M | 85.69% (84.41–87.09) | 71.92% | 0.54% | 97.23% | 82.68% | 98.59% |
| Meta Prompt Guard 2 · 22M | 84.12% (82.80–85.58) | 69.22% | 0.98% | 94.88% | 80.04% | 96.57% |
| deepset DeBERTa | 74.62% (73.78–75.43) | 99.53% | 50.29% | 34.28% | 50.99% | 92.19% |
| CodeIntegrity PromptGuard | 72.72% (71.17–74.28) | 53.26% | 7.82% | 64.23% | 58.24% | 82.72% |

CodeIntegrity catches **571 of 1,072 attacks**, misses **501**, and incorrectly flags **318 of 4,068 benign inputs**. Its ordinary accuracy is 84.07%; that larger number reflects the benign-heavy class distribution. These measurements are not directly comparable to its published 98.01% accuracy on a different internal test.

## Separate stress tests

| Model | NotInject false positives ↓ | BIPIA-derived attack recall ↑ | BIPIA-derived false positives ↓ | BIPIA-derived balanced accuracy ↑ |
| --- | ---: | ---: | ---: | ---: |
| Protect AI DeBERTa v2 | 42.77% | 19.08% | 20.71% | 49.18% |
| Meta Prompt Guard 2 · 86M | 6.49% | 1.15% | 0.00% | 50.57% |
| Meta Prompt Guard 2 · 22M | 7.67% | 4.60% | 0.00% | 52.30% |
| deepset DeBERTa | 70.50% | 100.00% | 100.00% | 50.00% |
| CodeIntegrity PromptGuard | 46.02% | 10.80% | 1.43% | 54.69% |

NotInject contains 339 benign prompts deliberately using words that can trigger detectors, including multilingual prompts. CodeIntegrity flags **156/339**. This is a false-alarm test, not an attack-recall test.

The BIPIA-derived suite contains 575 distinct inputs: 140 clean contexts and 435 attacked variants. It uses official email, table and code test contexts, with a test attack placed at the start, middle and end. All models struggle to distinguish clean and attacked external content at this threshold. In particular, deepset’s 100% attack recall comes with **100% false positives**, so it is not successful discrimination. This adapted classifier test is not the official BIPIA end-to-end attack-success benchmark.

## Measured local latency

Apple M5 Pro, 24 GB unified memory; PyTorch 2.14.0, Transformers 4.57.6, float32, MPS. Same seeded 100 SafeGuard prompts, sequential batch size 1, five warm-up prompts. Timing includes tokenization, all windows, transfer and device synchronization; excludes downloads and model loading.

| Model | Median (p50) | p95 | Total loaded parameters |
| --- | ---: | ---: | ---: |
| CodeIntegrity PromptGuard | 9.88 ms | 13.45 ms | 149,606,402 |
| Meta Prompt Guard 2 · 22M | 12.26 ms | 16.24 ms | 70,830,722 |
| Protect AI DeBERTa v2 | 13.44 ms | 16.19 ms | 184,423,682 |
| Meta Prompt Guard 2 · 86M | 13.64 ms | 28.43 ms | 278,810,882 |
| deepset DeBERTa | 16.57 ms | 19.24 ms | 184,423,682 |

Meta’s 86M and 22M names describe backbone sizes; the complete loaded parameter counts also include embeddings. These latency measurements apply to this laptop, software version and natural-length sample. They do not establish A100, production-server, sustained throughput or fixed-512-token performance. The GPU benchmark does not isolate other laptop activity.

## Methodology and limits

1. **Frozen inputs and checkpoints.** Model revisions, downloaded file hashes, source dataset revisions, input IDs, exclusions and every prediction are saved. A protocol was written before inference. No model was retrained and no thresholds were chosen using these test scores.
2. **Decontamination.** SafeGuard’s clean v2 root `test/data-00000-of-00001.arrow` has 5,425 rows, matching its dataset card. The other `data/test/` artifact has different counts and was not used. Removed 285 normalized matches against 480,637 distinct texts in CodeIntegrity’s available train/validation/test files. Normalization: Unicode NFKC, casefold, zero-width removal and collapsed whitespace. No NotInject overlaps were found. BIPIA preparation removed 25 within-suite duplicates.
3. **Scope of overlap checking.** Exact/normalized matches are removed, but shared templates, paraphrases, attacks embedded inside longer texts, undocumented training data and competitor training overlap can remain. The available local corpus is not proof of every historical training input. SafeGuard was published after CodeIntegrity’s current checkpoint, but its constituent instruction data can be older. Do not describe these as guaranteed unseen data for all five models.
4. **Complete input coverage.** Every model uses its native tokenizer and 512-token windows including special tokens, with 64-token overlap. A prompt is flagged if any window reaches 0.5 malicious probability. This standardizes scanning and retains trailing attacks, but does not evaluate CodeIntegrity’s native 8,192-token mode. Model-specific tokenization means character boundaries differ.
5. **Class mapping and numerics.** CodeIntegrity, Protect AI and deepset class names are verified from their configurations. Meta’s configs use generic names; index 1 is verified against Meta’s official inference utility. Probability ranking uses the maximum malicious-minus-benign logit margin to avoid softmax saturation; decisions still correspond to probability ≥0.5. CPU/MPS smoke logits agree for each model.
6. **Labels and sampling.** SafeGuard is largely a synthetic-attack/instruction dataset, not live attack traffic. Labels are accepted as supplied, without score-driven relabeling. Spot checks found some context-dependent/ambiguous synthetic examples. BIPIA sampling uses seed 20260911, up to 50 contexts per task and a shuffled test-attack pool; middle insertion occurs at a halfway line boundary. Neither test measures whether a downstream agent would actually be compromised.
7. **Uncertainty.** Primary error bars are 2,000 class-stratified empirical example-bootstrap replicates with a fixed seed. For balanced accuracy, resampling each class’s correctness indicators is equivalent to binomial draws. These intervals omit template clustering, label uncertainty and training contamination. No claim of statistically significant ordering between close scores is made. BIPIA variants share contexts and attack families.
8. **Verification.** A runnable integrity check reconstructs complete long-token inputs from overlapping windows, verifies identical IDs across all five predictions, checks SHA-256 hashes, independently recomputes confusion counts and confirms every summary score. All checks passed.

## Website interpretation

The evidence supports a transparent research graphic about detector tradeoffs. It does **not** support “best prompt-injection classifier,” “98% accurate against modern attacks,” or broad protection guarantees. A defensible caption is:

> We evaluated five open-weight prompt-injection classifiers on the same public test inputs. Results show a tradeoff between detection, false alarms and latency. CodeIntegrity PromptGuard achieved 72.7% balanced accuracy on this external test and a 9.9 ms median on our Apple M5 Pro latency sample.

Keep the dataset, date, threshold, hardware and limitations beside the chart. The main chart must not be relabeled as an industry-wide or independent benchmark.

## Reproduce and audit

The local benchmark package contains `download_models.py`, `prepare_data.py`, `benchmark.py`, `analyze.py`, `verify.py`, `render.py`, `report.py`, `requirements.lock.txt`, the frozen evaluation file, raw predictions and run metadata. `README.md` has commands. Models live in the ordinary Hugging Face cache. The website bundle includes only figures, aggregate results and methodology; no credentials or raw prompt texts.

| Model | Immutable Hugging Face revision |
| --- | --- |
| [protectai/deberta-v3-base-prompt-injection-v2](https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2/tree/90c9989b1a342275dd0d1a95aad283c04e075671) | `90c9989b1a342275dd0d1a95aad283c04e075671` |
| [meta-llama/Llama-Prompt-Guard-2-86M](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M/tree/a8ded8e697ce7c355e395a0df51f94adb4a2fd27) | `a8ded8e697ce7c355e395a0df51f94adb4a2fd27` |
| [meta-llama/Llama-Prompt-Guard-2-22M](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-22M/tree/11614a155199674a0a95e6602d6ab0417b790ed0) | `11614a155199674a0a95e6602d6ab0417b790ed0` |
| [deepset/deberta-v3-base-injection](https://huggingface.co/deepset/deberta-v3-base-injection/tree/80dda00d0b0d9a03917a7685e2ddbcd28e04dbb1) | `80dda00d0b0d9a03917a7685e2ddbcd28e04dbb1` |
| [codeintegrity-ai/promptguard](https://huggingface.co/codeintegrity-ai/promptguard/tree/c66a9b35c82079773c3119f48caf19be18c6ca22) | `c66a9b35c82079773c3119f48caf19be18c6ca22` |

## Sources

- [CodeIntegrity PromptGuard model card](https://huggingface.co/codeintegrity-ai/promptguard). Model identity and intended use; its reported scores are not used as benchmark results.
- [SafeGuard clean v2 dataset](https://huggingface.co/datasets/jcanode/safeguard-prompt-injection/tree/61fbe3588450fa9b47ac1176ca7b5d2cc932344c), jcanode, Apache-2.0.
- [NotInject dataset](https://huggingface.co/datasets/leolee99/NotInject/tree/847ae76cf8fea5ed325429e569ae8cfef022d2e0), Hao Li et al., MIT; [PIGuard paper](https://aclanthology.org/2025.acl-long.1468/).
- [Microsoft BIPIA](https://github.com/microsoft/BIPIA/tree/a004b69ec0dd446e0afd461d98cb5e96e120a5d0), Jingwei Yi et al. The code is MIT; underlying email, table and code contexts retain their original source terms. Context sources: OpenAI Evals, WikiTableQuestions and Stack Exchange. Added attacks at three positions; raw contexts are not included in the website bundle.
- [Meta’s official Prompt Guard 2 inference](https://github.com/meta-llama/llama-cookbook/blob/main/getting-started/responsible_ai/prompt_guard/inference.py), verifying malicious class index 1.
- [PINT benchmark notebook](https://github.com/CheckPointSW/pint-benchmark/blob/0efab3f463eae9c823130d8faffb71b2e7c06e63/benchmark/pint-benchmark.ipynb): the complete PINT dataset requires research access and was not available. Its public example YAML and third-party published scores were not substituted.