CODEINTEGRITY / RESEARCHMeasured 11 September 2026 · Local comparison
Balanced accuracy, attack detection and false positive rates for five classifiers. Exact data follows in a table.
Measured results on the same external test. No model-card scores are used. Download SVG · PNG

Read the score with its tradeoffs.

Balanced accuracy gives attacks and benign prompts equal weight. Attack recall shows what gets caught; false-positive rate shows legitimate traffic that gets blocked. These models solve related but different classification tasks, and results depend on the dataset.

ModelBalanced accuracyAttack recallFalse alarmsF1ROC-AUC
Protect AI DeBERTa v290.34%81.81%1.13%87.92%97.57%
Meta Prompt Guard 2 · 86M85.69%71.92%0.54%82.68%98.59%
Meta Prompt Guard 2 · 22M84.12%69.22%0.98%80.04%96.57%
deepset DeBERTa74.62%99.53%50.29%50.99%92.19%
CodeIntegrity PromptGuard72.72%53.26%7.82%58.24%82.72%
NotInject false alarms and BIPIA-derived indirect-injection detection, reported separately.
Stress tests reveal sensitivity to benign wording and indirect instructions. Download SVG
Median and 95th-percentile local classification latency on identical prompts.
Measured on one Apple M5 Pro laptop, with full input processing. Download SVG
Methodology and limitations
  • Primary test: SafeGuard clean v2 test, 5,140 retained prompts (1,072 attacks; 4,068 benign). Removed 285 normalized matches against 480,637 unique texts across CodeIntegrity’s available training, validation and test corpus.
  • Additional tests: all 339 NotInject prompts, plus 575 distinct BIPIA-derived inputs (140 clean, 435 attacked). BIPIA is adapted to classify external content; this is not its original end-to-end benchmark.
  • Inference: pinned Hugging Face releases; native tokenizers; float32; threshold 0.5; 512-token windows with 64-token overlap; maximum malicious score across all windows. No threshold optimization or retraining.
  • Uncertainty: whiskers show 95% class-stratified example-bootstrap intervals, 2,000 replicates. They do not account for repeated templates, label noise or unknown competitor training overlap. These are public test sets, not proven unseen data for every model.
  • Scope: synthetic attacks dominate the primary test; NotInject includes non-English content; the models’ label definitions differ. The shared window policy does not test CodeIntegrity’s native 8,192-token context.
  • Ownership: run by CodeIntegrity, not independently certified. Detection scores do not establish prevention of real-world agent compromise.

Full report and reproducibility · Measured JSON · CSV

Sources: SafeGuard dataset, NotInject, Microsoft BIPIA.