IndicPII-Bench v1
An open benchmark for Indian personal data: 2,500 documents across structured IDs, Hinglish, Hindi, Marathi, Tamil and Telugu, real bank codes, obfuscation, and look-alike numbers (order IDs, AWB, UTR) that should be left alone.
| System | Strict F1 | False alarms / 1k tokens | ms per KB | Cold start | Installed |
|---|---|---|---|---|---|
| Saaf | 0.711 | 0.35 | 0.94 | 12 ms | 0.1 MB |
| Presidio + Saaf pack | 0.620 | 2.08 | 41.6 | 1.3 s | 691 MB |
| Presidio, India recognizers on, tuned | 0.437 | 131.34 | 41.4 | 1.2 s | 691 MB |
| Bare regex | 0.466 | 245.66 | 0.15 | <1 ms | n/a |
| Presidio, default | 0.145 | 1.74 | 40.9 | 2.9 s | 691 MB |
Where every tool loses, including ours
Indian names in Hindi, Marathi, Tamil and Telugu text (slices drawn from the public Naamapadam and HiNER datasets): Saaf 0.00, Presidio 0.02 to 0.05. This is the next problem we are working on.
Read these numbers carefully. Most of the corpus is synthetic and generated by our own templates, and Saaf's rules were tuned on the same families, so Saaf's identifier scores are optimistic. Presidio was run as fairly as we could: version 2.2.364 with spaCy
en_core_web_lg, all six India recognizers enabled and its threshold tuned on a separate dev set. A human-labelled, held-out set is in progress; until then, treat this as a reproducible smoke test, not a leaderboard.