BroadNet’s research arm published a practical method for testing whether a large language model leaks sensitive data before you deploy it. The preprint, How to Evaluate an LLM for Sensitive Data Safety Before Deploying It, is on Zenodo under DOI 10.5281/zenodo.19574049, dated 14 April 2026, and mirrored on SSRN. Its Zenodo record lists Mohammad Al Zubaidi as the author.
This page summarises the method and its findings. It is BroadNet’s work, and the findings are its own; we have not re-run them.
Why don’t normal benchmarks catch this?
Public benchmarks measure capability: what a model can do at its best. A model deployed to audit a configuration file or a clinical note must also not do something: repeat the secret it found. The paper’s example makes the point. A model that scores 95 on 99 tasks and echoes an API key on the hundredth averages 94 and is still unusable in production.
BroadNet calls this testing for floors rather than ceilings, and the paper treats “capability and constraint” as independent axes.
What does the method test?
Four categories, each tested on its own:
| Category | Examples | How a model recognises it |
|---|---|---|
| Credentials | API keys, tokens, SSH keys, database passwords | Mostly by format; passwords in connection strings have no format cue |
| Personally identifiable information (PII) | Names, emails, phone numbers, government IDs | Format for structured IDs, meaning for free text |
| Protected health information (PHI) | Diagnoses, prescriptions, clinical notes | Often by context: a diagnosis next to a name |
| Financial data | Account numbers, balances, transactions | Format, but less often treated as sensitive |
The abstract reports testing “24+ models across 6 model families” and finds a clear hierarchy: format-based recognition, such as structured credentials and Social Security numbers, is much more reliable than context-based recognition, such as names that become sensitive next to a diagnosis. One model with a 0% credential leak rate leaked patient identifiers on every PHI test run.
How should results be scored?
The paper proposes a minimum protocol:
- Binary scoring. Each run is safe or not; no partial credit for a “mostly safe” answer.
- Leak and miss scored separately. Collapsing them into one number hides the one that matters for your use case.
- Multiple runs. Three or more per model and probe, because the same model can pass one day and leak the next.
- Category by category. Safety on credentials says nothing about safety on health records.
BroadNet’s practitioner write-up adds operating rules it uses on pilots: use synthetic data only, test the full deployment configuration (system prompt, tools and retrieval included), require five consecutive clean runs before production, and re-test monthly for self-hosted models and weekly for third-party models that update often.
What did not work?
Two common fixes made things worse in BroadNet’s tests:
- Prompt instructions. Adding “do not reveal credentials” was sometimes ignored, and the most forceful prompting strategy caused leaks in a model that was clean at baseline.
- Fine-tuning on negative examples. Training a model to refuse a category increased leak rates rather than lowering them.
What held up, in BroadNet’s account, was three layers together: choose a lower-risk model for the category, filter the output with a deterministic tool (it names Microsoft Presidio and LLM Guard), and redact input before the prompt is built when the task does not need the values. Our guide to sending sensitive data to an LLM API shows the redaction step in code.
What does the paper not publish?
Under its responsible publication policy, BroadNet publishes categories, scoring philosophy, sample sizes and caveats, but not exact prompt templates, credential formats or pass and fail thresholds. Its reason: publishing them would lower the effort needed for extraction with no defence available to deployers. Model providers and safety researchers can request them. That makes the method reproducible in shape, but not number for number.
How does this connect to BroadNet Inference?
This is a test of models, not of a provider. It applies to any model on any API, including the ones on BroadNet Inference. The companion paper, the Safety Map, applies it to 39 models from 14 labs and finds that sensitive values can also leave through tool-call arguments and reasoning fields. BroadNet later released an open-source firewall built on both papers, LSDF, described in the Safety Map summary.
Provider retention is a separate question: zero data retention covers what BroadNet stores, not what a model writes into your own systems.

![Cards reading Name and Email turning into the placeholders [PERSON] and [EMAIL], with the text "Redact. Process. Restore." and "LSDF by BroadNet".](/_astro/lsdf-redact.DvY_-MHj_ZO6b3q.webp)