BroadRouter

The Safety Map: how 39 LLMs handle sensitive data

By the BroadNet Inference team · 3 min read · Published · Updated

The Safety Map is a BroadNet preprint from April 2026 (DOI 10.5281/zenodo.19688433). It tested 39 models from 14 labs on 7 probes across credentials, PII, health data, account numbers and DLP scanning, in about 1,300 multi-run evaluations. Safety did not transfer between categories or output channels: 20 of 73 runs that filled tool-call arguments leaked through them.

The Safety Map is BroadNet’s survey of how large language models handle sensitive data. The preprint, The Safety Map: What Does and Doesn’t Transfer in LLM Sensitive-Data Handling, was published on Zenodo on 22 April 2026 under DOI 10.5281/zenodo.19688433, with Mohammad Al Zubaidi as author. It is the companion to BroadNet’s sensitive-data evaluation method.

What follows is our summary of the abstract and BroadNet’s research page. The verdicts are BroadNet’s.

What did the survey cover?

Dimension Coverage
Models 39, from 14 independent labs
Probe scenarios 7
Categories Credentials, PII, protected health information (PHI), financial account numbers, data-loss-prevention (DLP) scanning
Evidence 135+ classified model-probe combinations from about 1,300 multi-run evaluations

BroadNet’s catalogue says it has pushed about 90 remote and API models, which it never ran on its own hardware, through its safety, routing and Arabic test harnesses, and that those findings feed this map.

What are the main findings?

Safety is a surface, not a single score. It varies along three axes that stack:

  • Architecture. The paper reports a capacity floor of about 24B active parameters on credentials, across routing designs, with small-active mixture-of-experts models as the main case.
  • Generation. Alignment quality moves the boundary within a model family.
  • The lab’s alignment investment. A flagship label from a well-known lab is not a guarantee.

None of the three predicts safety on its own; they combine.

Safety does not transfer between categories. Handling credentials and Social Security numbers cleanly did not carry over to names, health records or account numbers in the same model.

Output channels leak independently. This is the finding with the most practical weight:

  • Of 73 runs that filled tool_calls.arguments, 20 sent sensitive values through the tool call. Among the models behind those runs was Moonshot’s Kimi K2.6, tested on its release day, a model the paper rates otherwise multi-run safe on credentials, PII, financial data and DLP.
  • Of 43 runs in which the provider filled reasoning_content with at least 20 characters, 29 leaked sensitive values into the reasoning while the chat reply was rated safe, missed or truncated.

Pruning can break safety. Two independent pruned versions of Meta’s Llama line, made by NVIDIA with neural architecture search, broke health-data safety through different output channels.

The paper ends with a five-tier ranking. Its top cluster, uniformly safe on every category tested, is Claude Opus 4.7, GPT-5.4 and GPT-5.4-mini, Gemini 3 Flash and Gemini 3.1 Pro. The abstract says xAI Grok and Meta Llama flagships do not make that cluster.

What should you change in your own stack?

  1. Filter every channel. A check on the visible reply misses tool-call arguments and reasoning fields. If you use tool calling or a reasoning model, inspect both.
  2. Test by category. A clean credential result says nothing about health or salary data.
  3. Retest on model updates. The Kimi K2.6 result came from a day-of-release model.
  4. Redact first where you can. A model cannot leak what it never saw. Our sensitive data guide shows placeholder redaction in Python.

What is LSDF?

LSDF, the LLM Sensitive Data Firewall, is the control BroadNet built from these two papers and released as open source under Apache-2.0 at github.com/brntech/lsdf. It is a reverse proxy for the OpenAI-compatible chat completions API: you point your client’s base URL at it, and it forwards to your model server or provider. BroadNet says it inspects requests, responses, streamed chunks, streamed tool-call arguments, retrieved context and reasoning fields, and sanitises traces and logs separately.

BroadNet’s May 2026 evaluation on its own corpora reports this recall:

Detector stack Credential replay Clinical PHI Broad English PII
Regex only 0.944 0.019 0.208
Microsoft Presidio 0.194 0.056 0.326
LSDF broad-PII detectors 1.000 0.944 0.685

These are a vendor’s numbers for its own tool, on corpora it chose. They are useful as a starting point, and the repository ships the replay so you can check them. LSDF works in front of any model, including ones BroadNet does not host.

What are the limits?

The paper is a preprint, not peer-reviewed. Under BroadNet’s responsible publication policy, exact prompts, credential formats and thresholds are withheld from the public version and shared with model providers and researchers on request. Model versions change fast, so a verdict from April 2026 is a reason to test, not a substitute for testing.

Questions

Which models did best in the Safety Map?

The abstract places a frontier-alignment cluster at the top, uniformly safe across every category tested: Claude Opus 4.7, GPT-5.4 and GPT-5.4-mini, Gemini 3 Flash and Gemini 3.1 Pro. Claude Sonnet 4.6 sits just below, with one date-of-birth-only health-data leak across 15 runs.

Is a model that never leaks API keys safe with patient data?

Not necessarily. The paper found that safety dissociates by category: a model that protected credentials, Social Security numbers and database passwords could still name employees in a salary document on every run.

Why do tool calls matter?

Because many filters only check the chat reply. Of 73 runs in which the model filled tool-call arguments, 20 sent sensitive values through that channel, including in a model that was otherwise multi-run safe.

Do small mixture-of-experts models leak more?

The paper reports a capacity floor of about 24B active parameters on credentials, with small-active mixture-of-experts designs the main expression of it. BroadNet's practitioner post says it has yet to find an MoE model under 24B active parameters that does not leak on its structured-data probes.

Are the models on BroadNet Inference in the Safety Map?

The abstract does not list per-model verdicts for them, and the preprint dates from April 2026, so a model released later cannot be in it. Gemma 4 VL 26B has about 4B active parameters, so test it on your own sensitive-data probes, or redact before sending, if it will see credentials or personal data.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity