Custodian Labs · Guardian Layer
Does the privacy swap
keep data usable?
这个隐私替换,
会不会让数据没法用?
A structure-preservation audit — 1,750 documents · 11 detectors · 7 benchmarks · 7 languages.
结构保持性审计 —— 1,750 文档 · 11 个检测器 · 7 个 benchmark · 7 种语言。
01 · What it does它做什么
It swaps real info for a realistic fake — not a black box.
把真实信息换成逼真的假数据 —— 不是涂黑。
Anna S. · April 12 2023 · Methodist Hospital
Maria S. · March 13 2021 · Methodist Hospital
Name → name, date → date. The sentence still reads normally, so anything that processed the original keeps working on the result.
名字换名字、日期换日期。句子读起来仍然正常,原来能处理它的系统照样能用。
02 · How we evaluate评测是怎么做的
Seven datasets, one unified input/output contract.
7 个数据集,用同一套输入输出协议评测。
Datasets数据集:
ASQ-PHI (EN)MEDDOCAN (ES)PII-300k EN
PII-300k NLPII-300k FRPII-300k DE
MultiCoNER v2 (EN)
Input per document每条输入: doc_id, text, gold_spans[start,end,label,text].doc_id、text、gold_spans[start,end,label,text]。
System output系统输出: predictions[system].spans + per-doc score(tp/fp/fn).predictions[system].spans + 文档级 score(tp/fp/fn)。
Aggregate output汇总输出: precision, recall, f1, leakage_rate, char_leakage_rate.precision、recall、f1、leakage_rate、char_leakage_rate。
Flow流程: benchmark loader → system predict → span match (type mode) → JSONL + summary.benchmark loader → 系统预测 → span 匹配(type 模式) → JSONL + summary。
Input输入
doc_id: "asq_001"
text: "Patient John Smith visited on 2024-03-09."
gold_spans: [
{start: 8, end: 18, label: "NAME", text: "John Smith"},
{start: 30, end: 40, label: "DATE", text: "2024-03-09"}
]
Per-doc output单条输出
predictions.openai_gpt.spans: [
{start: 8, end: 18, label: "NAME"},
{start: 30, end: 40, label: "DATE"}
]
score: {tp: 2, fp: 0, fn: 0}
Summary汇总
[{
system: "openai_gpt",
benchmark: "asq_phi",
n_docs: 250,
precision: 0.66,
recall: 0.84,
f1: 0.74
}]
Current public dashboard reports full 250-doc runs per benchmark for side-by-side model comparison.
当前公开看板展示的是每个 benchmark 的 250 条完整评测,用于模型横向比较。
03 · The result结果
±2 pts
statistically equivalent to zero与零统计等价
On the 57,112 masked spans, detector recall is 76.1% → 74.9% (−1.2 pts). An equivalence test (TOST, p≈3e-9) proves the change is within ±2 points of no change at all — too small to matter.
在 57,112 个脱敏 span 上,检测器召回率 76.1% → 74.9%(−1.2 个百分点)。等价检验(TOST,p≈3e-9)证明:变化在 ±2 个百分点内与零等价 —— 小到不具实际意义。
04 · Every detector agrees每个检测器都一致
| Detector检测器 |
Surrogate still found替身仍被找到 |
Recall on masked spans, transformed ÷ original. 93–100% across all 11 systems; ranking unchanged.
脱敏 span 上的召回保留率(transformed ÷ original)。11 个系统全部 93–100%,排名不变。
05 · Holds across formats & languages跨格式、跨语言都成立
70yo M, Dr. John L., Mt. Sinai, Feb 21 2023
73yo M, Dr. James L., Mt. Egypt, Nov 19 2021 clinical shorthand临床速记
DE: … Monsignore … 23/07/2011
… Fulgenzio … 24/07/2011 German德语
JSON: {"Date":"20/05/2022","City":"Saint-Priest"}
{"Date":"21/05/2023","City":"Saint-Priest"} structured data结构化数据
Only the private value changes — the format is preserved byte-for-byte.
只有敏感值变了 —— 格式逐字节保留。
06 · Where the small gap comes from那一点小差距来自哪里
It’s surrogate quality — not detector failure.
是替身质量问题 —— 不是检测器变差。
Chicago → Illino El Paso → El truncated / garbled残缺 / 乱码
Cedars-Sinai → Vidant famous → obscure著名 → 冷门
nachorutor@… → nxxxxxxxxx@… x-masked IDID 被 x 化
3,300 of 57k spans lost (50% same length — not a boundary effect). All three causes are fixable transform-side defects.
5.7 万个里丢失 3,300 个(50% 长度不变 —— 不是边界问题)。三类都是可修复的替身生成缺陷。
07 · How much it masks (honest)它脱敏了多少(诚实)
Strong on names & dates; ~half on IDs & addresses.
名字、日期藏得好;ID 和地址约一半。
Names名字75%
Dates日期73%
ID numbersID 号码51%
Addresses地址44%
Coverage on clinical PHI. Reported separately from utility — the ±2 pts above is measured only on spans it actually masks.
临床 PHI 上的覆盖率。与效用分开报告 —— 上面的 ±2 pts 只在它确实脱敏的 span 上测量。
08 · Bottom line结论
Protects privacy. Doesn’t break the data.
保护了隐私,又没破坏数据。
✓ Detectability statistically equivalent (±2 pts) ✓ Ranking unchanged ✓ Format preserved
Next: lift masking coverage on IDs & addresses.
✓ 可检测性统计等价(±2 pts) ✓ 排名不变 ✓ 格式保留
下一步:提升 ID 和地址的脱敏覆盖率。
Full data, report & per-system results → custodianai.pages.dev
完整数据、报告与逐系统结果 → custodianai.pages.dev