PII Redaction in Speech Transcripts
Short answer. PII redaction over a transcript is a span problem: find every match, resolve the overlaps, then replace. The order matters, because a phone pattern and a longer numeric-ID pattern will match the same digits and the shorter one must not win. Masking style matters too: replacing a span with a same-length mask keeps every character offset intact, so an alignment computed before redaction stays valid. A bracketed label such as [EMAIL] reads better but shifts every offset after it.
How to do it
Collect every match before replacing any
The naive implementation loops over patterns and calls sub on the string after each one. That is wrong in two ways. Later patterns run against already-modified text, so an offset computed by the first pattern is meaningless by the time the third runs. And a short pattern can match inside a longer match, producing a redaction that splits a value in half.
The correct shape is: find all matches from all patterns, collect them as spans, resolve overlaps, then apply every replacement in one pass from the end of the string backwards. Working backwards means each replacement is applied to text whose earlier offsets are still valid.
Resolving overlaps needs a stated rule
When two patterns match overlapping spans, something has to win. The rule that works in practice is: earliest start wins, and when two matches start at the same position, the longer one wins.
The example below has a domain pattern and an email pattern. The domain pattern matches "example.com" inside the email address; the email match starts earlier, so the email wins and the domain match is dropped. Without a rule, which one survives depends on the order of patterns in a list, which is not a specification.
Masking style is a decision about alignment
A bracketed label like [EMAIL] is readable and tells a downstream reader what was removed. It also changes the character count: in the example below, 165 characters become 121, and every offset after the first redaction is wrong by the difference.
A same-length mask of asterisks keeps the count identical: 165 characters in, 165 out, and every character offset still points where it did. That matters whenever the transcript is tied to something else by offset — a forced alignment, a word-level timestamp file, or a character-indexed annotation layer.
The tradeoff is that a same-length mask leaks the length of the removed value, which is itself a weak signal about what was removed. Decide which property you need and write it into the specification.
Where regex stops being enough
Patterns handle structured identifiers — emails, phone numbers, account numbers — reliably. They do not handle names, addresses, or anything context-dependent, because those have no fixed shape. The name pattern in the example only works because it is a hard-coded literal, which is not a solution that scales.
The practical division of labour is regex for the structured cases and a human pass for the rest, with the regex applied first so the annotator is not spending attention on phone numbers.
Code
Five patterns over one transcript, with overlap resolution and both masking styles, showing how each style affects character offsets.
import re
TRANSCRIPT = (
"My name is Dana Whitfield and my number is 415-555-0132. "
"Email dana.whitfield@example.com or call the office at (212) 555-0177. "
"The account number is 8823 4410 9921."
)
PATTERNS = [
("EMAIL", r"[\w.+-]+@[\w-]+\.[\w.]+"),
("PHONE", r"\(?\d{3}\)?[ .-]?\d{3}[ .-]?\d{4}"),
("ACCOUNT", r"\b\d{4}\s\d{4}\s\d{4}\b"),
("NAME", r"\bDana Whitfield\b"),
("DOMAIN", r"[\w-]+\.(?:com|org|net)\b"),
]
spans = [(m.start(), m.end(), label, m.group())
for label, pat in PATTERNS for m in re.finditer(pat, TRANSCRIPT)]
# earliest start wins; when two matches start at the same place, the longer one wins
spans.sort(key=lambda s: (s[0], -(s[1] - s[0])))
kept, cursor = [], 0
for s in spans:
if s[0] >= cursor:
kept.append(s)
cursor = s[1]
print(f"{len(spans)} raw matches, {len(kept)} kept after overlap resolution")
for s in spans:
if s not in kept:
print(f" dropped {s[2]} {s[3]!r} at {s[0]}-{s[1]}: inside a longer match")
print(f"\n{'span':>10} {'label':<9}{'matched text':<30}{'len':>4}")
for start, end, label, text in kept:
print(f"{f'{start}-{end}':>10} {label:<9}{text:<30}{len(text):>4}")
def redact(text, spans, char_preserving):
out = text
for start, end, label, original in reversed(spans):
out = out[:start] + ("*" * len(original) if char_preserving else f"[{label}]") + out[end:]
return out
token, chars = redact(TRANSCRIPT, kept, False), redact(TRANSCRIPT, kept, True)
print(f"\noriginal {len(TRANSCRIPT):>4} chars, {len(TRANSCRIPT.split()):>3} words")
print(f"[LABEL] style {len(token):>4} chars, {len(token.split()):>3} words -> offsets shift")
print(f"asterisk style {len(chars):>4} chars, {len(chars.split()):>3} words -> offsets survive")
print(f"\n[LABEL] {token}")
print(f"asterisk {chars}")- Standard library only. The transcript is the literal at the top of the script.
- Six raw matches, five kept: the DOMAIN match on "example.com" is dropped because it sits inside the earlier-starting EMAIL match.
- The [LABEL] style turns 165 characters into 121. Both styles drop the word count from 26 to 22, because each masked span collapses to a single token. Only the asterisk style keeps 165 characters and every offset.
Where this goes wrong
Redacting with successive substitutions
Calling sub once per pattern means each pattern runs on the output of the last. Offsets shift, and one value can be matched twice by two patterns and mangled. Collect all spans first, then replace once, from the end backwards.
Letting the shorter pattern win
A phone pattern will match the first digits of a longer account number. If it wins, the redaction splits one identifier into a masked part and a visible part, and the visible remainder is still identifying. Sort by start position and prefer the longer match.
Redacting the text and leaving the audio
If the transcript is redacted but the audio still contains the spoken phone number, the PII is still in the dataset. Decide whether the audio gets a beep, a cut, or nothing, and apply the same convention across the corpus.
Replacing with a fixed token and losing the alignment
Substituting [PHONE] changes the length. Any character-indexed annotation built before the redaction is now off by the difference, silently, and the error grows with every redaction earlier in the file. In the example on this page 165 characters become 121. Use a length-preserving mask when alignment has to survive, or recompute the alignment after redaction.
Assuming the patterns cover the language
A phone pattern written for US numbers does not match German or Indian formats. A corpus in five languages needs five pattern sets, and the ones you did not write are the ones that leak.
When to buy the data instead of building the pipeline
Build the redaction pass — it is the code above and it should run on your data before anything else sees it. Buy when the redaction has to be defensible rather than merely automated: an audit trail that records what was removed and why, a human review layer over the regex output, or PII in audio rather than in text. The regex is the cheap part. The review is the part that costs money, and it is the part that decides whether the redaction holds up.
More technical reference
-
Multilingual Speech Recognition
multilingual speech recognition
-
Signal-to-Noise Ratio (SNR)
signal to noise ratio
-
Mel Spectrogram
mel spectrogram
-
Transcription Accuracy
transcription accuracy
-
Speaker Verification
speaker verification
-
Speaker Identification vs Speaker Verification
speaker identification vs speaker verification
Need data for PII Redaction in Speech Transcripts?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.