Using disarm in LLM pipelines¶

The LLM ecosystem rebuilds disarm's functionality ad hoc — LiteLLM ships a hand-written normaliser, Haystack has ascii_only, every tokenizer wrapper has its own accent-stripping. The survey behind this page found the functionality gap is small; the documentation gap is the whole problem. normalize_confusables() is usually presented as a transliteration utility, when for this audience it is a guardrail primitive.

This page frames disarm's existing transforms for two LLM jobs — guardrail matching (filtering untrusted input) and ingestion (normalising content for ASCII indexes) — and is explicit about which path each recipe belongs to. Every snippet is executed and asserted in CI.

The NFKC-first convention¶

Matching frameworks normalise before they compare, because an attacker controls the encoding of a string, not just its letters. Most of disarm's defense functions follow the same convention — NFKC is their first step — so they are safe to call on raw, untrusted input:

from disarm import strip_obfuscation

# Fullwidth letters (NFKC-folded) and zero-width joiners (stripped):
assert strip_obfuscation("Ｈｅｌｌｏ") == "Hello"
assert strip_obfuscation("h\u200bi") == "hi"

use disarm::api;

// Fullwidth letters (NFKC-folded) and zero-width joiners (stripped):
assert_eq!(api::strip_obfuscation("Ｈｅｌｌｏ").unwrap(), "Hello");
assert_eq!(api::strip_obfuscation("h\u{200b}i").unwrap(), "hi");

require "disarm"

# Fullwidth letters (NFKC-folded) and zero-width joiners (stripped):
Disarm.strip_obfuscation("Ｈｅｌｌｏ")   # => "Hello"
Disarm.strip_obfuscation("h\u200bi")       # => "hi"

import { stripObfuscation } from 'disarm'

stripObfuscation('Ｈｅｌｌｏ') // => 'Hello'
stripObfuscation('h\u200Bi') // => 'hi'

You do not need to pre-normalise before handing text to strip_obfuscation() or canonicalize(); they start from NFKC themselves.

from disarm import normalize_confusables, strip_obfuscation

normalize_confusables("fi")  # 'fi'  — in the table
normalize_confusables("Ａ")  # 'A'   — in the table
normalize_confusables("²")  # '²'   — NOT in the table, and NFKC would give '2'

strip_obfuscation("²")  # '2'   — this one is NFKC-first

Guardrail primitives¶

For filtering untrusted input — denylist checks, prompt-injection screening, policy matching — the primitives are strip_obfuscation() (full deobfuscation) and normalize_confusables() (TR39 visual fold only).

The key difference from a LiteLLM-style hand-rolled normaliser is what disarm refuses to do: it does not apply leet/digit remapping. Digit remapping corrupts the numeric text that pervades an LLM stack — model names, versions, quantities — so 4, 0, 1 are left alone:

from disarm import normalize_confusables

# Identifiers and version numbers survive untouched:
assert normalize_confusables("gpt-4o") == "gpt-4o"
assert normalize_confusables("Llama-3.1-70B") == "Llama-3.1-70B"

# But cross-script homoglyph spoofs are folded to their Latin skeleton:
assert normalize_confusables("pаypаl") == "paypal"  # Cyrillic а → a

use disarm::api::{self, TargetScript};

// Identifiers and version numbers survive untouched:
assert_eq!(api::normalize_confusables("gpt-4o", TargetScript::Latin), "gpt-4o");
assert_eq!(api::normalize_confusables("Llama-3.1-70B", TargetScript::Latin), "Llama-3.1-70B");

// But cross-script homoglyph spoofs are folded to their Latin skeleton:
assert_eq!(api::normalize_confusables("pаypаl", TargetScript::Latin), "paypal"); // Cyrillic а → a

require "disarm"

# Identifiers and version numbers survive untouched:
Disarm.normalize_confusables("gpt-4o")        # => "gpt-4o"
Disarm.normalize_confusables("Llama-3.1-70B") # => "Llama-3.1-70B"

# But cross-script homoglyph spoofs are folded to their Latin skeleton:
Disarm.normalize_confusables("pаypаl")        # => "paypal"  (Cyrillic а → a)

import { normalizeConfusables } from 'disarm'

normalizeConfusables('gpt-4o') // => 'gpt-4o'
normalizeConfusables('Llama-3.1-70B') // => 'Llama-3.1-70B'
normalizeConfusables('pаypаl') // => 'paypal'

What TR39 covers instead of leet tables is visual confusability: a Cyrillic а that renders identically to Latin a folds to a. See Confusable Detection for the table and its limits.

Recipe — guardrail matching key:

from disarm import get_pipeline

guardrail = get_pipeline("llm_guardrail")
# NFKC → strip zalgo/bidi → demojize → strip accents → confusables →
# fold case → strip control/zero-width → collapse whitespace
assert guardrail("Ѕ𝗲𝗰𝗿𝗲𝘁  \u200bdata") == "secret data"

Compare the matched key, not the raw string, against your policy.

Convert, don't delete¶

The other common pattern is NFKD + encode("ascii", "ignore") (Haystack's ascii_only, Whisper's text cleaner). On non-Latin content this deletes the text — an ASCII index built that way simply loses the document:

from disarm import transliterate

passage = "Привет мир"
# ascii-ignore throws the whole passage away:
assert passage.encode("ascii", "ignore") == b" "
# transliterate keeps it, searchable, as readable romanisation:
assert transliterate(passage) == "Privet mir"

use disarm::api;

// transliterate keeps non-Latin content, searchable, as readable romanisation:
assert_eq!(api::transliterate("Привет мир"), "Privet mir");

require "disarm"

# transliterate keeps non-Latin content, searchable, as readable romanisation:
Disarm.transliterate("Привет мир")   # => "Privet mir"

import { transliterate } from 'disarm'

transliterate('Привет мир') // => 'Privet mir'

This is the wedge for index/retrieval: non-Latin content stays findable in an ASCII-normalised index without losing its semantics.

Recipe — ingestion / RAG index normalisation:

from disarm import get_pipeline

ingest = get_pipeline("rag_ingest")
# NFKC → strip bidi → strip accents → transliterate →
# strip control/zero-width → collapse whitespace
assert ingest("Café déjà vu") == "Cafe deja vu"
assert ingest("Привет, мир!") == "Privet, mir!"

A composed entry point¶

If you want one function, compose the two paths and keep transliteration optional — it belongs to the index/matching paths, never to a prompt you are about to send to a model that reads the original script fine:

from disarm import get_pipeline


def prepare_for_llm(text, *, romanize=False):
    """Normalize untrusted text for an LLM index / matching path.

    romanize=False → guardrail fold (homoglyph + obfuscation defense).
    romanize=True  → ingestion romanisation for an ASCII index.
    Either way this is for indexing/matching, not for rewriting prompts.
    """
    profile = "rag_ingest" if romanize else "llm_guardrail"
    return get_pipeline(profile)(text)


assert prepare_for_llm("pаypаl") == "paypal"  # guardrail
assert prepare_for_llm("Москва", romanize=True) == "Moskva"  # ingestion

Case, and what folding it costs a cased model¶

The ml_normalize preset folds case. That suits the uncased tokenizers most pipelines use, and it is the right default. In front of a cased model it is a measurable loss, and one an uncased evaluation harness cannot see — the fold happens before the model, so the harness scores text that has already lost the signal.

fold_case=False drops that one step and leaves every other stage running:

from disarm import ml_normalize, normalize_confusables

assert ml_normalize("José Martínez") == "jose martinez"
assert ml_normalize("José Martínez", fold_case=False) == "Jose Martinez"

Two things to be clear about before reaching for it.

It restores case, not diacritics. strip_accents is a separate step and still runs, which is why the second line reads Jose and not José. If the diacritics also have to survive, ml_normalize is the wrong entry point — use normalize_confusables, which folds homoglyphs and touches nothing else:

assert normalize_confusables("José Martínez") == "José Martínez"

ml_normalize is not a homoglyph defence. Its pipeline has no TR39 step, so it recovers nothing at either setting. This is the trap the name invites:

# Cyrillic С (U+0421) survives ml_normalize either way …
assert ml_normalize("fu\u0421k", fold_case=False) == "fu\u0421k"
# … and is recovered by the confusable primitive.
assert normalize_confusables("fu\u0421k") == "fuCk"

Put normalize_confusables (or the llm_guardrail profile) in front of ml_normalize when a cased model needs both. The full threat-model-to-entry-point table, with what each choice costs, is in what each entry point costs you.

Two more gaps behind the same name¶

Homoglyphs are the best-known of ml_normalize's blind spots, not the only ones. Its pipeline is NFKC, emoji, transliterate, strip_accents, emoji, fold_case, strip_control, strip_zero_width, collapse_whitespace. strip_control covers the C0 and C1 controls, which are category Cc. Bidi controls are Cf, so they fall straight through, and nothing in the list touches the Private Use Area.

BIDI = "".join(
    chr(c)
    for c in (
        0x202A,
        0x202B,
        0x202C,
        0x202D,
        0x202E,
        0x2066,
        0x2067,
        0x2068,
        0x2069,
        0x200E,
        0x200F,
        0x061C,
    )
)

# All twelve bidi controls survive.
assert [c for c in BIDI if c in ml_normalize(f"a{c}b")] == list(BIDI)

# So does a PUA code point.
assert ml_normalize("Summarize.\U000f0000") == "summarize.\U000f0000"

It does remove zero-width fragmentation, and most of the Tags block — but not all of it. U+E0061–U+E007A and the cancel tag go, because the emoji step consumes tag sequences; U+E0001 LANGUAGE TAG survives. Partial coverage of a class is what makes the whole look more complete than it is.

assert ml_normalize("Summarize.\U000e0061") == "summarize."  # tag letter: removed
assert ml_normalize("Summarize.\U000e0001") != "summarize."  # LANGUAGE TAG: survives

None of this makes ml_normalize broken. It is a tokenizer-hygiene preset, and THREAT_MODEL.md never lists it as a security mechanism. But the name reads as "the preset for ML input", which is exactly the pipeline position where a surviving bidi control or PUA code point matters. Reach for a profile when the text is untrusted.

from disarm import get_pipeline

guardrail = get_pipeline("llm_guardrail")
assert not any(c in guardrail(f"a{c}b") for c in BIDI)  # bidi handled
assert guardrail("Summarize.\U000f0000") == "summarize.\U000f0000"  # PUA still not

assert get_pipeline("rag_ingest")("Summarize.\U000f0000") == "Summarize."  # PUA handled

Code is not prose: code_context, and strip-and-report¶

Every preset and both LLM profiles above end in collapse_whitespace, which folds LF to a space by design. That is right for a prompt and wrong for a source file: measured over the 465 files of this repository, all thirteen collapse every file to a single line, and 147 of 287 Python files stop parsing.

code_context is the structure-preserving entry point. Line count, indentation and case are the contract, not a side effect:

from disarm import get_pipeline

code = get_pipeline("code_context")
snippet = "def check(user):\n    if user.is_admin:  # \u202egnp.exe\n        grant()\n"

cleaned = code(snippet)
assert cleaned.count("\n") == snippet.count("\n")  # line count preserved
assert "\u202e" not in cleaned  # the bidi control is gone
assert "    if user.is_admin:" in cleaned  # indentation intact

What it removes, and what it only reports¶

Byte-for-byte on 149 of 155 files (#745). Measured over this repository's own Python sources: canonicalize, strip_format and normalize_confusables round-trip 0 of them; code_context round-trips 149.

The six exceptions are one class, and it is worth knowing rather than working around: a ZWJ inside a string literal or a comment. A ZWJ-joined emoji ("👨‍👩‍👧‍👦") loses its joiners and a Sinhala conjunct ("ශ්‍රී") loses the one holding it together. That is code_context doing its job — U+200D is the Trojan Source carrier and the table above removes it unconditionally — but if your source embeds ZWJ-joined text in literals, the literal changes and the file still parses, so nothing will tell you. Compare before and after if that describes your corpus.

The confusable fold cannot run on code, and that is the design rather than an omission. Exactly three ASCII code points are TR39 confusable sources: " folds to two apostrophes, the backtick folds to one, and | folds to l. All three are load-bearing syntax, so normalize_confusables — the primitive adversarial defense describes as costing "nothing beyond the fold" — breaks 287 of 287 Python files in this repository while preserving every line.

So the homoglyph class is reported, not rewritten:

from disarm import get_pipeline, inspect_anomalies

code = get_pipeline("code_context")
src = "def p\u0251ypal():\n    pass\n"

assert code(src) == src  # unchanged
assert inspect_anomalies(src).kinds == ["confusable"]  # and reported

That split is what an AI-coding-assistant threat model needs. arXiv:2503.14281v4 (XOXO) shows assistants flattening repository snippets into one prompt with no origin differentiation; the useful answer is "this region of the gathered context is anomalous", not a rewritten prompt — and §E rules rewriting out on quality grounds anyway.

Which path, and when NOT to use disarm¶

Being explicit about the path is what earns credibility with this audience — the wrong path actively destroys signal:

from disarm import get_pipeline

# WRONG: the guardrail fold mangles legitimate Cyrillic — its confusable step
# rewrites real letters to Latin look-alikes, here producing the plausible-but-
# wrong all-Latin "mockba" instead of the romanisation "moskva":
assert get_pipeline("llm_guardrail")("Москва") == "mockba"

# RIGHT: the ingestion path romanises the same input cleanly:
assert get_pipeline("rag_ingest")("Москва") == "Moskva"

Do not reach for disarm when:

• The text goes straight to a multilingual model. Modern tokenizers already NFC/NFKC-normalise, and the model handles native script better than any romanisation. Transliterating the prompt throws away signal.
• You only need encoding repair. That is ftfy's job, not disarm's.
• You need lossless round-tripping. Compatibility-tier romanisation (CJK, Indic) is lossy; see Limitations.

Use disarm on the guardrail path (match untrusted input against policy) and the ingestion path (build an ASCII-normalised index) — not on the generation path.

See also¶

• Research: Transliteration for LLM Pre-Processing — the underlying survey.
• Precompiled Pipelines — every named profile and its steps.
• Confusable Detection, Adversarial-Text Defense.