disarm

PyPI Crates.io Documentation License: MIT

Identify malicious attacks hiding in text.

раypal.com — Cyrillic а (U+0430) and р (U+0440) — renders identically to paypal.com and is a different string. disarm finds that substitution and folds it back to its Unicode TR39 prototype, strips bidi overrides, zero-width and control characters, and flags spoofed hostnames — the Unicode layer your validation, dedup, moderation and logging code is missing.

One pure-Rust core, with bindings for Python, Rust, Ruby, Node.js, Java/Kotlin and C.

from disarm import canonicalize, is_suspicious_hostname

# U+202E is a right-to-left override and U+200B a zero-width space. Neither is
# visible, and both survive a copy-paste straight into your database.
assert canonicalize("\u202eexample\u200b.com") == "example.com"

# U+0397 is Greek capital eta and U+13D4 Cherokee letter wa. They render as H and W.
assert canonicalize("\u0397ello \u13d4orld") == "Hello World"

# Cyrillic small a (U+0430) standing in for Latin a: renders as "apple.com".
suspicious, analysis = is_suspicious_hostname("\u0430pple.com")
assert suspicious and analysis.canonical == "apple.com"

Try it in your browser · Documentation · API reference

Install

pip install disarm      # Python 3.10+   (wheels for Linux, macOS, Windows)
cargo add disarm        # Rust 1.81+     (pure Rust — no Python, no pyo3)
npm install disarm      # Node.js 14+
gem install disarm      # Ruby 3.1+

disarm capabilities

  • Confusable folding — TR39 visual mapping, plus what it misses
  • Obfuscation stripping — bidi controls, zero-width characters, zalgo, emoji
  • Hostname / IDN analysis — mixed-script, whole-script and bidi checks
  • Ready-made pipelines — canonicalize, catalog_key, search_key, LLM and RAG profiles
  • Transliteration — BGN/PCGN, ISO 9 and GOST; 83 language profiles
  • Normalization, slugs & filenames — case folding, graphemes, encoding detection 
    from disarm import canonicalize, collapse_whitespace, slugify, strip_obfuscation, transliterate

# Cyrillic er (U+0440) and es (U+0441) folded to Latin p and c — visual (TR39) mapping.
assert strip_obfuscation("\u0440rodu\u0441t") == "product"

# No-break space (U+00A0), ideographic space (U+3000), thin space (U+2009) and a
# line separator (U+2028) all collapse to one plain ASCII space.
assert collapse_whitespace("Ada\u00a0\u3000Lovelace\u2009\u2028King") == "Ada Lovelace King"

# Their zero-width look-alikes are not whitespace at all — U+200B and U+FEFF are
# format characters, so neither str.split() nor collapse_whitespace touches them.
assert collapse_whitespace("A\u200bB\ufeffC") == "A\u200bB\ufeffC"
assert canonicalize("A\u200bB\ufeffC") == "ABC"

# Phonetic romanization: a different mapping, and not a defence.
assert transliterate("Київ", lang="uk") == "Kyiv"
assert slugify("Héllo Wörld") == "hello-world"

Performance & benchmarks

Does it work? On the XMR confusable-recovery metric, disarm's visual mapping scores 0.63–0.68, against ≤ 0.19 for phonetic transliterators (unidecode, anyascii, uroman) and 0.10 for NFKC. → the evidence · what it misses

What does it cost? ~450M chars/sec on Latin (~38× Unidecode), ~106M on Cyrillic, ~712K slugs/sec (~10–24× python-slugify), ~65 ns for an already-ASCII call. Hardware-dependent and directional, not guarantees. → full results · how to read them · where disarm is slower

Both come from "Fire Extinguishers Full of Gasoline": 435,864 observations over eight tools, six attack types, three tasks and two model architectures. Zenodo · CITATION.cff

Bindings: one core, six languages

Each binding reads like its own ecosystem — snake_case in Ruby, camelCase and .d.ts in Node, builders in Java — over one shared core, so every language returns the same answer.

Language Package Getting started
Python 3.10+ disarm on PyPI guide
Rust 1.81+ disarm on crates.io guide · docs.rs
Ruby 3.1+, RubyGems 3.3.22+ disarm on RubyGems guide
Node.js 14+ disarm on npm guide
Java / Kotlin dev.disarm:disarm, dev.disarm:disarm-kotlin on Maven Central guide
C / other FFI C ABI and disarm.h bindings/cabi

Wheels, gems and addons are precompiled — no local Rust toolchain needed. The core crate is unsafe_code = "forbid" and stays pure Rust; BINDINGS.md is the bar a new binding has to meet.

Limitations: read this before deploying disarm

  • Defense in depth, not a complete control. disarm folds the confusables it bundles and strips the format characters it enumerates. The confusable space is larger than any table, so measure your residue with unmapped_confusables() rather than inferring it. Threat model.
  • Not an output sanitizer. disarm normalizes input. It performs no escaping — <script>alert(1)</script> passes through unchanged, and NFKC can even surface ASCII metacharacters from fullwidth look-alikes. Keep encoding at the output sink (framework auto-escaping, DOMPurify, parameterized queries); run disarm before it.
  • transliterate() is not a security control. It romanizes phonetically. For homoglyph defense use normalize_confusables() / strip_obfuscation().

CONFUSABLES_VERSION reports which confusables.txt release the bundled tables were folded from, so a deployment can answer "am I stale?" without inferring it from behaviour (provenance).

Found a bypass? Report it under the security policy rather than in a public issue.

User Guide

Core concepts and usage for each feature area.

  • Getting Started — install + quickstart for Python · Rust · Ruby · Node.js · Java & Kotlin
  • Adversarial-Text Defense — TR39 visual confusable mapping vs phonetic transliteration, the XMR benchmark, and why it matters
  • Transliteration — Unicode → ASCII with language profiles, plus reverse (Latin → native script)
  • Slugification — URL-safe slug generation, drop-in python-slugify replacement
  • Normalization — NFC / NFD / NFKC / NFKD Unicode normalization
  • Confusable Detection — TR39 homoglyph detection and normalization, and which sources the bundled table does not cover
  • Filename Sanitization — Cross-platform safe filenames
  • Text Cleaning — Accent stripping, case folding, whitespace collapse
  • Grapheme Clusters — User-perceived character counting, splitting, and truncation
  • Text Pipeline — Composable, pre-compiled multi-step processing
  • Language Support — Built-in profiles, auto-detection, custom profiles
  • Abjad Scripts — Context-aware Arabic, Persian, and Hebrew with dictionary-based vowel restoration
  • Language Detection — How lang="auto" works: script identification, character-level discrimination, fail-safe fallbacks
  • Policy Templates — Named institutional presets for libraries, web apps, ML, and more
  • CLI — Command-line usage, piping, and shell integration

API Reference

Complete function signatures, parameters, and return types.

  • Overview — API reference index
  • Core Transforms — transliterate, slugify, normalize, sanitize_filename, strip_accents, strip_zalgo, fold_case, collapse_whitespace, demojize, strip_bidi (all accept str or list[str])
  • Precompiled Pipelines — canonicalize, ml_normalize, catalog_key, strip_format, search_key, sort_key, canonicalize_strict, PRESETS, get_pipeline, list_profiles
  • Classes — Text, Slugifier, UniqueSlugifier, TextPipeline, compatibility aliases
  • Predicates & introspection — detect_scripts, inspect_auto_lang, is_mixed_script, is_confusable, is_ascii, is_normalized, is_zalgo, is_suspicious_hostname, unmapped_confusables, find_unmapped_confusables
  • Grapheme Clusters — grapheme_len, grapheme_split, grapheme_truncate
  • Encoding Detection — detect_encoding, decode_to_utf8
  • Language Profiles — list_langs, register_lang, register_replacements
  • Enums & Types — Script, NF, EmojiProvider, type aliases, language constants
  • Exceptions — DisarmError

Reference

  • Language Reference — All languages: codes, names, reference texts, and per-language transliteration rule tables
  • Provenance — Standards and sources behind every transliteration mapping

Architecture

Internal design documentation for contributors and advanced users.

  • Transliteration Engine — PHF lookup, language table chain, Indic virama handling
  • Data Tables — TSV format, build.rs code generation, compile-time PHF
  • Pipeline — TextPipeline internals, execution order, step bitflags
  • Emoji Engine — Emoji detection, the EmojiProvider protocol and custom providers, pure-Rust path
  • Security — Confusable detection, hostname validation, bidi stripping
  • Performance — Optimization strategies, PHF tables, batch amortization
  • Testing & Guarantees — Test philosophy, property-based testing, security invariants, CI matrix
  • Exhaustive Testing — Compile-time assertions, exhaustive domain coverage, stated invariants (I1–I7)
  • Transliteration Comparison — Character-level diff vs Unidecode and anyascii

Benchmarks

  • Performance Overview — Benchmark results: throughput and per-call speedups vs Unidecode, python-slugify, and pathvalidate
  • Benchmark Suite — How to run benchmarks, Criterion and timeit configurations

Migration Guides

Parameter-compatible replacements for existing libraries.

Other

  • Limitations — Known constraints, edge cases, and design trade-offs