Anomaly Detection¶

has_anomalies / inspect_anomalies flag text that carries out-of-place characters disguising a real word — a cross-script homoglyph, a bidi-direction conflict, leet, a single-letter segmentation, a zero-width / bidi control, or zalgo. Like is_suspicious_hostname, the detector reports a technical fact and leaves the malicious-or-not judgement to the caller — it never claims intent.

Detected classes¶

Eight branches fire. Six need no lexicon — only leet and segmentation do.

The table below is grouped by kind, not by evaluation order. control is checked first, ahead of the ASCII fast-path, because NUL, ESC, BEL and DEL are themselves ASCII: a check placed after that fast-path would never see the vectors it exists for. The remaining branches split on !tok.is_ascii(), so invisible, bidi, zalgo, bidi_mixed and mixed_script only run on non-ASCII tokens, and leet and segmentation run last on everything.

Most branches are script-agnostic and port across writing systems. mixed_script is the exception — it is anchored on Latin, and fires on Latin combined with Cyrillic or Greek.

The bidi kind is a judgement, not a census (#778)¶

The row above spares LRM everywhere and RLM/ALM outside a number run, on purpose: a lone directional mark is ordinary in right-to-left text, and reporting one would fire on any page that uses it. So the kind answers 9 of the 12 UAX #9 controls.

When the question is the census rather than the judgement — a filename, an identifier, a source file, anywhere the caller has already decided their input should carry no bidi control at all — use has_bidi_control, which answers all twelve and applies no context:

from disarm import has_bidi_conflict, has_bidi_control, inspect_anomalies

assert has_bidi_control("\u200e")  # a mark counts
assert inspect_anomalies("\u200e").kinds == []  # and is deliberately not an anomaly

assert has_bidi_control("invoice\u202egpj.exe")  # so does an override
assert not has_bidi_conflict("invoice\u202egpj.exe")  # which reads letters, not controls

The three predicates are disjoint answers to different questions: has_bidi_control is the raw set, inspect_anomalies is the judged subset, and has_bidi_conflict reads strong-direction letters and is structurally blind to controls altogether.

The leet and segmentation branches take a caller-supplied lexicon — a set of common words for the language being protected. The defining rule: a real leet attack substitutes a letter, whereas win32 carries a literal number that maps to no letter, so requiring every out-of-place character to be a real letter-substitution that yields a common word rejects the literals.

Usage¶

from disarm import has_anomalies, inspect_anomalies

words = {"free", "paypal"}

# leet: "fr33" decodes to "free"
assert has_anomalies("get fr33 now", words)
# a literal number is not a substitution, so "win32" is spared
assert not has_anomalies("the win32 api", words)

report = inspect_anomalies("log in to paypаl", {"paypal"})  # Cyrillic а
assert report.anomalous
assert report.kinds == ["mixed_script"]
assert report.findings[0].kind == "mixed_script"

use disarm::api::{self, AnomalyKind};
use std::collections::HashSet;

let words: HashSet<String> = ["free", "paypal"].iter().map(|s| s.to_string()).collect();

assert!(api::has_anomalies("get fr33 now", &words));
assert!(!api::has_anomalies("the win32 api", &words));

let report = api::inspect_anomalies("log in to paypаl", &words);
assert!(report.anomalous);
assert_eq!(report.kinds, vec![AnomalyKind::MixedScript]);

require "disarm"

# the lexicon is a common-word collection (Array or Set)
Disarm.has_anomalies?("get fr33 now", ["free"])  # => true
Disarm.has_anomalies?("the win32 api", ["free"]) # => false

Disarm.inspect_anomalies("log in to paypаl", ["paypal"])[:kinds] # => ["mixed_script"]

import { hasAnomalies, inspectAnomalies } from 'disarm'

// the lexicon is a Set or array of common words
hasAnomalies('get fr33 now', ['free'])  // => true
hasAnomalies('the win32 api', ['free']) // => false

inspectAnomalies('log in to paypаl', ['paypal']).kinds // => ['mixed_script']

The report¶

inspect_anomalies returns a report with anomalous, kinds (the anomaly kinds that fired, in first-appearance order), findings, and reason (the first finding's plain-language sentence). Each finding carries the offending kind, token, byte start/end span, detail (the codepoint, the scripts, or the decoded word), and its own reason.

A False result is not a safety guarantee — it means only that none of the six branches fired on the lexicon you supplied. Compose this with your own policy, as you would the hostname analysis.

Checking a transform at the seam¶

Run has_anomalies on the output of a transform and it tells you whether that transform left something behind. The check needs no new API and is the cheapest way to find out you picked the wrong function.

One input carrying three different hazards — a right-to-left override, a zero-width space and a Cyrillic ԁ standing in for Latin d:

from disarm import (
    canonicalize,
    has_anomalies,
    inspect_anomalies,
    ml_normalize,
    normalize_confusables,
)

hostile = "\u202eexample\u200b.com\u0501"

# canonicalize clears all three: nothing is left to report.
assert canonicalize(hostile) == "example.comd"
assert has_anomalies(canonicalize(hostile)) is False

# ml_normalize is not a security preset. The override survives, and the seam
# check is what tells you so.
assert has_anomalies(ml_normalize(hostile)) is True
assert inspect_anomalies(ml_normalize(hostile)).kinds == ["bidi"]

# normalize_confusables folds the homoglyph and nothing else.
assert inspect_anomalies(normalize_confusables(hostile)).kinds == ["invisible"]

The guidance only runs one way¶