Skip to main content

Preprocessing

Consistent preprocessing is critical for accurate fuzzy matching. reclink provides a comprehensive set of functions for cleaning, normalizing, tokenizing, and transliterating text -- all implemented in Rust for maximum throughput.

All functions are imported directly from reclink.

from reclink import fold_case, normalize_whitespace, clean_name

Basic Operations​

fold_case(s)​

Converts a string to lowercase using Unicode case folding (more thorough than Python's str.lower()).

from reclink import fold_case

fold_case("HELLO World")  # "hello world"
fold_case("Strasse")       # "strasse"

normalize_whitespace(s)​

Collapses all consecutive whitespace (including tabs, newlines, and Unicode whitespace) into single spaces and strips leading/trailing whitespace.

from reclink import normalize_whitespace

normalize_whitespace("  too   many   spaces  ")  # "too many spaces"
normalize_whitespace("line\none\ttwo")            # "line one two"

strip_punctuation(s)​

Removes all Unicode punctuation characters.

from reclink import strip_punctuation

strip_punctuation("hello, world!")      # "hello world"
strip_punctuation("O'Brien-Smith")      # "OBrienSmith"
strip_punctuation("john.doe@email.com") # "johndoeemailcom"

standardize_name(s)​

Combines multiple preprocessing steps into a single call: case folding, whitespace normalization, punctuation stripping, and common name standardization.

from reclink import standardize_name

standardize_name("  DR. John  Smith Jr.  ")  # "john smith"
standardize_name("MCDONALD, Jane A.")        # "mcdonald jane"

normalize_unicode(s, form="nfkc")​

Applies Unicode normalization. Supported forms: "nfc", "nfkc", "nfd", "nfkd".

from reclink import normalize_unicode

# NFKC converts compatibility characters to their canonical equivalents
normalize_unicode("half")        # normalizes using NFKC by default
normalize_unicode("cafe\u0301", form="nfc")  # "cafe" with composed accent

strip_diacritics(s)​

Removes diacritical marks (accents) from characters while preserving the base character.

from reclink import strip_diacritics

strip_diacritics("cafe")       # "cafe"
strip_diacritics("resume")     # "resume"
strip_diacritics("nino")       # "nino"

remove_stop_words(s)​

Removes common English stop words (the, a, an, of, and, etc.) from the string.

from reclink import remove_stop_words

remove_stop_words("the university of california")  # "university california"
remove_stop_words("bank of america")                # "bank america"

expand_abbreviations(s)​

Expands common abbreviations to their full forms.

from reclink import expand_abbreviations

expand_abbreviations("St. Louis")   # "Saint Louis"
expand_abbreviations("Dr. Smith")   # "Doctor Smith"
expand_abbreviations("Ave.")        # "Avenue"

regex_replace(s, pattern, replacement)​

Replaces all matches of a regular expression pattern with the given replacement string.

from reclink import regex_replace

regex_replace("abc-123-def", r"\d+", "")   # "abc--def"
regex_replace("hello world", r"\s+", "_")  # "hello_world"

Parameters​

FunctionParametersReturn
fold_case(s)s: strstr
normalize_whitespace(s)s: strstr
strip_punctuation(s)s: strstr
standardize_name(s)s: strstr
normalize_unicode(s, form)s: str, form: str = "nfkc"str
strip_diacritics(s)s: strstr
remove_stop_words(s)s: strstr
expand_abbreviations(s)s: strstr
regex_replace(s, pattern, replacement)s: str, pattern: str, replacement: strstr

Tokenization​

Tokenizers split strings into lists of tokens for use with token-based metrics or downstream processing.

ngram_tokenize(s, n=2)​

Splits a string into overlapping character n-grams.

from reclink import ngram_tokenize

ngram_tokenize("hello")        # ["he", "el", "ll", "lo"]
ngram_tokenize("hello", n=3)   # ["hel", "ell", "llo"]

whitespace_tokenize(s)​

Splits a string on whitespace boundaries.

from reclink import whitespace_tokenize

whitespace_tokenize("John Smith Jr")  # ["John", "Smith", "Jr"]

character_tokenize(s)​

Splits a string into individual characters.

from reclink import character_tokenize

character_tokenize("hello")  # ["h", "e", "l", "l", "o"]

smart_tokenize(s)​

CJK-aware tokenizer that handles Chinese, Japanese, and Korean characters alongside Latin text. CJK characters are tokenized individually while Latin text is split on whitespace.

from reclink import smart_tokenize

smart_tokenize("hello world")  # ["hello", "world"]
smart_tokenize("Tokyo")        # individual CJK characters tokenized

cjk_ngram_tokenize(s, n=2)​

Generates character n-grams specifically for CJK text.

from reclink import cjk_ngram_tokenize

cjk_ngram_tokenize("abcdef", n=2)  # CJK-aware bigrams

Parameters​

FunctionParametersReturn
ngram_tokenize(s, n)s: str, n: int = 2list[str]
whitespace_tokenize(s)s: strlist[str]
character_tokenize(s)s: strlist[str]
smart_tokenize(s)s: strlist[str]
cjk_ngram_tokenize(s, n)s: str, n: int = 2list[str]

Domain Preprocessors​

Higher-level functions tailored for specific data types. Each applies a sensible chain of normalization steps.

clean_name(s)​

Cleans personal names by removing titles, suffixes, punctuation, extra whitespace, and folding case.

from reclink import clean_name

clean_name("  DR. John  Smith Jr.  ")   # "john smith"
clean_name("Mrs. Jane O'Brien-Doe")     # "jane obrien doe"
clean_name("SMITH, JOHN A.")            # "smith john"

clean_address(s)​

Normalizes street addresses by expanding abbreviations, removing punctuation, and standardizing whitespace.

from reclink import clean_address

clean_address("123 N. Main St., Apt. 4B")  # "123 north main street apartment 4b"
clean_address("P.O. Box 456")              # "po box 456"

clean_company(s)​

Normalizes company names by removing legal suffixes (Inc., LLC, Ltd., etc.), punctuation, and folding case.

from reclink import clean_company

clean_company("Acme Corp., Inc.")      # "acme corp"
clean_company("Smith & Sons, LLC")     # "smith sons"
clean_company("THE BOEING COMPANY")    # "boeing"

normalize_email(s)​

Normalizes email addresses: lowercases, removes dots from the local part (Gmail-style), and strips plus-addressing.

from reclink import normalize_email

normalize_email("John.Doe+work@Gmail.com")  # "johndoe@gmail.com"

normalize_url(s)​

Normalizes URLs by lowercasing the scheme and host, removing trailing slashes, default ports, and common tracking parameters.

from reclink import normalize_url

normalize_url("HTTPS://WWW.Example.COM/path/")  # "https://www.example.com/path"

Parameters​

FunctionParametersReturn
clean_name(s)s: strstr
clean_address(s)s: strstr
clean_company(s)s: strstr
normalize_email(s)s: strstr
normalize_url(s)s: strstr

Transliteration​

Convert non-Latin scripts to their Latin equivalents. Essential for cross-script name matching.

Script-specific transliterators​

from reclink import (
    transliterate_cyrillic,
    transliterate_greek,
    transliterate_arabic,
    transliterate_hebrew,
    transliterate_devanagari,
    transliterate_hangul,
)

transliterate_cyrillic("Ivanov")       # "Ivanov" (Cyrillic to Latin)
transliterate_greek("alpha")           # Greek script to Latin
transliterate_arabic("text")           # Arabic script to Latin
transliterate_hebrew("text")           # Hebrew script to Latin
transliterate_devanagari("text")       # Devanagari script to Latin
transliterate_hangul("text")           # Korean Hangul to Latin

Parameters​

Each transliterator takes a single parameter:

ParameterTypeDefaultDescription
sstrrequiredInput string containing characters from the target script

Returns​

str -- transliterated string with non-Latin characters replaced by their Latin equivalents. Characters that are already Latin are passed through unchanged.

Arabic & Hebrew​

Specialized normalization functions for Arabic and Hebrew text.

normalize_arabic(s)​

Normalizes Arabic text by standardizing letter forms (e.g., various forms of Alef, Taa Marbuta).

from reclink import normalize_arabic

normalize_arabic("arabic text")  # normalized form

strip_arabic_diacritics(s)​

Removes Arabic diacritical marks (tashkeel/harakat) such as Fatha, Damma, Kasra, Shadda, and Sukun.

from reclink import strip_arabic_diacritics

strip_arabic_diacritics("arabic text with diacritics")  # diacritics removed

strip_hebrew_diacritics(s)​

Removes Hebrew diacritical marks (nikkud) such as vowel points and cantillation marks.

from reclink import strip_hebrew_diacritics

strip_hebrew_diacritics("hebrew text with nikkud")  # nikkud removed

strip_bidi_marks(s)​

Removes Unicode bidirectional control characters (LRM, RLM, LRE, RLE, etc.) that can interfere with string comparison.

from reclink import strip_bidi_marks

strip_bidi_marks("text\u200fwith\u200emarks")  # "textwithmarks"

Parameters​

FunctionParametersReturn
normalize_arabic(s)s: strstr
strip_arabic_diacritics(s)s: strstr
strip_hebrew_diacritics(s)s: strstr
strip_bidi_marks(s)s: strstr

Synonym Expansion​

synonym_expand(s, table)​

Replaces tokens in a string using a custom synonym lookup table. Useful for domain-specific abbreviations and aliases.

from reclink import synonym_expand

table = {
    "st": "street",
    "ave": "avenue",
    "blvd": "boulevard",
    "dr": "drive",
}

synonym_expand("123 main st", table)    # "123 main street"
synonym_expand("oak ave", table)        # "oak avenue"

Parameters​

ParameterTypeDefaultDescription
sstrrequiredInput string
tabledict[str, str]requiredMapping from tokens to their expansions. Matching is case-sensitive on the whitespace-tokenized input.

Returns​

str -- the string with matched tokens replaced.

Batch Processing​

Batch variants process lists of strings in parallel using Rayon thread pools. Use these when preprocessing entire columns of a DataFrame or large string collections.

Preprocessing batch​

from reclink import preprocess_batch

# Apply a chain of preprocessing steps to every string
cleaned = preprocess_batch(
    ["  DR. John Smith  ", "JANE DOE", "Bob O'Brien"],
    steps=["fold_case", "strip_punctuation", "normalize_whitespace"],
)
# ["dr john smith", "jane doe", "bob obrien"]

Tokenization batch​

from reclink import (
    ngram_tokenize_batch,
    whitespace_tokenize_batch,
    character_tokenize_batch,
    smart_tokenize_batch,
    cjk_ngram_tokenize_batch,
    smart_tokenize_ngram_batch,
)

strings = ["hello world", "foo bar"]

ngram_tokenize_batch(strings)            # [["he","el","ll",...], ["fo","oo",...]]
whitespace_tokenize_batch(strings)       # [["hello","world"], ["foo","bar"]]
character_tokenize_batch(strings)        # [["h","e","l",...], ["f","o","o",...]]
smart_tokenize_batch(strings)            # CJK-aware tokenization for each string
cjk_ngram_tokenize_batch(strings)        # CJK n-gram tokenization for each string
smart_tokenize_ngram_batch(strings)      # combined smart + n-gram tokenization

Parameters​

FunctionParametersReturn
preprocess_batch(strings, steps)strings: list[str], steps: list[str]list[str]
ngram_tokenize_batch(strings, n)strings: list[str], n: int = 2list[list[str]]
whitespace_tokenize_batch(strings)strings: list[str]list[list[str]]
character_tokenize_batch(strings)strings: list[str]list[list[str]]
smart_tokenize_batch(strings)strings: list[str]list[list[str]]
cjk_ngram_tokenize_batch(strings, n)strings: list[str], n: int = 2list[list[str]]
smart_tokenize_ngram_batch(strings)strings: list[str]list[list[str]]

:::tip Performance Batch functions use Rayon for automatic parallelism. For datasets with 1,000+ strings, batch functions are significantly faster than calling the single-string variant in a Python loop. :::

Chaining Preprocessors​

The pipeline API supports chaining multiple preprocessing steps declaratively:

from reclink.pipeline import ReclinkPipeline

pipeline = (
    ReclinkPipeline.builder()
    .preprocess("name", ["fold_case", "strip_punctuation", "normalize_whitespace"])
    .preprocess("address", ["fold_case", "expand_abbreviations", "normalize_whitespace"])
    .compare_string("name", metric="jaro_winkler")
    .compare_string("address", metric="token_sort_ratio")
    .classify_threshold(0.85)
    .build()
)

Valid step names for .preprocess() correspond to the function names on this page: "fold_case", "normalize_whitespace", "strip_punctuation", "standardize_name", "normalize_unicode", "strip_diacritics", "remove_stop_words", "expand_abbreviations", "clean_name", "clean_address", "clean_company".

See Also​