Skip to main content

Core Concepts

String similarity vs. distance​

reclink provides two families of functions for comparing strings:

  • Distance functions (e.g., levenshtein) return an integer count of operations needed to transform one string into another. Lower is more similar.
  • Similarity functions (e.g., jaro_winkler) return a float in [0, 1] where 1.0 means identical. Higher is more similar.

Most metrics have both variants:

from reclink import levenshtein, levenshtein_similarity

levenshtein("kitten", "sitting") # 3 (distance)
levenshtein_similarity("kitten", "sitting") # 0.571 (similarity)

Metric categories​

CategoryExamplesBest for
Edit distanceLevenshtein, Damerau-Levenshtein, HammingTypos, OCR errors
SimilarityJaro, Jaro-WinklerNames, short strings
Token-basedCosine, Jaccard, Token Sort/Set RatioWord-order differences
SubsequenceLCS, Longest Common SubstringPartial matches
AlignmentSmith-WatermanBiological sequences, flexible gaps
PhoneticSoundex, Metaphone, Double MetaphoneNames that sound alike
Hybridphonetic_hybridCombining phonetic + edit distance

Phonetic algorithms​

Phonetic algorithms encode strings by how they sound. Two names with the same phonetic code are likely the same name, even if spelled differently:

from reclink import soundex

soundex("Smith") # "S530"
soundex("Smyth") # "S530" -- same!

Different algorithms suit different languages and use cases. See the Phonetic API reference for details.

Record linkage pipeline​

Record linkage finds matching records across one or two datasets. reclink's pipeline has four stages:

1. Preprocessing​

Clean and normalize field values before comparison:

.preprocess("name", ["fold_case", "strip_punctuation", "normalize_whitespace"])

2. Blocking​

Blocking reduces comparisons from O(n^2) to roughly O(n) by grouping records that are likely to match:

.block_phonetic("last_name", algorithm="soundex")

Without blocking, 10,000 records would require 50 million comparisons. With phonetic blocking, typically only 1-5% of pairs are compared.

3. Comparison​

Each candidate pair is compared field-by-field to produce a similarity score vector:

.compare_string("first_name", metric="jaro_winkler")
.compare_string("last_name", metric="jaro_winkler")

4. Classification​

The score vector is classified into match/non-match (or a three-band match/possible-match/non-match):

.classify_threshold(0.85)
# or
.classify_fellegi_sunter_auto() # unsupervised probabilistic model

Optional: Clustering​

Group matched pairs into clusters of duplicate records:

.cluster_connected_components()

Index structures​

For large datasets, linear scanning is too slow. reclink provides specialized index structures for sub-linear search:

  • BK-tree — for edit-distance metrics (exact threshold search)
  • VP-tree — for any metric (k-nearest and range search)
  • N-gram index — for approximate matching via shared n-gram overlap
  • MmapNgramIndex — memory-mapped n-gram index for datasets larger than RAM
  • MinHash/LSH — approximate nearest-neighbor search for large collections

Composite scorer​

Combine multiple metrics with weights for more robust matching:

from reclink import CompositeScorer

scorer = CompositeScorer([
("jaro_winkler", 0.6),
("token_sort", 0.4),
])
scorer.similarity("Jon Smith", "John Smyth") # weighted blend

Pre-tuned presets are available for common use cases:

from reclink.presets import name_matching, address_matching

Edge cases​

reclink handles empty strings consistently:

  • Distance metrics return the length of the non-empty string when one input is empty
  • Similarity metrics return 0.0 when one input is empty, 1.0 when both are empty

Next steps​