Skip to content

Repository files navigation

soft_bigram

Build Status Go Reference Apache 2 License

Go implementation of Soft-Bidist, a character-bigram edit distance for personal-name matching (Hadwan, Al-Hagery, Al-Sanabani, Al-Hagree, PeerJ Computer Science 2021). Built for Watchman token scoring: short strings, [0, 1] similarity, no extra dependencies.

Soft-Bidist is Kondrak BI-DIST (SPIRE 2005) with a nine-case cost scale over adjacent character pairs. Transpositions, prefix/suffix mismatches, and doubled letters get distinct costs instead of a flat 0/1 substitution.

import "github.com/PhonoGrams/soft_bigram"

soft_bigram.Similarity("precede", "preceed")     // ~0.97
soft_bigram.SimilarityFold("Similar", "Similer") // ~0.94
soft_bigram.Distance("adam", "adams")            // raw distance
soft_bigram.KondrakSimilarity("toradol", "tegretol")

Similarity is case-sensitive on runes. Watchman already lowercases and Unicode-folds names; pass those tokens through Similarity. Use SimilarityFold only at the edges.

Default weights are the paper's best configuration (wt1…wt9) = (0, 1, 0, 0.2, 0.2, 1, 1, 0.5, 0.5). See docs/algorithm.md for the recurrence, cases, and Watchman notes.

Performance

The DP is two rows of len(b)+1 and stays on the stack for names up to 64 runes. Typical personal-name pairs allocate nothing on the hot path.

References

License

Apache License 2.0

About

Soft-Bidist: character-bigram edit distance for personal-name matching (Watchman / OFAC)

Topics

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Used by

Contributors

Languages