strutil

repository·master·Indexed 19 days ago

https://github.com/adrg/strutil

A Go library providing a collection of string metrics for calculating string similarity, including Levenshtein, Jaro-Winkler, Jaccard, Hamming, and Smith-Waterman-Gotoh. It features a StringMetric interface for custom comparisons, n-gram generation utilities, and string slice helper functions such as CommonPrefix and UniqueSlice.

Tokens
3K
Snippets
13
Records
14
Agent score
15%

What's inside strutil

  1. How StringMetric and Similarity work together

    master

    The package uses a StringMetric interface to define how two strings are compared. All specific metrics (like Hamming, Levenshtein, etc.) implement this interface.

    You use the strutil.Similarity function to calculate a similarity score between two strings by passing in a StringMetric implementation.

    • StringMetric interface: Defines a Compare(a, b string) float64 method.
    • strutil.Similarity(a, b string, metric StringMetric) float64: The primary entry point for calculating similarity scores.
    type StringMetric interface {
        Compare(a, b string) float64
    }
    
    func Similarity(a, b string, metric StringMetric) float64 {
    }
  2. Calculate Overlap Coefficient similarity

    master

    The Overlap Coefficient measures similarity based on n-grams. You can customize the NgramSize and CaseSensitive settings.

    Use metrics.NewOverlapCoefficient() to create the metric instance.

    // Calculate similarity using default options
    oc := metrics.NewOverlapCoefficient()
    similarity := strutil.Similarity("time to make haste", "no time to waste", oc)
    fmt.Printf("%.2f\n", similarity) // Output: 0.67
    
    // Customize n-gram size
    oc := metrics.NewOverlapCoefficient()
    oc.CaseSensitive = false
    oc.NgramSize = 3
    
    similarity := strutil.Similarity("Time to make haste", "no time to waste", oc)
    fmt.Printf("%.2f\n", similarity) // Output: 0.57
  3. Calculate Jaro-Winkler similarity

    master

    Use metrics.NewJaroWinkler() to calculate the Jaro-Winkler similarity score.

    similarity := strutil.Similarity("think", "tank", metrics.NewJaroWinkler())
    fmt.Printf("%.2f\n", similarity) // Output: 0.80
  4. Calculate Smith-Waterman-Gotoh similarity

    master

    The Smith-Waterman-Gotoh metric allows for local sequence alignment. You can customize the GapPenalty and the Substitution behavior (using metrics.MatchMismatch).

    Use metrics.NewSmithWatermanGotoh() to create the metric instance.

    // Calculate similarity using default options
    swg := metrics.NewSmithWatermanGotoh()
    similarity := strutil.Similarity("times roman", "times new roman", swg)
    fmt.Printf("%.2f\n", similarity) // Output: 0.82
    
    // Customize gap penalty and substitution function
    swg := metrics.NewSmithWatermanGotoh()
    swg.CaseSensitive = false
    swg.GapPenalty = -0.1
    swg.Substitution = metrics.MatchMismatch {
        Match:    1,
        Mismatch: -0.5,
    }
    
    similarity := strutil.Similarity("Times Roman", "times new roman", swg)
    fmt.Printf("%.2f\n", similarity) // Output: 0.96
  5. Calculate Sorensen-Dice similarity

    master

    The Sorensen-Dice coefficient measures the similarity between two sets of n-grams. You can customize the NgramSize and CaseSensitive settings.

    Use metrics.NewSorensenDice() to create the metric instance.

    // Calculate similarity using default options
    sd := metrics.NewSorensenDice()
    similarity := strutil.Similarity("time to make haste", "no time to waste", sd)
    fmt.Printf("%.2f\n", similarity) // Output: 0.62
    
    // Customize n-gram size
    sd := metrics.NewSorensenDice()
    sd.CaseSensitive = false
    sd.NgramSize = 3
    
    similarity := strutil.Similarity("Time to make haste", "no time to waste", sd)
    fmt.Printf("%.2f\n", similarity) // Output: 0.53
  6. Calculate Jaccard similarity

    master

    The Jaccard index measures similarity between sets of n-grams. You can customize the NgramSize and CaseSensitive settings.

    Use metrics.NewJaccard() to create the metric instance.

    // Calculate similarity using default options
    j := metrics.NewJaccard()
    similarity := strutil.Similarity("time to make haste", "no time to waste", j)
    fmt.Printf("%.2f\n", similarity) // Output: 0.45
    
    // Customize n-gram size
    j := metrics.NewJaccard()
    j.CaseSensitive = false
    j.NgramSize = 3
    
    similarity := strutil.Similarity("Time to make haste", "no time to waste", j)
    fmt.Printf("%.2f\n", similarity) // Output: 0.36
  7. Calculate Levenshtein similarity and distance

    master

    The Levenshtein metric calculates the minimum number of single-character edits required to change one word into another. You can configure the costs for insertion, deletion, and replacement, as well as case sensitivity.

    Use metrics.NewLevenshtein() to create the metric instance.

    // Calculate similarity with default options
    similarity := strutil.Similarity("graph", "giraffe", metrics.NewLevenshtein())
    fmt.Printf("%.2f\n", similarity) // Output: 0.43
    
    // Configure edit operation costs
    lev := metrics.NewLevenshtein()
    lev.CaseSensitive = false
    lev.InsertCost = 1
    lev.ReplaceCost = 2
    lev.DeleteCost = 1
    
    similarity := strutil.Similarity("make", "Cake", lev)
    fmt.Printf("%.2f\n", similarity) // Output: 0.50
    
    // Calculate distance
    lev := metrics.NewLevenshtein()
    fmt.Printf("%d\n", lev.Distance("graph", "giraffe")) // Output: 4
  8. Calculate Hamming similarity and distance

    master

    The Hamming metric can be used to calculate both a similarity score (0.0 to 1.0) and the edit distance (integer).

    Use metrics.NewHamming() to create the metric instance.

    // Calculate similarity
    similarity := strutil.Similarity("text", "test", metrics.NewHamming())
    fmt.Printf("%.2f\n", similarity) // Output: 0.75
    
    // Calculate distance
    ham := metrics.NewHamming()
    fmt.Printf("%d\n", ham.Distance("one", "once")) // Output: 2
  9. Calculate string similarity using Similarity()

    master

    The Similarity function computes the similarity between two strings using a provided StringMetric. The result is a float64 between 0 and 1, where higher values indicate closer matches.

    To use this, you must provide an implementation of the StringMetric interface (such as those found in the github.com/adrg/strutil/metrics package).

    import "github.com/adrg/strutil"
    // Assuming 'metric' is an implementation of StringMetric from the metrics package
    score := strutil.Similarity("apple", "appel", metric)
  10. Work with n-grams using Ngram functions

    master

    The strutil package provides several utilities for generating and analyzing n-grams (sequences of $n$ items from a given sample). If the provided size is $\le 0$, a size of 1 is used by default.

    • NgramCount(term string, size int) int: Returns the total count of n-grams.
    • Ngrams(term string, size int) []string: Returns a slice of all n-grams in their original order.
    • NgramMap(term string, size int) (map[string]int, int): Returns a map of n-gram frequencies and the total number of n-grams.
    • NgramIntersection(a, b string, size int) (map[string]int, int, int, int): Returns the intersection of n-grams between two strings. It returns the frequency map, the count of common n-grams, the total n-grams in string a, and the total n-grams in string b.
    // Get all 2-grams
    grams := strutil.Ngrams("hello", 2)
    
    // Get frequency map and total count
    freqMap, total := strutil.NgramMap("hello", 2)
    
    // Get intersection between two strings
    intersection, commonCount, totalA, totalB := strutil.NgramIntersection("hello", "help", 2)