repository·master·Indexed 21 days ago
https://github.com/jpmckinney/tf-idf-similarityA Ruby implementation of the Vector Space Model (VSM) used to calculate similarity between text documents. It supports tf*idf and Okapi BM25 weighting, providing tools to generate similarity matrices and retrieve specific term weights. The library allows for custom tokenization, term counts, and supports performance optimizations via NArray, GSL, or NMatrix.
By default, the library uses the Ruby Standard Library's Matrix class. For better performance, you can use the :library option when initializing a model to use narray, gsl, or nmatrix. narray is noted to have the best performance of the three.
require 'narray'
model = TfIdfSimilarity::TfIdfModel.new(corpus, :library => :narray)To calculate similarity between texts, create a collection of TfIdfSimilarity::Document objects and pass them to a model. You can use TfIdfSimilarity::TfIdfModel for standard tf*idf weights or TfIdfSimilarity::BM25Model for the Okapi BM25 ranking function. Once the model is initialized, you can generate a similarity matrix to compare documents.
require 'matrix'
require 'tf-idf-similarity'
document1 = TfIdfSimilarity::Document.new("Lorem ipsum dolor sit amet...")
document2 = TfIdfSimilarity::Document.new("Pellentesque sed ipsum dui...")
document3 = TfIdfSimilarity::Document.new("Nam scelerisque dui sed leo...")
corpus = [document1, document2, document3]
# Use tf*idf
model = TfIdfSimilarity::TfIdfModel.new(corpus)
# OR use BM25
# model = TfIdfSimilarity::BM25Model.new(corpus)
# Get similarity matrix
matrix = model.similarity_matrix
# Find similarity between two specific documents
similarity = matrix[model.document_index(document1), model.document_index(document2)]NoMethodError: undefined method '[]' for Matrix:Module, it is because the matrix gem conflicts with Ruby's internal Matrix module. To resolve this, do not use the matrix gem; instead, rely on the built-in module or use a high-performance library like NArray.You can retrieve the specific tf*idf weight for any term within a document using the tfidf method on the model instance.
tfidf_by_term = {}
document1.terms.each do |term|
tfidf_by_term[term] = model.tfidf(document1, term)
end
puts tfidf_by_term.sort_by{|_,tfidf| -tfidf}For maximum control, you can bypass the internal tokenizer by providing your own term frequency counts and the total number of tokens in the document using the :term_counts and :size keys.
term_counts = Hash.new(0)
size = 0
tokens.each do |token|
unless token[/\A\d+\z/]
term_counts[token.gsub(/\p{Punct}/, '')] += 1
size += 1
end
end
document1 = TfIdfSimilarity::Document.new(text, :term_counts => term_counts, :size => size)If you want to control tokenization (e.g., to exclude stop words), you can pass a pre-processed array of tokens to the TfIdfSimilarity::Document constructor using the :tokens key.
require 'unicode_utils'
text = "Lorem ipsum dolor sit amet..."
tokens = UnicodeUtils.each_word(text).to_a - ['and', 'the', 'to']
document1 = TfIdfSimilarity::Document.new(text, :tokens => tokens)Additional formulas for term frequency, document frequency, and normalization are available by requiring the extras modules.
require 'tf-idf-similarity/extras/document'
require 'tf-idf-similarity/extras/tf_idf_model'To perform similarity calculations, instantiate TfIdfSimilarity::Model by passing an array of Document objects. You can optionally specify a mathematical library to use for matrix operations via the :library option.
Supported libraries for the :library option:
:gsl (GSL::Matrix):narray (NArray):nmatrix (NMatrix):matrix (Standard Ruby Matrix, default)model = TfIdfSimilarity::Model.new(documents, library: :nmatrix)The TfIdfSimilarity::TfIdfModel class provides methods to normalize the term-document matrix.
Available normalization methods include:
no_normalization(matrix): Returns the matrix as-is.pivoted_unique_normalization(matrix): Currently raises NotImplementedError.MatrixMethods#normalize and is not a direct method of TfIdfModel.The TfIdfSimilarity::TfIdfModel class provides various methods to calculate Term Frequency (TF), which determines how a term's weight is scaled based on its occurrence within a document.
Available TF methods include:
binary_term_frequency(document, term) (alias: binary_tf): Returns 1 if the term exists in the document, otherwise 0.normalized_term_frequency(document, term, a = 0) (alias: normalized_tf): Scales frequency between a and 1 based on the document's maximum term count.augmented_normalized_term_frequency(document, term) (alias: augmented_normalized_tf): A variation of normalized TF using a 0.5 base.augmented_average_term_frequency(document, term) (alias: augmented_average_tf): Scales based on the document's average term count.changed_coefficient_augmented_normalized_term_frequency(document, term) (alias: changed_coefficient_augmented_normalized_tf): Uses a 0.2 base for scaling.log_term_frequency(document, term) (alias: log_tf): Uses 1 + log(count).normalized_log_term_frequency(document, term) (alias: normalized_log_tf): Logarithmic TF normalized by the document's average term count.augmented_log_term_frequency(document, term) (alias: augmented_log_tf): Uses a 0.2 base with log(count + 1).square_root_term_frequency(document, term) (alias: square_root_tf): Uses sqrt(count - 0.5) + 1.The TfIdfSimilarity::Tokenizer class converts a string into an enumerator of Token objects. It uses the UnicodeUtils library to identify word boundaries, ensuring robust handling of Unicode text.
To use it, instantiate the tokenizer and call the tokenize method with a string. The method returns an Enumerator of Token objects, which can be iterated over or converted to an array.
tokenizer = TfIdfSimilarity::Tokenizer.new
tokens = tokenizer.tokenize("Hello, world!") # Returns an Enumerator of Token objects
# To get an array of tokens:
token_array = tokenizer.tokenize("Hello, world!").to_aThe TfIdfSimilarity::TfIdfModel class provides several methods to calculate the Inverse Document Frequency (IDF), allowing you to choose different weighting schemes (e.g., SMART, Salton, or Chisholm variants).
Available IDF methods include:
plain_inverse_document_frequency(term, numerator = 0, denominator = 0) (alias: plain_idf): Standard IDF calculation using logarithms.probabilistic_inverse_document_frequency(term) (alias: probabilistic_idf): Probabilistic IDF calculation.global_frequency_inverse_document_frequency(term) (alias: gfidf): Based on global term frequency.log_global_frequency_inverse_document_frequency(term) (alias: log_gfidf): Logarithm of global frequency.incremented_global_frequency_inverse_document_frequency(term) (alias: incremented_gfidf): Global frequency plus one.square_root_global_frequency_inverse_document_frequency(term) (alias: square_root_gfidf): Square root of global frequency.entropy(term): Entropy-based weighting.no_collection_frequency(term): Returns a constant 1.0 (effectively disabling IDF weighting).