Pragmatic Segmenter

You can install Pragmatic Segmenter via RubyGem or by adding it to your Rails Gemfile.

Ruby (Supports Ruby 2.1.5 and above):

gem install pragmatic_segmenter

Ruby on Rails: Add this line to your application's Gemfile:

gem 'pragmatic_segmenter'

When initializing PragmaticSegmenter::Segmenter, you can pass language and doc_type to improve segmentation accuracy for specific contexts.

Example for Armenian (hy) and PDF documents:

text = "Այսօր երկուշաբթի է: Ես գնում եմ աշխատանքի:"
ps = PragmaticSegmenter::Segmenter.new(text: text, language: 'hy', doc_type: 'pdf')
ps.segment

text = "This is a sentence\ncut off in the middle because pdf."
ps = PragmaticSegmenter::Segmenter.new(text: text, language: 'en', doc_type: 'pdf')
ps.segment
# => ["This is a sentence cut off in the middle because pdf."]

If you only need to clean and preprocess text without actually performing sentence segmentation, you can use the segmenter for this purpose. This is useful for normalizing text before passing it to other NLP tools.

# Example cleaning only
segmenter = PragmaticSegmenter::Segmenter.new
segmenter.clean('This is a sentence cut off in the middle because pdf.')
# => "This is a sentence cut off in the middle because pdf."

If you only need to perform the text cleaning and preprocessing steps without performing sentence segmentation, use the PragmaticSegmenter::Cleaner class. This is useful for normalizing text (e.g., handling PDF line breaks or removing unwanted characters) before passing it to another tool.

text = "This is a sentence\ncut off in the middle because pdf."
ps = PragmaticSegmenter::Cleaner.new(text: text, doc_type: 'pdf')
ps.clean
# => "This is a sentence cut off in the middle because pdf."

The segmenter includes a cleaning and preprocessing step to normalize text. If you want to preserve the original text exactly as it is (including potential artifacts or specific formatting), you can disable this feature.

# Example disabling cleaning
segmenter = PragmaticSegmenter::Segmenter.new(cleaner: false)
segmenter.segment('This is a sentence cut off in the middle because pdf.')
# => ["This is a sentence cut", "off in the middle because pdf."]

When segmenting text extracted from PDF documents, you can set the pdf_document_type option. This helps the segmenter handle cases where sentences might be cut off in the middle due to PDF formatting artifacts.

# Example for PDF text
segmenter = PragmaticSegmenter::Segmenter.new(pdf_document_type: true)
segmenter.segment('This is a sentence cut off in the middle because pdf.')
# => ["This is a sentence cut off in the middle because pdf."]

By default, Pragmatic Segmenter performs text cleaning (e.g., removing 'table of contents' style periods or xhtml code). If you want to preserve the original text structure and prevent cleaning, set the clean option to false in the Segmenter constructor.

text = "This is a sentence\ncut off in the middle because pdf."
ps = PragmaticSegmenter::Segmenter.new(text: text, language: 'en', doc_type: 'pdf', clean: false)
ps.segment
# => ["This is a sentence cut", "off in the middle because pdf."]

The PragmaticSegmenter::Segmenter class is used for rule-based sentence boundary detection. By default, it uses English if no language is specified.

To specify a language, use its two-character ISO 639-1 code. You can also specify a doc_type (e.g., 'pdf') to handle specific formatting issues like line breaks in the middle of sentences. To prevent the library from performing its default text cleaning and preprocessing, set clean: false.

text = "Hello world. My name is Mr. Smith."
ps = PragmaticSegmenter::Segmenter.new(text: text)
ps.segment
# => ["Hello world.", "My name is Mr. Smith."]

By default, the segmenter uses English rules. To segment text in a different language, specify the language during initialization. This ensures the correct abbreviations and sentence-starting rules are applied for that specific language.

# Example for Armenian
segmenter = PragmaticSegmenter::Segmenter.new(language: 'Armenian')
segmenter.segment('Այսօր երկուշաբթի է: Ես գնում եմ աշխատանքի:')
# => ["Այսօր երկուշաբթի է:", "Ես գնում եմ աշխատանքի:"]

Pragmatic Segmenter provides specialized support for languages that use sentence boundary punctuation differently than English. The currently supported languages are:

• Amharic
• Arabic
• Armenian
• Burmese
• Chinese
• Greek
• Hindi
• Japanese
• Persian
• Urdu

To use the library, require pragmatic_segmenter and instantiate the PragmaticSegmenter::Segmenter class. You can pass an options hash to the constructor to configure behavior such as the target language or document type.

Basic usage involves creating a new instance and calling the .segment method on your text.

require 'pragmatic_segmenter'

segmenter = PragmaticSegmenter::Segmenter.new(text: "Your text here")
sentences = segmenter.segment