DOCSEARCHbyAlgolia
Submit Docs

markovify

repository·master·Indexed 25 days ago

https://github.com/jsvine/markovify

A simple, extensible Markov chain generator for building models from large text corpora and generating novel sentences. It provides tools for model compilation, combining multiple chains via markovify.combine(), and exporting/importing models using JSON. Supports handling large datasets through incremental building and memory management options like retain_original=False.

Tokens
1.5K
Snippets
7
Records
10
Agent score
36%

What's inside markovify

  1. Generate models from very large corpora

    master

    To manage memory when working with massive datasets, use retain_original=False to prevent Markovify from keeping the entire text corpus in memory. For extremely large datasets, you can build the model incrementally by reading files/lines and combining them using markovify.combine().

    # Option A: Don't retain the original text in memory
with open("path/to/my/huge/corpus.txt") as f:
    text_model = markovify.Text(f, retain_original=False)

# Option B: Incremental building via combining
combined_model = None
for (dirpath, _, filenames) in os.walk("path/to/my/huge/corpus"):
    for filename in filenames:
        with open(os.path.join(dirpath, filename)) as f:
            model = markovify.Text(f, retain_original=False)
            if combined_model:
                combined_model = markovify.combine(models=[combined_model, model])
            else:
                combined_model = model

  3. Handle messy text input

    master

    When instantiating markovify.Text, you can use these parameters to handle non-standard text:

    • well_formed=False: Skips rejecting sentences that contain 'bad characters' like ()[]'".
    • reject_reg: A regular expression used to define the input-sentence rejection pattern (only works if well_formed is True).

  4. Basic usage of markovify.Text

    master

    Use markovify.Text to build a model from a text corpus and generate random sentences. For text that uses newlines instead of periods to delineate sentences, use markovify.NewlineText instead.

    import markovify

# Get raw text as string.
with open("/path/to/my/corpus.txt") as f:
    text = f.read()

# Build the model.
text_model = markovify.Text(text)

# Print five randomly-generated sentences
for i in range(5):
    print(text_model.make_sentence())

# Print three randomly-generated sentences of no more than 280 characters
for i in range(3):
    print(text_model.make_short_sentence(280))

  5. Specify model state size

    master

    The state_size determines how many words the probability of a next word depends on. The default is 2. You can specify a different size during instantiation.

    text_model = markovify.Text(text, state_size=3)

  6. Combine multiple Markov models

    master

    Use markovify.combine() to merge two or more Markov chains of the same type (e.g., all markovify.Text instances).

    model_a = markovify.Text(text_a)
model_b = markovify.Text(text_b)

# Combine models with relative weights
model_combo = markovify.combine([ model_a, model_b ], [ 1.5, 1 ])

  7. Configure make_sentence parameters

    master

    The make_sentence method can be customized to control sentence generation:

    • tries: The number of attempts to make a sentence that does not overlap too much with the original text. Defaults to 10. Example: text_model.make_sentence(tries=100).
    • max_overlap_ratio: Suppress sentences that overlap the original text by more than this ratio of the sentence's word count.
    • max_overlap_total: Suppress sentences that overlap the original text by more than this many words.
    • test_output: Set to False to disable the overlap check entirely.

  8. Export and import models using JSON

    master

    To avoid re-generating models from large corpora, you can export them to JSON and reload them later.

    corpus = open("sherlock.txt").read()
text_model = markovify.Text(corpus, state_size=3)

# Export to JSON string
model_json = text_model.to_json()

# Reconstitute from JSON string
reconstituted_model = markovify.Text.from_json(model_json)

  9. Compile a model for performance

    master
    Compiling a model improves text generation speed and reduces model size. Note that compiled models cannot be combined using markovify.combine(); combine your models first, then compile the result.

  10. Extend markovify.Text via subclassing

    master

    You can customize how sentences and words are processed by overriding specific methods in a subclass of markovify.Text. The most useful methods to override are:

    • sentence_split / sentence_join
    • word_split / word_join
    • test_sentence_input
    • test_sentence_output
    import markovify
import re
import spacy

nlp = spacy.load("en_core_web_sm")

class POSifiedText(markovify.Text):
    def word_split(self, sentence):
        return ["::".join((word.orth_, word.pos_)) for word in nlp(sentence)]

    def word_join(self, words):
        sentence = " ".join(word.split("::")[0] for word in words)
        return sentence