DOCSEARCHbyAlgolia
Submit Docs

textgenrnn

repository·master·Indexed 26 days ago

https://github.com/minimaxir/textgenrnn

A Python 3 module built on Keras/TensorFlow for creating character-level or word-level recurrent neural networks (char-rns). It features a modern architecture with attention-weighting and skip-embeddings to accelerate training and improve text quality. The library supports fine-tuning pretrained models, training new architectures with bidirectional RNN layers, and generating sentence vectors from the Attention layer output. Requires TensorFlow 2.1.0 or higher.

Tokens
5.8K
Snippets
26
Records
34
Agent score
88%

What's inside textgenrnn

  2. Generate a Hacker News dataset for textgenrnn

    master

    You can create a dataset of the top Hacker News submissions by running a query in BigQuery.

    Important Formatting Requirement: When exporting your data, save it as a .tsv file. Do not use .csv, as CSVs enclose sentences with quotes that contain commas, which will corrupt the dataset structure for textgenrnn.

    #standardSQL
SELECT title
FROM `bigquery-public-data.hacker_news.full`
WHERE type = 'story'
ORDER BY score DESC
LIMIT 2000

  3. Use pretrained weights in textgenrnn

    master
    You can load pretrained weights into textgenrnn to generate text based on specific datasets. These weights are trained for 500 epochs on their respective datasets located in the /datasets directory. To use them, load the weight file into your model instance.

  4. Generate a Reddit Subreddit dataset for textgenrnn

    master

    You can create a dataset of top Reddit submissions for specific subreddits using BigQuery.

    Important Formatting Requirement: When exporting your data, save it as a .tsv file. Do not use .csv, as CSVs enclose sentences with quotes that contain commas, which will corrupt the dataset structure for textgenrnn.

    Constraints: Ensure the total number of rows does not exceed 10,000.

    #standardSQL 
SELECT title FROM (
SELECT title,
  ROW_NUMBER() OVER (PARTITION BY subreddit ORDER BY score DESC) as score_rank
  FROM `fh-bigquery.reddit_posts.*`
  WHERE (_TABLE_SUFFIX BETWEEN '2017_01' AND '2017_06')
  AND LOWER(subreddit) IN ("legaladvice", "relationship_advice")
  )
WHERE score_rank <= 1000

  5. Perform transfer learning with textgenrnn

    master

    You can perform transfer learning by training a textgenrnn instance on one dataset and then training the same instance on a second dataset.

    1. Initialize and train the first model: Use train_from_file with new_model=True to start training from scratch on your initial dataset.
    2. Transfer learning: Call train_from_file again on the second dataset without specifying new_model=True.

    Best Practice: When performing the second stage of training (the transfer), use fewer num_epochs than the initial training to avoid overwriting the previous knowledge too aggressively.

  6. Avoid overfitting using train_size and dropout

    master

    To prevent the neural network from learning exact sequences (overfitting), use the following parameters in train_from_file():

    • train_size: A float representing the proportion of sequences used for training. The remaining data is used as a validation set. Monitor validation loss; it should ideally not increase after each epoch.
    • dropout: A float representing the proportion of characters to drop out of a sequence for a given epoch. This forces the model to weigh remaining characters more efficiently.

    Warning on dropout:

    • Do not use if max_length is low.
    • Do not set higher than 0.2 or the model may fail to converge.
    from textgenrnn import textgenrnn

textgen = textgenrnn()
file_path = "../datasets/reddit_rarepuppers_politics_2000.txt"

textgen.reset()
# Using train_size=0.8 and dropout=0.2 to mitigate overfitting
textgen.train_from_file(
    file_path, 
    new_model=True, 
    num_epochs=5, 
    gen_epochs=5, 
    train_size=0.8, 
    dropout=0.2
)

  7. Control model weights and switching behavior in synthesis

    master

    You can control the influence of different models in synthesize by manipulating the input list:

    • Weighted Averaging: To create a weighted influence (e.g., 1/2 model1, 1/4 model2, 1/4 model3), pass a list containing the models in those proportions: [model1, model1, model2, model3].
    • Increased Persistence: To allow a model to generate for multiple tokens before switching, double or triple the models in the list (e.g., models_list * 3).
    • Switching Frequency: For character-level models, pass stop_tokens=[] to force a model switch after every single character.

  9. Initialize and use textgenrnn for text generation

    master

    To use the default pretrained model, import textgenrnn and instantiate the class. Calling .generate() will produce text based on the internal pretrained weights.

    from textgenrnn import textgenrnn

textgen = textgenrnn()
textgen.generate()

  10. Train textgenrnn from a text file

    master

    You can fine-tune the model on a new dataset by using the train_from_file method. Use the num_epochs parameter to control how many passes the model makes through the data. For small datasets, you may need to increase num_epochs to achieve better results.

    textgen.train_from_file('hacker_news_2000.txt', num_epochs=1)

  11. Use Interactive Mode for text generation

    master

    Interactive mode allows you to manually pick from the top N options for the next character or word, providing a human-in-the-loop experience. Pass interactive=True and top_n=N to the generate method.

    from textgenrnn import textgenrnn

textgen = textgenrnn()
textgen.generate(interactive=True, top_n=5)