pyLDAvis Documentation

pyLDAvis is a Python library designed for the interactive web-based visualization of topic models (specifically Latent Dirichlet Allocation - LDA). It extracts information from a fitted LDA topic model to help users interpret topics within a corpus of text data.

Key features include:

• Interactive visualizations designed for use within IPython notebooks.
• Ability to save visualizations as stand-alone HTML files for sharing.
• A Python port of the original R package LDAvis.

To install the stable version of pyLDAvis, use the standard Python package manager pip.

pip install pyldavis

pyLDAvis is model-agnostic. To visualize a custom topic model, you must transform your data into the format required by the pyLDAvis.prepare function.

To use the prepare function, you need to provide the following components:

• topic-term distributions (often denoted as $\phi$)
• document-topic distributions (often denoted as $\theta$)
• document lengths
• vocabulary
• term frequencies

Once prepared, you can use pyLDAvis.display(vis_data) to show the visualization in a notebook, pyLDAvis.save_html(vis_data, filename) to save it as a standalone HTML file, or pyLDAvis.show(vis_data) to serve it.

You can install the library using pip:

pip install pyldavis

To avoid calling pyLDAvis.display(vis_data) manually every time you want to see a visualization in a Jupyter notebook, you can enable automatic display with pyLDAvis.enable_notebook().

import pyLDAvis

pyLDAvis.enable_notebook()
# After this, visualizations will display automatically in the notebook output

To ensure that pyLDAvis visualizations render automatically within a Jupyter Notebook cell without needing to explicitly call the object, use pyLDAvis.enable_notebook().

import pyLDAvis
pyLDAvis.enable_notebook()

You can visualize a GraphLab Create topic_model using the pyLDAvis.graphlab module. This requires passing both the trained topic_model object and the original document collection (docs) to the pyLDAvis.graphlab.prepare() function. This produces an interactive visualization of the topics and their relationships.

import graphlab as gl
import pyLDAvis
import pyLDAvis.graphlab

# Assuming topic_model and docs are already defined
vis = pyLDAvis.graphlab.prepare(topic_model, docs)
vis

Instead of using the generic prepare function, you can use the pyLDAvis.gensim_models module to visualize models trained with Gensim. This helper function accepts a trained Gensim LDA model, the corpus, and the dictionary.

Example usage:

import pyLDAvis.gensim_models as gensimvis

# lda: trained Gensim LdaModel
# corpus: Gensim corpus
# dictionary: Gensim Dictionary
vis_data = gensimvis.prepare(lda, corpus, dictionary)

By default, pyLDAvis projects topics onto a 2D plane using PCoA (Principal Coordinate Analysis) based on a distance matrix created using Jensen-Shannon divergence on the topic-term distributions.

You can change the dimensionality reduction method using the mds parameter in pyLDAvis.prepare(). Supported options include:

• 'pcoa' (default)
• 'tsne' (requires sklearn installed)
• 'mmds' (requires sklearn installed)

Note that tsne and mmds still operate on the same JS-divergence distance matrix.

import pyLDAvis

# Using t-SNE for projection
pyLDAvis.prepare(mds='tsne', **movies_model_data)

# Using MMDs for projection
pyLDAvis.prepare(mds='mmds', **movies_model_data)