ConvoKit Documentation

The GenAI module provides a unified interface for integrating Large Language Models (LLMs) into ConvoKit conversational analysis workflows. It abstracts the differences between various LLM providers (like OpenAI GPT and Google Gemini) into a consistent interface, handling API key management and response formatting.

Key components include:

• LLMClient: The abstract base class defining the interface for all clients.
• LLMResponse: A unified wrapper that standardizes outputs from different providers.
• Factory Pattern: A factory function used to instantiate the appropriate client.
• GenAIConfigManager: Manages API keys and provider-specific configurations.
• LLMPromptTransformer: A ConvoKit transformer that applies custom prompts to corpus objects (utterances, conversations, speakers, or the entire corpus).

The politenessStrategies module allows you to extract politeness features and markers from conversations. It currently supports three distinct strategy collections across two languages:

• politeness_api: English politeness strategies based on the computational approach described in A computational approach to politeness with application to social factors.
• politeness_local: English politeness strategies realized through local markers, as used in Facilitating the Communication of Politeness through Fine-Grained Paraphrasing.
• politeness_cscw_zh: Chinese politeness strategies adapted from Studying Politeness across Cultures using English Twitter and Mandarin Weibo.

You can use these features to analyze conversational dynamics, such as understanding the misuse of politeness in 'conversations gone awry' or assessing how politeness markers behave in machine-translated text.

The data model follows a specific hierarchy where each component is identified by a unique ID (any arbitrary string):

• Corpus: The top-level container holding conversations.
• Conversation: A grouping of one or more utterances.
• Utterance: An individual piece of text spoken by a participant. Each utterance stores its text and timestamp.
• Speaker: The entity that authored an utterance. A single Speaker can participate in multiple Conversations.

Relationships:

• Corpus $\rightarrow$ Conversation $\rightarrow$ Utterance $\rightarrow$ Speaker.

If you do not provide a config object, the following default hyperparameters are used based on the model class:

TransformerEncoderModel

• per_device_batch_size=4
• gradient_accumulation_steps=1
• num_train_epochs=1
• learning_rate=6.7e-6

TransformerDecoderModel

• per_device_batch_size=2
• gradient_accumulation_steps=32
• num_train_epochs=1
• learning_rate=1e-4

The LLMPromptTransformer is a ConvoKit transformer designed to apply custom LLM prompts to different levels of a corpus. You can target:

• Utterances
• Conversations
• Speakers
• The entire corpus

It allows for fine-grained control over how data is formatted before being sent to the LLM and determines where the resulting analysis is stored within the ConvoKit objects.

SCD (Summary of Conversation Dynamics) is a ConvoKit Transformer that generates structured representations of conversational dynamics from transcripts. It produces two types of outputs:

1. Summary of Conversation Dynamics (SCD): A high-level summary describing the overall dynamics within a conversation.
2. Sequence of Patterns (SoP): A structured sequence of interaction patterns extracted from the SCD, used for comparing conversational dynamics.

Prerequisite: LLM Setup SCD computation requires access to a Large Language Model (LLM). Before using the SCD Transformer, you must configure an LLM using the convokit.genai module. It is recommended to set up your GenAI models in that module first to ensure the SCD Transformer can successfully call the required model APIs.

A Corpus is composed of three primary interconnected components:

1. Conversations: A sequence of Utterances.
2. Utterances: Individual units of speech/text, each made by a Speaker and belonging to a Conversation.
3. Speakers: The entities making the Utterances.

Data Structure

Each component contains:

• Primary data fields: Core information like id, text, timestamp, and reply_to (for Utterances).
• Metadata attributes: Additional features or labels (e.g., Reddit scores, speaker demographics) accessible via the .meta attribute.

# Inspecting an utterance's primary fields and metadata
utt = corpus.random_utterance()
print(utt.id)
print(utt.text)
print(utt.meta['subreddit'])

ConvoKit offers two backend options for data handling:

• Native Python Backend (Default): Keeps all data in memory during runtime. This is suitable for most use cases and requires no additional setup.
• MongoDB Backend: Recommended for low-memory environments or real-time applications. This requires additional setup and configuration.

ConvoKit provides a ForecasterModel base class that can be subclassed to implement various forecasting architectures. To add a new model to the ConvoKit ecosystem (and potentially the official leaderboard), you should create a new subclass of ForecasterModel implementing the required forecasting logic.

Available model types include:

• DecisionPolicy
• CRAFTModel
• TransformerEncoderBasedModel
• TransformerDecoderBasedModel

To include a new model in the project's performance leaderboard, you must provide the ForecasterModel class and a version of the Run Transformer Fine-tuned Models.ipynb notebook that generates the corresponding leaderboard line.