repository·master·Indexed 18 days ago
https://github.com/dmarx/psawA minimalist Python wrapper for the Pushshift.io API used to search public Reddit comments and submissions. PSAW handles rate limiting and paging, provides integration with PRAW, and includes a Command Line Interface (CLI). Key features include generator-based search methods (search_comments, search_submissions), result aggregation via the aggs parameter, and raw data access through the .d_ attribute for easy conversion to pandas DataFrames.
Result objects (comments/submissions) provide a .d_ attribute. This attribute is a dictionary containing all the raw data attributes associated with the object. This is particularly useful for converting search results directly into a pandas.DataFrame.
import pandas as pd
# Assuming 'gen' is a generator from search_submissions or search_comments
df = pd.DataFrame([thing.d_ for thing in gen])When you provide an aggs (aggregations) parameter to a search method, the first item yielded by the generator contains the aggregation results. Subsequent items in the generator are the actual search results.
api = PushshiftAPI()
# The first result from this generator is the aggregation dict
gen = api.search_comments(author='nasa', aggs='subreddit')
aggregation_results = next(gen)
# The rest of the generator yields the comments
for comment in gen:
print(comment.body)To debug API requests or see the URLs being called, configure the psaw logger. Set the level to INFO for basic request visibility or DEBUG for detailed messages.
import logging
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
logger = logging.getLogger('psaw')
logger.setLevel(logging.INFO)
logger.addHandler(handler)To use PSAW, import PushshiftAPI from psaw. You can initialize it in two ways:
praw.Reddit instance to the constructor. This allows PSAW to fetch full Reddit objects (using IDs from Pushshift) instead of just Pushshift data objects.Note: This repository is stale. Consider using PMAW instead.
# Standalone usage
from psaw import PushshiftAPI
api = PushshiftAPI()
# Integration with PRAW
import praw
from psaw import PushshiftAPI
r = praw.Reddit(...)
api = PushshiftAPI(r)Install the psaw package using pip to begin using the Pushshift.io API wrapper.
pip install psawUse api.search_submissions() and api.search_comments() to query Reddit data. These methods return generator objects that yield results.
By default, omitting the limit parameter will attempt to return all historical results, which may take a long time.
limit: The maximum number of results to return.subreddit: Filter by a specific subreddit.q: Search for specific text.after: A Unix timestamp to search for posts after this time.filter: A list of specific fields to return (e.g., ['url', 'author']).stop_condition: A callable (lambda) that takes a result object and returns True to stop the generator.# Get 100 most recent submissions
gen = api.search_submissions(limit=100)
results = list(gen)
# Search for specific text in a subreddit
gen = api.search_comments(q='OP', subreddit='askreddit')
# Search with a stop condition (e.g., stop when an author contains 'bot')
gen = api.search_submissions(stop_condition=lambda x: 'bot' in x.author)When you provide an aggs argument to a search method, the first item yielded by the generator will be the aggregation result (a dictionary), rather than a comment or submission object. Subsequent items in the generator will be the actual search results.
Example: Aggregating by subreddit for a specific author.
api = PushshiftAPI()
gen = api.search_comments(author='nasa', aggs='subreddit')
# The first call to next() retrieves the aggregation dictionary
agg_results = next(gen)
print(agg_results)
# Subsequent items are the actual comments
for comment in gen:
print(comment.body)Use api.search_comments() and api.search_submissions() to query the Pushshift API. These methods return generator objects, meaning they yield results one by one and support paging automatically.
limit parameter to stop after a certain number of results. Omitting it performs a full historical search.filter parameter to return only specific fields (e.g., ['url', 'author']).after or before with Unix timestamps to scope searches.stop_condition argument to stop yielding results based on custom logic (e.g., finding the first bot account).# Get 100 most recent submissions
gen = api.search_submissions(limit=100)
results = list(gen)
# Search with specific filters and time range
import datetime as dt
start_epoch = int(dt.datetime(2017, 1, 1).timestamp())
results = list(api.search_submissions(
after=start_epoch,
subreddit='politics',
filter=['url', 'author', 'title', 'subreddit'],
limit=10
))
# Use a stop_condition
gen = api.search_submissions(stop_condition=lambda x: 'bot' in x.author)
for subm in gen:
passThe redditor_subreddit_activity method is a convenience shorthand to profile a user's activity. It returns a dictionary containing two collections.Counter objects: one for comment activity and one for submission activity, keyed by subreddit name.
api = PushshiftAPI()
result = api.redditor_subreddit_activity('nasa')
# Returns: {'comment': Counter({...}), 'submission': Counter({...})}You can access metadata from the most recent successful Pushshift request via api.metadata_. Useful keys include:
api.metadata_.get('shards'): Check if any shards are down (can impact result count).api.metadata_.get('total_results'): The database-side count of total items found for the query.PSAW provides several ways to access the underlying data:
thing.d_: Every result object (comment/submission) has a .d_ attribute which is a dictionary containing all the data attributes. This is highly useful for converting results directly into a pandas.DataFrame.api.metadata_: Contains metadata from the most recent successful API request. Useful keys include:api.metadata_.get('shards'): Check if any shards are down.api.metadata_.get('total_results'): The database-side count of total items found for the query.import pandas as pd
# Convert search results to a DataFrame
df = pd.DataFrame([thing.d_ for thing in gen])
# Access metadata
total = api.metadata_.get('total_results')PSAW includes a CLI for usage outside of a Python environment. To see available commands and options, run:
psaw --help