repository·master·Indexed 23 days ago
https://github.com/arthurheitmann/arctic_shiftProject Arctic Shift provides researchers and moderators with access to Reddit data through large data dumps, web interfaces, and a specialized API. The API allows for querying posts, comments, subreddits, and users, retrieving comment trees, resolving short links, and accessing time series data for global or subreddit-specific counts and scores.
Arctic Shift provides several ways to interact with Reddit data depending on your needs:
The project provides auto-generated JSON schemas and TypeScript interfaces based on data dumps (from Pushshift and subsequent sources). These schemas are organized into three categories:
RX/20XX/.... These provide high-fidelity TypeScript interfaces for individual dumps.RX/20XX.ts. These merge individual dump schemas for a specific year.RX.ts. These merge all available dumps into a single schema.Note that while original file schemas include detailed usage statistics, merged schemas (yearly and global) do not.
When using the original dump schemas (found in RX/20XX/...), the TypeScript interfaces include metadata to help you understand data distribution:
crosspost_parent?: string, // 628166/16445391 (3.82%). The numbers represent the absolute and relative number of objects containing that field.Accompanying each TypeScript interface is a non-standard JSON file containing detailed statistics, including:
Data is archived immediately upon posting, meaning score and num_comments may initially be 1 or 0. After approximately 36 hours, all data is updated and should match the values found in the .zst dumps.
Aggregated data in the /api/time_series endpoint might not be 100% accurate and can take a few hours or days to fully update.
If you need to request data removal or require generic support, use the official Google Form.
You can verify if your data is currently in the dataset by searching for your username using the Search Tool.
To process Reddit data dumps locally using the provided helper scripts, follow these steps. Note that you need at least Python 3.10 installed.
git clone --recursive https://github.com/ArthurHeitmann/arctic_shift.git
cd arctic_shiftzstandard to handle compressed files:pip install zstandardscripts/processFiles.py in your editor.fileOrFolderPath variable and enter the path to your data file or folder (use raw strings to avoid escaping backslashes).processFile function.git clone --recursive https://github.com/ArthurHeitmann/arctic_shift.git
cd arctic_shift
pip install zstandardArctic Shift archives are available via torrents on Academic Torrents in two primary formats:
zst_blocks format. Note that zst_blocks availability was discontinued starting from 2024-04.It is requested that users seed the torrents for as long as possible to assist others, as download speeds may be slow shortly after a new release.
For specific questions regarding zst files, contact u/Watchful1 on Reddit.
Arctic Shift provides an API for querying Reddit data.
Base URL: https://arctic-shift.photon-reddit.com
Status Page: https://status.arctic-shift.photon-reddit.com
Search UI: https://arctic-shift.photon-reddit.com/search
Note: There are no uptime or performance guarantees.
If you receive a 429 error code, you are being rate limited. Limits are calculated dynamically based on load and complexity. To know when you can retry, check the following headers:
X-RateLimit-Reset: Seconds until reset.X-RateLimit-Reset-At: Timestamp of the next reset.Note: For massive data processing, use the monthly dumps instead of the API.
If you receive the message "Query timed out", your parameter combination is likely unoptimized. To fix this:
limit parameter.after, before, subreddit, or author.body, selftext, or title if possible.Use the /api/time_series endpoint to retrieve aggregated data over time (e.g., post counts or subscriber counts). Note that data may take a few hours or days to fully update and might not be 100% accurate.
| Parameter | Type | Default | Notes |
|---|---|---|---|
key | string | Time series type or category. See available keys below. | |
precision | string | One of year, quarter, month, week, day, hour, minute. | |
after | Date | Start date. If not set, the earliest date of the data is used. | |
before | Date | End date. If not set, the latest date of the data is used. |
global/posts/count, global/comments/count (Total posts/comments across Reddit)global/posts/sum_score, global/comments/sum_score (Sum of all upvotes)global/posts/sum_retrieved_after_seconds, global/comments/sum_retrieved_after_seconds (Sum of time between creation and archive)r/<subreddit>/posts/count, r/<subreddit>/comments/count (Posts/comments in a specific subreddit)r/<subreddit>/posts/sum_score, r/<subreddit>/comments/sum_score (Sum of upvotes in a subreddit)r/<subreddit>/subscribers (Number of subscribers in a subredditUse the /api/users/interactions/users endpoint to get aggregated counts of interactions between users. An interaction is defined as:
Use /api/users/interactions/users/list to list all individual interactions between users.
Warning: For very active users (e.g., bots), this endpoint may time out. If it does, try narrowing the time range using after and before parameters.
| Parameter | Type | Default | Notes |
|---|---|---|---|
author | string | required | |
subreddit | string | ||
after | Date | ||
before | Date | ||
min_count | int (>= 0) | Minimum number of interactions | |
limit | int (>= 1) | 100 | empty means no limit (limit=) |
Use /api/users/interactions/subreddits to see which subreddits a user has been active in. This combines post and comment activity with weighting options.
| Parameter | Type | Default | Notes |
|---|---|---|---|
author | string | required | |
weight_posts | float | 1.0 | |
weight_comments | float | 1.0 | |
before | Date | ||
after | Date | ||
min_count | int (>= 0) | ||
limit | int (>= 1) | 100 | empty means no limit (limit=) |
# Get all interactions of u/spez, with more than 10 interactions, up until 2017
/api/users/interactions/users?author=spez&before=2017-01-01&min_count=10Use the ID lookup endpoints to retrieve specific Reddit objects based on their unique identifiers.
Endpoints:
/api/posts/ids/api/comments/ids/api/subreddits/ids/api/users/idsParameters:
ids (Required): A comma-separated list of IDs. Maximum limit is 500.md2html (boolean): If true, adds auto-generated selftext_html or body_html fields.fields (string): A comma-separated list of specific fields to return.