DOCSEARCHbyAlgolia
Submit Docs

ThreatExchange Documentation

repository·main·Indexed 23 days ago

https://github.com/facebook/threatexchange

A platform for exchanging security threat information such as malware and phishing details. This repository provides tools for hashing media, a content moderation platform (HMA), and client libraries for interacting with ThreatExchange RESTful APIs in Go, Java, Node.js, and Python (via Jupyter Notebooks). It includes guides for managing threat descriptors, querying malware analyses, and using the te-tag-query tools for hash metadata retrieval.

Tokens
64.1K
Snippets
128
Records
358
Agent score
80%

What's inside ThreatExchange

  1. Overview of python-threatexchange

    main

    The python-threatexchange library is a Python-based tool designed to simplify the exchange and use of trust and safety information, with a specific focus on media hash exchanges. It is built to be extensible, allowing developers to add new functionality easily.

    In addition to the library, it includes a CLI tool named threatexchange that demonstrates the library's capabilities. For users looking for a production-ready, deployable service version of this functionality, refer to the Hasher-Matcher-Actioner (HMA) platform.

  2. Overview of PDQ, MD5, and TMK Hashing with Web Assembly

    main

    This project provides a method to generate PDQ, MD5, and TMK hashes of content directly in a client browser using Web Assembly (WASM). This approach allows for local hashing without uploading content to a server.

    • PDQ Hashes: Used for photos (e.g., JPEG, GIF, TIFF, PNG).
    • MD5 Hashes: Used by default for all other non-video files.
    • TMK Hashes: Used for video files (e.g., MP4, MOV, WMV).

    The WASM binaries are built from modified C++ code using the emscripten tool and can be hosted on a Node.js server or any standard web server.

  3. Overview of the Vendored PDQ CPP Implementation

    main
    The PDQ C++ implementation is vendored within the vpdq directory as a standalone source tree. This separation is maintained to simplify the VPDQ build system and prevent version conflicts between the main project and the PDQ implementation. This structure ensures that the VPDQ build system remains functional even if PDQ or VPDQ are migrated to independent repositories.

  4. Overview of ThreatExchange sub-projects

    main

    The ThreatExchange repository contains several specialized tools for content moderation, hashing, and signal exchange:

    • PDQ: A photo hashing algorithm that generates 256-bit signatures for image similarity matching.
    • TMK+PDQF (TMK): A video hashing algorithm that generates 256KB signatures for video similarity matching.
    • Video PDQ (vPDQ): A video hashing algorithm that determines similarity based on shared similar frames.
    • Hasher-Matcher-Actioner (HMA): A ready-to-deploy content moderation platform for AWS. It manages lists of known content for scanning and supports connecting to hash exchange programs.
    • Open Media Match: A cloud-agnostic, Docker-based version of the HMA platform currently under construction.
    • python-threatexchange: A Python library and CLI tool available on PyPI as threatexchange. It provides implementations for downloading hashes from the ThreatExchange API and scanning images with PDQ.

  5. Overview of Hashing Algorithms in ThreatExchange

    main

    ThreatExchange provides open-source algorithms to convert content into anonymous digital fingerprints known as "hashes". These hashes are used for similarity matching and content identification. The available algorithms include:

    • PDQ: A perceptual, spectral hashing algorithm that uses Discrete Cosine Transform (DCT). It is designed to match images that are perceived as similar and provides a quality metric to filter out featureless or low-quality images.
    • vPDQ (Video PDQ): A video-similarity-detection algorithm that applies the PDQ image similarity algorithm to video frames to measure video similarity.
    • TMK (Temporal Match Kernel): A video-similarity-detection algorithm developed in collaboration with Facebook AI Research (FAIR).

  7. Deployment and interaction model for HMA

    main

    The Hasher-Matcher-Actioner (HMA) is designed for flexible deployment and integration into existing partner platform stacks.

    Deployment Characteristics:

    • Cloud Agnostic: No dependencies on specific cloud providers; supports AWS, Google Cloud, or on-premises environments.
    • Single-Box Deployment: Supports deployment on a single machine (including developer laptops) for testing and development.
    • Horizontal Scaling: Supports multi-instance deployment for high-throughput requirements.

    Interaction Model:

    • API-First: The primary method of interaction is via APIs to facilitate integration with existing platform stacks.

  8. Understand the Hasher-Matcher-Actioner (HMA) stack

    main

    The HMA platform is built around the python-threatexchange package, which provides the core logic for media matching. You can consume this stack in several ways:

    • As a Library: Use the python-threatexchange package directly within your own Python applications for image hashing (PDQ), signature computation (MD5), database management, and similarity searching.
    • As a Service (API & CLI): The Python package includes a Flask-based REST API and a CLI tool. Installing the package provides all three components (library, API, and CLI) in one.
    • As a Docker Container: For rapid deployment, HMA is distributed as a Docker image containing the API service and a recommended Python stack.

    Core Capabilities:

    • Computation of PDQ image hashes.
    • Computation of other signatures (e.g., MD5 for video).
    • Management of a local database of hashed content.
    • Exchange of hash lists with third parties (NCMEC, GIFCT, StopNCII, ThreatExchange, etc.).
    • Similarity search of candidate hashes against known content.

  9. Understand the Hasher Matcher Actioner (HMA) Roadmap

    main

    The Hasher Matcher Actioner (HMA) is a subproject within the ThreatExchange repository designed for photo/video similarity hashing and content moderation. The current roadmap (as of 9/2025) focuses on hardening the system for production, improving documentation, and expanding capabilities for interacting with Exchanges.

    Key development areas include:

    • Production Hardening: Scaling to support ~10M indexed hashes with 4k lookups per second and addressing memory leaks in the Flask/Python stack.
    • Exchange Signal Management: Implementing features to confirm or deconfirm signals received from third parties to prevent false positive floods.
    • Explainability: Developing UI tools to debug why content matches or fails to match.
    • Advanced Matching: Exploring retroactive matching via in-memory index ring buffers for recently seen content clustering.