DOCSEARCHbyAlgolia
Submit Docs

X Algorithm (Phoenix Recommendation System)

repository·main·Indexed 12 days ago

https://github.com/xai-org/x-algorithm

The core recommendation system for the X 'For You' feed. It features Phoenix, a Grok-based transformer model implemented in JAX that uses a two-stage pipeline: a Two-Tower Model for efficient retrieval of out-of-network content via approximate nearest neighbor (ANN) search, and a Transformer with Candidate Isolation for ranking candidates based on predicted engagement probabilities (likes, reposts, replies).

Tokens
9.4K
Snippets
17
Records
37
Agent score
98%

What's inside X Algorithm

  1. Overview of the X For You Feed Algorithm

    main
    The X For You feed algorithm is a recommendation system that combines in-network content (from followed accounts) with out-of-network content (discovered via ML-based retrieval). The system uses Phoenix, a Grok-based transformer model, to rank content by predicting engagement probabilities. The final feed score is a weighted combination of these predicted engagements. The architecture aims to replace hand-engineered features and heuristics with a transformer-based approach that understands user engagement history (likes, replies, shares, etc.) directly.

  2. Overview of the Phoenix Recommendation System

    main

    Phoenix is a recommendation system designed to predict user engagement (such as likes, reposts, and replies) for content. It is implemented using JAX and follows a two-stage pipeline to handle content at scale:

    1. Retrieval: Uses a Two-Tower Model to efficiently narrow down millions of potential candidates to a few hundred using approximate nearest neighbor (ANN) search.
    2. Ranking: Uses a Transformer with Candidate Isolation to score and order the small set of retrieved candidates based on predicted engagement.

    This repository provides a mini version of the model (128-dim, 4-layer transformer) and a frozen checkpoint for demonstration purposes.

  3. How the Phoenix Two-Stage Recommendation Pipeline works

    main

    Phoenix operates using a two-stage architecture to balance efficiency and precision:

    1. Retrieval (Two-Tower Model)

    At this stage, the system must find relevant candidates from a massive pool (millions of items). It uses a Two-Tower Model architecture, which is optimized for speed and allows for approximate nearest neighbor (ANN) search to quickly select a subset of hundreds of candidates.

    2. Ranking (Transformer with Candidate Isolation)

    Once the candidate pool is narrowed down, the Ranking stage applies a more expressive transformer model. This model uses Candidate Isolation (via custom attention masking) to score and order the retrieved candidates by their predicted engagement levels.

  4. Understand the Grox content-understanding pipeline

    main

    The grox/ service provides a suite of tools for content understanding workloads. It includes:

    • Classifiers
    • Embedders
    • Task-execution engine

    These components are used for tasks such as spam detection, post-category classification, and PTOS policy enforcement.

  5. How the X For You Feed System Architecture works

    main

    The X For You Feed algorithm follows a multi-stage pipeline orchestrated by the HOME MIXER. The process transforms a raw feed request into a ranked response through the following stages:

    1. Query Hydration: The system gathers context by combining the user's User Action Sequence (engagement history) and User Features (following list, preferences, etc.).
    2. Candidate Sources: Potential posts are retrieved from two primary sources:
      • THUNDER: Provides in-network posts from accounts the user follows.
      • PHOENIX RETRIEVAL: Provides out-of-network posts using ML-based similarity search across the global corpus.
    3. Hydration: Fetches necessary metadata for the candidates, such as core post metadata, author info, and media entities.
    4. Filtering: Removes duplicates, old posts, self-posts, blocked authors, and posts containing muted keywords.
    5. Scoring: A three-step scoring process:
      • Phoenix Scorer: Uses a Grok-based Transformer to predict probabilities for various actions: P(like), P(reply), P(repost), P(click), etc.
      • Weighted Scorer: Combines these predictions into a single value using the formula: Weighted Score = Σ (weight × P(action)).
      • Author Diversity Scorer: Attenuates scores for repeated authors to ensure the feed remains diverse.
    6. Selection: Sorts candidates by their final score and selects the top $K$ candidates.
    7. Post-Selection Filtering: A final safety layer that filters for visibility (removing deleted posts, spam, violence, gore, etc.) before returning the RANKED FEED RESPONSE.

  6. Configure Ads blending in the Home Mixer

    main
    The home-mixer/ads/ module manages the injection and positioning of advertisements within the feed. It includes brand-safety tracking capabilities designed to respect sensitive content boundaries.

  7. How the Candidate Pipeline framework works

    main

    The candidate-pipeline crate provides a structured framework for building recommendation systems. It uses a series of traits to define the lifecycle of a recommendation request. The framework is designed to run sources and hydrators in parallel where possible, providing configurable error handling and logging.

    Pipeline Traits

    TraitPurpose
    SourceFetch candidates from a data source
    HydratorEnrich candidates with additional features
    FilterRemove candidates that shouldn't be shown
    ScorerCompute scores for ranking
    SelectorSort and select top candidates
    SideEffectRun async side effects (e.g., caching, logging)

  8. Understand the X For You Feed System Architecture

    main

    The X For You Feed algorithm is composed of several specialized components that work together to assemble a personalized feed. The orchestration is handled by the Home Mixer, which uses the Candidate Pipeline framework to move through stages of hydration, sourcing, filtering, and scoring.

    Core Components

    • Home Mixer: The orchestration layer. It exposes a gRPC endpoint (ScoredPostsService) to return ranked posts for a user.
    • Thunder: An in-memory post store and real-time ingestion pipeline. It consumes Kafka events to maintain per-user stores (original posts, replies, videos) and provides sub-millisecond lookups for "in-network" content (posts from accounts the user follows).
    • Phoenix: The Machine Learning component. It handles both Retrieval (using a Two-Tower model to find out-of-network posts via similarity search) and Ranking (using a Transformer model to predict engagement probabilities).
    • Candidate Pipeline: A reusable framework that defines the traits used to build recommendation pipelines, such as Source, Hydrator, Filter, Scorer, Selector, and SideEffect.

  9. How scoring and ranking are calculated

    main

    The system uses a Phoenix Grok-based transformer model to predict probabilities for a wide range of engagement actions.

    Predicted Actions

    The model outputs probabilities for:

    • P(favorite), P(reply), P(repost), P(quote), P(click), P(profile_click), P(video_view), P(photo_expand), P(share), P(dwell), P(follow_author), P(not_interested), P(block_author), P(mute_author), P(report)

    Final Score Calculation

    The Weighted Scorer computes the final relevance score by summing the products of each action's probability and its assigned weight:

    Final Score = Σ (weight_i × P(action_i))

    Positive actions (e.g., like, repost) have positive weights, while negative actions (e.g., block, mute, report) have negative weights to suppress undesirable content.

  10. How Phoenix ranking and retrieval work

    main

    Phoenix uses a two-stage recommendation pipeline:

    1. Retrieval (Two-Tower Model): Uses a retrieval user tower (sharing the same transformer architecture as the ranking model) to find the top-N most relevant posts from a large corpus using dot-product similarity.
    2. Ranking (Transformer with Candidate Isolation): Takes the retrieved candidates and predicts multiple engagement types simultaneously.

    Key Design Features:

    • Hash-Based Embeddings: Both models use multiple hash functions for embedding lookups.
    • Multi-Action Prediction: The ranking model outputs a tensor of shape [B, num_candidates, num_actions], allowing it to predict probabilities for various engagement types (e.g., Like, Repost, Reply, Click) at once.

  11. Ranking using Transformer with Candidate Isolation

    main

    The ranking stage uses a transformer architecture to score candidates. A critical design feature is Candidate Isolation: during inference, candidates are prevented from attending to one another via a specialized attention mask.

    This ensures that the score assigned to a specific candidate is independent of which other candidates are present in the current batch, preventing batch-dependent bias in ranking.

    Model Inputs:

    • User Embedding: [B, 1]
    • History Embeddings: [B, S, D] (includes Posts, Authors, Actions, and Product Surface data)
    • Candidate Embeddings: [B, C, D] (includes Posts, Authors, and Product Surface data)

    Model Output:

    • Output Logits: [B, num_candidates, num_actions] (produced via an unembedding projection after extracting candidate outputs from the positions following the history).
    PHOENIX RANKING MODEL
    ┌────────────────────────────────────────────────────────────────────────────┐
    │                                                                            │
    │                              OUTPUT LOGITS                                 │
    │                        [B, num_candidates, num_actions]                    │
    │                                    │                                       │
    │                                    │ Unembedding                           │
    │                                    │ Projection                            │
    │                                    │                                       │
    │                    ┌───────────────┴───────────────┐                       │
    │                    │                               │                       │
    │                    │    Extract Candidate Outputs  │                       │
    │                    │    (positions after history)   │                       │
    │                    │                               │                       │
    │                    └───────────────┬───────────────┘                       │
    │                                    │                                       │
    │                    ┌───────────────┴───────────────┐                       │
    │                    │                               │                       │
    │                    │         Transformer           │                       │
    │                    │     (with special masking)   │                       │
    │                    │                               │                       │
    │                    │   Candidates CANNOT attend   │                       │
    │                    │   to each other               │                       │
    │                    └───────────────┬───────────────┘                       │
    │                                    │                                       │
    │    ┌───────────────────────────────┼───────────────────────────────┐       │
    │    │                               │                               │       │
    │    ▼                               ▼                               ▼       │
    │ ┌──────────┐              ┌─────────────────┐              ┌────────────┐      │
    │ │   User   │              │     History     │              │ Candidates │      │
    │ │Embedding │              │   Embeddings    │              │ Embeddings │      │
    │ │  [B, 1]  │              │    [B, S, D]    │              │  [B, C, D] │      │
    │ │ User     │              │                 │              │            │      │
    │ │ Hashes   │              │ Posts + Authors │              │ Posts +    │      │
    │ │          │              │ + Actions +     │              │ Authors +  │      │
    │ │          │              │ Product Surface │              │ Product    │      │
    │ └──────────┘              └─────────────────┘              │ Surface    │      │
    │                                                            └────────────┘      │
    │                                                                            │
    └────────────────────────────────────────────────────────────────────────────┘

  12. Retrieval using the Two-Tower Model

    main

    The retrieval stage is designed for efficient similarity search at scale using a two-tower architecture. It works by mapping users and items into the same embedding space:

    1. User Tower: Processes user features and engagement history through a transformer to generate a normalized user embedding of shape [B, D].
    2. Candidate Tower: Computes normalized embeddings for all items in the corpus, resulting in shape [N, D].
    3. Similarity Search: The top-K candidates are selected based on the dot product similarity between the user embedding and the candidate embeddings.