DOCSEARCHbyAlgolia
Submit Docs

Test Lab Documentation

website·Indexed 29 days ago

https://testl-ab.github.io/

An open-source, self-hosted feature management and A/B testing platform. Test Lab provides a backend server for managing features, experiment variants, users, and events, with native SDKs for Node, React, Ruby, and Python. It utilizes a PostgreSQL database and is Dockerized for deployment to local servers or AWS Elastic Container Service.

Tokens
9.7K
Snippets
27
Records
76
Agent score
74%

What's inside Test Lab

  1. Overview of Test Lab Architecture Components

    Test Lab is a self-hosted, open-source A/B testing and feature management platform modeled after Unleash. It enables users to manage features and collect, retrieve, and analyze event data from experiments. The architecture consists of four primary components:

    1. Backend Server: A Node.js with Express server that provides API endpoints for feature lifecycle management (create, edit, delete), configuration retrieval, and data collection.
    2. Admin User Interface (Admin UI): A React-based dashboard used to manage current, scheduled, and past features and visualize experiment results.
    3. Database: A PostgreSQL instance used to persist feature definitions, user base block allocations for experiments, and event data for analytics.
    4. Native SDKs: Client libraries for Node.js, React, Ruby, and Python that communicate with the Test Lab API to allocate requests to specific features.

  2. Overview of Test Lab SDK functionality

    Test Lab provides native SDKs for Node, React, Ruby, and Python. The SDK maintains an in-memory representation of feature configurations, allowing it to determine if a feature is active or which variant a user should see without making a network request for every evaluation. By providing a unique user_id, the SDK deterministically calculates feature values based on local state.

  3. Overview of Test Lab feature management and A/B testing

    Test Lab is an open-source, self-hosted platform providing infrastructure for feature management and A/B testing. It supports three primary feature types:

    • Toggles: Features that are either active or inactive for all users.
    • Rollouts: Features active for a specified percentage of users.
    • Experiments: Features with two or more variants, where users are assigned a specific version to measure changes in behavior.

    Test Lab provides native SDKs for both server-side and client-side rendered applications to evaluate features and render variants. It also includes capabilities for collecting, analyzing, and visualizing event data to determine experiment outcomes.

  4. Overview of Test Lab Feature Management

    Test Lab is an open-source, self-hosted platform for feature management and A/B testing. It provides infrastructure for three primary types of feature control:

    • Toggles: Features that are either active or inactive for all users.
    • Rollouts: Features active for a specified percentage of the user base.
    • Experiments: Features with two or more variants, where users are assigned to a specific version to measure impact.

  5. Understand server-side A/B testing

    Server-side A/B testing occurs before the page is served to the user. This method can be used to test any aspect of the application controlled by the server, including content, layout, and functionality, avoiding the client-side 'flicker' and Virtual DOM synchronization issues.

  6. Test Lab System Architecture

    The Test Lab platform consists of four main components:

    1. Backend Server: A Node.js and Express server that provides API endpoints for managing features, retrieving configurations, and collecting data.
    2. Admin User Interface (Admin UI): A React-based dashboard for creating, editing, and deleting features, as well as analyzing experiment results.
    3. Database: A PostgreSQL database used to persist feature definitions, user-block allocations for experiments, and event data for analytics.
    4. SDKs: Native libraries for Node.js, React, Ruby, and Python that communicate with the API to allocate requests to the correct feature variants.

  7. Future statistical analysis and experiment conversion features

    While Test Lab currently provides raw and normalized event data via the API and UI for flexible client-side analysis, future plans include:

    1. Providing built-in measures of statistical significance for common experiment types.
    2. Allowing users to convert a winning experiment variant directly into a toggle or a rollout.

  8. Understand Test Lab PostgreSQL database data storage

    Test Lab utilizes a PostgreSQL database to maintain persistence for the following data types:

    • Feature Configuration: Stores data for toggle, rollout, and experiment features, including the feature name, description, and the proportion of users enrolled.
    • Variant Configuration: Stores experiment variant details, including the variant name, value, and the proportion of users within a specific experiment assigned to that variant.
    • User Data: Tracks users enrolled in experiments, storing a unique user identifier, the user's IP address, and the specific variant they experienced.
    • Event Data: Records events triggered by experiments, including the timestamp, the user who triggered the event, and the variant the user was experiencing at the time.
    • User-block Data: Manages the allocation of user blocks to specific experiments at any given point in time.

  9. Planned real-time experiment health monitoring via SSEs

    To improve visibility into experiment errors and overall 'health', Test Lab plans to implement Server-Sent Events (SSEs). This will enable the backend server to communicate errors and health metrics directly to the Admin UI dashboard, ensuring clients can verify that experiments are performing as designed.

  10. Understand the role of the Test Lab Backend Server

    The Test Lab Backend Server manages the core logic and data flow for the Test Lab application. Its primary responsibilities include:

    • API Management: Providing endpoints to create, edit, delete, and retrieve feature data, user data for experiment analysis, and event data from experiments.
    • Data Integrity: Validating and ensuring the integrity of data before it is committed to the Test Lab Database.
    • Experiment Enrollment: Assigning user-blocks to experiments to prevent over-enrollment (capping at 100%) and ensuring users remain consistent within their assigned experiments.
    • SDK Synchronization: Tracking modification timestamps of feature data to allow SDKs to retrieve updates efficiently.

    Developers can interact with the backend server via the Admin UI, official SDKs, or by calling the API directly.

  11. Prevent experiment contamination with user-blocks

    To avoid the complexity and sample size requirements of multivariate analysis, Test Lab uses a 'user-block' approach to ensure a user is exposed to only one experiment at a time. The Admin UI limits total experiment participation to a maximum of 100% of the user base. The backend server provides the SDK with a list of user-blocks; once a block of users is allocated to an experiment, they remain in that experiment until it is completed, preventing them from being assigned to concurrent tests.

  12. Understand the User-Block strategy for experiment isolation

    To prevent multivariate interference and ensure data validity, the Admin UI uses a user-block strategy. The user base is segmented into 20 permanent chunks (user-blocks), each representing 5% of the total user base. Each block is allocated to only one experiment at a time. The Admin UI dynamically calculates and limits the maximum available percentage of users for a new experiment based on existing schedules for the selected date range.