DOCSEARCHbyAlgolia
Submit Docs

WebPageTest Documentation

repository·master·Indexed 25 days ago

https://github.com/catchpoint/webpagetest

The official performance-testing engine used at webpagetest.org. This repository provides tools for running web performance tests, bulk testing via REST API, and local development setup using Docker Compose with Nginx, PHP 7.4-FPM, and WPT Agents. It includes documentation on configuring environment settings, debugging PHP with XDebug, running unit tests with PHPUnit, and integrating with the AWS SDK for PHP and PHPStats.

Tokens
12.6K
Snippets
22
Records
63
Agent score
84%

What's inside WebPageTest

  1. Overview of RequestCore

    master
    RequestCore is a lightweight PHP HTTP request/response class built on top of cURL. It is designed as a lightweight alternative to PEAR HTTP_Request, focusing on essential features. A key capability is its support for parallel requests by leveraging cURL's curl_multi_exec() via MultiCurl, allowing multiple HTTP requests to be executed in parallel.

  3. AWS SDK for PHP Features

    master

    The AWS SDK for PHP provides several high-level features:

    • HTTP Clients: Easy-to-use clients for all supported AWS services, regions, and authentication protocols (built on Guzzle).
    • Convenience Tools: Includes Paginators for result pagination, Waiters, and simple Result objects.
    • S3 Multipart Uploader: A tool for Amazon S3 and Amazon Glacier that supports pausing and resuming uploads.
    • S3 Stream Wrapper: Allows using native PHP file handling functions to interact with S3 buckets like a local filesystem.
    • DynamoDB Session Handler: Enables scaling sessions using Amazon DynamoDB.
    • IAM Integration: Automatically uses IAM Instance Profile Credentials when running on configured Amazon EC2 instances.

  4. Understand the Laravel adoption architecture

    master

    WebPageTest is migrating its www codebase to the Laravel framework using a three-phase approach to minimize disruption. The migration moves the application logic from a legacy structure into a Laravel root directory (referred to in documentation as possum/ or [LARAVELROOT]).

    Key architectural shifts include:

    • Routing: Moving from server configurations and /www/index.php to [LARAVELROOT]/routes/web.php.
    • Templating: Transitioning from legacy templates to the Blade templating engine located in [LARAVELROOT]/resources/views/.
    • Logic: Moving legacy code into [LARAVELROOT]/app/wpt/ and eventually into formal Laravel Controllers and Models.
    • Public Access: Only the [LARAVELROOT]/public directory is intended to be publicly facing.

  5. Understand Time to Interactive (TTI) variations

    master

    WebPageTest provides two variations of the Time to Interactive (TTI) metric to measure when a page becomes usable and responsive to user input:

    1. Time to Consistently Interactive: Measures the point where the page is likely complete and will consistently respond quickly. It follows the canonical WICG definition.
    2. Time to First CPU Idle (formerly First Interactive): A newer metric reporting when the page is first expected to be usable and respond to input quickly, even if subsequent content loading might cause temporary slowness.

  6. Hardware requirements for Raspberry Pi Mobile Agent host

    master

    To use a Raspberry Pi as a tethered host for driving the WebPageTest mobile agent, the following hardware is recommended:

    • Raspberry Pi: 1GB RAM minimum. Raspberry Pi 3 Model B is recommended (Model B 2 also works).
    • MicroSD Card: 8GB minimum (32GB Samsung Evo+ recommended for performance/endurance).
    • USB Cables (2.0 A Male to Micro B):
      • One to connect the phone to the Pi.
      • One to connect the Pi to power (if not using a direct AC adapter).
    • Ethernet Cable: Recommended for hard-wired connections to support reverse-tethering.
    • Power Supply: Optional (e.g., 10-way USB power chargers for multiple devices).
    • Switch: Optional (e.g., Netgear ProSafe 8-port switches for grouping devices).

    For initial setup, a USB keyboard and an HDMI monitor are required.

  7. Run a standalone WPT Agent with the Dockerized Server

    master

    If you want to run a standalone agent instead of the one bundled in the Docker Compose setup, follow these steps:

    1. Disable the bundled agent: In docker-compose.yml, comment out the agent: section:
      #### DOCKER WPTAGENT - comment this out to run a standalone agent ####
agent:
  cap_add:
    - NET_ADMIN
  build:
    context: .
    dockerfile: docker/local/Dockerfile-wptagent
  environment:
    - SERVER_URL=http://web/work/
    - LOCATION=Test
    - KEY=123456789
  init: true
#### ####
    2. Install WPT Agent: Follow the official WPT Agent installation guide.
    3. Verify Agent locally: Run a test without the server connection first:
      python3 wptagent.py -vvvv --xvfb --testurl www.google.com
    4. Connect Agent to Docker Server: Point the agent to your local Docker container using the --server flag. The default configuration values (Location and Key) are defined in docker/local/wptconfig/locations.ini:
      python3 wptagent.py -vvvv --xvfb --server http://127.0.0.1:80/work/ --location Test --key 123456789
    python3 wptagent.py -vvvv --xvfb --server http://127.0.0.1:80/work/ --location Test --key 123456789

  8. Configure PC setup requirements for Microsoft Edge on private instances

    master

    To run Microsoft Edge tests on a private WebPageTest instance, the host machine must meet the following requirements:

    1. Operating System: Windows 10
    2. Python Environment: Python 2.7 must be installed.
    3. Selenium: Install the Selenium library via pip.
    4. Windows Performance Toolkit: Install the Windows Performance Toolkit and ensure xperf is added to the system PATH.
    pip install selenium

  9. Set up WebPageTest locally using Docker Compose

    master

    To run a local development environment for WebPageTest including Nginx, PHP 7.4-FPM, and a WPT Agent, follow these steps:

    1. Clone the repository:
      git clone https://github.com/catchpoint/WebPageTest.git
cd webpagetest
    2. Build and start the containers:
      sudo docker-compose up
    3. Access the WebPageTest homepage at http://localhost.

    Note on Platforms:

    • Macintosh/Linux: Tested.
    • Windows: Untested but expected to work.
    • WSL2: Known to fail because WSL2 lacks the network interface required for Traffic Shaping used by the WPT Agent.

    Traffic Shaping: To ensure traffic shaping works on supported platforms, you must run docker-compose with sudo privileges.

    git clone https://github.com/catchpoint/WebPageTest.git
cd webpagetest
sudo docker-compose up

  10. Define custom Hero Elements in WebPageTest

    master

    WebPageTest allows you to measure when specific critical elements appear on screen. You can define custom hero elements using two methods:

    1. Via HTML attribute: Add the elementtiming attribute to any element (based on the Hero Text Element Timing proposal).
    2. Via WebPageTest UI: In the Custom tab, enter a JSON string in the Custom Hero Element Selectors text box. The format must be a JSON object mapping a name to a CSS selector: { "heroElementName": "elementSelector" }.
    { "intro": "p.introduction", "buyButton": ".item .buy" }

  11. Configure Reverse-Tethered Networking (RNDIS) for Mobile Agent

    master

    Reverse-tethering allows the Raspberry Pi to provide wired network connectivity to the phone via USB. This improves scaling and consistency.

    Requirements:

    • Phone supporting USB Tethering (e.g., Moto G Gen 1, Moto E, Nexus 5).
    • Phone running Android KitKat 4.4.4 (current agent support).
    • Static IP configuration for both the phone and the Raspberry Pi.

    Setup Steps:

    1. Install dependencies:
      sudo apt-get remove avahi-daemon dhcpcd5
sudo apt-get install ifplugd bridge-utils
    2. Configure ifplugd: Run sudo dpkg-reconfigure ifplugd, add usb0 to the dynamic interface list, and add the -b flag to the command-line options.
    3. Configure Network Interfaces: Edit /etc/network/interfaces to set up a bridge (br0) between eth0 and usb0 with a static IP.
    4. Update Agent Configuration: Modify your agent.sh script to include the --rndis444 flag in the pidriver.sh command. The format is: "<IP>/,<gateway>,<dns1>,<dns2>".

    Example Flag: --rndis444 "192.168.0.yy/24,192.168.0.1,192.168.0.1,192.168.0.1"

  12. Deploy a private WebPageTest instance using EC2 AMIs

    master

    You can quickly spin up a private WebPageTest instance using pre-configured Amazon Machine Images (AMIs).

    Available AMIs by Region

    • us-east-1: ami-fcfd6194
    • us-west-1: ami-e44853a1
    • us-west-2: ami-d7bde6e7
    • sa-east-1: ami-0fce7112
    • eu-west-1: ami-9978f6ee
    • eu-central-1: ami-22cefd3f
    • ap-southeast-1: ami-88bd97da
    • ap-southeast-2: ami-eb3542d1
    • ap-northeast-1: ami-66233967

    Setup Requirements

    1. Firewall: Ensure HTTP traffic is allowed through your EC2 security group.
    2. SSH Keys: Check /home/ubuntu/.ssh/authorized_keys to ensure old keys are removed.
    3. User Data: You must provide an ec2_key and ec2_secret via EC2 User Data at launch.

    Security Warning: Remove your ec2_key and ec2_secret from the instance user data after the initial launch to prevent security leaks.

    Key Features

    • Automatically starts/stops test agents in EC2 regions as needed.
    • Defaults to a headless instance (API only). Set headless=0 in user data to enable a UI.
    • Automatically updates server and agent code hourly.
    • Supports S3 archiving if a bucket is configured in user data.
    • Allows providing a default no-limit API key via user data. Additional keys must be added manually to /var/www/webpagetest/www/settings/keys.ini.
    ec2_key=AKIAJKP75OFSROV5GWEQ
ec2_secret=<secret for the key>
api_key=MyAPIKey
headless=0