DOCSEARCHbyAlgolia
Submit Docs

Keycloak Benchmark

repository·main·Indexed 41 days ago

https://github.com/keycloak/keycloak-benchmark

A benchmarking tool for Keycloak featuring Ansible roles for AWS EC2 provisioning and execution, OpenTofu modules for infrastructure management, and scripts for initializing benchmark entities like realms, clients, and users.

Tokens
60.4K
Snippets
212
Records
338
Agent score
51%

What's inside Keycloak Benchmark

  2. Overview of Keycloak Benchmark tools

    main

    Keycloak Benchmark is a collection of tools designed to perform performance testing on Keycloak servers. It aims to provide reproducible results, the ability to run load tests against any Keycloak instance, and an observability stack for analyzing test runs.

    The project is composed of three main functional modules:

    • benchmark module: Contains load tests implemented using Gatling.
    • provisioning module: Provides environment setup options. It includes an advanced Kubernetes setup using minikube (which includes a Grafana-based observability stack) and docker-compose scripts for a simpler, non-observability setup.
    • dataset module: Provides a Keycloak add-on capable of creating entities within a Keycloak data store to prepare the environment for load testing.

  3. Overview of Keycloak Benchmark modules

    main

    The Keycloak Benchmark project consists of several modules designed to load test Keycloak. These modules can be used in isolation to test specific custom setups or existing installations, or combined to create a standardized environment for load testing.

    The primary workflows involve:

    1. Deployment: Setting up Keycloak in a Kubernetes-like environment (e.g., minikube).
    2. Data Seeding: Preparing the necessary datasets for the benchmark tests.
    3. Execution: Running the actual benchmark tests against a provisioned Keycloak instance.

  4. Overview of production running strategies

    main

    This project provides logic and guidance for running Keycloak in production environments, which is often condensed into Helm charts and scripts. The documentation covers high availability, deployment patterns, and performance tuning.

    Note on Documentation Location: Many of these guides have migrated to the official Keycloak website:

  5. Overview of the Keycloak Benchmark Kubernetes setup

    main

    The Keycloak Benchmark Kubernetes guide provides a method to install Keycloak alongside a monitoring stack consisting of Grafana, Prometheus, and Jaeger.

    Key components used in this setup:

    • minikube: Runs the containers.
    • Helm: Provisions the containers.
    • Keycloak Operator: A Java-based operator used to provision Keycloak.
    • kubectl: Command-line tool to interact with minikube.
    • Kubebox: Provides a minimal UI to inspect containers within minikube.

    Monitoring capabilities include:

    • OpenTelemetry (OTEL): An optional setup to gather additional metrics and publish traces from Keycloak to Jaeger. This enables measuring latency per endpoint and tracing database statements executed by REST endpoints.
    • Loki: Collects logs from all containers, which can be queried via Grafana.
    • Tempo: An alternative tracing option to Jaeger (note: trace searching is currently in beta and may not return all traces, such as user deletions).

  6. Use the _task_ tool for Kubernetes automation

    main

    The project uses task to manage the dependencies and execution order required for Kubernetes provisioning and configuration. It acts as a modern alternative to make, supporting file check-summing and parallel task execution.

    All automation logic is defined in the Taskfile.yaml file.

    Key behaviors:

    • If a command within a task fails, the entire task fails.
    • If a task fails while other tasks are running in parallel, _task_ will kill the parallel tasks.
    • Environment-specific settings can be managed by adding a .env file to your environment.

  7. Available test scenarios in keycloak-benchmark

    main

    The keycloak-benchmark tool provides several pre-defined test scenarios categorized by authentication types, administrative actions, and basic HTTP operations. These scenarios allow you to benchmark different Keycloak workflows such as OAuth2 flows, user/client management, and realm operations.

    Authentication Scenarios

    • keycloak.scenario.authentication.AuthorizationCode: Benchmarks the Authorization Code Grant Type.
    • keycloak.scenario.authentication.LoginUserPassword: Benchmarks browser login via the Authorization Endpoint (note: this does not include the exchange of the OAuth2 'code' for tokens).
    • keycloak.scenario.authentication.ClientSecret: Benchmarks the Client Credentials Grant.

    Administrative Scenarios

    Many administrative scenarios require specific credentials to perform actions via the Keycloak Admin API:

    Scenarios requiring --client-secret (for the Gatling client):

    • keycloak.scenario.admin.CreateClients / keycloak.scenario.admin.CreateDeleteClients
    • keycloak.scenario.admin.CreateUsers / keycloak.scenario.admin.CreateDeleteUsers
    • keycloak.scenario.admin.CreateRoles / keycloak.scenario.admin.CreateDeleteRoles
    • keycloak.scenario.admin.CreateGroups / keycloak.scenario.admin.CreateDeleteGroups
    • keycloak.scenario.admin.CreateClientScopes / keycloak.scenario.admin.CreateDeleteClientScopes
    • keycloak.scenario.admin.UserCrawl: Crawls all users page by page.

    Scenarios requiring --admin-username and --admin-password:

    • keycloak.scenario.admin.ListSessions: Creates User and Client sessions.
    • keycloak.scenario.admin.CreateRealms: Creates realms.
    • keycloak.scenario.admin.CreateDeleteRealms: Creates and immediately deletes realms.

    Basic Scenarios

    • keycloak.scenario.basic.Get: Repeatedly calls a specific URL using HTTP GET and reports/checks the status code.

  8. Use the Dataset Provider to populate Keycloak test environments

    main

    The Dataset Provider is a tool designed to quickly populate a target Keycloak or RHSSO server with large volumes of objects (users, clients, realms, etc.) for performance testing. It is significantly faster than using the standard Keycloak Admin REST API directly.

    Use Cases

    • Populate many users: Enables performance testing with high concurrency where each login uses a unique user.
    • Populate many clients: Useful for testing service account logins and other client-based authentication flows.
    • Populate many realms: Each new realm can be automatically filled with a specified number of roles, clients, groups, and users, which is often sufficient for most performance testing scenarios.

    ⚠️ Security Warning

    Never install the Dataset Provider in a production environment. It allows the creation and deletion of entities without authentication, posing a severe security risk.

  9. Understand the Kubernetes benchmark runtime architecture

    main

    The benchmark environment runs within a minikube virtual machine. The architecture consists of the following components and interactions:

    • Access: A configured ingress allows a local browser to access services like Keycloak and Grafana.
    • Keycloak: Connects to a PostgreSQL database running inside minikube. The database is accessible from the host via a node port.
    • Observability Stack:
      • Metrics: Collected by Prometheus.
      • Traces: Collected by Jaeger from Keycloak (and optionally from local test applications).
      • Logs: Collected by Promtail and sent to Loki for storage.
      • Visualization: Grafana is used to view the collected data.
    • Load Generation: Gatling can run locally and sends Graphite metrics via a node port to a collector inside minikube.
    • Profiling: Cryostat can connect to Keycloak instances to create Java Flight Recorder (JFR) recordings.

  10. When to use Ansible and EC2 for load generation

    main

    Use Ansible and EC2 to scale your load generation beyond a single instance. This approach is necessary in the following scenarios:

    1. Overcoming Connection Limits: When a single instance cannot establish network connections fast enough because too many connections are stuck in the TIMED_WAIT status.
    2. Latency Simulation: When you need to place a load driver in the same or a different AWS region to either minimize latency or simulate the specific latency experienced by real users.

  11. Understand the `results.json` output format

    main

    The kcb.sh script generates a results.json file for every benchmark run. This file contains the input parameters used for Gatling, the resulting output, and metadata to identify the run. The filename includes a timestamp and a UUID, allowing multiple result files to be stored in a single directory for pipeline processing.

    Key fields in the JSON include:

    • uuid: A unique identifier for the run.
    • name: A descriptive name for the run.
    • grafana_input: The timing and input parameters used for Grafana visualization.
    • grafana_output: The resulting metrics/output.
    • system_under_test: Metadata describing the environment (if provided via SUT_DESCRIBE).
    {
  "uuid": "...",
  "name": "...",
  "grafana_input": {
    "start": "...",
    "end": "...",
    "input": "..."
  },
  "grafana_output": "...",
  "system_under_test": "..."
}

  12. How the CreateRealms/CreateDeleteRealms scenario works

    main

    The CreateRealms/CreateDeleteRealms scenario performs realm operations (creation and deletion) using the root admin account via the built-in admin-cli client.

    Unlike other scenarios that might use service account tokens, this scenario requires direct root admin credentials because:

    1. Real-world accuracy: Real-world realm operations are typically performed using root admin credentials.
    2. Permission requirements: Deleting a newly created realm requires realm-specific permissions. Using a service account would require a logout/login cycle to switch to the realm-specific client.
    3. Token size constraints: Using a single token with all realm permissions would cause the token to exceed maximum header lengths, resulting in 431 Request Header Fields Too Large errors.