DOCSEARCHbyAlgolia
Submit Docs

OTelBin

repository·main·Indexed 19 days ago

https://github.com/dash0hq/otelbin

A web-based configuration tool for OpenTelemetry Collector pipelines providing pipeline visualization, configuration validation, and online sharing. It includes a validation backend deployed via AWS CDK that uses AWS Lambda and API Gateway to validate YAML configurations against multiple OpenTelemetry collector distributions and versions.

Tokens
8.4K
Snippets
37
Records
41
Agent score
68%

What's inside otelbin

  1. What is OTelBin

    main

    OTelBin is a configuration tool designed to help users manage and optimize OpenTelemetry Collector pipelines. It provides three primary capabilities:

    1. Pipeline Visualization: Visualizes configured OpenTelemetry Collector pipelines using a swimlane layout.
    2. Configuration Validation: Validates your OpenTelemetry configurations and highlights errors.
    3. Online Sharing: Allows you to share your OpenTelemetry Collector configurations online (requires authentication via GitHub or Google).

    The service is hosted at otelbin.io.

  2. How the OTelBin validation infrastructure works

    main

    The validation backend is an AWS CDK v2 application that provides validation services for multiple OpenTelemetry collector distributions.

    Architecture Overview:

    • API Gateway: Exposes validation routes via the pattern /validation/${distribution_name}/${version}. It is secured using an API Key.
    • AWS Lambda: Each route is mapped to a dedicated Lambda function running a Docker image (otelcol-validator).
    • Validation Process: The Lambda function receives the API Gateway event, extracts the YAML configuration, saves it to a file, starts the specific collector version with that configuration, and returns the outcome as the API response.
    • Data Source: Supported distributions and versions are managed in ./src/assets/supported-distributions.json and are assumed to be available as GitHub releases.

  3. Deploy the validation backend to AWS

    main

    The validation backend is deployed using AWS CDK.

    Prerequisites:

    • AWS CLI configured with a profile containing aws_access_key_id, aws_secret_access_key, and region.
    • The AWS Access Key must have sufficient permissions to bootstrap CDK (e.g., AdministratorAccess).
    • A GitHub classic token to allow the gh utility to download release artifacts.

    Deployment Command: Run the following command from the repository root:

    GH_TOKEN=<token> npm run deploy --profile <your-profile-name>

    Post-Deployment: After deployment, the CDK output will provide the necessary API details:

    • apikeyid: The API Key ID
    • apiname: The API Name
    • apiurl: The base URL for the API Gateway
    • validationapiEndpoint...: The specific endpoint for validation.
    GH_TOKEN=<token> npm run deploy --profile otelbin-dev

  4. How to add support for a new distribution

    main

    To add a new OpenTelemetry collector distribution to the validation service, follow these steps:

    1. Verify Requirements: Ensure the distribution provides RPM packages as artifacts or links within a GitHub release in a <github_org>/<repo_name> repository.
    2. Add Metadata: Add the distribution entry to ./src/assets/supported-distributions.json.
      • Note: The icon field must match an identifier used in ValidationTile.tsx, pointing to an SVG (24px x 24px) in the distro-icons folder.
    3. Add a Test Release: Manually add at least one release to the releases array in the JSON file to enable testing.
    4. Handle Custom Validation (If needed): If the distribution does not support the standard community validate command, you must implement bespoke logic to detect if the collector bootstrapped correctly (see otelbin-validation-handler for an example like ADOT).
    5. Deploy and Test: Deploy the backend to AWS and run API tests.
    6. Automate: Extend the nightly automation via .github/workflows/discover-new-releases.yaml to track future releases automatically.
    // Example metadata for a new distribution
{
   "my-distro": {
      "provider": "My company",
      "description": "Much wow",
      "icon": "my-icon",
      "website": "https://my-company.io",
      "repository": "<github_org>/<repo_name>",
      "releases": [
         {
            "version": "v0.88.0",
            "artifact": "otelcol_0.88.0_linux_amd64.rpm",
            "released_at": "2023-10-24T11:24:58.000Z"
         }
      ]
   }
}

  5. Structure of Distributions and Releases

    main

    The Distributions interface defines a map where keys are unique identifiers (strings) and values are Distribution objects. Each Distribution contains metadata about a software package and a list of available Release objects.

    A Release represents a specific version of a distribution and points to its downloadable artifact URL.

    // Example shape of a Distribution object
const distribution: Distribution = {
  name: "OpenTelemetry Collector",
  provider: "OpenTelemetry",
  description: "A vendor-agnable observability pipeline",
  icon: "otel-icon-url",
  website: "https://opentelemetry.io",
  repository: "https://github.com/open-telemetry/opentelemetry-collector",
  releases: [
    { version: "0.90.0", artifact: "https://example.com/otel-0.90.0.tar.gz" }
  ]
};

  6. Configure the OTelBin Validation API endpoints

    main

    The deployed stack exposes the following API structure via AWS API Gateway:

    1. GET /validation/supported-distributions: Returns a JSON list of all supported distributions and their metadata. This is backed by an S3 bucket.
    2. POST /validation/{distributionName}/{version}: Triggers a configuration validation for a specific distribution and version. This endpoint requires an API Key passed in the header.

    Security Note: All validation endpoints require an API Key (apiKeyRequired: true).

  7. Manage editor state with EditorProvider

    main

    The EditorProvider component acts as the central state manager for the Monaco editor instance within the application. It provides a suite of React contexts that allow child components to access the editor instance, the Monaco engine, view modes, breadcrumbs (YAML path), and focus state.

    Wrap your editor-related component tree with EditorProvider to enable access to these hooks.

    Available Contexts and Hooks

    HookContextPurpose
    useEditorRef()EditorContextAccess the editor.IStandaloneCodeEditor instance.
    useMonacoRef()MonacoContextAccess the Monaco engine instance.
    useEditorDidMount()EditorDidMountAccess the onMount callback to initialize the editor.
    useFocus()FocusContextManage and track which element is currently focused.
    useViewMode()ViewModeContextControl the UI layout mode (both, code, or pipeline).
    useBreadcrumbs()BreadcrumbsContextAccess and update the current YAML path string.
    import { EditorProvider } from './contexts/EditorContext';

function App() {
  return (
    <EditorProvider>
      <MyEditorComponent />
    </EditorProvider>
  );
}

  9. Embed OTelBin badges in documentation

    main

    You can use shields.io-powered badges to reference a collector configuration within your documentation. Use the following snippets depending on your target format:

    Markdown

    ![OpenTelemetry collector configuration on OTelBin](https://www.otelbin.io/badges/collector-config)

    HTML

    <img src="https://www.otelbin.io/badges/collector-config" alt="OpenTelemetry collector configuration on OTelBin">

    URL

    https://www.otelbin.io/badges/collector-config

    ![OpenTelemetry collector configuration on OTelBin](https://www.otelbin.io/badges/collector-config)

  10. Validate an OpenTelemetry configuration

    main

    To validate a specific configuration against a specific distribution and version, send a POST request to the endpoint /validation/${DISTRO_NAME}/${VERSION}. The configuration YAML should be sent in the request body.

    Request:

    curl https://${API_ID}.execute-api.${AWS_REGION}.amazonaws.com/prod/validation/${DISTRO_NAME}/${VERSION} --header "X-Api-Key: ${API_KEY}" --data "${CONFIGURATION}"

    Example Error Response: If the configuration is empty or invalid, the API returns a JSON error:

    {"message":"The provided configuration is invalid","error":"the provided configuration is empty"}
    $ curl https://${API_ID}.execute-api.${AWS_REGION}.amazonaws.com/prod/validation/${DISTRO_NAME}/${VERSION} --header "X-Api-Key: ${API_KEY}" --data "${CONFIGURATION}"

  11. Retrieve supported distribution versions

    main

    You can query the API to get a list of all supported distributions and their available versions using the /validation/supported-distributions endpoint. This requires the X-Api-Key header.

    Request:

    curl https://${API_ID}.execute-api.${AWS_REGION}.amazonaws.com/prod/validation/supported-distributions --header "X-Api-Key: ${API_KEY}"
    $ curl https://${API_ID}.execute-api.${AWS_REGION}.amazonaws.com/prod/validation/supported-distributions --header "X-Api-Key: ${API_KEY}"

  12. Configure required environment variables for OTelBin

    main

    To run OTelBin, you must provide credentials for Upstash Redis (used for caching and short link persistence) and Clerk (used for user authentication).

    # Upstash Redis
UPSTASH_REDIS_REST_URL=
UPSTASH_REDIS_REST_TOKEN=

# Clerk authentication
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=
CLERK_SECRET_KEY=