DOCSEARCHbyAlgolia
Submit Docs

retraced

repository·main·Indexed 18 days ago

https://github.com/retracedhq/retraced

A fully open source audit log service and embeddable UI for Kubernetes clusters. Retraced provides a compliant, searchable, and exportable record of read/write events for applications, featuring integration via Go and JavaScript client libraries, a publisher API for event ingestion, and CLI tools for database migrations, reindexing between PostgreSQL and Elasticsearch, and GeoIP management.

Tokens
18K
Snippets
71
Records
88
Agent score
66%

What's inside retraced

  1. Run Retraced locally with Skaffold

    main

    If you prefer using Skaffold instead of docker-compose for local development, use the following commands:

    npm run run:skaffold
# or
skaffold run -f skaffold-dev.yaml --status-check=false --force=true

  2. Run Retraced locally with docker-compose

    main

    To start Retraced locally using Docker Compose, run the following command. Note that for local development, the ADMIN_ROOT_TOKEN is set to dev. You must change this and other sensitive environment variables/secrets when deploying to production.

    docker-compose up -d
# or
npm run dev

  3. Define DeletionConfirmation data structures

    main

    The deletion_confirmation module provides several TypeScript interfaces for handling user deletion confirmations at different stages of the data lifecycle: raw values, hydrated objects with user details, and sanitized versions for public consumption.

    • DeletionConfirmationValues: The base properties required for a confirmation.
    • DeletionConfirmation: Extends values with a unique id.
    • DeletionConfirmationHydrated: Extends DeletionConfirmation by including the associated retracedUser object (or null).
    • DeletionConfirmationSanitized: A stripped-down version containing only id, userId, email, and approved status, typically used for API responses.
    export interface DeletionConfirmationValues {
  deletionRequestId: string;
  retracedUserId: string;
  received?: moment.Moment;
  visibleCode: string;
}

export interface DeletionConfirmation extends DeletionConfirmationValues {
  id: string;
}

export interface DeletionConfirmationHydrated extends DeletionConfirmation {
  retracedUser: RetracedUser | null;
}

export interface DeletionConfirmationSanitized {
  id: string;
  userId: string;
  email: string;
  approved: boolean;
}

  4. Define Template data structures

    main

    Retraced uses several interfaces to represent audit log templates.

    • TemplateValues: The base configuration for a template, containing the name, the rule (or a string representation), and the template string.
    • rule: Defines a specific condition for a template, consisting of a comparator, a path to the data, and a value (which can be a string, number, boolean, or Date).
    • Template: The internal representation of a template, including metadata like id, project_id, environment_id, and moment.Moment objects for created and updated timestamps.
    • TemplateResponse: The serialized version of a template used for API responses, where created and updated are converted to ISO strings.
    export interface TemplateValues {
  id?: string;
  name: string;
  rule: rule[] | string;
  template: string;
}

export interface rule {
  comparator: string;
  path: string;
  value: string | number | boolean | Date;
}

export interface Template {
  id: string;
  project_id: string;
  environment_id: string;
  created: moment.Moment;
  updated: null | moment.Moment;
}

export interface TemplateResponse extends TemplateValues {
  id: string;
  project_id: string;
  environment_id: string;
  created: string;
  updated: null | string;
}

  5. Define Environment data structures

    main

    The environment module provides TypeScript interfaces for representing environments within the system at different stages of data processing (raw database rows, hydrated objects, and API responses).

    • EnvironmentValues: The base interface containing common fields like name.
    • Environment: Represents a standard environment, extending EnvironmentValues with id and projectId.
    • EnvironmentHydrated: An extended version of Environment that may include an optional deletionRequest of type DeletionRequestHydrated.
    • EnvironmentResponse: The shape of the data returned in API responses, using snake_case for project_id.

  6. Understand the GraphQL search response structure

    main

    The GraphQL API returns data wrapped in a GraphQLResp object. A successful search returns a GraphQLSearch object containing an EventsConn (Events Connection).

    Response Components

    • data.search.edges: An array of EventEdge objects. Each edge contains a node (of type RawEventNode) and a cursor string used for pagination.
    • data.search.pageInfo: Contains hasNextPage and hasPreviousPage booleans to indicate if more data is available.
    • data.search.totalCount: The total number of events matching the query.
    • errors: An array of GraphQLError objects containing a message, locations (line/column), and path if the request failed.
    // Example of the expected shape of a successful response
const response: GraphQLResp = {
  data: {
    search: {
      edges: [
        { node: { /* RawEventNode data */ }, cursor: 'cursor_1' }
      ],
      pageInfo: { hasNextPage: true, hasPreviousPage: false },
      totalCount: 123
    }
  }
};

  7. Run Retraced with Docker Compose

    main

    Retraced can be deployed using Docker Compose. The configuration includes an Elasticsearch service used for data storage. The setup relies on a base configuration file (docker-compose-base.yaml) which must be present in the same directory.

    To run the stack, use the standard Docker Compose command in the directory containing the docker-compose.yaml file.

    docker-compose up

  8. Ingest a custom audit log event via curl

    main

    You can manually ingest a custom audit log event by sending a POST request to the publisher endpoint. For local development, use Authorization: token=dev and ensure the project ID in the URL matches your local setup.

    curl -X POST -H "Content-Type: application/json" -H "Authorization: token=dev" -d '{
  "action": "some.record.created",
  "teamId": "boxyhq",
  "group": {
    "id": "dev",
    "name": "dev"
  },
  "crud": "c",
  "created": "2023-01-16T15:48:44.573Z",
  "source_ip": "127.0.0.1",
  "actor": {
    "id": "jackson@boxyhq.com",
    "name": "Jackson"
  },
  "target": {
    "id": "100",
    "name": "tasks",
    "type": "Tasks"
  }
}' http://localhost:3000/auditlog/publisher/v1/project/dev/event

  9. Configure the Elasticsearch service in Docker Compose

    main

    The elasticsearch service is part of the Retraced stack. It uses image elasticsearch:8.14.3.

    Key configuration settings include:

    • Environment Variables:
      • discovery.type=single-node: Configures Elasticsearch to run in single-node mode.
      • ES_JAVA_OPTS=-Xms1g -Xmx1g: Sets the initial and maximum heap size to 1GB.
      • xpack.security.enabled=false: Disables X-Pack security features.
    • Ports: Maps host port 9200 to container port 9200.
    • Network: Connects to the retraced network.
    services:
  elasticsearch:
    image: elasticsearch:8.14.3
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
    networks:
      - retraced
    restart: "always"

  10. Configure Retraced API base path and SSL

    main

    The Retraced API behavior is controlled via configuration settings.

    • Base Path: Set API_BASE_URL_PATH to define the URL prefix for all API routes. The application will listen on this basePath.
    • SSL/HTTPS: To enable HTTPS, provide paths to your SSL certificate and private key using SSL_SERVER_CERT_PATH and SSL_SERVER_KEY_PATH. If these are unset, the server defaults to HTTP on port 3000.
    • Admin Access: If ADMIN_ROOT_TOKEN is configured, an additional administrative login route is enabled at /admin/v1/user/_login.

  11. Generate and access Swagger documentation

    main

    Retraced uses TSOA to generate its Swagger specification.

    • Generating the spec: Run npm run swagger to write the output to build/swagger.json.
    • Accessing the spec: By default, the spec is served by Express at /publisher/v1/swagger.json.
    npm run swagger

  12. Query events using GraphQL

    main

    For complex queries and data retrieval, use the GraphQL endpoint. This allows you to fetch specific event data and relationships efficiently.

    Endpoint: POST /publisher/v1/project/{projectId}/graphql

    Request Body: A GraphQLRequest object containing the query, variables, and operationName.

    Authentication: Requires an Authorization header in the form token=....

    Refer to the Retraced GraphQL documentation for schema details.

    # Example: GraphQL query
curl -X POST "https://<your-domain>/publisher/v1/project/my-project-id/graphql" \
     -H "Authorization: token=my-auth-token" \
     -H "Content-Type: application/json" \
     -d '{
       "query": "query { events(limit: 10) { id timestamp } }"
     }'