NATS By Example

repository·main·Indexed 42 days ago

https://github.com/connecteverything/nats-by-example

A collection of functional reference examples for NATS patterns across different programming languages. It includes the nbe CLI to run examples in reproducible Docker environments and demonstrates operational patterns such as rotating NATS cluster nodes with minimal downtime.

Tokens
5.9K
Snippets
16
Records
30
Agent score
52%

What's inside NATS By Example

  1. Understand the NATS By Example directory structure

    main

    The repository is organized by NATS concepts. Under the examples/ directory, examples are grouped into categories. Each example contains one or more client implementations (e.g., go, python, cli, or shell).

    Structure Pattern: examples/<category>/<example>/<client>/main.[ext]

    • Category: A high-level NATS concept (e.g., messaging).
    • Example: A specific pattern or feature (e.g., pub-sub).
    • Client: The language or tool used to implement the example (e.g., go, cli).
    • Entrypoint: The file main.[ext] (e.g., main.go, main.sh) is the required entrypoint for the client.
    examples/
      meta.yaml
      messaging/
        meta.yaml
        pub-sub/
          meta.yaml
          cli/
            main.sh
          go/
            main.go
          python/
            main.py
  2. How client environments and Docker overrides work

    main

    Each client directory in an example can define its own environment:

    1. Entrypoint: The file must be named main.[ext] (e.g., main.go, main.py, main.sh).
    2. Custom Dockerfile: If a client directory contains a Dockerfile, it will be used to build the container for that example. If not, the CLI uses the default language-specific Dockerfile from the docker/ directory.
    3. Custom Docker Compose: If an example directory contains a docker-compose.yaml file, it will override the default NATS server configuration provided in the docker/ directory.
  3. Set up the NATS cluster node replacement demo locally

    main

    To run the cluster node rotation example locally, follow these setup steps. Ensure you are in the examples/operations/replace-cluster-nodes/shell/ directory.

    Dependencies

    • nats-server binary
    • nats CLI
    • watch command (native to Linux; brew install watch on macOS)

    Setup Steps

    1. Configure NATS Context: Ensure your nats context is set to localhost without credentials.
    2. Start Cluster: Run ./scripts/start-cluster.sh. Servers run in the background; PIDs are in pids/, logs in logs/, and data in data/.
    3. Monitor Servers: Run ./scripts/watch-servers.sh in a separate shell.
    4. Initialize Streams: Run ./scripts/create-streams.sh. This creates foo (file-based, 3 replicas) and bar (memory-based, 100k max messages, 3 replicas).
    5. Initialize Consumers: Run ./scripts/create-consumers.sh to create push consumers foo-c and bar-c.
    6. Start Traffic:
      • Start publishers: ./scripts/start-foo-publisher.sh and ./scripts/start-bar-publisher.sh (in separate shells).
      • Start subscribers: ./scripts/start-foo-subscriber.sh and ./scripts/start-bar-subscriber.sh (in separate shells).
    7. Execute Migration: Run ./scripts/migrate-cluster.sh. This script automates the rotation with 60-second wait times for lame-ducking and node removal.
    8. Cleanup: Run ./scripts/stop-cluster.sh to stop the cluster.
    # Summary of execution sequence
    ./scripts/start-cluster.sh
    ./scripts/create-streams.sh
    ./scripts/create-consumers.sh
    ./scripts/start-foo-publisher.sh
    ./scripts/start-bar-publisher.sh
    ./scripts/start-foo-subscriber.sh
    ./scripts/start-bar-subscriber.sh
    ./scripts/migrate-cluster.sh
    ./scripts/stop-cluster.sh
  4. Run NATS examples using the nbe CLI

    main

    To execute a NATS example in a reproducible environment, use the nbe CLI. The CLI runs a set of containers hosting both the NATS server and the specific client implementation for the example.

    Prerequisites

    Execution

    Run the command from the root of the repository using the path to the example following the pattern <category>/<example>/<client>.

    Example: To run the NATS CLI implementation of the core publish-subscribe example:

    $ nbe run messaging/pub-sub/cli
  5. Rotate NATS cluster nodes with minimal downtime

    main

    This operational pattern demonstrates how to replace underlying hardware or configuration of NATS cluster nodes without requiring client configuration updates. The strategy involves adding a new node, putting an old node into 'lame duck' mode, and then removing it to force replicas to reschedule onto the new node.

    By following a specific rotation cycle, the original connection URLs remain addressable, allowing clients to reconnect automatically to the existing cluster endpoints even as the underlying nodes change.

    ### General Rotation Algorithm
    
    1. **Add new node** (e.g., `n3`):
       `scripts/add-node.sh n3`
    2. **Signal lame duck mode** on old node (e.g., `n0`):
       `scripts/signal-lameduck.sh n0`
    3. **Remove old node** (e.g., `n0`) to force replica rescheduling:
       `scripts/remove-peer.sh n0`
    4. **Repeat cycle** for remaining nodes to ensure connection URLs remain valid for clients.
  6. Data structures for documentation generation

    main

    The documentation generator uses several internal data structures to map the repository's example hierarchy into HTML pages. These structures represent the relationship between categories, examples, and specific language implementations (clients).

    • indexData: Represents the root index page, containing a list of indexCategory objects.
    • indexCategory: A grouping of examples under a specific title and description.
    • categoryData: Represents a category-specific landing page.
    • exampleData: Represents the landing page for a specific NATS example, containing links to various language implementations.
    • clientData: The most granular level, representing a specific language implementation of an example (e.g., the Go version of a 'Pub/Sub' example). It includes source code, rendered blocks, and execution output.
  7. Understand the NATS By Example versioning and matrix system

    main

    The nbe CLI tool manages a compatibility matrix for NATS examples. It ensures that various client libraries (Go, Python, Node, Rust, etc.) work correctly across different versions of the NATS Server and the NATS CLI.

    Internally, the tool uses YAML configuration files to define:

    1. Versions: Specific versions for the NATS Server, CLI, and various client languages.
    2. Matrix: A set of version combinations that must be tested to ensure compatibility.

    When running the matrix, the tool automates the process of updating dependency files (like go.mod, package.json, requirements.txt, or Cargo.toml) within example directories, building Docker images, and running the examples via Docker Compose to verify they pass (returning OK) or fail (returning ERR).

  8. Render code and comments for documentation

    main

    The documentation engine processes raw example files into formatted HTML using two main mechanisms:

    1. Code Blocks: Uses the chroma library for syntax highlighting. It maps specific language names to appropriate lexers (e.g., Deno is treated as ts, WebSocket as js).
    2. Comment Blocks: Extracts comments from source files, cleans them of leading whitespace and comment delimiters (like // or /* */), and renders the remaining text as Markdown using blackfriday.

    Results are encapsulated in the RenderedBlock struct, which specifies whether the content is code or comment and provides the resulting HTML.

  9. Structure of an Example and its Clients

    main

    An Example represents a specific NATS pattern and contains a map of Clients. Each Client represents a specific language implementation of that example. A Client includes:

    • Language: The identifier for the SDK (e.g., go, python).
    • MainFile: The entry point file (e.g., main.go, main.py).
    • Blocks: A collection of Block objects representing segments of code or comments.
    • Source: The full raw source code string.
  10. How code blocks are parsed

    main

    The parser divides source code into Block types to distinguish between different types of content. This allows the system to treat code and comments differently when rendering or processing examples.

    Supported BlockType values:

    • EmptyBlock: Initial state.
    • BreakBlock: A boundary marker.
    • CodeBlock: Actual executable code.
    • SingleLineCommentBlock: Comments starting with //, #, etc.
    • MultiLineCommentBlock: Comments wrapped in /* ... */ or /** ... */.

    Special boundary marker: The parser recognizes <!break> at the end of a line as a BreakLine to explicitly separate blocks.

  11. Run the NATS infrastructure with Docker Compose

    main

    The docker-compose.yaml file defines a local development environment consisting of a NATS server and an application service.

    NATS Service Configuration

    • Image: docker.io/nats:2.10.4
    • Startup Commands:
      • --debug: Enables debug logging.
      • --http_port=8222: Enables the NATS HTTP monitoring server on port 8222.
      • --js: Enables JetStream capabilities.

    App Service Configuration

    • Image: Uses the ${IMAGE_TAG} environment variable to determine the application image version.
    • Environment Variables:
      • NATS_URL: Set to nats://nats:4222 to allow the application to connect to the NATS service within the Docker network.
    • Dependencies:
      • The app service depends on the nats service being started.
    services:
      nats:
        image: docker.io/nats:2.10.4
        command:
          - "--debug"
          - "--http_port=8222"
          - "--js"
    
      app:
        image: ${IMAGE_TAG}
        environment:
          - NATS_URL=nats://nats:4222
        depends_on:
          - nats