DOCSEARCHbyAlgolia
Submit Docs

Magistrala

repository·main·Indexed 25 days ago

https://github.com/absmach/magistrala

A modern, event-driven IoT platform framework designed for scalability and flexibility. Built on the FluxMQ message broker, it provides a unified control plane for device management, connectivity (MQTT, HTTP), and data processing. The platform includes specialized services for authentication (Auth), domain management, and alarm handling, featuring a REST API, gRPC support, and Personal Access Tokens (PATs) for secure automation.

Tokens
70.3K
Snippets
133
Records
318
Agent score
82%

What's inside magistrala

  1. Overview of Magistrala Writers

    main

    Writers are optional plugin services in Magistrala that consume messages from the message broker, normalize them (into SenML or JSON formats), and persist them to a storage backend.

    There are two available writer services:

    • Postgres writer: Stores data in PostgreSQL.
    • Timescale writer: Stores data in TimescaleDB using hypertables optimized for time-series workloads.

    Note: Core services and the message broker must be running before starting writers.

  2. Overview of the Magistrala Rules Engine

    main
    The Magistrala Rules Engine (RE) processes incoming messages using user-defined scripts (Lua or Go) and routes the results to various outputs such as channels, alarms, email, SenML writers, PostgreSQL, or Slack. It also supports scheduled rule execution and publishes rule events to the event store.

  3. Overview of the Notifications Service

    main

    The Notifications Service is an event-driven component of the Magistrala platform responsible for sending email notifications related to domain invitation events. It listens to events from the event store (NATS or RabbitMQ) and interacts with Atom to fetch user details (name, email) before sending styled HTML emails via SMTP.

    It handles three specific event types:

    • invitation.send: When a user is invited to join a domain.
    • invitation.accept: When a domain invitation is accepted.
    • invitation.reject: When a domain invitation is rejected.

  4. Overview of Magistrala pkg packages

    main

    The pkg directory contains standalone packages for building Magistrala back-end applications. The available packages are categorized below:

    Identity & Access

    • authn, authz, oauth2, policies: Authentication and authorization helpers, middleware, and policy utilities.

    Networking & Servers

    • grpcclient: TLS-aware gRPC client setup with health checks and timeouts.
    • server: HTTP, gRPC, and COAP server bootstrap utilities (TLS, graceful shutdown).

    Data & Storage

    • postgres: PostgreSQL connector with migrations helpers.
    • events: Event store client abstractions and subscriber utilities.

    Observability

    • prometheus: Metrics collectors for request counts/latency.
    • jaeger, tracing: OpenTelemetry tracing configuration and instrumentation helpers.

    Domain Services & Types

    • channels, clients, groups, domains, roles: Shared types and helpers for Magistrala domain services.
    • messaging, connections, callout: Messaging DTOs, connection types, and outbound callout helpers.
    • sdk: Go SDK for interacting with Magistrala services.

    Utilities

    • errors: Error wrappers with consistent error typing.
    • uuid, ulid, sid: ID generators.
    • transformers, svcutil: Generic data transformation and service utilities.

  5. Overview of Magistrala IoT Framework

    main

    Magistrala is an open-source, event-driven IoT middleware platform designed for scalability and full control over messaging, device management, and data pipelines. It is built on top of FluxMQ and serves as a unified control plane for devices, users, and data.

    Core Concepts:

    • Users: Human entities interacting with the system.
    • Clients: Devices or applications that send/receive data.
    • Channels: Messaging/data paths for publishing and subscribing.
    • Messages: The data units flowing through the system.
    • Policies: Rules governing access and behavior.

    Key Features:

    • Connectivity: Native support for MQTT, HTTP, WebSocket, and CoAP.
    • Identity & Access: Multi-tenant domains, RBAC/ABAC, X.509/JWT authentication, and Personal Access Tokens (PATs).
    • Automation: Rules engine, alarms, triggers, and scheduled actions.
    • Observability: Audit logs, Prometheus metrics, and OpenTelemetry tracing.

  6. Overview of the Notifiers service

    main

    The Notifiers service manages notification subscriptions and dispatches alerts for incoming messages. It stores subscription records (mapping a topic to a contact), exposes an HTTP API for CRUD operations, and consumes Magistrala messages to fan out notifications via specific implementations like SMTP (email) or SMPP (SMS).

    Note: Notifiers are dependencies used by the service, not standalone services.

  7. Overview of the Reports service

    main

    The Reports service generates time-series reports from stored messages. It fetches data from the readers gRPC service, formats results as JSON, CSV, or PDF, and supports scheduled report delivery via email.

    Key features include:

    • Report generation: Build report data from time-series messages.
    • Multiple formats: JSON responses, CSV exports, and PDF rendering.
    • Scheduling: Periodic report delivery via email.
    • Template support: Custom HTML templates for PDF reports.
    • Observability: /metrics Prometheus endpoint and Jaeger tracing support.

  8. Use the UUID identity provider

    main
    The uuid package provides a UUID identity provider that generates random, universally unique identifiers (UUIDs). These identifiers are unique for all practical purposes and can be used to provide unique identities within the Magistrala framework.

  9. Understand the Reports service runtime flow

    main

    The Reports service operates through the following lifecycle:

    1. Trigger: The Reports API receives a report request or a scheduled run triggers generation.
    2. Data Fetching: The service expands requested metrics and fetches messages via the readers gRPC API in batches of 1000.
    3. Grouping: Results are grouped by publisher if client_ids are not specified.
    4. Output Generation: Output is returned as JSON, rendered to CSV, or converted to PDF via the MG_PDF_CONVERTER_URL endpoint.
    5. Delivery: For scheduled/email actions, the report is sent as an email attachment.

  10. Use ULID for sortable unique identifiers

    main
    The ulid package provides a ULID (Universally Unique Lexicographically Sortable Identifier) identity provider. It generates 128-bit identifiers that are string-encoded, unique for all practical purposes, and lexicographically sortable. This makes them suitable for use as primary keys or identifiers where chronological ordering is required.

  11. Understand Authentication Key Types

    main

    The Auth service manages several types of authentication keys, which are distributed as JWTs. Each key contains an ID, Type, IssuerID, Subject, IssuedAt, and ExpiresAt.

    Key types include:

    • Access key: Issued to the user upon login.
    • Refresh key: Used to generate new access keys.
    • Recovery key: Short-lived token used for the password recovery process.
    • API key: Issued upon user request. These have a configurable expiration time (can be set to never expire). Unlike other keys, API keys can be revoked and require a database query for validation. They allow the user to act on behalf of a Client, Channel, or user profile, but cannot be used to issue new API keys.
    • Invitation key: Used to invite new users.

  12. Use the Messaging interfaces: Publisher, Subscriber, and Pubsub

    main

    The messaging package provides three primary interfaces for interacting with message brokers (such as MQTT, NATS, or RabbitMQ):

    • Publisher: Defines methods for sending messages to a broker.
    • Subscriber: Defines methods for receiving messages from a broker.
    • Pubsub: An aggregate interface that combines both Publisher and Subscriber, allowing a single client to both send and receive messages.