DOCSEARCHbyAlgolia
Submit Docs

Layline Documentation

website·Indexed 32 days ago

https://doc.layline.io

Documentation for layline.io, a workflow automation platform. Includes guides on core concepts like Projects and Assets, cluster operations, and an Alarm Center for monitoring. It provides detailed configuration for various asset sources (S3, Kafka, GCS, SharePoint, OneDrive, NFS, SMB) and services (HTTP, DynamoDB, Cassandra, Teams, KVS), along with language references for JavaScript and Python APIs.

Tokens
244.5K
Snippets
585
Records
1.9K
Agent score
56%

What's inside Layline

  1. Overview of QuickScript

    QuickScript is a lightweight, limited scripting language used within layline.io Projects to implement custom functions and minor data transformations where full-scale scripting would be excessive. It is specifically designed for use in configuration areas such as mapping assets, routing rules, and grammar definitions.

  2. Overview of Layline Python API components

    The Layline Python API is used to build workflow processors, services, and integrations. It consists of several core components:

    • Enumerations: Constants for Java type mappings (JavaType) and log/alert severity levels (Severity).
    • Classes: Core building blocks including Workflow (orchestration), Message (data manipulation), PythonProcessor (base class for processors), Stream (stream processing), and Email (composition and sending).
    • Interfaces: Type definitions for structured development, such as EmailMessage and various timer configurations (TimerCron, TimerFixedRate, TimerOnce).
    • Global Variables: Pre-available objects in the Python environment for quick access to context, such as message, workflow, processor, stream, metrics, dataDictionary, and statusRegistry.

  3. Overview of Virtual File System (VFS) Connection

    The Virtual File System (VFS) Connection provides an abstraction layer over multiple storage backends, including local filesystems, SMB/CIFS shares, NFS mounts, and cloud storage. It allows VFS Sources and Sinks to read from or write to these backends using logical paths without needing to change their specific configurations. The VFS Connection itself performs no I/O; it only resolves mount point paths for child assets during workflow execution.

  4. Overview of Google Cloud Storage (GCS) Asset Source

    The Google Cloud Storage (GCS) Asset Source polls one or more GCS buckets for objects and provides them to downstream processors, such as the Stream Input Processor. It supports filtering objects by prefix, suffix, and regular expression patterns, and can be configured with housekeeping rules to delete processed objects after a specific age threshold. Authentication is managed via a Google Cloud Connection using OAuth 2.0.

  6. Overview of the AI Model Resource

    The AI Model Resource is a shared technical specification used by the AI Trainer (to train a model) and the AI Classifier (to apply the model at runtime). It defines the input and output attributes, the machine learning algorithm to be used, and the hyperparameters for training. A model must be defined in this resource before it can be trained or used for classification.

  7. Overview of the layline.io Alarm Center

    The Alarm Center is the central hub for monitoring, managing, and configuring cluster alarms. It is located under Operations → Cluster → Alarm Center. Alarms are notifications triggered by the cluster for events such as engine/node errors, workflow failures, resource exhaustion, or connection issues. Each alarm contains a severity level (Error, Warning, or Info), a descriptive name, and the originating node.

  8. Overview of Layline JavaScript API components

    The Layline JavaScript API provides a toolkit for building workflow processors, services, and integrations. The API is organized into the following categories:

    • Enumerations: Constants and typed values (e.g., JavaType for cross-platform compatibility, Severity for logging).
    • Classes: Core building blocks for data handling, workflow orchestration, and utility operations (e.g., Workflow, Message, Email, DateTime, StringUtils).
    • Interfaces: Contracts and type definitions for structured development, particularly for timer configurations (TimerCron, TimerFixedRate, TimerOnce) and email structures.
    • Global Variables: Objects instantly accessible within the JavaScript environment, such as message, workflow, processor, and metrics.

  9. Overview of layline.io platform architecture

    layline.io is a low-code platform for real-time and batch event data processing. The system is organized into four primary functional areas within the Configuration Center:

    1. Project: The build environment where Assets and Workflows are defined to determine data ingestion, processing, and output.
    2. Operations: The monitoring and management hub for live systems, clusters, and engine states.
    3. Shelf: A reusable asset library for sharing standardized configurations across different projects.
    4. Settings: The administrative area for user management, role assignment, and system-wide cluster configuration.

  10. Overview of the Virtual File System (VFS) Source

    The Virtual File System (VFS) Source polls a VFS for files and makes them available to downstream processors. Because it uses a VFS abstraction, it can read from local filesystems, SMB shares, NFS mounts, or any backend supported by the configured VFS Connection. It is primarily used by the Stream Input Processor.

  12. Overview of the Timer Service for delayed message processing

    The Timer Service allows workflows to schedule a message's payload (or any other payload) to be processed at a later time or periodically. This is useful when external systems are unavailable, specific conditions are not yet met, or a predefined delay is required.

    Key operational flow:

    1. Scheduling: A workflow (e.g., Workflow A) uses Timer Service methods to schedule a payload within a specific Timer Group. The payload is stored internally by the Reactive Cluster.
    2. Consumption: A separate workflow (e.g., Workflow B) subscribes to the Timer Service via a Timer Source and an Input Processor. When the timer expires, the service presents the payload to this workflow.
    3. Completion: If the payload reaches a Frame Committer Processor, it is removed from internal storage and marked as processed. If it does not reach a committer, it is rescheduled for the configured period and presented again upon expiration.