DOCSEARCHbyAlgolia
Submit Docs

AGEIPort Framework

repository·master·Indexed 20 days ago

https://github.com/alibaba/ageiport

A high-performance, event-driven framework for large-scale data import and export (Excel/CSV) in complex B2B scenarios. It supports single-machine and cluster execution, featuring a decentralized architecture with three primary modules: ageiport-processor for business logic, ageiport-task for persistence, and ageiport-web for frontend integration. The framework utilizes a system of Main Tasks, Sub-tasks, Slices, and Offsets to handle task distribution and progress tracking.

Tokens
1K
Snippets
0
Records
5
Agent score
23%

What's inside AGEIPort

  1. Overview of AGEIPort Framework

    master

    AGEIPort is a high-performance, scalable, and extensible data import/export framework designed for complex B2B business scenarios (e.g., uploading/downloading Excel or CSV files). It is built on an event-driven architecture and supports both single-machine and cluster execution, as well as serial and parallel processing.

    Key features include:

    • Transparent Execution: Developers focus on business logic while the framework handles task slicing, distribution, and progress tracking.
    • Real-time Progress: Supports real-time task progress calculation and feedback.
    • Decentralized Architecture: Business applications manage their own cluster resources, ensuring isolation and scalability.
    • Standardized Workflow: Defines a clear lifecycle for data processing tasks to improve maintainability.

  2. Core Concepts: Generic Type Parameters (QUERY, VIEW, DATA)

    master

    AGEIPort uses generic interfaces to define domain models, avoiding the use of generic Map or JSON objects and improving type safety.

    • QUERY (QueryObject): Maps to the query parameters. For example, if the input is {"name":"tom"}, the QueryObject should be a class containing a name field.
    • VIEW (ViewObject): Maps to a single row of data in the file (e.g., one row in an Excel sheet).
    • DATA (DataObject): Maps to the type actually written to the data source or returned by an API. This is typically the result of transforming a ViewObject. For example, you might read a row from a CSV (ViewObject), enrich it with extra fields, and then convert it to a DataObject before inserting it into a database.

  3. Core Modules of AGEIPort

    master

    The framework is divided into three primary modules. A developer must implement logic within the processor module and deploy the task module for persistence.

    1. ageiport-processor (Required): The core engine. Developers implement business logic by overriding specific interfaces in this module. It provides both Local APIs and HTTP APIs to create and execute task instances.
    2. ageiport-task (Required): The task management module. It interacts with the database to persist task definitions and instances. It exposes HTTP interfaces that the processor module calls to perform CRUD operations on tasks. You must deploy this module and ensure the processor can reach it via network (DNS/K8S Service).
    3. ageiport-web (Optional): Provides HTTP/Websocket interfaces for frontend integration (e.g., task creation, querying, file uploads/downloads). If the built-in web features are insufficient, you can write custom Controllers that call the processor module's HTTP APIs.

  4. Core Concepts: Tasks, Slices, and Offsets

    master

    Understanding how AGEIPort decomposes work is essential for implementing custom logic:

    • Main Task: The orchestrator. One machine is selected as the 'Main Task Machine' to handle task acceptance, slicing, distribution, progress statistics, and file merging.
    • Sub Task: The worker. Sub-tasks execute the actual user-defined business logic. A single Main Task is divided into multiple Sub-tasks. For example, exporting 1000 rows might be split into 10 Sub-tasks of 100 rows each.
    • Slice: A segment of the total data or condition set. The framework calculates the size and offset for each slice.
    • Offset: The starting position of a slice. For a total of 10,000 rows and a slice size of 1,000, the first slice has an offset of 0, the second 1000, and so on.

  5. Quick Start with AGEIPort

    master

    To get started with AGEIPort, you can run the provided unit tests in the ageiport-test module.

    Note on Testing vs. Production:

    • Unit Tests: The framework stores tasks in local memory and saves generated files to the local disk.
    • Production: You must follow the "Production Deployment" documentation to configure database persistence and distributed execution.

    For more detailed information, refer to the following guides in the repository: