CGraph Documentation

repository·main·Indexed 25 days ago

https://github.com/chunelfeng/cgraph

A high-performance, dependency-free Directed Acyclic Graph (DAG) execution framework written in C++11. CGraph supports parallelism, conditional logic, and complex task pipelines via GNode and GGroup abstractions. It provides a native C++ implementation and a Python wrapper called pycgraph, with additional support for C#, Java, and Go. Key features include eDAG scheduling, control features like pausing and timeouts, and a specialized CSTATUS class for error handling.

Tokens
8.4K
Snippets
9
Records
42
Agent score
80%

What's inside CGraph

  1. What is CGraph?

    main

    CGraph is a cross-platform Directed Acyclic Graph (DAG) framework written in pure C++11 with no third-party dependencies. It is designed to allow developers to build custom operators and describe complex execution schedules, including dependencies, parallelism, aggregation, and conditional logic.

    Key features include:

    • eDAG Scheduling: Supports sequential execution of dependent elements and concurrent execution of non-dependent elements.
    • Control Features: Supports pausing, resuming, and timeout settings.
    • Abstraction Layers: Uses GNode for individual tasks and GGroup for controlling logic like loops and conditional branching.
    • Multi-language Support: Native C++ implementation with a Python wrapper called pycgraph.
  2. Explore related projects and ecosystem

    main

    CGraph is part of a broader ecosystem of high-performance computing and graph-based tools. Key related projects include:

    • GraphANNS: Graph-based Approximate Nearest Neighbor Search built on top of CGraph.
    • PyCGraph-example: A repository containing practical examples of how to use the Python interface.
    • CThreadPool: A high-performance, cross-platform C++ thread pool used by CGraph.
    • CGraph-lite: A lightweight version of CGraph providing simple DAG construction and parameter passing, with a fully compatible interface for seamless migration.
  3. Explore Python and C++ hybrid programming

    main
    CGraph supports hybrid programming between Python and C++. This allows developers to leverage the high performance of C++ graph execution while using the ease of use of Python for orchestration and high-level logic.
  4. How to use pycgraph in Python

    main

    To use the Python version of CGraph (pycgraph):

    1. Inherit from GNode: Create a subclass of GNode and implement the run() method. The method should return a CStatus() object.
    2. Initialize Pipeline: Create a GPipeline() instance.
    3. Register Elements: Use pipeline.registerGElement(node_instance, dependency_set, name) to define nodes and their dependencies. Dependencies are passed as a set of node instances.
    4. Execute: Call pipeline.process() to run the graph.
    import time
    from datetime import datetime
    from pycgraph import GNode, GPipeline, CStatus
    
    class MyNode1(GNode):
        def run(self):
            print("[{0}] {1}, enter MyNode1 run function. Sleep for 1 second ... ".format(datetime.now(), self.getName()))
            time.sleep(1)
            return CStatus()
    
    class MyNode2(GNode):
        def run(self):
            print("[{0}] {1}, enter MyNode2 run function. Sleep for 2 second ... ".format(datetime.now(), self.getName()))
            time.sleep(2)
            return CStatus()
    
    if __name__ == '__main__':
        pipeline = GPipeline()
        a, b, c, d = MyNode1(), MyNode2(), MyNode1(), MyNode2()
    
        pipeline.registerGElement(a, set(), "nodeA")
        pipeline.registerGElement(b, {a}, "nodeB")
        pipeline.registerGElement(c, {a}, "nodeC")
        pipeline.registerGElement(d, {b, c}, "nodeD")
    
        pipeline.process()
  5. Install pycgraph via pip

    main

    The Python version of CGraph, pycgraph, can be installed using pip. This provides access to the full-featured Python version of the graph execution engine.

    Note: In versions prior to v3.2.2, the package was named PyCGraph. For the current version, use pycgraph.

    pip3 install pycgraph
  6. Compile the C++ version of CGraph

    main

    CGraph supports MacOS, Linux, Windows, and Android with no third-party dependencies. It defaults to C++11, but C++17 is recommended. It does not support versions below C++11.

    IDE Setup

    • CLion (Recommended): Open the CMakeLists.txt file directly as a project.
    • Visual Studio (Windows): Use CMake to generate a .sln file.
    • Xcode (MacOS): Use CMake to generate an .xcodeproj file.

    Build Systems

    You can build CGraph using CMake, Bazel, or Xmake.

    ### Windows (Visual Studio)
    $ git clone https://github.com/ChunelFeng/CGraph.git
    $ cd CGraph
    $ cmake . -Bbuild
    
    ### MacOS (Xcode)
    $ git clone https://github.com/ChunelFeng/CGraph.git
    $ cd CGraph
    $ mkdir build && cd build
    $ cmake .. -G Xcode
    
    ### CMake (Linux/MacOS/Windows)
    $ git clone https://github.com/ChunelFeng/CGraph.git
    $ cd CGraph
    $ cmake . -Bbuild
    $ cd build && make -j8
    $ ./tutorial/T00-HelloCGraph
    
    ### Bazel (Linux/MacOS/Windows)
    $ git clone https://github.com/ChunelFeng/CGraph.git
    $ cd CGraph
    $ bazel build //tutorial/...
    $ bazel run //tutorial:T00-HelloCGraph
    
    ### Xmake (Linux/MacOS/Windows)
    $ git clone https://github.com/ChunelFeng/CGraph.git
    $ cd CGraph
    $ xmake build
    $ xmake run T00-HelloCGraph
  7. Integrate CGraph into a CMake project via FetchContent

    main

    If you are using a project that already uses CMakeLists.txt, you can include CGraph as a third-party library using FetchContent. It is recommended to disable tutorials, examples, and tests to reduce build time.

    set(CGRAPH_BUILD_TUTORIALS OFF CACHE BOOL "" FORCE)
    set(CGRAPH_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
    set(CGRAPH_BUILD_FUNCTIONAL_TESTS OFF CACHE BOOL "" FORCE)
    set(CGRAPH_BUILD_PERFORMANCE_TESTS OFF CACHE BOOL "" FORCE)
    
    Include(FetchContent)
    FetchContent_Declare(
      CGraph
      GIT_REPOSITORY https://github.com/ChunelFeng/CGraph.git
      GIT_TAG main
      GIT_SHALLOW true
    )
    
    FetchContent_MakeAvailable(CGraph)
    target_include_directories(${PROJECT_NAME} PRIVATE ${CGraph_SOURCE_DIR}/src)
    target_link_libraries(${PROJECT_NAME} PRIVATE CGraph)
  8. How to use CGraph in C++

    main

    To use CGraph in C++, follow these steps:

    1. Inherit from GNode: Create a subclass of GNode and implement the run() method. The run() method should return a CStatus object.
    2. Create a Pipeline: Use GPipelineFactory::create() to instantiate a GPipelinePtr.
    3. Register Elements and Dependencies: Use pipeline->registerGElement<T>(element_ptr, dependencies, name) to add nodes to the graph and define their execution order. Dependencies are passed as a list of pointers to previously registered elements.
    4. Execute: Call pipeline->process() to start the graph execution.
    5. Cleanup: Call GPipelineFactory::remove(pipeline) to release resources.
    #include "CGraph.h"
    
    using namespace CGraph;
    
    class MyNode1 : public GNode {
    public:
        CStatus run() override {
            printf("[%s], sleep for 1 second ...\n", this->getName().c_str());
            CGRAPH_SLEEP_SECOND(1)
            return CStatus();
        }
    };
    
    class MyNode2 : public GNode {
    public:
        CStatus run() override {
            printf("[%s], sleep for 2 second ...\n", this->getName().c_str());
            CGRAPH_SLEEP_SECOND(2)
            return CStatus();
        }
    };
    
    int main() {
        /* Create a pipeline for configuring and executing graph flow information */
        GPipelinePtr pipeline = GPipelineFactory::create();
        GElementPtr a, b, c, d = nullptr;
    
        /* Register dependencies between nodes */
        pipeline->registerGElement<MyNode1>(&a, {}, "nodeA");
        pipeline->registerGElement<MyNode2>(&b, {a}, "nodeB");
        pipeline->registerGElement<MyNode1>(&c, {a}, "nodeC");
        pipeline->registerGElement<MyNode2>(&d, {b, c}, "nodeD");
    
        /* Execute the graph flow framework */
        pipeline->process();
    
        /* Clear all resources in the pipeline */
        GPipelineFactory::remove(pipeline);
    
        return 0;
    }
  9. Install the Python version (pycgraph)

    main

    You can install the Python bindings pycgraph via pip, from source, or using uv.

    Prerequisites for source installation: python3, pybind11, and setuptools must be installed.

    ### Via pip (Recommended)
    $ pip3 install pycgraph
    $ python3 -c "import pycgraph"
    
    ### Via Source (Wheel - Recommended)
    $ git clone https://github.com/ChunelFeng/CGraph.git
    $ cd CGraph/python
    $ python3 setup.py bdist_wheel
    $ pip3 install dist/pycgraph-xxx.whl
    
    ### Via Source (Direct Install)
    $ git clone https://github.com/ChunelFeng/CGraph.git
    $ cd CGraph/python
    $ python3 setup.py install
    
    ### Via uv
    $ uv init cgraph_env
    $ cd cgraph_env
    $ uv add "pycgraph @ git+https://github.com/ChunelFeng/CGraph.git@main#subdirectory=python" --marker "sys_platform == 'linux'"
    $ python3 -c "import pycgraph"
  10. Use the CGraph Online Compilation Environment

    main

    An online environment is available via Gitpod. After logging in with a GitHub account, you can use the provided build script to compile and run the project.

    $ sudo apt-get install cmake -y
    $ ./CGraph-build.sh
    $ ./build/tutorial/T00-HelloCGraph
  11. Manage parallel execution with UThreadPool

    main

    The UThreadPool class provides a mechanism for managing a pool of threads to execute tasks in parallel. It supports different task submission strategies, including priority-based execution, specific thread targeting, and task groups.

    Lifecycle

    1. Construction: Create an instance using UThreadPool(CBool autoInit = true, const UThreadPoolConfig& config = UThreadPoolConfig()). If autoInit is true, the pool starts immediately.
    2. Configuration: Use setConfig(const UThreadPoolConfig &config) to modify settings. Note: This must be called before init(). If using the UThreadPoolSingleton, you must call destroy() before setting new parameters.
    3. Initialization: Call init() to start all threads. Check status with isInit().
    4. Execution: Submit tasks using commit, execute, or submit (see below).
    5. Cleanup: Call destroy() to release all thread resources.
  12. Manage graph processing workflows with GPipeline

    main

    The GPipeline class is the central controller for managing graph-based processing workflows. It allows you to define a directed acyclic graph (DAG) of elements (nodes and groups), manage their dependencies, and execute them either synchronously or asynchronously.

    Key capabilities include:

    • Lifecycle Management: Initialize, run, suspend, resume, and destroy the pipeline.
    • Element Registration: Create and register GNode and GGroup objects with specific dependencies and loop counts.
    • Execution Control: Run the pipeline multiple times (process), run it asynchronously (asyncRun, asyncProcess), or cancel/suspend execution.
    • Observability: Dump the graph structure to Graphviz format for visualization and generate performance analysis reports.
    • Extensions: Add aspects, daemons, events, and stages to the pipeline to extend its behavior.