DOCSEARCHbyAlgolia
Submit Docs

WeFe Secure Computing Framework

repository·main·Indexed 40 days ago

https://github.com/tianmiantech/wefe

An open-source secure computing framework for federated learning. WeFe supports horizontal, vertical, and hybrid federated learning, featuring blockchain-based evidence sharing, visual workflow management, and a dedicated board module for administrative and modeling tasks. The framework includes specialized storage scenarios (Persistent, FC, and Middleware) and integrates CGBN for high-performance GPU-accelerated multiple-precision integer arithmetic.

Tokens
139.3K
Snippets
351
Records
538
Agent score
50%

What's inside WeFe

  1. Overview of the fusion data fusion system

    main

    fusion is a lightweight data fusion system designed for sample alignment, which is typically the initial step in Vertical Federated Learning. It uses the RSA-based PSI (Private Set Intersection) algorithm to provide secure, reliable, and fast data fusion services.

    Key features include:

    • Sample alignment using the RSA-PSI algorithm.
    • Data security guaranteed by the RSA algorithm.
    • Data preprocessing that generates and persists Bloom filters to disk for reuse.
    • Support for time-based backtracking of data sample features.
    • Extensible architecture to support future algorithms.

  2. Overview of the WEFE Gateway

    main

    The gateway component is responsible for data interaction between members and various parties in the WEFE system. It is built on a SpringBoot + gRPC architecture and provides a unified entry and exit point for federated member data, reducing the complexity of interactions between modules and members.

    Key Features:

    • Multi-language Support: Supports clients in various languages, including Java and Python.
    • Unified Interface: Simplifies data exchange by centralizing member data access.
    • Security Protections: Implements signature authentication, IP whitelisting, data anti-tampering, and anti-replay mechanisms.

  3. Overview of the Union module

    main

    The union module is a decentralized public service designed to provide shared services among members within a Wefe Federated System. It consists of two primary components:

    1. union-service: The backend service for the union module that provides the necessary APIs for service calls.
    2. blockchain-data-sync: A module responsible for the real-time synchronization of blockchain data to a business database. This addresses the inherent limitations of blockchain technology regarding query performance and optimizes business-related queries.

  4. Overview of WeFe (WeLab Federated Learning)

    main

    WeFe is an open-source secure computing framework designed for the federated learning ecosystem. It provides a reliable infrastructure for collaborative machine learning while maintaining data privacy.

    Key features include:

    • Federated Ecosystem: Tools to find collaborators and intended datasets.
    • User-Friendly Interface: A visual interaction page that follows traditional modeling workflows.
    • Multi-Engine Support: Supports both Function Computing and Spark engines.
    • Hybrid Modeling: Supports industry solutions combining Vertical Federated Learning (VFL) and Horizontal Federated Learning (HFL).
    • Performance & Security: Supports GPU acceleration and blockchain-based accounting/logging.
    • Deployment: Supports one-click deployment via Docker.

  5. Overview of the board module

    main

    The board module in the WeFe system provides a user-friendly interface for several key administrative and modeling tasks, including:

    • Visual Modeling Operations: Perform modeling through a graphical interface.
    • User Management: Manage system users.
    • System Management: Handle general system administration.

    The module is split into two primary components:

    1. board-service: The backend service that exposes APIs for all operations.
    2. board-website: The frontend module that provides the visual user interface.

  6. Overview of Java Common Modules

    main

    The Java service common modules provide shared functionality for data storage and synchronization.

    Key modules include:

    • common-data-storage: Handles persistence for data and intermediate algorithm data. Supported storage backends include ClickHouse, LMDB, and cloud storage services like OTS and OSS.
    • common-data-mongodb: Implements storage for synchronizing blockchain data into business databases using MongoDB.

  7. Overview of serving-service features

    main

    The serving-service is an independent online service platform for federated learning joint prediction. Key capabilities include:

    • Prediction Modes: Supports both single-record and batch prediction.
    • Security: Uses RSA key signing to secure exposed APIs and ensures multi-party member data never leaves its original storage to maintain privacy.
    • Monitoring: Real-time logging of model calls and statistics, with multi-dimensional visual monitoring via serving-website.
    • Model Management: One-click model deployment (online/offline) and one-click synchronization of models trained by other WeFe services.
    • Debugging: Supports online model debugging.

  8. Overview of WeFe Federated Learning Framework

    main

    WeFe (WeLab Federated Learning) is an open-source secure computing framework designed for the federated learning ecosystem. It provides a reliable infrastructure for performing privacy-preserving machine learning across distributed entities.

    Key features include:

    • Hybrid Federated Learning: Combines horizontal and vertical federated learning into industry-specific solutions.
    • Blockchain Integration: Uses consortium chains as a decentralized solution for evidence sharing and storage.
    • Visual Workflow: Supports drag-and-drop interactive flow editing for modeling processes.
    • Scalability: Implements dynamic resource expansion based on function computing and cloud storage objects.
    • Hardware Acceleration: Supports GPU acceleration for homomorphic encryption operations (experimental).

    Supported algorithms include Horizontal Federated Learning, Vertical Federated Learning, Hybrid Federated Learning, and Deep Learning. For specific algorithm details, refer to the kernel/README.md documentation.

  9. Overview of Python Common Modules

    main

    The Python service common modules provide a foundation for tool layers, data storage interaction, and distributed computing engine integration.

    The module structure is organized as follows:

    • calculation: Implements computation logic. Supports Spark distributed computing, Function Compute, and experimental GPU hardware acceleration.
    • common: Defines system constants, including unified exception definitions, constants, and enums.
    • db: Provides specific business database operations. Supports MySQL and SQLite.
    • dto: Contains Data Transfer Objects.
    • p_federation: Implements data interaction between members, providing both single-machine and cluster-based solutions.
    • p_session: Implements session management for storage and computation functions.
    • protobuf: Contains protocol files for data transmission and certain storage structures.
    • storage: Handles persistence for data and intermediate algorithm data. Supported backends include ClickHouse, LMDB, and cloud storage services like OTS and OSS.
    • utils: Provides utility methods for caching, serialization, configuration, and logging.

  10. Overview of WEFE core modules

    main

    WEFE (Wireless Edge Federated Learning) is composed of several functional modules categorized into Basic and Advanced modules. A federation is made up of individual participants called 'members'.

    Basic Modules

    • board: Provides a user-friendly visual interface for modeling operations and user management.
    • gateway: Handles data interaction between different services.
    • flow: Responsible for task scheduling and monitoring.
    • kernel: Contains the underlying federated learning algorithms.
    • common: Provides foundational services used by all other modules.

    Advanced Modules

    • fusion: Provides data fusion services.
    • serving: Provides model invocation/inference services.

  11. Overview of union-service

    main
    The union-service is the backend service for the Federated Learning (FL) Union component. It utilizes a decentralized data storage approach to store public information about federated members, such as member profiles and public dataset information. It provides HTTP interfaces that allow members to modify their own public information and retrieve public information about other members in the federation. The architecture is built using Spring Boot, MongoDB, and Fisco BCOS to ensure decentralization, immutability, and traceability.

  12. Benchmark pybind11 against Boost.Python

    main

    The pybind11 library is benchmarked against Boost.Python using a synthetic test suite that generates dummy C++ classes with varying numbers of methods (from 1 to 2048 classes in powers of two). Each class contains four methods with randomly generated signatures.

    Key findings from the benchmark:

    • Compilation Time: pybind11 generally offers faster compilation because it includes fewer headers. For a large file with 2048 classes and 8192 methods, pybind11 was approximately 1.2x faster (19.8s vs 116.35s for Boost.Python).
    • Module Size: pybind11 produces significantly smaller binaries. For the largest test case, the pybind11 output was 2.17x smaller (9.1 MiB smaller) than the Boost.Python binary.