Autonomi Network

repository·main·Indexed 41 days ago

https://github.com/maidsafe/autonomi

A decentralized, autonomous data and communications network providing lifetime storage, private-by-design encryption, and blockchainless data architecture using Kademlia and libp2p. Includes the `ant` CLI for managing permanent files, mutable key-value registers, shared scratchpads, and EVM-compatible wallets.

Tokens
126.6K
Snippets
383
Records
578
Agent score
45%

What's inside Autonomi

  1. Overview of Autonomi Network EVM data payments

    main
    The ant-evm crate provides the logic for data payments on the Autonomi Network using the Ethereum protocol. It offers abstractions for interacting with EVM-based networks, including handling addresses, wallets, payments, and network configurations.
  2. Overview of the Bootstrap Cache crate

    main
    The ant-bootstrap crate provides a peer caching system for the Autonomi Network. It is designed to provide persistent storage and management of network peer addresses. It handles peer discovery and implements a FIFO (First-In-First-Out) caching strategy. The system is built to support concurrent access across multiple processes safely.
  3. Security models for Registers, Scratchpads, and Pointers

    main

    Different data structures have different access control models:

    Registers

    • Read Access: Anyone with the address can read the value.
    • Write Access: Only the owner with the register key can write.
    • Warning: Do not store sensitive data in registers as they are publicly readable.

    Scratchpads

    • Access: Secret keys grant both read and write access.
    • Permanence: Anyone with the secret key has permanent access. Once shared, a key cannot be revoked.
    • Best Practice: Only share scratchpad keys with trusted parties; consider using application-specific scratchpads.

    Pointers

    • Read Access: Anyone with the pointer address can read what it points to. Pointers reveal the target address.
    • Write Access: Only the owner with the pointer key can update the target. Secret keys grant both read and write access to update the pointer.
    • Warning: Updating a pointer affects everyone who uses that pointer. Be cautious when changing targets.
  4. Understand file privacy and permanence

    main

    When using the Autonomi network, keep these privacy principles in mind:

    • Default Privacy: Files are encrypted by default.
    • Permanence: Public files are permanent. Once uploaded, anyone with the address can access them forever.
    • Public Addresses: Network addresses are public; treat file addresses like public URLs.
    • No Deletion: There is no deletion mechanism. Only upload data you are comfortable storing permanently.
  5. Key features of Bootstrap Cache

    main

    The Bootstrap Cache system includes the following capabilities:

    Storage and Accessibility

    • System-wide accessibility: The cache location is accessible system-wide.
    • Configurable location: The primary cache location can be configured.
    • Cross-process safety: Uses file locking to ensure safety when multiple processes access the cache simultaneously.
    • Atomic operations: Uses atomic write operations to prevent cache corruption.

    Data Management

    • Automatic cleanup: Stale and unreliable peers are automatically removed from the cache.
    • Peer limits: Supports a configurable maximum peer limit.
    • Data integrity: Employs atomic file operations to maintain data integrity.
  6. When to create a new NodeServiceData version

    main

    The NodeServiceData structure uses a versioned schema to manage backward compatibility. You need to decide whether to increment the version based on the nature of your changes:

    Do NOT create a new version if:

    • You are deprecating or removing a field (handled by existing logic).
    • You are adding a new field using #[serde(default)].

    DO create a new version if:

    • You are renaming a field.
    • You are changing the logic or behavior of an existing field.
    • You are adding a new field that requires specific initialization logic (not just a simple serde default).
  7. How NodeServiceData version routing works

    main

    The system uses a custom Deserialize implementation for the main NodeServiceData type to route incoming data to the correct versioned struct based on the schema_version field.

    • Latest versions typically use a custom deserialize_vX() method to handle complex logic.
    • Older versions use standard serde::Deserialize via serde_json::from_value().
    • Routing logic matches the schema_version and then performs a conversion (using Into or From) to upgrade the data to the current latest version.
    impl<'de> Deserialize<'de> for NodeServiceData {
        fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
        where
            D: Deserializer<'de>,
        {
            // Check schema_version and route to appropriate deserializer
            match schema_version {
                Some(2) => {
                    // Use custom deserialize_v2() instead of derived Deserialize
                    match super::node_service_data_v2::NodeServiceDataV2::deserialize_v2(
                        &mut serde_json::de::Deserializer::from_str(&json_value.to_string()),
                    ) {
                        Ok(v2) => Ok(v2),
                        Err(e) => Err(D::Error::custom(format("Failed to deserialize as V2: {}", e))),
                    }
                }
                // ... handle other versions
            }
        }
    }
  8. How Autonomi data types work

    main

    The Autonomi CLI manages different types of data depending on whether you need permanence or mutability:

    • Files (file): Used for permanent, end-to-end encrypted data storage. Data flows from Upload (encryption/storage) $\rightarrow$ Vault (organization) $\rightarrow$ Download (retrieval).
    • Vaults (vault): A personal encrypted storage system used to organize and track your file uploads.
    • Registers (register): Used for mutable key-value data. Ideal for data that needs to change over time.
    • Scratchpads (scratchpad): Used for shared mutable data (up to 4MB). Designed for collaborative data editing.
    • Pointers (pointer): Used to create named references to other data (graphs, scratchpads, or chunks), allowing for updatable links.
    • Wallets (wallet): Manages the EVM-compatible tokens required to fund storage operations.
  9. Export metrics via OpenMetrics

    main

    To enable metrics collection, use the open-metrics feature flag on the node or client. This starts an OpenMetrics exporter on a random port.

    1. Check the logs or stdout for the URL: Metrics server on http://127.0.0.1:xxxx/metrics.
    2. Use a collector like Prometheus to scrape the data.
    3. Use a visualization tool like Grafana to analyze the metrics.
  10. Manage Mutable Encrypted Data with Scratchpad

    main

    The Scratchpad allows for managing mutable, encrypted data on the network. It uses a counter to handle updates and versioning.

    Core Concepts

    • Counter Lifecycle: When creating a scratchpad via scratchpad_create, the counter starts at 0. When using scratchpad_update, the counter is automatically incremented by 1. To set a specific counter value, you must create a new Scratchpad instance and use scratchpad_put.
    • Conflict Resolution: If multiple versions exist, only the scratchpad with the highest counter is kept on the network.

    Scratchpad Class

    • Scratchpad(owner: SecretKey, data_encoding: int, unencrypted_data: bytes, counter: int): Constructor for a new instance.
    • address() -> ScratchpadAddress: Returns the scratchpad's address.
    • decrypt_data(sk: SecretKey) -> bytes: Decrypts the content using the provided secret key.

    Client Methods for Scratchpad

    • scratchpad_get_from_public_key(public_key: PublicKey) -> Scratchpad: Retrieve by owner's public key.
    • scratchpad_get(addr: ScratchpadAddress) -> Scratchpad: Retrieve by address.
    • scratchpad_check_existance(addr: ScratchpadAddress) -> bool: Check if an address exists.
    • scratchpad_put(scratchpad: Scratchpad, payment: PaymentOption) -> Tuple[str, ScratchpadAddress]: Store a scratchpad instance.
    • scratchpad_create(owner: SecretKey, content_type: int, initial_data: bytes, payment: PaymentOption) -> Tuple[str, ScratchpadAddress]: Create a new scratchpad (counter starts at 0).
    • scratchpad_update(owner: SecretKey, content_type: int, data: bytes) -> None: Updates an existing scratchpad (increments counter).
    • scratchpad_cost(public_key: PublicKey) -> str: Returns cost in atto tokens.
  11. How antnode services work with antctl

    main

    The primary purpose of antctl is to manage antnode as a long-running background service using the host operating system's service infrastructure.

    Platform Behavior

    • Linux & macOS: Supports user-mode services, which can be managed without sudo (provided an active user session exists). To run system-wide services instead, prefix all antctl commands with sudo.
    • Windows: Does not support user-mode services. antctl must always be run in an elevated, administrative session.

    Service Lifecycle

    1. Create: Use add to download the antnode binary and define new services.
    2. Status: Use status to view the registry of added, running, or stopped services.
    3. Start/Stop: Use start or stop to control the execution of the services.
    4. Remove: Use remove to delete a service from the registry (the service must be stopped first).
  12. Run integration tests in a local setup

    main

    System-wide integration tests require root access and create real services. To avoid this on a development machine, you can use the provided Vagrant environment.

    Prerequisites:

    • libvirt installed on your host.

    Steps:

    1. Start the VM: vagrant up
    2. Access the VM: vagrant ssh
    3. Switch to root: sudo su -
    4. Navigate to the code directory: cd autonomi
    5. Execute tests: just node-man-integration-tests
    # Inside the Vagrant VM as root
    cd autonomi
    just node-man-integration-tests