DOCSEARCHbyAlgolia
Submit Docs

go-libp2p Documentation

repository·master·Indexed 27 days ago

https://github.com/libp2p/go-libp2p

The Go implementation of the libp2p networking stack, a modular protocol suite for building large-scale peer-to-peer systems. It provides tools for building interoperable P2P applications, including a Network Resource Manager to constrain memory, file descriptors, connections, and streams using a DAG of resource scopes. The library includes built-in support for Prometheus metrics and provides prebuilt Grafana dashboards for monitoring components such as AutoNAT, Auto Relay, Eventbus, Identify, Relay Service, and Swarm.

Tokens
10K
Snippets
12
Records
73
Agent score
90%

What's inside go-libp2p

  1. Understand libp2p Resource Scopes

    master

    The libp2p Resource Manager uses a Directed Acyclic Graph (DAG) of Resource Scopes to enforce multi-resolution resource accounting. Resource usage is aggregated from leaf nodes (like Streams or Connections) up through intermediate nodes (Peers, Protocols, Services) to the top-level System scope. This allows for granular control, such as limiting the impact of a single peer or a specific protocol without affecting the entire system.

    Scope Hierarchy

    • System Scope: The top-level scope that enforces global hard limits across the entire system.
    • Transient Scope: A 'DMZ' scope for resources in the process of being established (e.g., a connection undergoing a handshake or a stream before protocol negotiation). This prevents attacks targeting unestablished resources.
    • Service Scopes: Logical groupings of streams (e.g., DHT, PubSub, or ambient services like Identify) used to isolate and tune resource usage for specific application components.
    • Protocol Scopes: Intermediate scopes used to restrict specific protocols. This is useful for limiting legacy or inefficient protocols (e.g., limiting floodsub within a gossipsub service) to protect the application.
    • Peer Scopes: Constrains resources (connections and streams) used by an individual remote peer to limit the 'blast radius' of a single peer's consumption.
    • Connection Scopes: Leaf nodes representing the duration of a single connection.
    • Stream Scopes: Leaf nodes representing the duration of a single stream.

    Specialized Scopes

    • Allowlist System/Transient Scopes: Used when the standard System or Transient scopes have reached their limits, but the resource originates from an allowlisted peer.
    • User Transaction Scopes: Custom scopes created by a programmer as a child of any existing scope. These are used to account for delimited control flows, such as a specific Request/Response interaction within a stream.

  3. Configure Resource Manager Limits

    master

    The libp2p Resource Manager accounts for basic resource usage across the stack to prevent resource exhaustion and OOM (Out of Memory) conditions.

    Supported Basic Resources:

    • Memory
    • File Descriptors
    • Connections
    • Streams

    Users can set both static (fixed) and dynamic limits. The design allows for seamless integration where existing applications benefit from protection automatically, or opt-in to explicit resource usage accounting to inform and constrain their own usage.

  4. Understand managed libp2p resources

    master

    The Resource Manager tracks and constrains several key resource types:

    • Memory: Primarily buffers used for network operations. Components should attempt a memory reservation before allocation to handle potential limit exhaustion.
    • File Descriptors: Critical system-level resources used for sockets. Exhaustion can prevent the application from opening new connections or files.
    • Connections: High-level libp2p constructs.
      • Inbound Connections: Initiated by remote peers; tightly controlled to prevent overload/attacks.
      • Outbound Connections: Initiated by the local application; constrained to prevent resource leaks from faulty logic.
    • Streams: The fundamental unit of interaction.
      • Inbound Streams: Initiated by remote peers; paramount for system protection.
      • Outbound Streams: Initiated by the local application/service; constrained to manage memory and goroutine usage.

  5. Understand the IPFS Network package architecture

    master

    The IPFS Network package manages peer-to-peer networking, including host connections, communication encryption, and message multiplexing between client services and target hosts. The architecture is composed of the following subcomponents:

    • Conn: A connection to a single Peer.
      • MultiConn: A set of connections to a single Peer.
      • SecureConn: An encrypted (TLS-like) connection.
    • Swarm: Manages connections to multiple Peers and multiplexes data to/from each MultiConn.
    • Muxer: Multiplexes between Services and the Swarm. It handles Request/Reply patterns.
      • Service: Acts as the bridge between an external client service and the Network.
      • Handler: The component within a client service that processes incoming requests.

  6. Install and use go-libp2p in your Go application

    master

    To use the Go implementation of the libp2p networking stack, import the main package into your Go project. This repository serves as the entrypoint to the various packages that compose the stack.

    import "github.com/libp2p/go-libp2p"

  7. Run all interop tests locally with Docker Compose

    master

    To test compatibility against all released libp2p versions, you must have the libp2p/test-plans repository checked out. This process involves building a Docker image for the current go-libp2p head and then running the test suite via npm from the test-plans directory.

    Prerequisites:

    • The libp2p/test-plans repository must be present on your local machine.
    • Docker and make must be installed.

  11. Monitor resource manager metrics

    master

    Use Prometheus metrics to monitor resource usage and identify if limits are being hit too frequently.

    • Key Metric: rcmgr_blocked_resources indicates what was blocked and for which scope.
    • Protocol Monitoring: If streams are being blocked, check the rcmgr_streams metric to understand usage patterns per protocol.
    • Implementation: Refer to obs/stats_test.go for examples on enabling metrics and DefaultViews in stats.go for recommended views.

  12. Expose Prometheus metrics in your Go application

    master

    To use the provided Grafana dashboards for local debugging, your application must expose a metrics collection endpoint. You can do this by using the promhttp handler from the Prometheus client library. This exposes metrics at http://localhost:5001/debug/metrics/prometheus, matching the endpoint used by Kubo.

    import "github.com/prometheus/client_golang/prometheus/promhttp"

go func() {
    http.Handle("/debug/metrics/prometheus", promhttp.Handler())
    log.Fatal(http.ListenAndServe(":5001", nil))
})()