gotd/td

repository·main·Indexed 25 days ago

https://github.com/gotd/td

A high-performance, pure Go implementation of the Telegram MTProto 2.0 protocol. It enables developers to build Telegram clients for users and bots with low memory overhead and high concurrency support. The library provides a high-level telegram package for connection lifecycle and authentication, a generated tg.Client for direct MTProto method calls, and support for JSON-formatted API interactions.

Tokens
12.1K
Snippets
21
Records
77
Agent score
81%

What's inside gotd/td

  1. Understand the request lifecycle of an RPC call

    main

    When calling a method like client.API().SomeMethod(ctx, ...), the request flows through the following layers:

    1. Serialization: The generated tg.Client method serializes the request using the bin package and calls Invoke on the telegram.Client.
    2. Middleware: Middlewares (such as rate limiting, flood-wait retry, or tracing) process the request.
    3. Connection Pooling: telegram.Client acquires a connection from the pool for the target Data Center (DC), handling migrations or redirects to CDN DCs if necessary.
    4. MTProto Layer: mtproto.Conn wraps the payload in an MTProto message (proto), encrypts it using the session auth_key (crypto), and writes it via the transport codec.
    5. RPC Engine: The rpc engine manages message IDs, sends acknowledgments (acks), and handles retransmissions.
    6. Response Handling: The response is decrypted, decoded, and routed back to the caller. Simultaneously, updates are dispatched to the telegram/updates manager and then to the user's UpdateHandler.
  2. Handle large file uploads with automatic part sizing

    main
    The uploader automatically manages part sizes to prevent FILE_PARTS_INVALID errors on large files. It grows the part size from the file size to keep parts within the 3999 limit. However, if you provide an explicit size using WithPartSize, that value will still be respected.
  3. Understand the layers of the `gotd` architecture

    main

    The gotd library is organized into a layered architecture where each layer depends only on the layers below it. This separation of concerns allows for modularity and specialized handling at different levels of the protocol stack:

    1. High level (telegram): Authentication, updates, DC management, and convenience helpers.
    2. Pool (pool): Per-datacenter connection pooling (pool.DC).
    3. Connection (mtproto): Manages a single MTProto connection lifecycle (key exchange, pings, salts, and (de)serialization).
    4. Support Layers:
      • RPC engine (rpc): Handles request/response matching, acknowledgements, and retries.
      • Crypto (crypto & exchange): Implements AES-IGE, RSA, Diffie-Hellman, and the auth key generation protocol.
      • Binary protocol (bin & proto): Handles TL/MTProto wire types and message primitives using non-streaming, reflection-free serialization.
    5. Transport (transport): Manplements TCP/WebSocket connections, codecs (abridged, full, etc.), and obfuscation.
  4. Connection-loss recovery and transparent retries

    main

    The library implements connection-loss recovery to handle requests that are not processed by the server (either not sent or sent but not acknowledged). These requests are transparently retried on a new connection. This mechanism involves:

    • rpc close cause handling
    • telegram.Client.invokeConn waiting for a reconnect
    • pool.DC.Invoke re-acquiring the connection
  5. How video uploading and processing works in bot-bigbuckbunny

    main

    This example demonstrates several patterns for handling large file uploads in gotd:

    • Multi-connection Upload Pool: Uses client.Pool(N) to create a pool of N sub-connections to the current Data Center (DC). This allows part uploads to be balanced across multiple connections for higher throughput.
    • Thread Management: Uses uploader.WithThreads(N) to set a per-upload goroutine limit. This should be matched to the pool size to ensure part uploads fan out across all available connections.
    • Progress Tracking: Uses uploader.WithProgress to implement a custom uploader.Progress interface. This allows logging upload speed (instantaneous and average) and progress at specific intervals (e.g., every 5 seconds).
    • FFmpeg Integration:
      • Probing: Uses ffprobe to determine video width, height, and duration, which are then set via Resolution and Duration.
      • Thumbnails: Extracts a JPEG thumbnail using ffmpeg and attaches it via UploadedDocumentBuilder.Thumb.
      • Trimming: Uses the -cut flag to apply the ffmpeg -t command to limit video duration.
      • Compression: Uses the -max-size flag to re-encode the video to fit a target size budget by deriving a target bitrate from the requested size and duration.
    • Video Replies: Uses message.UploadedDocument(...).Video() to ensure the uploaded file is sent as a video reply.
  6. Use testing infrastructure for Telegram development

    main

    The project provides several specialized tools for testing different layers of a Telegram client implementation:

    • tgtest: An in-process Telegram server written in pure Go. Use tgtest/cluster to spin up multi-DC setups and tgtest/services to provide server-side behavior for end-to-end tests without a real network.
    • tgmock: A mock tg.Invoker designed for unit-testing code that issues RPC calls.
    • testutil and clock: Provides deterministic time (backed by gotd/neo) to test timeouts, pings, and retries reliably.
    • _fuzz: Contains fuzzing corpora for testing message handling, the key-exchange flow, and RSA.
  7. Prepare credentials for connclose

    main

    Before running the reproduction environment, you must configure your Telegram bot and application credentials. Copy the example secret file to secret.yml and edit it with your actual credentials.

    cp secret.example.yml secret.yml
  8. Run the bot-bigbuckbunny example

    main

    The bot-bigbuckbunny example is a bot that replies to every incoming message with a video. It demonstrates advanced video uploading techniques including multi-connection upload pools, progress logging, and optional ffmpeg integration for probing and thumbnail generation.

    Prerequisites

    Set the following environment variables:

    • BOT_TOKEN
    • APP_ID
    • APP_HASH
    • SESSION_FILE or SESSION_DIR

    ffmpeg and ffprobe are optional dependencies. If they are in your PATH, the bot will use them to probe video dimensions and extract thumbnails. If they are missing, the bot will still function using default MIME types and no thumbnails.

    go run ./bot-bigbuckbunny
  9. Regenerate generated code from TL schemas

    main

    A large portion of the tg package is generated from TL schemas. If you need to regenerate the code from the schemas located in _schema/*.tl, use the go generate command or the provided make target.

    go generate ./...
    # or
    make generate
  10. Run examples using environment variables

    main

    Most examples in this repository are designed to be run using environment variable client builders. To run an example, follow these steps:

    1. Obtain your api_id and api_hash from Telegram.
    2. Set the APP_ID and APP_HASH environment variables.
    3. Set SESSION_FILE to a path (e.g., ~/session.yourbot.json) to ensure persistent authentication.
    4. Execute the example binary.

    Warning: Do not share your APP_ID or APP_HASH, as they cannot be easily rotated.

    For manual authentication setup without environment variables, refer to the bot-auth-manual example.

  11. Authenticate as a User

    main

    Use td/telegram/auth.Flow to handle the user authentication process. You can define a custom CodeAuthenticatorFunc to prompt the user for the login code (e.g., via terminal). For accounts without 2FA, use telegram.CodeOnlyAuth. For accounts with 2FA, use telegram.ConstantAuth to provide the password.

    codePrompt := func(ctx context.Context, sentCode *tg.AuthSentCode) (string, error) {
        // NB: Use "golang.org/x/crypto/ssh/terminal" to prompt password.
        fmt.Print("Enter code: ")
        code, err := bufio.NewReader(os.Stdin).ReadString('\n')
        if err != nil {
            return "", err
        }
        return strings.TrimSpace(code), nil
    }
    // This will setup and perform authentication flow.
    // If account does not require 2FA password, use telegram.CodeOnlyAuth
    // instead of telegram.ConstantAuth.
    if err := auth.NewFlow(
        auth.Constant(phone, password, auth.CodeAuthenticatorFunc(codePrompt)),
        auth.SendCodeOptions{},
    ).Run(ctx, client.Auth()); err != nil {
        panic(err)
    }