broadcaster

repository·master·Indexed 22 days ago

https://github.com/encode/broadcaster

A Python library providing a simple, unified API for real-time streaming and broadcasting functionality. It supports multiple backends including Redis (PUB/SUB and Streams), Kafka, Postgres (LISTEN/NOTIFY), and an in-memory backend for local development. The library allows users to publish messages to channels and subscribe to them using an asynchronous iterator to receive Event objects.

Tokens
3.5K
Snippets
9
Records
21
Agent score
79%

What's inside broadcaster

  1. Publish and Subscribe to channels

    master

    Broadcaster provides a simple API for real-time messaging.

    Publishing

    Use broadcast.publish(channel, message) to send a message to a specific channel.

    Subscribing

    Use async with broadcast.subscribe(channel) as subscriber: to listen for messages. The subscriber object is an async iterator that yields events. Each event has a .message attribute.

    Lifecycle Management

    Ensure you call broadcast.connect() on application startup and broadcast.disconnect() on application shutdown to manage backend connections properly.

  2. Install Broadcaster

    master

    Install the core package using pip:

    pip install broadcaster

    To use specific backends, install the corresponding extras:

    pip install broadcaster[redis]
    pip install broadcaster[postgres]
    pip install broadcaster[kafka]
  3. Configure Broadcaster backends

    master

    Broadcaster uses connection strings to initialize different backends. Use the following formats to select your backend:

    • In-memory (for local development/testing): Broadcast('memory://')
    • Redis PUB/SUB: Broadcast("redis://localhost:6379")
    • Redis Streams: Broadcast("redis-stream://localhost:6379")
    • Postgres LISTEN/NOTIFY: Broadcast("postgres://localhost:5432/broadcaster")
    • Apache Kafka: Broadcast("kafka://localhost:9092")
    from broadcaster import Broadcast
    
    # Example: Redis backend
    broadcast = Broadcast("redis://localhost:6379")
  4. Configure Broadcaster backends via BROADCAST_URL

    master

    To use backends other than the default in-memory backend, set the BROADCAST_URL environment variable and ensure the corresponding service is running (e.g., via Docker Compose).

    BackendEnvironment VariableService Command
    kafkaexport BROADCAST_URL=kafka://localhost:9092docker-compose up kafka
    redisexport BROADCAST_URL=redis://localhost:6379docker-compose up redis
    postgresexport BROADCAST_URL=postgres://localhost:5432/broadcasterdocker-compose up postgres
  5. Example: Simple WebSocket Chat App

    master

    This example demonstrates how to integrate Broadcaster with Starlette to create a real-time chat application. It uses broadcast.publish to receive messages from a WebSocket and broadcast.subscribe to send messages from the broadcast backend back to the WebSocket.

    Requirements: starlette, uvicorn, jinja2 Run command: uvicorn example:app

    import anyio
    from broadcaster import Broadcast
    from starlette.applications import Starlette
    from starlette.routing import Route, WebSocketRoute
    from starlette.templating import Jinja2Templates
    
    broadcast = Broadcast("redis://localhost:6379")
    templates = Jinja2Templates("templates")
    
    
    async def homepage(request):
        template = "index.html"
        context = {"request": request}
        return templates.TemplateResponse(template, context)
    
    
    async def chatroom_ws(websocket):
        await websocket.accept()
    
        async with anyio.create_task_group() as task_group:
            # run until first is complete
            async def run_chatroom_ws_receiver() -> None:
                await chatroom_ws_receiver(websocket=websocket)
                task_group.cancel_scope.cancel()
    
            task_group.start_soon(run_chatroom_ws_receiver)
            await chatroom_ws_sender(websocket)
    
    
    async def chatroom_ws_receiver(websocket):
        async for message in websocket.iter_text():
            await broadcast.publish(channel="chatroom", message=message)
    
    
    async def chatroom_ws_sender(websocket):
        async with broadcast.subscribe(channel="chatroom") as subscriber:
            async for event in subscriber:
                await websocket.send_text(event.message)
    
    
    routes = [
        Route("/", homepage),
        WebSocketRoute("/", chatroom_ws, name='chatroom_ws'),
    ]
    
    
    app = Starlette(
        routes=routes, on_startup=[broadcast.connect], on_shutdown=[broadcast.disconnect],
    )
    # Requires: `starlette`, `uvicorn`, `jinja2`
    # Run with `uvicorn example:app`
    import anyio
    from broadcaster import Broadcast
    from starlette.applications import Starlette
    from starlette.routing import Route, WebSocketRoute
    from starlette.templating import Jinja2Templates
    
    
    broadcast = Broadcast("redis://localhost:6379")
    templates = Jinja2Templates("templates")
    
    
    async def homepage(request):
        template = "index.html"
        context = {"request": request}
        return templates.TemplateResponse(template, context)
    
    
    async def chatroom_ws(websocket):
        await websocket.accept()
    
        async with anyio.create_task_group() as task_group:
            # run until first is complete
            async def run_chatroom_ws_receiver() -> None:
                await chatroom_ws_receiver(websocket=websocket)
                task_group.cancel_scope.cancel()
    
            task_group.start_soon(run_chatroom_ws_receiver)
            await chatroom_ws_sender(websocket)
    
    
    async def chatroom_ws_receiver(websocket):
        async for message in websocket.iter_text():
            await broadcast.publish(channel="chatroom", message=message)
    
    
    async def chatroom_ws_sender(websocket):
        async with broadcast.subscribe(channel="chatroom") as subscriber:
            async for event in subscriber:
                await websocket.send_text(event.message)
    
    
    routes = [
        Route("/", homepage),
        WebSocketRoute("/", chatroom_ws, name='chatroom_ws'),
    ]
    
    
    app = Starlette(
        routes=routes, on_startup=[broadcast.connect], on_shutdown=[broadcast.disconnect],
    )
  6. Implement a custom Broadcaster backend

    master

    To extend Broadcaster with a new backend, create a class that inherits from BroadcastBackend and pass an instance of it to the Broadcaster constructor using the backend argument.

    from broadcaster import Broadcaster, BroadcastBackend
    
    class MyBackend(BroadcastBackend):
        # Implement required BroadcastBackend methods
        pass
    
    broadcaster = Broadcaster(backend=MyBackend())
    from broadcaster import Broadcaster, BroadcastBackend
    
    class MyBackend(BroadcastBackend):
        pass
    
    broadcaster = Broadcaster(backend=MyBackend())
  7. Configure infrastructure with docker-compose.yaml

    master

    The docker-compose.yaml file provides a pre-configured environment for running Broadcaster with various backends. It includes services for Zookeeper, Kafka, Redis, and PostgreSQL.

    Note that these services are configured with specific environment variables and ports that may be required for the Broadcaster backends to connect successfully.

    version: '3'
    services:
      zookeeper:
        image: "confluentinc/cp-zookeeper"
        hostname: zookeeper
        ports:
          - 32181:32181
        environment:
          - ZOOKEEPER_CLIENT_PORT=32181
          - ALLOW_ANONYMOUS_LOGIN=yes
      kafka:
        image: confluentinc/cp-kafka
        hostname: kafka
        ports:
        - 9092:9092
        - 29092:29092
        depends_on:
        - zookeeper
        environment:
          - KAFKA_ZOOKEEPER_CONNECT=zookeeper:32181
          - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
          - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
          - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT_HOST://localhost:29092,PLAINTEXT://localhost:9092
          - KAFKA_BROKER_ID=1
          - ALLOW_PLAINTEXT_LISTENER=yes
      redis:
        image: "redis:alpine"
        ports:
          - 6379:6379
      postgres:
        image: "postgres:12"
        environment:
          - POSTGRES_DB=broadcaster
          - POSTGRES_PASSWORD=postgres
          - POSTGRES_HOST_AUTH_METHOD=trust
          - POSTGRES_USER=postgres
        ports:
          - 5432:5432
  8. Subscribe to a channel using the subscribe() context manager

    master

    To receive messages, use the subscribe asynchronous context manager on a Broadcast instance. This returns a Subscriber object. The context manager handles the lifecycle of the subscription: it registers the subscriber with the backend and automatically unsubscribes/cleans up when the block is exited.

    When the subscription context is exited, the Subscriber will raise an Unsubscribed exception when attempting to retrieve further messages.

  9. Use the Subscriber class to consume events

    master

    A Subscriber instance provides two ways to consume messages:

    1. Asynchronous Iteration: Iterating over the Subscriber using async for will yield Event objects. The loop terminates gracefully when the subscription is closed.
    2. Manual Retrieval: Calling await subscriber.get() returns the next Event. If the subscription has been closed, it raises an Unsubscribed exception.

    An Event object contains the channel (str) and the message (Any).