DOCSEARCHbyAlgolia
Submit Docs

SIPSorcery

repository·master·Indexed 23 days ago

https://github.com/sipsorcery-org/sipsorcery

A comprehensive .NET library for real-time communications, enabling the implementation of VoIP (SIP) and WebRTC capabilities. It supports protocols including SIP, RTP, WebRTC, ICE, STUN, and SDP. The library includes specialized implementations for integrating with the OpenAI Realtime API, featuring examples for audio routing, local function calling, and ASP.NET Core bridges.

Tokens
59K
Snippets
100
Records
304
Agent score
81%

What's inside SIPSorcery

  1. Overview of SIPSorcery.Cli

    master

    The sipsorcery command-line tool is used to route and bridge live media streams between SIP, WebRTC, and major realtime fabrics (such as Cloudflare Realtime and LiveKit) or AI agents (such as OpenAI Realtime).

    Unlike the sipsorcery-diags tool which is designed for probing and benchmarking, SIPSorcery.Cli treats a stream as the primitive, allowing you to attach various edges to it. It is designed to be human and agent-friendly, supporting --json output and meaningful exit codes for every verb.

  2. Overview of SIPSorcery Core Packages

    master

    SIPSorcery is modular. Depending on your requirements, you may need to install additional packages:

    • SIPSorcery: Core library (SIP, WebRTC, RTP, ICE, STUN, SDP).
    • SIPSorceryMedia.Abstractions: Interfaces for audio/video encoders and device access.
    • SIPSorceryMedia.Windows: Windows-specific audio capture, playback, and video capture.
    • SIPSorceryMedia.FFmpeg: Cross-platform media support using FFmpeg (high performance).
    • SIPSorcery.OpenAI.Realtime: Support for OpenAI's Realtime WebRTC and SIP endpoints.
    • SIPSorcery.VP8: Pure C# implementation of the VP8 video codec (experimental, no native dependencies).

  3. Available diagnostic capabilities in SIPSorcery.Diagnostics

    master

    The SIPSorcery.Diagnostics package is in early preview and provides tools for testing various protocols and services. Current capabilities include:

    • SIP: Testing OPTIONS requests (ping), making calls with audio testing, account registration, and creating digest store files for passwordless registration.
    • WebRTC: Testing WHEP, WHIP publish, and WHIP server (with optional self-publish), echo test peers, and video send benchmarks.
    • Connectivity: STUN/TURN/ICE connectivity checks.
    • Service Integrations: Testing Cloudflare Realtime (TURN, SFU), LiveKit, and the OpenAI Realtime API.

    Note: SIP DNS resolution and additional SIP verbs are planned for future updates.

  4. Key features of the OpenAI Realtime WebRTC implementation

    master

    The SIPSorcery.OpenAI.WebRTC library provides the following capabilities in this example:

    • WebRTC Connectivity: Establishes a connection with OpenAI's realtime endpoint.
    • Audio Streaming: Streams audio directly from default Windows input devices.
    • Conversation Control: Sends response prompts to trigger and manage conversations.
    • Transcription Handling: Logs transcription deltas and completions for both the user and the assistant (utilizing the Whisper1 model).

  5. Explore SIP and VoIP usage examples

    master

    The examples/SIPExamples directory contains sample code for various SIP/VoIP implementations. Use these examples to understand how to implement specific telephony features using the SIPSorcery library.

    Key implementation patterns available in the examples include:

    • Basic Calling: Simple SIP call placement (GetStarted) and adding video support (GetStartedVideo).
    • User Agents:
      • Registering a client with a registrar using SIPRegistrationUserAgent (UserAgentRegister).
      • Placing calls to a server using SIPClientUserAgent (UserAgentClient).
      • Receiving calls using SIPServerUserAgent (UserAgentServer).
    • Infrastructure:
      • Implementing a rudimentary SIP Proxy and Registrar (SIPProxy).
      • Creating a basic STUN server (RFC3849) (StunServer).
      • Creating a WebSocket listener to transport SIP messages (GetStartedWebSocket).
    • Call Control:
      • Performing Call Hold and Blind Transfers using REFER requests (RFC3515) (CallHoldAndTransfer).
      • Performing Attended Transfers (AttendedTransfer).
    • Media & Signaling:
      • Sending DTMF tones via RTP events (RFC2833) (SendDtmf).
    • Applications: A rudimentary SIP softphone implementation (Softphone).

  6. How OpenAI WebRTC Function Calling works in SIPSorcery

    master

    This example demonstrates a pattern for handling local LLM agent behavior triggered via live transcription using OpenAI's real-time WebRTC API.

    The Workflow

    1. Session Initialization: The app establishes a WebRTC connection and registers tools (e.g., get_weather) during the session.update event.
    2. Function Call Trigger: When OpenAI requests a function call, the application receives a function_call_arguments_done message.
    3. Local Execution: The application extracts arguments from the message and executes a local method.
    4. Response: The result is sent back to OpenAI using a function_call_output message.
    5. Continuation: A follow-up message is sent to allow the AI to continue the conversation using the returned data.

    Function Calling Flow

    [User Speaks] ──► [Transcription] ──► [OpenAI Function Call Request] ──► [get_weather(location)] ──► [Local Function Execution] ──► [Return Result to AI] ──► [AI Continues Response]

  7. How WebRTC Nostr Signalling works

    master

    This pattern uses the Nostr protocol as a federated signalling channel for WebRTC, replacing the need for a custom WebSocket server. Instead of a dedicated server, peers use a public Nostr relay to exchange SDP (Session Description Protocol) and ICE candidates.

    The Signalling Flow:

    1. Connection: Both peers connect to a Nostr relay (e.g., wss://nos.lol).
    2. Subscription: Each peer subscribes to the relay using a #p filter for events tagged with its own pubkey.
    3. Exchange:
      • One peer creates a WebRTC offer and publishes it as a Nostr event of kind 25555 with a p tag pointing to the recipient's pubkey.
      • The recipient receives the event, decrypts it, and publishes an answer.
      • ICE candidates are trickled via additional kind 25555 events.
    4. P2P Media: Once the ICE pair is established, media flows directly between peers via RTP/SRTP, bypassing the relay entirely.
    |
| Layer                    | Carried by                             |
| ------------------------ | -------------------------------------- |
| Identity                 | secp256k1 keypair (Nostr pubkey)       |
| Per-peer addressing      | `p` tag on the Nostr event             |
| Discovery                | Relay subscription with `#p` filter    |
| Confidentiality          | NIP-44 v2 encryption of `event.content`| 
| Wire transport          | Any public Nostr relay (default `wss://nos.lol`) |
| Media                    | Direct WebRTC peer-to-peer (RTP / SRTP) |
|

  8. How the OpenAI WebRTC bridge works

    master

    The application acts as a bridge between a browser-based WebRTC client and OpenAI's real-time API using the following architecture:

    • Dual WebRTC Sessions: The browser connects to the ASP.NET app via WebSocket/WebRTC, and the ASP.NET app establishes a separate WebRTC session with OpenAI.
    • Audio/Data Relay: Audio is piped between the browser and OpenAI. Transcription events and other metadata are relayed from OpenAI to the browser via a WebRTC data channel.
    • Local Execution: Logic that requires external data (like weather or database lookups) is executed in the ASP.NET backend, with results sent back to OpenAI to inform the AI's response.

  9. Use the bridge verb for full-duplex connections

    master

    The bridge verb connects two duplex endpoints in both directions (full duplex). Unlike route, which is directional (--from/--to), bridge is symmetric and order-agnostic.

    Supported v0.1 endpoints include:

    • web: A browser-based microphone and speaker page.
    • agent: An Azure speech-to-text -> LLM -> Azure text-to-speech voice agent.
    • openai: The OpenAI Realtime API.
    • sip:<uri>: A phone call (transcoded G.711 <-> Opus).

  10. Configure the runtime environment for FFmpeg shared libraries

    master

    When using FFmpeg shared libraries, avoid using Microsoft dotnet images (e.g., mcr.microsoft.com/dotnet/aspnet:8.0) as base images directly, as they often lack the necessary shared library dependencies for FFmpeg.

    Instead, use an Ubuntu 24.04 (ubuntu:24.04) base image to ensure all FFmpeg dependencies are met via apt, and then install the required .NET runtime manually using the aspnetcore-runtime-8.0 package.

  11. Understand the OpenAI WebRTC SIP Gateway Workflow

    master

    The gateway acts as a bridge between traditional SIP telephony and OpenAI's WebRTC-based Realtime API. The lifecycle of a call is as follows:

    1. Listening: The application starts a SIP server on UDP port 5060.
    2. Call Reception: When a SIP call is received, the application automatically answers it.
    3. WebRTC Establishment: A WebRTC connection is established with OpenAI's real-time endpoint.
    4. Bidirectional Bridging:
      • The caller's voice is routed to OpenAI.
      • OpenAI's generated responses are routed back to the caller via the SIP call.
    5. Transcription: Conversation transcripts are logged to the console in real-time.
    6. Cleanup: When the caller hangs up, the application performs resource cleanup for both the SIP and WebRTC connections.

  12. Understand SIPSorcery.VP8 performance characteristics

    master

    When choosing between SIPSorcery.VP8 and SIPSorceryMedia.FFmpeg, consider these performance notes:

    • Encoder: Optimized for allocation hygiene (effectively zero allocations per frame after warmup). It is single-threaded and can handle 30 fps at typical webcam resolutions on a modern CPU.
    • Decoder: Currently not performance-tuned. For high-resolution (e.g., 1080p) at 30 fps, it is recommended to use the FFmpeg-based decoder instead.
    • Compatibility: Verified end-to-end against Chrome.