OpenTok Android SDK Samples

repository·main·Indexed 42 days ago

https://github.com/opentok/opentok-android-sdk-samples

A collection of Android sample applications demonstrating the OpenTok (now Vonage) Android SDK. Includes implementations for custom audio drivers using BaseAudioDevice and AudioBus, ARCore integration via CustomVideoCapturer, and managing session archiving through a backend web service.

Tokens
33.5K
Snippets
101
Records
123
Agent score
51%

What's inside OpenTok

  1. Overview of OpenTok Android SDK sample use cases

    main

    This repository contains numerous Android projects demonstrating various features of the OpenTok Android SDK. Key categories include:

    • Core Video/Audio: Basic video chat (Java/Kotlin), multiparty layouts using ConstraintLayout, and managing audio/video streams.
    • Advanced Media: Custom video/audio drivers, custom video capturers (using Camera or Camera2), and custom video renderers.
    • Specialized Features: Screen sharing (Device or WebView), ARCore integration, and End-to-End Encryption (E2EE).
    • System Integration: Phone call detection, Picture-in-Picture mode, Foreground Services for seamless UX, and Android ConnectionService integration.
    • Media Processing: Archiving (recording sessions), signaling (text messages), and Video Transformers (using Vonage Media Processor).
  2. Implement a custom video renderer using BaseVideoRenderer

    main

    The BaseVideoRenderer class allows you to intercept video frames. To capture a screenshot, you can monitor a property (like saveScreenshot) inside the displayFrame(Frame frame) method. When the property is true, you can extract the ByteBuffer from the Frame object, decode the YUV data into a Bitmap, and save it to storage.

    Key steps within displayFrame:

    • Access the frame buffer: frame.getBuffer().
    • Retrieve dimensions: frame.getWidth() and frame.getHeight().
    • Decode the YUV data to an integer array (e.g., using a helper like decodeYUV420).
    • Create a Bitmap using Bitmap.createBitmap with Bitmap.Config.ARGB_8888.
    • Save the bitmap to the device storage using Bitmap.compress.
    @Override
    public void displayFrame(Frame frame) {
        // ... frame management logic ...
    
        if (saveScreenshot) {
            ByteBuffer bb = frame.getBuffer();
            bb.clear();
    
            int width = frame.getWidth();
            int height = frame.getHeight();
            
            // ... YUV decoding and Bitmap creation logic ...
    
            Bitmap bmp = Bitmap.createBitmap(intArray, width, height, Bitmap.Config.ARGB_8888);
            
            // Save to file
            File file = new File(Environment.getExternalStorageDirectory().toString(), "opentok-capture-" + System.currentTimeMillis() + ".png");
            // ... stream and compress ...
            
            saveScreenshot = false; // Reset flag
        }
    }
  3. How the AudioBus facilitates audio data transfer

    main

    The AudioBus is the central mechanism for moving audio samples between your custom driver and the OpenTok session.

    • For Publishers: Your custom capturer thread produces audio samples and sends them to the AudioBus using writeCaptureData. The publisher then uses these samples to transmit audio in the stream.
    • For Subscribers: The AudioBus provides audio samples from subscribed streams. Your custom renderer thread retrieves these samples using readRenderData to process or play them back.

    Access the AudioBus instance via the getAudioBus() method within your audio device implementation.

    ```java
    // Sending data to the publisher
    getAudioBus().writeCaptureData(capturerBuffer, SAMPLING_RATE);
    
    // Receiving data from a subscriber
    getAudioBus().readRenderData(rendererBuffer, SAMPLING_RATE);
    ```埋
  4. How the app retrieves credentials from a server

    main

    The application uses an apiService to perform an asynchronous network call to the CHAT_SERVER_URL. Upon a successful response, it receives a GetSessionResponse object containing the apiKey, sessionId, and token, which are then passed to initializeSession() to start the OpenTok session.

    private void getSession() {
        Log.i(TAG, "getSession");
    
        Call<GetSessionResponse> call = apiService.getSession();
    
        call.enqueue(new Callback<GetSessionResponse>() {
            @Override
            public void onResponse(Call<GetSessionResponse> call, Response<GetSessionResponse> response) {
                GetSessionResponse body = response.body();
                initializeSession(body.apiKey, body.sessionId, body.token);
            }
    
            @Override
            public void onFailure(Call<GetSessionResponse> call, Throwable t) {
                throw new RuntimeException(t.getMessage());
            }
        });
    }
  5. How ConnectionService and Connection work together

    main

    A ConnectionService allows your app to manage VoIP calls by integrating with Android's Telecom APIs. This enables features like call switching, Bluetooth support, and smartwatch integration.

    Self-Managed vs System-Managed

    • Self-Managed: Your app handles the connection logic and call management independently. This is achieved by assigning the CAPABILITY_SELF_MANAGED capability to your PhoneAccount.
    • System-Managed: The Android system manages the calls. You must use CAPABILITY_CALL_PROVIDER and the user must manually enable the account in system settings. Calls will appear in the device's default call history.

    Implementation Lifecycle

    To initiate calls, use TelecomManager.placeCall() (outgoing) or TelecomManager.addNewIncomingCall() (incoming). The Telecom framework then binds to your ConnectionService, which must override:

    • onCreateOutgoingConnection(): Create a Connection for an outgoing call.
    • onCreateIncomingConnection(): Create a Connection for an incoming call.
    • onCreateOutgoingConnectionFailed(): Handle failed outgoing call attempts.
    • onCreateIncomingConnectionFailed(): Handle failed incoming call attempts (reject silently).
  6. How signaling works in OpenTok

    main

    Signaling allows clients connected to an OpenTok session to exchange text messages. You can send a signal to every client in a session or target a specific client. To use signaling, you must manage the lifecycle of the Session by setting a SessionListener and a SignalListener. Signals can only be sent while a client is actively connected to a session (after onConnected is called and before onDisconnected is called).

    session = new Session.Builder(this, OpenTokConfig.API_KEY, OpenTokConfig.SESSION_ID).build();
    session.setSessionListener(sessionListener);
    session.setSignalListener(signalListener);
    session.connect(OpenTokConfig.TOKEN);
  7. Manage view constraints with ConstraintSet

    main

    To programmatically define relationships between video views (e.g., 'View A is above View B'), use the ConstraintSet class. You define these relations using connect() and then apply the entire set to your ConstraintLayout instance. You can also use TransitionManager.beginDelayedTransition(layout) to animate the changes when constraints are applied.

    // Example of specifying that one view is above another
    set.constrainHeight(aboveViewId, 0);
    set.constrainHeight(belowViewId, 0);
    set.connect(aboveViewId, ConstraintSet.BOTTOM, belowViewId, ConstraintSet.TOP);
    set.connect(belowViewId, ConstraintSet.TOP, aboveViewId, ConstraintSet.BOTTOM);
    
    // Applying the constraints to the layout with animation support
    if (animated && Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
        TransitionManager.beginDelayedTransition(layout);
    }
    set.applyTo(layout);
  8. How custom video drivers work

    main

    A custom video driver in the OpenTok Android SDK is the combination of a custom video capturer and a custom video renderer.

    • Custom Video Capturer: Used to intercept and modify the video stream before it is published (e.g., adding effects to the camera feed).
    • Custom Video Renderer: Used to intercept and modify how video is displayed on the device for both publishers and subscribers (e.g., applying color filters to the incoming stream).

    While the default capturer and renderer work for most standard applications, implementing a custom driver is necessary when you need to add specific visual effects or custom processing to the video pipeline.

  9. How the app retrieves credentials from a web server

    main

    The application uses a Retrofit-style apiService to call the getSession() method on your configured CHAT_SERVER_URL. The server is expected to return a GetSessionResponse containing the apiKey, sessionId, and token. Once received, the app calls initializeSession() to start the OpenTok session.

    private void getSession() {
        Log.i(TAG, "getSession");
    
        Call<GetSessionResponse> call = apiService.getSession();
    
        call.enqueue(new Callback<GetSessionResponse>() {
            @Override
            public void onResponse(Call<GetSessionResponse> call, Response<GetSessionResponse> response) {
                GetSessionResponse body = response.body();
                initializeSession(body.apiKey, body.sessionId, body.token);
            }
    
            @Override
            public void onFailure(Call<GetSessionResponse> call, Throwable t) {
                throw new RuntimeException(t.getMessage());
            }
        });
    }
  10. Implement a custom video renderer by extending BaseVideoRenderer

    main

    To create a custom renderer, extend BaseVideoRenderer. This class provides the onFrame(frame: Frame) method, which is triggered whenever the SDK renders a video frame.

    Important Lifecycle Note: The Frame object passed to onFrame becomes the property of your Renderer. You are responsible for destroying the frame when it is no longer needed to prevent memory leaks. A common pattern is to destroy the previous frame when a new one arrives.

    override fun onFrame(frame: Frame) {
        renderer.displayFrame(frame)
        view.requestRender()
    }
    
    // Example of managing frame lifecycle
    fun displayFrame(frame: Frame) {
        frameLock.lock()
        currentFrame?.destroy() // Disposes previous frame
        currentFrame = frame
        frameLock.unlock()
    }
  11. Manage view relations with ConstraintSet and ConstraintSetHelper

    main

    The sample uses the ConstraintSet class to define relationships between views (e.g., 'View A is above View B'). A helper class, ConstraintSetHelper, is used to encapsulate these constraints.

    Defining relations

    To position one view above another, use set.connect() to link the bottom of the upper view to the top of the lower view. Note that you should set the height of the constrained views to 0 to allow the constraints to control their dimensions.

    Applying constraints

    After defining all constraints in the ConstraintSet, apply them to your ConstraintLayout. You can use TransitionManager.beginDelayedTransition(layout) before applying to enable smooth animations when the layout changes.

    // Example of positioning a view above another
    public void layoutViewAboveView(int aboveViewId, int belowViewId) {
        set.constrainHeight(aboveViewId, 0);
        set.constrainHeight(belowViewId, 0);
        set.connect(aboveViewId, ConstraintSet.BOTTOM, belowViewId, ConstraintSet.TOP);
        set.connect(belowViewId, ConstraintSet.TOP, aboveViewId, ConstraintSet.BOTTOM);
    }
    
    // Applying the set to the layout with animation support
    public void applyToLayout(ConstraintLayout layout, boolean animated) {
        if (animated) {
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
                TransitionManager.beginDelayedTransition(layout);
            }
        }
        set.applyTo(layout);
    }