sceneform-android

repository·master·Indexed 20 days ago

https://github.com/sceneview/sceneform-android

A maintained ARCore Android SDK that uses Google Filament as its 3D engine. It provides tools for rendering 3D models, managing augmented reality experiences, and handling animated glTF models, depth occlusion, cloud anchors, and environment light estimation.

Tokens
11.8K
Snippets
41
Records
54
Agent score
71%

What's inside sceneform-android

  1. How SceneView and ArSceneView work

    master

    SceneView provides two primary view types for 3D and AR experiences:

    • SceneView: Used for 3D-only experiences using the Google Filament engine.
    • ArSceneView: Used for ARCore + 3D experiences.

    Unlike previous implementations (like ArFragment), all core functionality is accessible directly at the SceneView/ArSceneView level. You no longer need to traverse through sceneFragment.sceneview.scene or sceneFragment.session.config.

    Key improvements include:

    • Automatic Handling: Camera permissions and ARCore installation/updates are handled automatically by the view.
    • Lifecycle Awareness: Components are lifecycle-aware for better memory management, and resource loading via LifecycleScope starts on view creation and is cancelled on destruction.
    • Simplified API: Properties like positionX, rotationY, and scale are directly accessible, replacing the need for manual local/world Vector3 conversions.
  2. How ModelAnimator handles morphing and bone animations

    master

    In a glTF context, ModelAnimator updates matrices according to glTF animation and skin definitions. It performs two primary roles:

    1. TransformManager: Updates matrices in components according to model animation definitions.
    2. RenderableManager: Updates bone matrices.

    Morph Targets: If a model has morph targets (e.g., facial expressions), ModelAnimator handles the tracks that define how the influence of each morph target changes over the animation clip.

    Important Note on Time Modification: When creating a PropertyValuesHolder to modify the time position of an animation, you must use ModelAnimation.TIME_POSITION instead of a custom property. This ensures that the modification can correctly cancel any other ObjectAnimator currently operating on the time of that same ModelAnimation.

  3. Configure DepthOcclusionMode for CameraStream

    master

    While DepthMode is set via the Session configuration, DepthOcclusionMode is a user-settable property on the CameraStream. It controls whether virtual objects are occluded by real-world objects based on the depth data.

    • DEPTH_OCCLUSION_DISABLED: The default value. The standard camera material is applied to the CameraStream Renderable even if the Session is using AUTOMATIC or RAW_DEPTH_ONLY depth modes. Use this if you want to access depth data (like DepthImage or DepthPoints) without the visual occlusion effect.
    • DEPTH_OCCLUSION_ENABLED: Enables the occlusion material. This requires the Session to be properly configured with either Config.DepthMode.AUTOMATIC or Config.DepthMode.RAW_DEPTH_ONLY. If the Session is not configured for depth, the standard camera material will be used instead.
    arSceneView.getCameraStream().setDepthOcclusionMode(CameraStream.DepthOcclusionMode.DEPTH_OCCLUSION_ENABLED);
  4. How morphing and bone animations work

    master

    In a glTF context, ModelAnimator updates matrices according to glTF animation and skin definitions. It handles two primary tasks:

    1. Updating matrices in TransformManager components based on model animation definitions.
    2. Updating bone matrices in RenderableManager.

    This includes animating skeleton tracks (position, rotation, or scale of bones) and morph targets (e.g., transitioning a face from 'friendly' to 'angry').

    Important: When using PropertyValuesHolder to modify animation time, you must use ModelAnimation.TIME_POSITION instead of its own property to ensure it can correctly cancel other ObjectAnimator instances operating on the same ModelAnimation.

  5. Understand DepthMode in ARCore Sessions

    master

    The DepthMode is automatically determined by the ARCore Session configuration. It defines how the depth information is processed by the session. The available modes are:

    • NO_DEPTH: The default mode. The Session is not configured to use the Depth-API.
    • DEPTH: The Session is configured to use Config.DepthMode.AUTOMATIC.
    • RAW_DEPTH: The Session is configured to use Config.DepthMode.RAW_DEPTH_ONLY.
  6. Configure AR Required vs AR Optional in Android Manifest

    master

    To ensure your app is only visible in the Google Play Store on devices that support ARCore, you must declare the app as 'AR Required'. This involves requesting camera permissions, declaring the AR hardware feature as required, and setting the ARCore metadata value to required in your AndroidManifest.xml.

    <uses-permission android:name="android.permission.CAMERA" />
    <uses-feature android:name="android.hardware.camera.ar" android:required="true"/>
    
    <application>
        ...
        <meta-data android:name="com.google.ar.core" android:value="required" />
    </application>
  7. Convert SFA and SFB assets to glTF

    master
    Starting with version 1.16.0, support for proprietary .SFA and .SFB file formats has been removed in favor of the open glTF standard. To maintain compatibility, you must convert your existing assets to glTF files. To load these new assets, use the glTF loading patterns (refer to GltfActivity in the samples for implementation details).
  8. Enable Depth Occlusion in AR

    master

    Depth occlusion allows virtual objects to be occluded by real-world objects. To enable it, first check if the session supports Config.DepthMode.AUTOMATIC via setOnSessionConfigurationListener. Then, set the depthOcclusionMode on the arSceneView.cameraStream to CameraStream.DepthOcclusionMode.DEPTH_OCCLUSION_ENABLED within the setOnViewCreatedListener.

    arFragment.apply {
        setOnSessionConfigurationListener { session, config ->
            if (session.isDepthModeSupported(Config.DepthMode.AUTOMATIC)) {
                config.depthMode = Config.DepthMode.AUTOMATIC
            }
        }
        setOnViewCreatedListener { arSceneView ->
            // Available modes: DEPTH_OCCLUSION_DISABLED, DEPTH_OCCLUSION_ENABLED
            arSceneView.cameraStream.depthOcclusionMode =
                CameraStream.DepthOcclusionMode.DEPTH_OCCLUSION_ENABLED
        }
    }
  9. Configure AndroidManifest.xml for ARCore and Camera access

    master

    To use Sceneform, you must update your AndroidManifest.xml to request camera permissions and declare ARCore requirements.

    Depending on whether AR is a core requirement of your app or an optional feature, you have two configuration paths:

    1. AR Required (App requires ARCore to function)

    Use this if your app's primary purpose is AR. This ensures the app is only visible in the Google Play Store on ARCore-supported devices and triggers the automatic installation of Google Play Services for AR.

    2. AR Optional (App can function without AR)

    Use this if AR is just one of many features. Remove the <uses-feature android:name="android.hardware.camera.ar" /> line and change the com.google.ar.core meta-data value to optional.

    Note: Both configurations require the android.permission.CAMERA permission and OpenGL ES 3.0 support.

    <!-- Both "AR Optional" and "AR Required" apps require CAMERA permission. -->
    <uses-permission android:name="android.permission.CAMERA" />
    
    <!-- Sceneform requires OpenGL ES 3.0 or later. -->
    <uses-feature android:glEsVersion="0x00030000" android:required="true" />
    
    <!-- Indicates that app requires ARCore ("AR Required"). Ensures the app is
         visible only in the Google Play Store on devices that support ARCore.
         For "AR Optional" apps remove this line. -->
    <uses-feature android:name="android.hardware.camera.ar" />
    
    <application>
        …
        <!-- Indicates that app requires ARCore ("AR Required"). Causes the Google
             Play Store to download and install Google Play Services for AR
             along with the app. For an "AR Optional" app, specify "optional" instead of
             "required".
        -->
        <meta-data android:name="com.google.ar.core" android:value="required" />
    </application>