DOCSEARCHbyAlgolia
Submit Docs

Flutter Scene

repository·master·Indexed 19 days ago

https://github.com/bdero/flutter_scene

A realtime 3D engine for high-performance 3D applications that leverages Flutter GPU and Impeller. It provides rendering, physics, audio, and a full asset pipeline within the Flutter ecosystem. The project includes physics backends like flutter_scene_box3d and flutter_scene_rapier_native, as well as audio integration via flutter_scene_fmod.

Tokens
30.8K
Snippets
83
Records
126
Agent score
64%

What's inside flutter_scene

  1. Overview of Flutter Scene features

    master

    Flutter Scene provides a comprehensive toolkit for 3D development in Flutter, including:

    Rendering

    • PBR Materials: Physically based materials with image-based lighting (IBL) and a procedural studio environment.
    • Lighting: Directional, point, and spot lights with shadow casting (cached shadow tiles for static geometry).
    • Post-processing: HDR tone mapping, bloom, fog, god rays, screen-space reflections, depth of field, and anti-aliasing.
    • Advanced Rendering: 3D Gaussian splatting (.ply and .splat), instanced rendering, and automatic geometry LODs.

    Materials and Shaders

    • Custom Materials: .fmat workflow for fragment and vertex stages with shader hot reload.
    • Shader Inputs: Access to per-frame scene data like depth and shadow data.

    Assets and Animation

    • Importing: Runtime glTF (.glb) import or build-time conversion to .fsceneb.
    • Animation: Skinned meshes and a blended animation system with declarative per-clip control.
    • Textures: KTX2 compressed textures and HDR/EXR environment decoding.

    App Integration

    • SceneView Widget: Offers both an imperative scene-graph API and a declarative widget API (SceneNode, SceneMesh, SceneModel).
    • Interactivity: Embed Flutter widgets on 3D surfaces with pointer raycasting into the scene.

  2. Overview of flutter_scene_net

    master

    flutter_scene_net provides multiplayer capabilities for flutter_scene using dashwire. It binds replicated state to the scene graph, allowing networked entities to be represented as scene nodes that render smoothly.

    Key components:

    • SceneReplication: Maps replicas to scene nodes using per-type builders. It automatically spawns and despawns nodes based on server instructions.
    • TransformReplica: Replicas extending this class receive a NetworkTransformComponent. This component uses an interpolation buffer to render remote poses with a fixed delay, ensuring smooth motion even under network jitter.
    • SceneHost: Enables an app to run a dashwire room and WebSocket listener internally. This allows a device to host a game and join via loopback without requiring a separate server process.

    Note: SceneHost is native-only; it provides a throwing stub on the web.

  3. Overview of the scene package

    master

    The scene package provides an engine-agnostic scene document core. It defines the .fscene (JSON) and .fsceneb (binary) document models used to represent a tree of node specifications.

    Key capabilities include:

    • Managing a tree of node specs with stable IDs and typed component properties.
    • Handling resources and payloads.
    • Coordinator-free ID allocation.
    • Prefab composition with overrides.
    • Structural diffing using stable IDs.
    • Parsing and writing .fscene (JSON) and .fsceneb (binary) formats.

    This package is written in pure Dart and has no dependency on Flutter, making it suitable for editors, asset pipelines, and servers running under dart run. Note that this package does not handle rendering; rendering (turning a document into live nodes) is the responsibility of the flutter_scene package.

    import 'package:scene/scene.dart';

final document = SceneDocument();
final id = document.allocator.mint();
// ... build specs, then:
final text = writeFscene(document);
final reread = readFscene(text);

  4. Understand the flutter_scene ecosystem and packages

    master

    The flutter_scene project is organized as a pub workspace. Depending on your needs (physics, audio, or editing), you may need to include specific companion packages alongside the core engine.

    Core Engine

    • flutter_scene: The primary 3D engine. It includes the glTF importer, the .fscene format, and the web (WebGL2) backend. This is the main package published to pub.dev.

    Physics Backends

    • flutter_scene_rapier: Uses the Rapier physics engine. Ships prebuilt native binaries and a WASM module. Published to pub.dev.
    • flutter_scene_box3d: Uses the box3d physics engine. Published to pub.dev.

    Audio Backends

    • flutter_scene_soloud: Uses the SoLoud audio engine (not yet published).
    • flutter_scene_fmod: Uses the FMOD Studio audio engine (not yet published).

    Editor Stack

    • flutter_scene_editor_core, flutter_scene_editor, and flutter_scene_mcp: The components for the scene editor (headless core, Flutter UI, and MCP tool surface). Currently in development and not yet published.

  5. Understand the third-party dependencies in flutter_scene_rapier

    master

    flutter_scene_rapier includes precompiled native libraries and a WebAssembly module. These artifacts statically link several third-party Rust crates, primarily the Dimforge Rapier physics engine.

    Key components include:

    • rapier3d, parry3d: Rigid-body solver and collision/geometry library (Dimforge), licensed under Apache License 2.0.
    • nalgebra, simba, approx, glam: Linear algebra libraries, licensed under Apache License 2.0 and/or MIT License.

    Because these are statically linked, the licenses for these crates are bundled within the distributed artifacts.

  6. Choose between .fmat and ShaderMaterial

    master

    flutter_scene provides two ways to implement custom materials:

    1. The .fmat declarative format (Recommended): You declare parameters in a metadata block and implement a Surface() function in GLSL. A build hook handles compilation, providing typed, name-addressed parameters without manual std140 packing. It also provides built-in physically based lighting (PBR) for lit materials. This is the preferred path for most use cases.

    2. ShaderMaterial (Low-level escape hatch): You write raw GLSL for fragment (and optionally vertex) stages. You must manually declare uniform blocks and samplers and handle std140 packing when binding from Dart. Use this for full control or shader shapes not yet supported by .fmat.

  7. Configure WidgetUpdatePolicy for performance

    master

    The WidgetUpdatePolicy determines how often the widget is re-recorded into a texture. Captures are asynchronous and throttled to one in flight.

    • WidgetUpdatePolicy.everyFrame (Default): Re-records the widget every frame. This is necessary to observe all changes (animations, scrolls, etc.). It reuses retained layers to minimize cost.
    • WidgetUpdatePolicy.interval(duration): Throttles the capture cadence to a specific duration.
    • WidgetUpdatePolicy.manual: Captures only when controller.requestCapture() is explicitly called. Use this for static panels to save resources.

  8. Choose the insertion point for a custom PostEffect

    master

    The PostEffect.insertion property determines when your custom shader runs in the rendering pipeline:

    • PostInsertion.beforeTonemap (Default): Runs on linear HDR scene color. The output must be linear HDR premultiplied by alpha. This is the correct slot for effects that require high dynamic range (e.g., bloom-like effects).
    • PostInsertion.afterTonemap: Runs on the display-referred image after tone mapping. The output is a display color.

  9. Use backend-agnostic audio features with FMOD

    master

    The FmodAudioEngine implements the standard flutter_scene audio contract. This means you can use generic audio components that route through FMOD Core channels:

    • ClipAudioSource: Use this for standard audio clips.
    • AudioEngine.playOneShot: Use this to trigger one-off sounds.
    • createBus: Use this to create core channel groups.

    To access FMOD Studio-specific features like Studio buses, use engine.studioBus('bus:/NAME').

  11. Understand the shader output contract

    master

    When writing raw ShaderMaterial (instead of .fmat), you must adhere to the engine's vertex and fragment output contracts.

    Vertex Outputs (World Space):

    • in vec3 v_position: world space position.
    • in vec3 v_normal: world space normal (not necessarily unit length).
    • in vec3 v_viewvector: camera_position - vertex_position.
    • in vec2 v_texture_coords: UV coordinates.
    • in vec4 v_color: per-vertex color (white if none).

    Fragment Output:

    • out vec4 frag_color: The output must be linear radiance and premultiplied by alpha (RGB * A).
    • Do not apply tone-mapping or gamma encoding in the shader; the engine handles this in a post-process pass.
    • When sampling sRGB textures, you must linearize them (e.g., pow(color, vec3(2.2)) or SRGBToLinear).

  12. Follow std140 packing rules for ShaderMaterial

    master

    When using ShaderMaterial, you must manually fill byte buffers for uniform blocks following the std140 layout. Incorrect packing is a common cause of visual bugs (e.g., black geometry).

    TypeSizeAlignmentNotes
    bool / int / float44
    vec288
    vec31216pads to 16
    vec41616
    mat46416four vec4 columns
    array elementvaries16each element strides to a 16-byte boundary

    Best Practice: To avoid alignment errors, declare blocks using vec4s and group trailing scalars into vec4-aligned rows of four.