DOCSEARCHbyAlgolia
Submit Docs

SHARP

repository·main·Indexed 27 days ago

https://github.com/apple/ml-sharp

A tool for photorealistic monocular view synthesis that regresses 3D Gaussian parameters from a single image in less than a second. It provides a metric 3D representation rendered in real-time for nearby views. The package includes a CLI for inference via the `predict` command and video rendering via the `render` command (which requires a CUDA GPU). It supports CPU, MPS, and CUDA devices and outputs 3D Gaussian Splatting (3DGS) .ply files.

Tokens
1.7K
Snippets
1
Records
18
Agent score
92%

What's inside ml-sharp

  1. Install SHARP

    main

    To set up the SHARP environment, create a new Python 3.13 environment using conda and install the required dependencies via pip.

    conda create -n sharp python=3.13
pip install -r requirements.txt

    To verify the installation, run the CLI help command:

    sharp --help

  2. Render video trajectories with SHARP

    main

    You can render videos using a camera trajectory. Note that while prediction works on CPU, CUDA, and MPS, the --render option currently requires a CUDA GPU.

    You can render directly during prediction or from previously generated intermediate gaussians.

    Note: The gsplat renderer may take time to initialize during its first launch.

  3. Configure TrajectoryParams for eye movement

    main

    Use the TrajectoryParams dataclass to define the movement pattern and constraints for eye trajectories.

    Available type options:

    • "swipe": Left to right movement.
    • "shake": Horizontal shake followed by vertical shake.
    • "rotate": Circular rotation.
    • "rotate_forward": Rotation with depth (Z-axis) oscillation.

    Other parameters:

    • lookat_mode: Either "point" (look at a fixed point) or "ahead" (look straight ahead).
    • max_disparity: Maximum lateral offset.
    • max_zoom: Maximum medial (Z) offset.
    • distance_m: The Z-plane where the eye trajectory is placed.
    • num_steps: Number of steps in the trajectory.
    • num_repeats: Number of times the pattern repeats.

  4. Reference: SHARP CLI flags and output format

    main

    CLI Flags

    • -i, --input: Path to input images.
    • -o, --output: Path to output directory.
    • -c, --checkpoint: Path to a manually downloaded model checkpoint.
    • --render: Enables video rendering via camera trajectory (requires CUDA GPU).

    Output Format

    • Files: The output consists of 3D Gaussian Splatting (3DGS) .ply files.
    • Coordinate System: Follows the OpenCV convention:
      • x: right
      • y: down
      • z: forward
    • Scene Center: The 3DGS scene center is approximately at (0, 0, +z). When using 3rd-party renderers, you may need to scale and rotate the scene to re-center it.

  5. Run SHARP predictions via CLI

    main

    Use the sharp predict command to generate 3D Gaussian splats from input images. The model checkpoint is automatically downloaded to ~/.cache/torch/hub/checkpoints/ on the first run.

    If you have manually downloaded the checkpoint, use the -c flag to specify its path.

  6. Compute CameraInfo from eye position

    main

    Use the compute method of a PinholeCameraModel instance to calculate the specific CameraInfo (intrinsics, extrinsics, width, height) for a given 3D eye position.

    This handles the transformation from the eye position to the camera's view matrix, accounting for the lookat_mode and depth focusing.

  7. Generate eye trajectories with create_eye_trajectory

    main

    The create_eye_trajectory function generates a list of camera positions (torch.Tensor) based on a specified TrajectoryParams configuration and the scene geometry.

    Arguments:

    • scene: A Gaussians3D object representing the scene.
    • params: A TrajectoryParams instance defining the movement type and scale.
    • resolution_px: A (width, height) tuple in pixels.
    • f_px: Focal length in pixels.

  10. Reference the `sharp predict` CLI flags

    main

    The following flags are available for the sharp predict command:

    FlagLong FlagDescription
    -i--input-pathRequired. Path to an image file or a directory containing images.
    -o--output-pathRequired. Directory path where predicted Gaussians (.ply) and renderings (.mp4) will be saved.
    -c--checkpoint-pathPath to a .pt checkpoint file. If omitted, the default model is downloaded automatically.
    N/A--render / --no-renderFlag to enable or disable trajectory rendering. Note: Rendering requires a CUDA device.
    N/A--deviceDevice to run on. Options: cpu, mps, cuda. Default is default (auto-detects CUDA > MPS > CPU).
    -v--verboseActivates debug-level logging.

  11. Available feature encoders for Gaussian predictor

    main

    The sharp.models.encoders package provides several encoder implementations that inherit from BaseEncoder:

    • create_vit: Factory function for Vision Transformer encoders.
    • create_monodepth_encoder: Factory function for Monodepth feature encoders.
    • UNetEncoder: A UNet-based encoder implementation.
    • SlidingPyramidNetwork: An encoder implementation based on a sliding pyramid network.
    • MonodepthFeatureEncoder: The specific class for monodepth features.