loaders.gl

loaders.gl is a collection of open source loaders and writers for various file formats, primarily focused on supporting visualization and analytics of big data. It provides composable modules with consistent APIs for tabular, geospatial, and 3D file formats.

Key features include:

• Framework Agnostic: Parses files into plain data structures (objects + typed arrays) usable with any JS framework.
• Streaming Support: Supports incremental parsing for

import {load} from '@loaders.gl/core';
import {CSVLoader} from '@loaders.gl/csv';

const data = await load('data.csv', CSVLoader);

for (const row of data) {
  console.log(JSON.stringify(row)); // => '{header1: value1, header2: value2}'
}

loaders.gl provides support for both Apache Parquet and GeoParquet formats. These are binary formats used to store data as Tables or Geometry Tables.

Key characteristics:

• File Extension: .parquet
• MIME Type: application/octet-stream
• Streaming: Supported
• Worker Thread: Not supported
• Loader APIs: load, parse, parseInBatches
• Source APIs: createDataSource, getMetadata, getSchema, read, close
• Writer APIs: encode, encodeSync

The @loaders.gl/math library provides classes and utilities designed for working with 3D geometries (vertex arrays) stored in typed arrays. It follows WebGL/OpenGL layout rules to facilitate efficient processing of vertex data.

Note: This library is currently being developed to support 3D tiles and is planned to be moved to the math.gl repository once it stabilizes.

The @loaders.gl/zip module provides utilities for handling compressed archives. It supports:

• ZIP format decompression and compression.
• TAR format decompression and compression.

Note that ZipLoader is a wrapper around the JSZip module, meaning it inherits JSZip's capabilities, though it may not expose all of JSZip's advanced options directly.

The @loaders.gl/terrain module is designed to reconstruct mesh surfaces from two primary source formats:

1. Height map images: Formats like Mapzen Terrain Tiles that encode elevation data into R,G,B pixel values.
2. Quantized mesh: The [quantized-mesh][quantized_mesh] format.

Reconstructed data is typically output as a Mesh or a Mesh Arrow table.

loaders.gl organizes loaders into categories that return parsed data in standardized forms. This allows applications to handle multiple similar file formats easily.

• Table Loaders: Streaming tabular loaders for CSV, JSON, Parquet, Arrow, etc.
• Geospatial Loaders: Formats like GeoJSON, KML, WKT/WKB, Mapbox Vector Tiles.
• Image Loaders: Standard images and utilities like parsed/binary image APIs.
• Texture Loaders: Compressed textures, Basis textures, and manifest-driven texture loaders.
• Pointcloud and Mesh Loaders: Draco, LAS, PCD, PLY, OBJ, and Terrain.
• Scenegraph Loaders: glTF and OpenUSD.
• Tiled Data Loaders: 3D Tiles, I3S, and potree.

The @loaders.gl/parquet module is an experimental loader and writer for the Apache Parquet format. It provides multiple ways to interact with Parquet data depending on your performance and data structure needs:

• WASM-backed plain-row APIs: Use ParquetLoader and ParquetWriter for high-performance operations.
• Selective Arrow reads: Use ParquetSourceLoader for reusable, selective Arrow reads that utilize cached schema and footer metadata.
• ParquetJS APIs: Use ParquetJSLoader and ParquetJSWriter for experimental plain-row and plain-table APIs based on parquetjs.

Data Formats Support:

• ParquetLoader can output Arrow tables by setting the option parquet.shape: 'arrow-table'.
• ParquetWriter accepts loaders.gl Arrow tables as input.

The Apache Arrow JS library provides a JavaScript API designed for working with binary columnar data in the Apache Arrow format. It is used within loaders.gl to handle high-performance data loading. The API provides core classes for:

• Batched loading and writing: Efficiently handling large datasets in chunks.
• Column and row access: Navigating data structures.
• Schema management: Defining and interacting with data types and structures.

The @loaders.gl/graphs module provides parsers for various graph data formats. It is designed to handle graph-based serialization formats and convert them into usable data structures within the loaders.gl ecosystem.

Currently, the module supports:

• DOT: A graph description language used for visualizing structural information.
• GraphML: An XML-based format for expressing graph structures.

Future updates will include additional graph serialization formats as they are added to the module.