DOCSEARCHbyAlgolia
Submit Docs

Dremio Open Source Core

repository·master·Indexed 23 days ago

https://github.com/dremio/dremio-oss

The open-source core of the Dremio data acceleration platform. This documentation covers building from source using Maven and JDKs 11, 17, and 21, deploying via Docker in single or multi-node configurations, and developing the Dremio Analyst Center (DAC) backend and React-based UI. It includes details on the DAC REST API for dataset management, the Pseudo-Distributed FileSystem (PDFS) plugin, and shared ESLint configurations for JS/TS projects.

Tokens
100K
Snippets
96
Records
337
Agent score
80%

What's inside dremio-oss

  1. Overview of UI Translations

    master

    The UI Translations module provides a centralized collection of localized strings used throughout the Dremio user interface. It includes:

    • Common translations: Standard UI elements like "Save" and "Cancel" used across all Dremio editions.
    • OSS-specific translations: Feature-specific strings used in the Open Source Software (OSS) edition, such as "Run job" and "Rename script".

  2. Overview of Dremio Common modules

    master

    Dremio Common is organized into two primary modules:

    1. core: Contains lightweight utilities designed to be safely added as a dependency to any other module within the Dremio ecosystem.
    2. legacy: Contains older code that was previously the sole module. These components are slated for migration into more appropriately scoped submodules.

  3. Overview of the Dremio Pseudo-Distributed FileSystem (PDFS) Plugin

    master

    The Dremio Pseudo-Distributed FileSystem (PDFS) plugin is a storage plugin designed to allow querying local filesystem metadata on remote Dremio node instances.

    Key Capabilities and Limitations:

    • Metadata Querying: It enables querying of metadata from the local filesystem of remote nodes.
    • File Operations: It is restricted to opening and/or creating files only on the local filesystem of the node being queried.

  4. Overview of the Embedded Catalog module

    master

    The embedded-catalog module provides a specialized implementation of a subset of Nessie gRPC APIs designed for "Embedded Nessie" use cases. It is primarily used to manage Iceberg metadata for Reflections and Infinite Splits.

    Key characteristics:

    • Storage: Catalog data is stored directly in KVStore.
    • Data Model: The standard Nessie Data Model is not used in the backend, as it is not required for these specific embedded use cases.

  9. Understand the Dremio Free Software License 1.0

    master

    The Dremio Free Software License 1.0 governs the use of free libraries, plug-ins, drivers, and other software provided by Dremio in binary code form in connection with Dremio Community.

    Key Terms

    • License Grant: Dremio grants a non-exclusive, non-transferable, and non-sublicenseable right to install and use the software solely for internal processing needs in conjunction with Dremio Community.
    • Use Restrictions: You may not:
      • Decompile, reverse engineer, or attempt to discover source code or algorithms (subject to local law and Open Source Licenses).
      • Disclose or allow access to the software to any third party.
      • Use the software for timesharing, service bureau purposes, or on an application service provider basis.
      • Remove or alter copyright, trademark, or proprietary notices.
      • Incorporate the software into, or use it to develop, a competing product or service.
    • Third Party Open Source Components: The software may contain Open Source Components. These are governed by their respective Open Source Licenses. In the event of a conflict between this Agreement and an Open Source License that the Open Source License does not permit resolving, the terms of the Open Source License will apply to that component.
    • Termination: The agreement terminates automatically upon any breach. Upon termination, you must cease use and destroy all copies of the software.
    • Limitation of Liability: Dremio's liability to the licensee is limited to a maximum of $100.

  10. Understand the Dremio Analyst Center (DAC) architecture

    master

    The Dremio Analyst Center (DAC) is the management component of Dremio. It consists of two primary parts:

    1. Java-based server: The backend management services.
    2. React-based client: The web-based user interface.

    Important: The DAC web server cannot function independently of the query engine. To run the full system (including the query engine, web server, and other required services), you must use the daemon module instead of launching this module directly.

  11. Understand the CDDL 1.1 License terms

    master

    The Common Development and Distribution License (CDDL) version 1.1 is a file-level copyleft license.

    Key obligations for users:

    • Source Code Availability: If you distribute Covered Software in Executable form, you must also make the Source Code available under the terms of this License and include a copy of the License.
    • Modifications: Modifications you create are governed by this License. You must include a notice in each modification identifying you as the contributor.
    • Larger Works: You may combine Covered Software with code not governed by this License to create a 'Larger Work' and distribute it as a single product, provided the requirements of this License are fulfilled for the Covered Software portion.
    • No Warranty: The software is provided "AS IS" without warranty of any kind.

  12. Use `bg` classes for background and text color synchronization

    master

    The .bg utility classes are designed to package a background color and a matching color (text) property into a single class. This ensures that text remains legible by always using the correct text color for a specific background.

    Additionally, these classes provide contextual CSS variables to descendants, allowing them to access associated properties like border colors or shadows that are contextually appropriate for that specific background color.