DOCSEARCHbyAlgolia
Submit Docs

Apache Zeppelin Documentation

repository·master·Indexed 27 days ago

https://github.com/apache/zeppelin

A web-based notebook for interactive data analytics and collaborative data-driven documents. It features built-in Apache Spark integration and supports multiple languages and interpreters, including SQL, Scala, Python, Java, Groovy, MongoDB, InfluxDB 2.0, JDBC, Livy, and BigQuery.

Tokens
99.1K
Snippets
239
Records
608
Agent score
92%

What's inside Apache Zeppelin

  1. Overview of Apache Zeppelin

    master
    Apache Zeppelin is a web-based notebook designed for interactive data analytics. It allows users to create collaborative, data-driven documents using various languages including SQL and Scala. Key features include a web-based notebook editor and built-in support for Apache Spark.

  2. Overview of Zeppelin Visualization Transformations

    master

    Zeppelin provides four types of transformations to handle how data is rendered and transformed for visualizations. Transformations allow you to set columns and transform table rows according to configured columns.

    1. PassthroughTransformation: Does not convert the original table data.
    2. ColumnselectorTransformation: Used when you need multiple axes but do not require aggregation.
    3. PivotTransformation: Provides grouping and aggregation using three axes: Keys, Groups, and Values.
    4. AdvancedTransformation: A highly configurable transformation supporting multiple sub-charts, configurable axes, parameter widgets, and multiple transformation methods.

  4. Overview of the Spark Interpreter

    master
    The Spark interpreter is the primary interpreter in Apache Zeppelin. It provides support for multiple versions of Apache Spark and multiple versions of Scala. It is responsible for creating the SparkContext and SparkSession.

  5. Understand the Flink Interpreter project structure

    master

    The Flink interpreter is organized into modules to handle different Flink and Scala versions. It is divided into two main groups:

    1. Flink Shims: Used to adapt to different Flink versions due to API changes. flink-shims acts as the parent module. At runtime, the interpreter loads specific shims using FlinkShims#loadShims based on the detected Flink version.

      • flink-shims (Parent)
      • flink1.20-shims (Shared by Flink 1.19 and 1.20)

    2. Scala Adapters: Used to adapt to different Scala versions. Since Flink 1.15, only Scala 2.12 is supported.

      • flink-scala-2.12

  6. Use the MongoDB interpreter in Apache Zeppelin

    master
    The MongoDB interpreter allows you to execute MongoDB scripts within Apache Zeppelin notebooks. It utilizes the mongo shell to execute scripts. To use it, you must configure the MongoDB interpreter in your Zeppelin instance and ensure you are familiar with MongoDB aggregate functions for data manipulation.

  7. Understand Helium plugin types

    master

    Helium is a plugin system used to extend Apache Zeppelin. It supports four primary types of packages:

    • Helium Visualization: Used for adding new chart types.
    • Helium Spell: Used for adding new interpreters or display systems that run in the browser.
    • Helium Application: Used for adding full applications.
    • Helium Interpreter: Used for adding new custom interpreters.

  8. Understand EmbeddingSearch indexing and capabilities

    master

    Unlike the standard LuceneSearch, EmbeddingSearch uses ONNX-based sentence embeddings (all-MiniLM-L6-v2) to provide semantic understanding. This allows users to find notebooks using natural language queries (e.g., searching for "yesterday's spending" can find SQL queries using current_date - 1).

    Indexed Content:

    • Paragraph text: Full text of the paragraph.
    • Paragraph title: The title of the paragraph.
    • Notebook name: Included in the embedding context.
    • Paragraph output: Both TABLE and TEXT outputs are indexed.
    • SQL table names: Extracted from FROM and JOIN clauses and used for result boosting.
    • Interpreter prefix: Stripped from the text before indexing.

    Technical Details:

    • Model: all-MiniLM-L6-v2 (384-dimensional embeddings).
    • Model Storage: The 86MB ONNX model is automatically downloaded to zeppelin.search.index.path/models/ on first use.
    • Performance: Brute-force cosine similarity is used, providing < 50ms search time for up to 50,000 paragraphs.

  9. Understand the Apache Zeppelin Security Model

    master

    Apache Zeppelin is a web-based, multi-user notebook server designed for interactive data analytics. Its primary function is to execute user-supplied code (via pluggable interpreters like Spark, Flink, Python, etc.) on a server.

    Key Security Concept: The security model is not about preventing code execution (which is the product's core purpose), but about controlling who is allowed to reach that capability and providing isolation between users and processes.

    Deployment Shape:

    • A long-running JVM server (zeppelin-server).
    • One or more interpreter processes.
    • Communication via HTTP/Websockets (browser to server) and Thrift IPC (server to interpreter).
    • Often deployed behind a reverse proxy like NGINX.

  10. Understand Zeppelin's Security Model (HTTP and WebSocket)

    master

    Zeppelin secures its web application and remote interpreter communication using Apache Shiro. Security is applied across two main channels:

    HTTP Endpoint Security

    Apache Shiro operates as a servlet filter between the browser and exposed services, handling authentication automatically.

    WebSocket Security

    Because Zeppelin uses WebSockets for communication, a ticket-based mechanism is used to maintain security:

    1. Ticket Request: On startup, the browser requests a ticket via an HTTP request.
    2. Authentication: The Apache Shiro servlet filter authenticates the user.
    3. Ticket Assignment: Once authenticated, a ticket is assigned to the user and returned to the browser.
    4. Message Validation: Every WebSocket message must include both the username and the ticket. The server validates that the provided ticket matches the one assigned to that username during the initial HTTP request.

  11. Use the Groovy Interpreter

    master
    Apache Zeppelin supports a Groovy interpreter, allowing you to execute Groovy code within notebooks. For detailed configuration, usage patterns, and interpreter-specific settings, refer to the official Groovy interpreter documentation.

  12. Use Zeppelin-Context for data exchange and utilities

    master
    The zeppelin-context (accessible via the predefined variable z in many programming-language interpreters like Spark and Python) is a system-wide container for user-specific data and utility functions. It allows for data exchange between different cells and even different notebooks by storing and retrieving objects.