DOCSEARCHbyAlgolia
Submit Docs

Apache Hop Documentation

repository·main·Indexed 23 days ago

https://github.com/apache/hop

An orchestration platform for data and metadata orchestration. Documentation includes guides for managing the Hop plugin marketplace via local Nexus repositories, deploying Hop server and web components using Helm on Kubernetes, running Iceberg and Lakehouse (Delta) integration tests with Java 21, and executing native Spark cluster demos using Docker Compose.

Tokens
235.1K
Snippets
291
Records
1.2K
Agent score
81%

What's inside Apache Hop

  1. Overview of Apache Hop Docker Containers

    main

    Apache Hop provides an official Docker image (available on Docker Hub) that supports two primary deployment modes:

    1. Short-lived containers: Designed to execute a specific pipeline or workflow and then terminate immediately. This is ideal for scheduled batch jobs.
    2. Long-lived containers: Designed to run a Hop Server that stays active and waits for work via API or UI.

    The container runs on a minimal Alpine Linux system using OpenJDK 21. The process runs under the Linux user hop (group hop).

  2. Overview of Neo4j support in Apache Hop

    main

    Apache Hop provides extensive support for Neo4j graph databases through several integration points:

    Metadata Types

    • Neo4j Connection: Used to create or manage connections to a Neo4j database.
    • Neo4j Graph Model: Used to create or manage Neo4j (sub)graph models.

    Workflow Actions

    • Neo4j Check Connection: Validates a connection to a Neo4j database.
    • Neo4j Cypher script: Executes a Cypher script or query.
    • Neo4j index: Manages (creates or drops) Neo4j indexes.
    • Neo4j constraint: Manages (creates or drops) Neo4j constraints.

    Pipeline Transforms

    • Neo4j Cypher: Executes a Cypher query using parameter information from input fields.
    • Neo4j generate CSVs: Generates CSV files for nodes and relationships compatible with neo4j-import.
    • Neo4j Get Log Info: Queries the Neo4j logging graph for execution information.
    • Neo4j graph output: Writes to a Neo4j graph using an input fields mapping.
    • Neo4j import: Runs a Neo4j import command using provided CSV files.
    • Neo4j output: Writes nodes and/or relationships to a Neo4j graph.
    • Neo4j split graph: Splits the nodes and relationships of a Neo4j graph.

  3. Overview of Hop Gui

    main

    Hop Gui is the visual development environment for Apache Hop. It is used primarily for designing, running, and debugging workflows and pipelines. Beyond development, it serves as a central hub for:

    • Metadata Configuration: Setting up database connections, run configurations, and other metadata items.
    • Search: Finding specific metadata items via the search perspective.
    • File Management: Managing files and version control through the file explorer perspective.
    • Version Control: Integrating with Git to manage your work.

    For detailed instructions on creating and executing logic, refer to the documentation for [pipelines] and [workflows].

  4. Overview of Apache Hop Modules and Capabilities

    main

    Apache Hop is a modular data orchestration platform. The user manual is organized into several key functional areas:

    • Core Concepts & Setup: Includes getting started guides, installation, configuration, and supported Java versions.
    • Pipelines: The primary unit for data processing. Includes the Pipeline Editor, error handling, unit testing, and various execution engines like Apache Beam and Native Spark.
    • Workflows: Used for orchestration and control flow. Includes workflow actions (e.g., file operations, SQL execution, shell commands) and run configurations.
    • Projects & Metadata: Management of environments, metadata types (connections, data sets, schemas), and advanced configuration.
    • Hop Server: Provides Web Services and REST APIs for remote execution and management.
    • Hop Tools: A suite of command-line utilities including hop-run, hop-conf, hop-encrypt, and hop-import.
    • Virtual File System (VFS): Support for various storage backends like AWS S3, Azure Blob Storage, Google Cloud Storage, and more.
    • Database Connections: Extensive support for relational databases (PostgreSQL, MySQL, Oracle, etc.) and NoSQL systems (MongoDB, Cassandra, Neo4j).

  5. Use the Language model chat transform

    main

    The Language model chat transform (powered by LangChain4j) allows you to interact with various language model endpoints such as OpenAI, Anthropic, Hugging Face, Mistral, and Ollama. It provides a unified interface to integrate different models into a single pipeline and supports prompt engineering techniques to refine model responses.

    Supported Engines:

    • Single Threaded: Supported
    • Native Spark: Supported
    • Beam Spark/Flink/Dataflow: Maybe Supported

  6. Use the Neo4j import transform

    main

    The Neo4j import transform executes a neo4j-admin import command using provided CSV files to populate a Neo4j database. It supports both Neo4j 4.x and 5.x syntax, automatically adjusting the command format based on the selected version.

    Key Capabilities:

    • Version Support: Select between Neo4j 4.x and 5.x via the Neo4j version option.
    • Docker Support: Can run neo4j-admin commands inside Docker containers if configured.
    • Performance Tuning: Supports high IO settings, heap/off-heap cache management, and processor allocation.

    For complete details on the underlying command, refer to the official Neo4j documentation.

  8. Stanford NLP Transform capabilities and engine support

    main

    The Stanford NLP Transform uses Stanford CoreNLP to parse raw English text. It provides foundational building blocks for text understanding, including:

    • Word base forms (lemmatization)
    • Parts of speech tagging
    • Named entity recognition (e.g., companies, people)
    • Normalization of dates, times, and numeric quantities
    • Sentence structure markup (phrases and word dependencies)
    • Coreference resolution (identifying noun phrases that refer to the same entities)

    Engine Compatibility

    EngineSupport
    Single ThreadedSupported
    Native SparkSupported
    Beam SparkMaybe Supported
    Beam FlinkMaybe Supported
    Beam DataflowMaybe Supported

  9. Use the CSV file input transform

    main

    The CSV file input transform reads data from a delimited file. Despite its name, it supports any delimiter (e.g., pipes, tabs, semicolons).

    Key Advantages

    • NIO (Native I/O): Uses native system calls for faster performance, but is currently limited to local files only (no VFS support).
    • Parallel Running: When configured with multiple transform copies or in clustered mode, each copy reads a separate block of a single file to distribute the workload across threads or nodes.
    • Lazy Conversion: Avoids unnecessary data type conversions (e.g., converting strings to dates or numbers) if the fields are merely being passed through the pipeline to another text-based destination. This provides significant performance improvements.

    Supported Engines

    EngineSupported
    Single ThreadedYes
    Native SparkYes
    Beam SparkNo
    Beam FlinkNo
    Beam DataflowNo

  11. Use the REST Client transform to consume RESTful services

    main

    The REST Client transform allows you to interact with RESTful APIs using various HTTP methods. You can use a pre-defined REST connection to manage base URLs and authentication, or specify full URLs directly.

    Key Behaviors

    • With REST Connection: The URL provided in the transform is treated as a path relative to the connection's base URL. Headers defined in the transform's Headers tab will overwrite matching headers from the connection.
    • Without REST Connection: You must provide the full URL.
    • Pagination: If pagination is enabled and the REST connection uses a non-NONE strategy, Hop will issue multiple HTTP requests per incoming row and can split pages into multiple output rows using JsonPath or XPath.

    Supported Engines

    • Single Threaded: Supported
    • Native Spark: Supported
    • Beam (Spark, Flink, Dataflow): Maybe Supported

  12. Use the Salesforce Input transform

    main

    The Salesforce Input transform reads data directly from Salesforce using the Salesforce Web Service. It is used to pull data into a Hop pipeline for processing.

    Related transforms for modifying Salesforce data:

    • Salesforce Insert: Inserts records into Salesforce.
    • Salesforce Update: Updates existing records in Salesforce.
    • Salesforce Upsert: Updates existing records and inserts new ones.
    • Salesforce Delete: Deletes records from Salesforce.