DOCSEARCHbyAlgolia
Submit Docs

Apache StreamPark Documentation

repository·dev·Indexed 26 days ago

https://github.com/apache/streampark

A development framework and cloud-native platform designed to simplify the end-to-end lifecycle of real-time streaming applications for Apache Flink and Apache Spark. It provides unified batch and streaming support, multi-engine and multi-version management, and compatibility with Standalone, YARN, and Kubernetes environments.

Tokens
8.6K
Snippets
22
Records
87
Agent score
87%

What's inside Apache StreamPark

  1. Overview of Apache StreamPark

    dev

    Apache StreamPark is a streaming application development framework and cloud-native real-time computing platform. It simplifies the end-to-end lifecycle of stream processing for Apache Flink and Apache Spark.

    Key capabilities include:

    • Unified Batch & Streaming: Supports both Apache Flink and Apache Spark.
    • Multi-Engine/Version Support: Manage multiple versions of Flink and Spark.
    • Multi-Environment Compatibility: Works on Standalone, YARN (Hadoop 2.x/3.x), and Kubernetes.
    • Cloud-Native Platform: Provides a one-stop platform for development, deployment, monitoring, and operations.

  3. Understand JDK requirements for StreamPark Console and Jobs

    dev

    StreamPark 3.0 requires JDK 11 or later to run the Console process. Note that the Console JDK and the Job (Flink/Spark) JDK are independent; upgrading the Console to JDK 11 does not require upgrading your Flink or Spark cluster JDK.

    ComponentJDK requirementConfiguration
    StreamPark ConsoleJDK 11+JAVA_HOME in streampark-env.sh or system environment
    Flink jobsDepends on Flink version$FLINK_HOME/conf/flink-env.sh → JAVA_HOME
    Spark jobsDepends on Spark version$SPARK_HOME/conf/spark-env.sh → JAVA_HOME

  5. Initialize StreamPark database with MySQL or PostgreSQL

    dev

    StreamPark supports MySQL and PostgreSQL. To initialize the database, you must execute the schema script first, followed by the data script. The scripts are located in the schema and data directories respectively.

    For MySQL:

    1. Execute mysql-schema.sql to create the schema.
    2. Execute mysql-data.sql to initialize the data.

    For PostgreSQL:

    1. Execute pgsql-schema.sql to create the schema.
    2. Execute pgsql-data.sql to initialize the data.

  8. Build the PyFlink virtual environment using Docker

    dev

    You can automate the creation of the venv.zip file using a Docker container (specifically quay.io/pypa/manylinux2014_x86_64) to ensure a consistent build environment. This requires a setup-pyflink-virtual-env.sh script and a build.sh script located in a /build directory.

    # 1. Create /build directory and place setup-pyflink-virtual-env.sh and build.sh inside.

# 2. build.sh content:
#!/bin/bash
set -e -x
yum install -y zip wget

cd /root/
bash /build/setup-pyflink-virtual-env.sh
mv venv.zip /build/

# 3. Execute the build command:
docker run -it --rm -v $PWD:/build  -w /build quay.io/pypa/manylinux2014_x86_64 ./build.sh

  10. Deploy PyFlink virtual environment and dependencies

    dev

    After generating venv.zip, follow these steps to deploy it to your Flink environment:

    1. Upload to HDFS: Upload the zip file to your HDFS path.
    2. Copy to Workspace: Copy the zip file to your $WORKSPACE/python directory.
    3. Install Flink Python Dependencies: Copy the Python dependencies and libraries to $FLINK_HOME/lib.
    4. Connectors: If your PyFlink job requires a Flink connector dependency, ensure it is also placed in $FLINK_HOME/lib.

  11. Set up the Test Environment with Testcontainers

    dev

    The testing framework uses testcontainers and Docker Compose to manage the environment.

    To automate environment setup, use the @StreamPark annotation on your test class and provide the path to the required docker-compose.yaml file via the composeFiles parameter.

    To interact with the browser, declare a field of type RemoteWebDriver in your test class; the framework will automatically inject the driver instance.

    @StreamPark(composeFiles = "docker/basic/docker-compose.yaml")
class UserManagementTest {
    private RemoteWebDriver browser;

    @Test
    @Order(10)
    void testCreateUser() {
        final UserManagementPage userManagementPage = new UserManagementPage(browser);
        userManagementPage.createUser(newUserName, "test", password, newUserEmail, UserManagementUserType.ADMIN);

        Awaitility.await().untilAsserted(() -> assertThat(userManagementPage.userList())
            .as("User list should contain newly-created user")
            .extracting(WebElement::getText)
            .anyMatch(it -> it.contains(newUserName)));
    }
}

  12. Implement Page Object Model for StreamPark E2E Tests

    dev

    StreamPark End-to-End tests use the Page Object Model (POM) design pattern. To maintain tests, abstract each StreamPark page into a class. Use Selenium's @FindBy annotation to locate elements by id, className, css, tagName, or xpath.

    To initialize these elements, pass a RemoteWebDriver into the class constructor and call PageFactory.initElements(driver, this). For better maintainability, prefer using unique id or class attributes on UI elements over complex XPath or CSS selectors.

    public final class LoginPage {
    @FindBy(id = "form_item_account")
    private WebElement inputUsername;

    @FindBy(id = "form_item_password")
    private WebElement inputPassword;

    @FindBy(xpath = "//button[contains(@classnames, 'login-button')]")
    private WebElement buttonLogin;

    public LoginPage(RemoteWebDriver driver) {
        this.driver = driver;
        PageFactory.initElements(driver, this);
    }
}