scaleph

Scaleph is a one-stop data platform designed for cloud environments, focusing on simplifying the development of data applications. It provides capabilities across data integration, development, and warehousing.

Core Capabilities

• Data Integration: Features a Web-UI with click-and-drag capabilities for data integration, powered by Apache SeaTunnel on a Flink engine. It supports the latest 2.3.8 V2 connectors and transforms out-of-the-box and includes DataSource management.
• Data Development:
  • Jar: Users can develop Flink jobs using the DataStream or Table API and package them as JAR files.
  • SQL: Provides an online Flink SQL editor backed by the Flink SQL Gateway.
• Data Warehouse: Supports Doris cluster management on Kubernetes via the doris-operator.
• Orchestration & Management: Leverages Kubernetes to manage Flink (via the Flink Kubernetes Operator) and Doris (via the doris-operator).

After starting the cluster, you can access the Hadoop web interfaces.

Note for macOS users: Direct access via Docker IP addresses may not work due to networking limitations. See this issue for details.

Method 1: Using Docker IP Addresses

First, find the network name and inspect it to get the container IP addresses:

docker network list
docker inspect network hadoop_hadoop

Then use the following URLs (replace <dockerhadoop_IP_address> with the IPv4 address found via docker inspect):

• Namenode: http://<dockerhadoop_IP_address>:9870/dfshealth.html#tab-overview
• History server: http://<dockerhadoop_IP_address>:8188/applicationhistory
• Datanode: http://<dockerhadoop_IP_address>:9864/
• Nodemanager: http://<dockerhadoop_IP_address>:8042/node
• Resource manager: http://<dockerhadoop_IP_address>:8088/

Method 2: Using Local Hosts File (Recommended for macOS/Local Access)

To access the cluster via localhost, you must map the container IDs to your local hosts file:

1. Run docker ps -a to find the container IDs for namenode, datanode, and resourcemanager.
2. Add an entry to your local /etc/hosts file mapping 127.0.0.1 to localhost and the specific container IDs.

Example hosts entry:

127.0.0.1 localhost namenode datanode nodemanager resourcemanager historyserver <namenode_id> <datanode_id> <nodemanager_id> <resourcemanager_id> <historyserver_id>

Once configured, use these URLs:

• Namenode: http://localhost:9870/dfshealth.html#tab-overview
• History server: http://localhost:8188/applicationhistory
• Datanode: http://localhost:9864/
• Nodemanager: http://localhost:8042/node
• Resource manager: http://localhost:8088/

To successfully deploy a Flink job on the Hadoop cluster, you must ensure proper hostname resolution and client configuration.

Configuration Requirements

Set the following client configuration parameters:

• dfs.client.use.datanode.hostname = true
• dfs.datanode.use.datanode.hostname = true

Hostname Resolution Strategies

Option A: Local Hosts File Mapping Map the container IDs (found via docker ps -a) to hostnames in your local /etc/hosts file.

Option B: Custom Hostnames in docker-compose.yaml To avoid using long container IDs, you can explicitly set hostnames in your docker-compose.yaml:

namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    hostname: namenode
...
datanode:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    hostname: datanode

If you use Option B, you must still add the following to your local hosts file: ${host ip} datanode namenode nodemanager resourcemanager historyserver

Warning: DNS Resolve Exceptions

Be aware that Flink clients may attempt to resolve hostnames using container IDs (e.g., 1e3974bf4bd8) retrieved from the deployed cluster. If these IDs are not in your hosts file, you will encounter a java.net.UnknownHostException.

The DataIntegrationSeaTunnelWeb component provides a user interface for managing SeaTunnel artifacts within a project workspace. It allows users to list, create, edit, and delete SeaTunnel configurations, and navigate to the DAG (Directed Acyclic Graph) view for a specific artifact.

Key Capabilities

• List Artifacts: Displays a table of SeaTunnel artifacts including engine type, Flink version, SeaTunnel version, and metadata.
• Create/Edit: Uses the DataIntegrationSeaTunnelForm component to handle the creation and modification of artifact data.
• Delete: Removes an artifact via WsArtifactSeaTunnelService.deleteArtifact after user confirmation.
• DAG Navigation: Navigates to the DAG definition page for a specific record using history.push("/workspace/data-integration/seatunnel/dag", record).

Permissions Required

Access to specific actions is controlled via useAccess and the following privilege codes:

• PRIVILEGE_CODE.datadevProjectEdit: Allows editing existing artifacts.
• PRIVILEGE_CODE.datadevJobEdit: Allows navigating to the DAG definition.
• PRIVILEGE_CODE.datadevDatasourceDelete: Allows deleting artifacts.
• PRIVILEGE_CODE.datadevResourceAdd: Allows creating new artifacts.

// This component is the default export for the SeaTunnel Data Integration page.
import DataIntegrationSeaTunnelWeb from './index';
// Usage is typically handled by the router in the application.