DataSphere Studio (DSS)

DataSphere Studio (DSS) is an integrated framework for managing data application development, developed by WeBank. It uses a pluggable architecture and the Linkis computing middleware to connect various data application systems.

Key capabilities include:

• Workflow-based development: A graphical drag-and-drop experience for the full data lifecycle (exchange, cleaning, analysis, quality testing, visualization, scheduling, and output).
• Pluggable components: Users can easily replace existing integrated components or add new ones.
• Enterprise-grade execution: Leverages Linkis for high concurrency, high availability, multi-tenant isolation, and resource control.
• Workspace management: Uses 'Workspace' as the management unit to organize business applications and provide user role management.

In DSS, the Workspace serves as the primary management unit. It is used to:

• Organize and manage business applications across various data application systems.
• Define common standards for collaborative development within workspaces.
• Provide user role management capabilities.

AppConn is the core application connector concept in DSS that enables the integration of various upper-layer web systems. It defines a set of unified three-level integration protocols:

1. First-level: SSO (Single Sign-On) specification.
2. Second-level: Organizational structure specification.
3. Third-level: Development process specification.

By arranging multiple AppConns in series, DSS forms workflows that support both real-time and scheduled execution. Because AppConn is integrated with Linkis, external systems benefit from shared resource management, concurrent limiting, and sharable context across the system level.

The web project is organized into a monorepo structure where core logic and individual applications reside in the packages directory. The dist directory contains the built static assets after compilation.

├─dist              # 构建后静态资源
├─docs              # 文档
├─node_modules
└─packages          # 各应用模块
    ├─apiServices
    ├─dataGovernance
    ├─dataService
    ├─dolphinScheduler
    ├─dss
    ├─scheduleCenter
    ├─scriptis
    ├─shared
    │  ├─common
    │  └─components
    ├─workflows
    └─workspace

Scriptis provides the following capabilities for data engineering and analysis:

• Script Editor: Supports multi-language development with auto-completion, syntax highlighting, and SQL syntax error-correction.
• Computation Engines: Connects to multiple engines (Spark, Hive, Python, etc.) based on Linkis.
• Runtime Functionality: Provides complete job life cycle display and intelligent diagnosis.
• Result Set Management: Supports multiple result sets, customized result set aliases, and one-click visualization.
• Database Services: Includes functionalities for database management and importing/exporting files (CSV, Excel) to/from tables.
• Context Management: Management and sharing of UDFs, custom variables, and functions.
• Console: Customized settings for engine parameters, task/engine management, and resource isolation/display.

DataSphere Studio (DSS) is designed for the following scenarios:

1. Organizations building or starting to develop big data platform capabilities but lacking data application tools.
2. Organizations with existing big data infrastructure but only a limited number of data application tools.
3. Organizations with complete data application tools that are currently siloed, causing high learning costs and a fragmented user experience.
4. Organizations with complete data application tools where some integration exists, but unified standards and specifications have not yet been defined.

The project is organized into the following directory structure:

• dist/: Static assets after build.
• node_modules/: Project dependencies.
• exts/: Extension plugins.
• packages/: Core application modules, including:
  • dss: DSS core module.
  • shared: Shared utilities and components.
  • workspace: Workspace management module.

The @dataspherestudio/exts/open-source package provides the following extended functionalities:

• Workflow Development History: A bottom TAB panel in the workflow development interface that allows for copying/accessing history.
• Scriptis Data Service API: Enables the publication of data services via the Scriptis script publishing mechanism.