XL-LightHouse Documentation

repository·master·Indexed 18 days ago

https://github.com/xl-xueling/xl-lighthouse

A real-time business monitoring system for massive-scale streaming data statistics, capable of handling up to 10M metrics and 500B messages per day. It provides a full pipeline from data ingestion and complex stream computation to multi-dimensional visualization. The system supports statistical operations (count, sum, avg, topN, etc.), multi-dimensional analysis, and offers both a Big Data version and a lightweight Single-Machine Version for smaller enterprises.

Tokens
101.8K
Snippets
258
Records
330
Agent score
63%

What's inside XL-LightHouse

  1. Overview of XL-LightHouse

    master

    XL-LightHouse is a next-generation panoramic real-time business monitoring system designed for complex data statistics. It integrates data writing, data computation, and data visualization to handle massive data volumes and high concurrency.

    Key Capabilities:

    • Statistical Operations: Supports count, sum, max, min, avg, distinct, topN, and lastN.
    • Multi-dimensional Analysis: Supports multi-dimensional calculations and various time granularities (minute, hour, day) with customizable statistical cycles.
    • Complex Logic: Includes built-in transformation functions and expression parsing for complex conditional filtering and logical judgments.
    • Data Management: Provides API query interfaces, metric management, permission management, and statistical rate limiting.
    • Scalability: Supports time-series data storage and querying. It can handle up to 10 million data metrics and 500 billion messages per day.
  2. Understand the licensing for XL-LightHouse Open Source vs Commercial

    master

    XL-LightHouse is available in two versions with different usage terms:

    Open Source Version

    • License: Based on Apache 2.0 (this agreement takes precedence in case of conflict).
    • Usage: Unrestricted for internal use within enterprises or organizations.
    • Restrictions: You must not remove copyright notices, project logos, original author email addresses, or the project website from the program.
    • Cost: Free to use as long as you are not selling related products or services.

    Commercial Version

    • License: Closed-source.
    • Restrictions: Unauthorized cracking, distribution, or use is strictly prohibited and subject to legal action.

    When to pay a licensing fee (0.2%~2%)

    You are required to pay a licensing fee to the Developer if you sell products or services that:

    1. Directly or indirectly include more than 1000 lines of XL-LightHouse source code.
    2. Are based on XL-LightHouse design methods (system design, implementation plans, or UI/UX design).
    3. Incorporate the XL-Formula design standard (including modified versions).
    4. Rely on XL-LightHouse or XL-Formula for data visualization (e.g., plugins, terminals, or other visualization services).
  3. Review legal liabilities and disclaimers

    master

    By installing and using XL-LightHouse, you agree to the following legal terms:

    • User Responsibility: Users are solely responsible for ensuring their use of the system complies with local and international laws, especially regarding cross-border data transfer and privacy.
    • Indemnification: If illegal use leads to third-party claims, the user must bear all responsibility and compensate the Developer for losses.
    • Developer Disclaimers:
      • The Developer is not liable for data leakage, data loss, or any direct/indirect losses caused by system issues or defects.
      • The Developer is not liable for losses resulting from the user's violation of laws.
    • Data Security: While the Developer implements reasonable security measures and provides backup/recovery mechanisms, the Developer does not assume liability for data loss or damage.
  4. Use the XL-LightHouse Single-Machine Version

    master

    In addition to the Big Data version, XL-LightHouse supports a Single-Machine Version designed for low-cost, lightweight usage.

    Specifications & Use Cases:

    • Minimum Requirements: A cloud server with at least 4 cores and 8GB RAM.
    • Ideal For:
      • Small to medium-sized enterprises or business teams.
      • Temporary/ephemeral use cases (e.g., stress testing a new interface, troubleshooting data anomalies, or investigating sudden database pressure spikes).
      • Initial product experience or as a local environment for secondary development and debugging.
  5. Access technical support and updates

    master

    System Updates

    To ensure security and stability, users should promptly install updates released by the Developer, which include bug fixes and feature enhancements.

    Support Tiers

    • Basic Support: Includes assistance with system deployment, configuration, and general usage questions.
    • Advanced Support: Includes custom feature development and other specialized services. These require additional fees and are governed by a separate service agreement.
  6. Application Entrypoint and Initialization

    master

    The application is initialized in main.tsx using React and Redux. It sets up a global configuration environment including:

    • Routing: Uses react-router-dom with routes for /login, /register, /license, and a root / route wrapped in PageLayout.
    • State Management: A Redux store is created using rootReducer and provided via <Provider store={store}>.
    • Theming and Localization: Uses @arco-design/web-react's ConfigProvider to manage global locale (zh-CN or en-US) and component configurations (e.g., removing borders from Card, List, and Table).
    • Context: A GlobalContext is provided to the application tree to allow components to access and modify lang and theme settings.
    • Lifecycle: On mount, the app checks authentication via checkLogin(). If not logged in and not on a public route (/login, /register, or /license), it redirects to /login.
    ReactDOM.render(
        <AliveScope>
        <Index />
        </AliveScope>
        , document.getElementById('root'));
  7. Access dashboard data via HomePageContext

    master

    The Workplace component provides a context named HomePageContext that allows child components to access the current dashboard's state without prop drilling.

    Context Shape

    The context provides an object containing:

    • homeData: An object of type HomeData representing the overview information.
    • statInfo: An object of type Stat representing statistical data.

    To use this in a child component, consume the context using the useContext hook.

    import { useContext } from 'react';
    import { HomePageContext } from '@/pages/common/context';
    
    const MyChildComponent = () => {
      const { homeData, statInfo } = useContext(HomePageContext);
      // Use homeData and statInfo here
      return <div>...</div>;
    };
  8. Understand the standalone logging structure

    master

    The standalone version of Lighthouse uses a Log4j2 configuration that categorizes logs into several distinct files based on their severity level. This allows for easier troubleshooting by isolating specific types of events.

    Logs are stored in the directory defined by the ${ldp_lighthouse_home}/logs/lighthouse-standalone/ path. Each log file uses a rolling policy where files are archived (compressed as .gz) when they reach a size of 500MB or based on a time pattern.

    Key log files include:

    • standalone_trace.log: Contains only TRACE level logs.
    • standalone_debug.log: Contains DEBUG level logs.
    • standalone_info.log: Contains INFO level logs.
    • standalone_warn.log: Contains WARN level logs.
    • standalone_error.log: Contains ERROR level logs.
    • Console output: Displays INFO level and above.
  9. Understand Hadoop environment variable precedence

    master

    The hadoop-env.sh file acts as the master configuration file for all Hadoop projects (YARN, HDFS, MapReduce). When configuring settings, be aware of the following precedence rules:

    1. Component-specific files override master file: {yarn-env.sh|hdfs-env.sh} > hadoop-env.sh > hard-coded defaults.
    2. Variable specificity overrides general variables: {YARN_xyz|HDFS_xyz} > HADOOP_xyz > hard-coded defaults.

    Most defaults are configured for substitution rather than appending, allowing you to overwrite them via the command line (e.g., JAVA_HOME=/usr/java/testing hdfs dfs -ls).

  10. Global Context for Language and Theme

    master

    The application uses a GlobalContext to manage user preferences for language and visual themes. This context is accessible to all components within the BrowserRouter.

    Context Values:

    • lang: The current language string (e.g., 'zh-CN', 'en-US').
    • setLang: Function to update the language.
    • theme: The current theme string (e.g., 'light').
    • setTheme: Function to update the theme.

    These values are persisted using the useStorage hook with keys arco-lang and arco-theme.