bboss-elasticsearch

repository·master·Indexed 19 days ago

https://github.com/bbossgroups/bboss-elasticsearch

A high-performance Java ORM and REST client for Elasticsearch, Easysearch, and Opensearch. It provides a MyBatis-like experience for managing CRUD operations, aggregations, and complex queries. Supports JDK 1.6+ and Elasticsearch versions 2.x, 5.x, 6.x, 7.x, and above, with dedicated starters for Spring Boot projects.

Tokens
353.4K
Snippets
766
Records
939
Agent score
60%

What's inside bboss-elasticsearch

  1. Overview of Elasticsearch Aggregation Use Cases

    master

    The bboss aggregation documentation provides patterns for three primary types of aggregation scenarios:

    1. Multi-level Aggregations: Calculating total visits, successful visits, and failed visits within a specific time range per application, combined with response time bucket statistics (e.g., <1s, 1-3s, 3-5s, >5s).
    2. Simple Term Aggregations: Basic term-based statistics.
    3. Cardinality Aggregations: Estimating the count of unique values.
  2. Overview of bboss-ai Agent Framework

    master

    What is bboss-ai?

    bboss-ai is a lightweight Java AI Agent development client built on Apache HttpClient5, HttpCore5, and Project Reactor. It provides a unified interface for interacting with Large Language Models (LLM) and multi-modal models, supporting both synchronous and streaming (reactive) invocation modes.

    Core Capabilities

    • Model Integration: Unified access to major MaaS platforms including DeepSeek, Kimi, Zhipu, Alibaba Bailian (Qwen), ByteDance Doubao, MiniMax, Tencent Hunyuan, and OpenAI-compatible interfaces.
    • Tooling & MCP: Supports Function Calling, Model Context Protocol (MCP) service discovery (via SSE and Streamable HTTP), and multi-step autonomous task execution (Loop Tool Call). It also includes execution tools for Shell/CLI and Python/NodeJS.
    • Multi-modal Support: Capabilities for text, image (recognition/generation), audio (STT/TTS), and video (understanding/generation).
    • RAG (Retrieval Augmented Generation): Built-in support for text embedding and reranking, compatible with vector databases like Elasticsearch and Milvus.
    • Multi-Agent Orchestration: Uses directed cyclic graphs to implement complex workflows including serial, parallel, routing, conditional branching, and judge evaluation. This is powered by bboss jobflow.
    • Session Management: Supports long/short-term memory, message compression, and multiple persistence backends (Memory, MySQL, Oracle, PostgreSQL, SQLite, and production-grade ClickHouse).
    • Observability: Full-link tracing (Trace) covering LLM calls, tool execution, and workflow orchestration.
  3. What is BBoss?

    master

    BBoss is an open-source ecosystem consisting of five main components designed for high-performance data operations and AI integration:

    1. Multi-modal Agent Framework: Supports synchronous and streaming calls. Integrates models like Deepseek and Tongyi Qianwen for tasks like Q&A, image/video recognition, and generation. Supports workflow-based multi-agent collaboration and MCP tool calling.
    2. Elasticsearch Highlevel Java Restclient: A high-performance, high-compatibility Java client framework for Elasticsearch, Easysearch, and Opensearch.
    3. Data Collection & ETL: A Java-based ETL tool with extensible input/output plugins, supporting data vectorization.
    4. Stream-Batch Integrated Computing Framework: A framework for flexible data metric statistics and processing, suitable for enterprise-scale data analysis.
    5. Universal Distributed Job Scheduling Workflow: A workflow orchestration model for scheduling tasks like data collection, ETL, business logic, and LLM inference with support for conditional triggers and parameter sharing via context.
  4. Overview of bboss-datatran ETL capabilities

    master

    bboss-datatran is an open-source data collection, synchronization, and ETL (Extract, Transform, Load) tool. It provides capabilities for:

    • Data collection
    • Data cleaning and transformation processing
    • Data ingestion (loading into databases)
    • Unified stream and batch processing for data metric statistical calculations.
  5. What is bboss Elasticsearch ORM?

    master

    bboss is a high-performance Elasticsearch ORM (Object-Relational Mapping) library for Java. It allows developers to manage Elasticsearch DSL (Domain Specific Language) using XML files, similar to how MyBatis manages SQL.

    Key features include:

    • DSL Management: Use XML to manage DSL scripts with support for variables, script fragments, foreach loops, logical judgments, and comments.
    • Dynamic Loading: Supports online modification and automatic hot-reloading of DSL configuration files.
    • Extensibility: DSL management can be extended to use databases, Redis, etc.
    • Broad Compatibility: Supports native Elasticsearch RESTful API, Java ORM API, and Elasticsearch SQL. It does not depend on official Elasticsearch JAR files, making it compatible with a wide range of versions.
  6. Overview of bboss-datatran ETL tool

    master

    bboss-datatran is an open-source data collection and ETL (Extract, Transform, Load) tool designed for high-performance data synchronization. It functions as a "simplified Flink," allowing developers to write data synchronization jobs directly in Java.

    Key capabilities include:

    • Data Processing: Supports data collection, cleaning, transformation, and loading.
    • Stream & Batch Integration: Provides unified processing for both historical batch data and real-time incremental data.
    • Customization: Since jobs are written in Java, you can use any existing Java libraries or frameworks to process massive datasets.
    • Job Control: Provides APIs for job monitoring and control, including start, pause, resume, and stop.
    • Workflow Orchestration: Uses bboss jobflow for distributed job scheduling and complex workflow orchestration.
    • Multi-Output: Supports an asynchronous pipeline architecture where data can be sent to multiple output plugins simultaneously.
  7. Overview of Elasticsearch Bboss

    master

    Elasticsearch Bboss is a high-performance Java ORM (Object-Relational Mapping) and REST client for Elasticsearch. It is designed to operate similarly to MyBatis, providing a simplified way to interact with Elasticsearch.

    Key features include:

    • Support for multiple clusters and multiple data sources.
    • Automatic index management.
    • Various pagination mechanisms.
    • Simplified CRUD operations.
    • Support for scripts, SQL, JDBC, highlighting, weighting, aggregations, IP, and GEO locations.
    • Support for parent-child and nested document structures.
    • Compatibility with Elasticsearch, Easysearch, and Opensearch.
  8. Overview of bboss Data Synchronization

    master

    bboss is a Java-based data synchronization tool designed for high-concurrency and distributed task scheduling. It is similar to Logstash but allows developers to write synchronization logic directly in Java, enabling complex data processing (e.g., transforming XML to Java objects) before ingestion.

    Key Capabilities

    • Supported Data Sources: MySQL, MariaDB, PostgreSQL, Oracle, SQLServer, DB2, TiDB, Hive, MongoDB, and Kafka/RocketMQ.
    • Supported Destinations: Elasticsearch (versions 1.x through 7.x+) and various databases.
    • Import Modes:
      • Single-row import
      • Batch import
      • Multi-threaded parallel batch import
      • Scheduled full data import (serial or parallel)
      • Scheduled incremental data import (serial or parallel)
    • Scheduling Engines:
      • jdk timer (built-in)
      • quartz
      • xxl-job (supports distributed sharding for massive data synchronization)
  9. Overview of BBoss Elasticsearch Client

    master

    BBoss is a high-performance Object-Relational (O/R) mapping REST client for Elasticsearch, Easysearch, and Opensearch. It provides a high-level Java API designed to simplify complex operations and handle large-scale data tasks.

    Key Features:

    • High-level API: Simplifies CRUD operations, scripts, SQL, JDBC, highlighting, weighting, aggregations, IP/GEO location, and parent-child/nested mappings.
    • Cluster Management: Built-in client cluster node load balancing and disaster recovery.
    • Multi-source Support: Supports multiple clusters and multiple data sources.
    • Automated Management: Includes automatic index hosting and various pagination mechanisms.
    • Data Processing: Functions as a 'lightweight Flink' for data collection, ETL, and unified stream/batch big data analysis processing.

    Supported Engines:

    • Elasticsearch: 1.x, 2.x, 5.x, 6.x, 7.x, 8.x, 9.x+
    • Easysearch: 1.x, 2.x
    • Opensearch: 1.x, 2.x, 3.x+

    Licensing: Apache License 2.0

  10. Summary of MongoDB to Elasticsearch synchronization features

    master

    The MongoDB to Elasticsearch synchronization tool provides the following capabilities:

    • Data Transformation and Filtering: Ability to transform and filter data during the sync process.
    • Incremental Parallel Sync: Supports high-concurrency, incremental data synchronization.
    • Distributed Scheduling: Supports various scheduling mechanisms including JDK Timer, Quartz, and XXL-JOB (including XXL-JOB distributed sharding).
    • Performance Tuning: Allows for JVM memory adjustment and tuning of parallel mechanisms to optimize data throughput and processing performance.
  11. Key features of the bboss file & log collection plugin

    master

    The bboss-datatran-fileftp plugin provides the following capabilities for data synchronization and log collection:

    • Collection Modes: Supports both full (all data) and incremental (new data only) collection.
    • ETL Support: Integrated ETL (Extract, Transform, Load) capabilities for stream and batch processing.
    • Multi-Protocol Support: Collects data from local directories, FTP, and SFTP.
    • Parallel Processing: Supports multi-threaded parallel downloading and processing of remote files.
    • File Management:
      • Backs up collected log files with configurable retention periods and automatic cleanup.
      • Automatically cleans up files on FTP servers after successful download.
    • Flow Control: Includes a mechanism to limit the maximum number of concurrent file collections to prevent resource exhaustion. When the threshold is reached, the plugin waits for active tasks to complete before starting new ones.
  12. Manage Elasticsearch Indices and Templates via ClientInterface

    master

    The ClientInterface provides high-level Java REST API methods to manage Elasticsearch indices and index templates. You can create, modify, retrieve, and delete indices, as well as manage index templates and aliases. Most operations are performed by defining DSL scripts in an XML configuration file (e.g., esmapper/demo.xml) and then calling the corresponding methods on a ClientInterface instance.

    // Initialize ClientInterface using a configuration file
    ClientInterface clientUtil = ElasticSearchHelper.getConfigRestClientUtil("esmapper/demo.xml");
    
    // Example: Create an index using a predefined DSL script name
    clientUtil.createIndiceMapping("demo", "createDemoIndice");