DOCSEARCHbyAlgolia
Submit Docs

MinIO Plus

website·Indexed 32 days ago

https://minioplus.liuxp.me

A Java SDK designed as a companion for MinIO, providing a StorageService interface for object storage operations. It includes features for instant and chunked uploads with breakpoint resumption, file metadata management via a customizable MetadataRepository (with a default MySQL implementation), automatic thumbnail generation, and bucket strategies based on file types.

Tokens
20.3K
Snippets
80
Records
123
Agent score
59%

What's inside MinIO Plus

  1. Overview of MinIO Plus features

    MinIO Plus is a secondary encapsulation and enhancement tool for MinIO. It is designed to be non-intrusive, meaning it enhances MinIO's capabilities without modifying the original MinIO code. Key features include:

    • Instant File Transmission: Uses hash digest recognition to avoid re-uploading identical files.
    • Concurrent & Resumable Uploads: Splits files into small blocks for concurrent uploading to maximize bandwidth and allows resuming failed transfers from the last unfinished block.
    • Automatic Thumbnail Generation: Automatically generates thumbnails for images upon upload with configurable sizes.
    • Automatic Bucket Policy: Organizes files into buckets by type (e.g., documents, audio, video) and paths by date (/year/month) to prevent OS file system performance degradation.
    • Access Control: Supports user- and group-based file access control.
    • Link Validity: Provides pre-signed upload and download addresses with configurable expiration periods.
    • Client Direct Connection: Allows the frontend to connect directly to MinIO, bypassing the application server's file stream handling to maintain native performance.

  2. Overview of MinIO Plus features and capabilities

    MinIO Plus is a non-intrusive wrapper and enhancement tool for MinIO designed to simplify development and improve efficiency. It provides several advanced file management features without modifying the core MinIO code. Key capabilities include:

    • Fast File Upload (Instant Upload): Uses hash digests to identify files; if a file already exists, it avoids actual data transmission.
    • Concurrent Uploads: Splits files into small chunks and uploads them concurrently to maximize bandwidth utilization.
    • Resumable Uploads (Breakpoint Transmission): Allows resuming failed transfers by only uploading the remaining unfinished chunks.
    • Automatic Thumbnail Generation: Automatically generates thumbnails for images with configurable sizes.
    • Automatic Bucket Strategy: Automatically creates buckets based on file types (e.g., documents, archives, audio, video, images) and organizes paths by /year/month to prevent OS directory performance degradation.
    • Access Control: Supports file permission control based on users and groups.
    • Timed Access Links: Provides pre-signed upload and download URLs with expiration dates based on MinIO's temporary link strategy.
    • Client Direct Connection: Enables front-end clients to connect directly to MinIO, avoiding the need for the application server to proxy file streams while maintaining these enhanced features.
    • Compatibility: Supports JDK 1.8 through 17+ and is compatible with both Spring Boot 2 and Spring Boot 3.

  3. MinIO-Plus Project Architecture Overview

    MinIO-Plus is a multi-module Maven project designed for modularity and extensibility. It is built with JDK 8+ and follows the base package naming convention org.liuxp.minioplus.*. The project is structured to isolate core logic from API definitions and integration starters.
    Group ID: me.liuxp
Artifact ID: minio-plus-parent
Version: 1.0.5
Java Version: JDK 8+
Build Tool: Maven

  4. Understand the MinIO Plus project architecture and module division

    MinIO Plus is a Java language SDK for MinIO designed for automatic assembly with Spring Boot. It provides the business logic and interface definitions, while users are responsible for data storage (though a default MySQL implementation is provided).

    Module Breakdown:

    • minio-plus-api: Service layer interface definitions for external capabilities.
    • minio-plus-core: Implementation of the business logic defined in minio-plus-api.
    • minio-plus-extension: Controller layer encapsulation of minio-plus-api for out-of-the-box usage.
    • minio-plus-common: Utility and configuration classes.
    • minio-s3-api: S3 specification handling, split into minio-s3-api-definition (interfaces), minio-s3-api-official (native SDK implementation), and minio-s3-api-custom (custom implementations).
    • minio-plus-application: Example projects demonstrating usage with native SDKs (minio-plus-application-official) or custom S3 specifications (minio-plus-application-custom), both using MySQL for metadata.

  5. Understand automatic thumbnail generation for image previews

    Thumbnails are automatically generated when a user first accesses the image preview interface. The system maintains the original aspect ratio during compression and stores the resulting file in the thumbnail bucket using the same MD5 filename as the original image. By default, thumbnails are compressed to a width of 300 pixels. If the original image dimensions are smaller than the target thumbnail size, the original image is stored instead of being upscaled.

  6. Understand how Instant Upload (文件秒传) works

    Instant Upload (文件秒传) optimizes the upload process by performing a hash digest identification for every uploaded file. If a user attempts to upload a file that has already been uploaded to the system, the actual data transmission process is skipped, resulting in an 'instant' upload experience.

  7. Implement the file metadata database table

    The file metadata table (file_metadata_info) stores essential information about uploaded files, including their unique keys, MD5 hashes, MIME types, storage bucket locations, and upload status. The schema is designed to be customizable and expandable based on project requirements.
    CREATE TABLE `file_metadata_info` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
  `file_key` varchar(50) NOT NULL COMMENT 'File Key',
  `file_md5` varchar(50) DEFAULT NULL COMMENT 'File MD5',
  `file_name` varchar(255) NOT NULL COMMENT 'File Name',
  `file_mime_type` varchar(128) DEFAULT NULL COMMENT 'MIME Type',
  `file_suffix` varchar(20) DEFAULT NULL COMMENT 'File Suffix',
  `file_size` bigint(20) DEFAULT NULL COMMENT 'File Size',
  `is_preview` tinyint(1) DEFAULT '0' COMMENT 'Preview Available 0: No 1: Yes',
  `is_private` tinyint(1) DEFAULT '0' COMMENT 'Is Private 0: No 1: Yes',
  `bucket` varchar(20) NOT NULL COMMENT 'Bucket',
  `bucket_path` varchar(20) NOT NULL COMMENT 'Bucket Path',
  `upload_id` varchar(255) DEFAULT NULL COMMENT 'Upload ID',
  `is_finished` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'Status 0: Incomplete 1: Complete',
  `is_part` tinyint(1) DEFAULT NULL COMMENT 'Is Part 0: No 1: Yes',
  `part_number` int(4) DEFAULT NULL COMMENT 'Number of Parts',
  `create_time` datetime NOT NULL COMMENT 'Creation Time',
  `create_user` varchar(255) NOT NULL COMMENT 'Created By',
  `update_time` datetime NOT NULL ON UPDATE CURRENT_TIMESTAMP COMMENT 'Update Time',
  `update_user` varchar(255) NOT NULL COMMENT 'Updated By',
  PRIMARY KEY (`id`) USING BTREE,
  KEY `INDEX_KEY` (`file_key`),
  KEY `INDEX_MD5` (`file_md5`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='File Metadata Information Table';

  8. Accelerate file transfers using hash-based identification

    The system uses hash summary identification for every uploaded file. If a user attempts to upload a file that has already been uploaded by another user, the system recognizes the file via its hash and skips the actual transfer process, enabling near-instantaneous 'second transfers'.

  9. Calculate MinIO ETAG for shard (multipart) uploads

    When a file is uploaded in shards, MinIO does not re-read the entire file stream to calculate the final ETAG. Instead, it follows this process:

    1. Each shard is assigned its own hidden ETAG (MD5 of that shard).
    2. Upon calling the merge interface, MinIO concatenates the binary byte arrays of all shard ETAGs into a single byte array.
    3. The final file ETAG is the MD5 hash of this concatenated byte array.

    Note: The resulting ETAG often follows the format {MD5_hash}-{shard_count}. Because of this mechanism, a file uploaded via shards will have a different ETAG than the same file uploaded via a regular upload.

    // Example: Simulating MinIO Shard ETAG calculation
String part1 = "101faaef4cc4567091dbf5c62d0f2a66";
String part2 = "54b96a8a5cf6164c1e197ff9e97b9e3c";
String part3 = "f647e6824a99b6ca06805dbc10756870";
String part4 = "3c2f9b262e71c6fbf336432e9dbfe89b";

// IMPORTANT: Convert hex strings to byte arrays first
byte[] part1byte = HexUtil.decodeHex(part1);
byte[] part2byte = HexUtil.decodeHex(part2);
byte[] part3byte = HexUtil.decodeHex(part3);
byte[] part4byte = HexUtil.decodeHex(part4);

// Concatenate bytes
byte[] result = new byte[part1byte.length + part2byte.length + part3byte.length + part4byte.length];
System.arraycopy(part1byte, 0, result, 0, part1byte.length);
System.arraycopy(part2byte, 0, result, part1byte.length, part2byte.length);
System.arraycopy(part3byte, 0, result, part1byte.length + part2byte.length, part3byte.length);
System.arraycopy(part4byte, 0, result, part1byte.length + part2byte.length + part3byte.length, part4byte.length);

// The final ETAG is the MD5 of the concatenated byte array
String finalEtag = SecureUtil.md5(IoUtil.toStream(result));

  10. Implement resumable upload for reliable file transfers

    Resumable upload leverages chunking technology to ensure high availability during large file transfers. It provides the following benefits:

    • Efficiency: Only unfinished chunks are retransmitted after a failure, reducing network and disk IO.
    • Reliability: Prevents the need to restart the entire upload process due to network fluctuations or disconnections.
    • Control: Allows users to pause and resume transfer tasks at any time.

  11. Use the MinIO Plus Controller Layer Interface

    For standard Spring projects, the minio-plus-extension provides a pre-defined Controller layer interface, eliminating the need to write custom controllers for basic file operations. The available endpoints include:

    • Initialize File Upload Task
    • Upload Complete
    • File Download (returns a download address for 302 redirect)
    • Image Preview - Original Image (returns original image address for 302 redirect)
    • Image Preview - Thumbnail (returns thumbnail address for 302 redirect)

  12. Implement user authentication for MinIO-Plus controllers

    MinIO-Plus controllers retrieve the current user's information using UserHolder.get(). To implement this, you must set the user information in the UserHolder before the request is processed and clear it after completion to ensure thread safety (as it uses ThreadLocal).
    @Component
public class UserContextInterceptor implements HandlerInterceptor {
    
    @Override
    public boolean preHandle(HttpServletRequest request, 
                           HttpServletResponse response, 
                           Object handler) {
        // Extract user info (e.g., from JWT token)
        String userId = extractUserIdFromRequest(request);
        UserHolder.set(userId);
        return true;
    }
    
    @Override
    public void afterCompletion(HttpServletRequest request, 
                              HttpServletResponse response, 
                              Object handler, Exception ex) {
        UserHolder.clear();
    }
}