x-file-storage Documentation

repository·main·Indexed 25 days ago

https://github.com/dromara/x-file-storage

A unified file storage abstraction for Java applications that allows developers to manage files across various storage providers with minimal code changes. It supports local storage, FTP, SFTP, WebDAV, and cloud providers including Alibaba Cloud OSS, Huawei Cloud OBS, Tencent Cloud COS, Amazon S3, Google Cloud Storage, Azure Blob Storage, and MinIO. Key features include file migration, image processing (resizing and thumbnails), ACL management, and file hash calculation (MD5, SHA256) during upload and download.

Tokens
31.4K
Snippets
50
Records
99
Agent score
80%

What's inside x-file-storage

  1. Overview of x-file-storage

    main
    x-file-storage is a unified file storage abstraction that allows you to upload, download, and manage files across a wide variety of storage platforms with minimal code changes. It supports local storage, FTP, SFTP, WebDAV, and numerous cloud providers including Alibaba Cloud OSS, Huawei Cloud OBS, Tencent Cloud COS, Amazon S3, Google Cloud Storage, Azure Blob Storage, MinIO, and many others. It also supports file migration between different platforms and can be used to connect to services like Baidu Netdisk or Aliyun Pan via WebDAV (using Alist).
  2. Supported storage platforms and capabilities

    main

    x-file-storage supports a wide variety of storage platforms including local file systems, FTP/SFTP, WebDAV, Amazon S3 (and S3-compatible providers), various cloud providers (Aliyun OSS, Huawei OBS, Tencent COS, etc.), and more.

    Key behaviors for unsupported features:

    • Copying: If a platform does not support native same-platform copying, the system automatically performs a cross-platform copy by downloading the file and then uploading it.
    • Moving/Renaming: If a platform does not support native same-platform moving/renaming, the system performs a cross-platform move by copying the file and then deleting the source.
  3. Move (Rename) files

    main

    Moving files works similarly to copying, with two modes:

    1. Same-platform move: Uses native provider methods (fast, no local bandwidth/disk usage). Supported by: Local, FTP, SFTP, WebDAV, Qiniu Kodo, Upyos USS, Mongo GridFS, and Volcengine TOS.
    2. Cross-platform move: Implemented as a Copy followed by a Delete. This is used for different platforms or unsupported providers.

    Important Notes on ACL and Metadata:

    • ACL and Metadata must be present in the FileInfo object for them to be moved. If the target platform doesn't support them, an exception will be thrown unless configured to ignore them.
  4. Copy files (Same-platform vs Cross-platform)

    main

    The service supports two modes of copying:

    1. Same-platform copy: Uses the storage provider's native copy method. It is fast and doesn't use local network/disk space. Supported by most major providers.
    2. Cross-platform copy: Downloads the file and then uploads it to the destination. This is used when moving between different storage providers (e.g., Aliyun to Local) or when the provider doesn't support native copying (e.g., FTP, SFTP, FastDFS). It uses network bandwidth but avoids local disk usage.

    Important Notes on ACL and Metadata:

    • ACL and Metadata are copied if they exist in the FileInfo object.
    • If the target platform doesn't support the source's ACL/Metadata, an exception is thrown. You can configure parameters to ignore these errors.
  5. Configure Mongo GridFS for directory simulation

    main

    Mongo GridFS does not natively support directory hierarchies. x-file-storage simulates this by using the entire file path as the filename.

    Important Considerations:

    • File Management: Because filenames contain / characters, tools like Navicat may fail to preview or download files directly. You may need to rename them to remove slashes first.
    • Performance: Listing files can be slow in large collections because the library must fetch all files with a matching prefix to simulate the directory structure.
    • Duplicate Filenames: While GridFS supports duplicate filenames, x-file-storage will overwrite existing files with the same name to maintain consistency with other storage platforms.
  6. What are Presigned URLs and when to use them

    main

    Presigned URLs allow temporary access to files stored in private buckets or files with private ACLs that cannot be accessed directly. They are useful for:

    • Providing temporary access for downloading or viewing files.
    • Enabling client-side uploads without exposing permanent credentials.

    Supported storage platforms include:

    • Huawei Cloud OBS
    • Alibaba Cloud OSS
    • Qiniu Kodo
    • Tencent Cloud COS
    • Baidu Cloud BOS
    • MinIO
    • Amazon S3 (including S3 V2)
    • Google Cloud Storage
    • Azure Blob Storage
    • Volcengine TOS
  7. Use Aspects to intercept file operations

    main
    In x-file-storage, Aspects allow you to intercept and intervene in file operations such as uploading and deleting files. This mechanism provides a way to inject custom logic (e.g., logging, security checks, or metadata processing) during the lifecycle of a file operation.
  8. Understand the difference between various paths in configuration and FileInfo

    main

    When using x-file-storage, it is important to distinguish between the logical paths used for access and the physical paths used for storage.

    Path Definitions in FileInfo

    • url: The full access URL, constructed as domain + basePath + path + filename.
    • domain: The access domain (e.g., https://file.abc.com/). If not needed, leave it empty.
    • basePath: A prefix used to distinguish environments or projects sharing the same storage platform (e.g., dev/ or test/).
    • path: A sub-directory used to categorize files (e.g., cover/ for article covers, avatar/ for user avatars).
    • filename: The actual name of the file on the storage medium. Can be customized using .setSaveFilename().
    • originalFilename: The original name of the uploaded file. If uploading via InputStream or byte[], you must manually set this using .setOriginalFilename().
    • storagePath: The physical location on the disk/server (e.g., /www/wwwroot/file.abc.com/). This is used by platforms like SFTP, FTP, or Local Storage to map the logical structure to the physical disk. It is used for Nginx configuration and is not exposed in the FileInfo object.

    Local Storage Access via path-patterns

    For local-plus (Local Storage Upgrade), you can use SpringWeb to serve files directly without Nginx by using path-patterns.

    Requirement: The domain in your configuration must end with a / and match the path-patterns structure.

    Example configuration for direct SpringWeb access:

    dromara:
      x-file-storage:
        local-plus:
          - platform: local-plus-1
            enable-storage: true
            enable-access: true
            domain: http://127.0.0.1:8030/file/
            base-path: local-plus/
            path-patterns: /file/**
            storage-path: D:/Temp/
  9. Manage manual multipart upload chunk information

    main

    When performing manual multipart uploads, you need to track individual chunks (parts). This is handled via FilePartInfo and can be persisted using a service that manages FilePartDetail entities.

    Required operations for chunk management:

    • saveFilePart(FilePartInfo info): Stores details like platform, uploadId, eTag, partNumber, partSize, and hashInfo.
    • deleteFilePartByUploadId(String uploadId): Removes all chunk records associated with a specific uploadId, useful for cleaning up failed or abandoned uploads.
    @Service
    public class FilePartDetailService extends ServiceImpl<FilePartDetailMapper, FilePartDetail> {
        public void saveFilePart(FilePartInfo info) {
            // Logic to save FilePartInfo
        }
    
        public void deleteFilePartByUploadId(String uploadId) {
            // Logic to remove parts by uploadId
        }
    }
  10. Directly upload HttpServletRequest for high performance

    main

    To achieve high-performance uploads where files are streamed directly from the request without being saved to the local disk (non-disk-landing), you can pass the HttpServletRequest directly to fileStorageService.of().

    Critical Configuration: You MUST enable lazy multipart resolution in your Spring configuration, otherwise the input stream will be consumed before it reaches the service:

    spring.servlet.multipart.resolve-lazily: true

    Accessing Request Parameters: To access non-file parameters from the request, wrap the request using fileStorageService.wrapper(request) to get a HttpServletRequestFileWrapper. This allows you to retrieve multipart form data and parameters without prematurely reading the input stream.

    Warning: Do not define individual request parameters (e.g., @RequestParam String aaa) in your Controller method signature, as this will trigger the input stream to be read early and cause the upload to fail.

    @PostMapping("/upload-request")
    public FileInfo uploadRequest(HttpServletRequest request) {
        return fileStorageService.of(request).upload();
    }
    
    @PostMapping("/upload-request2")
    public FileInfo uploadRequest2(HttpServletRequest request) {
        HttpServletRequestFileWrapper wrapper = (HttpServletRequestFileWrapper) fileStorageService.wrapper(request);
        // Access parameters via wrapper
        String aaa = wrapper.getParameter("aaa");
        // Access multipart form data
        MultipartFormDataReader.MultipartFormData formData = wrapper.getMultipartFormData();
        Map<String, String[]> parameterMap = formData.getParameterMap();
        
        return fileStorageService.of(wrapper).upload();
    }
  11. Install x-file-storage in a Spring Boot project

    main

    To use x-file-storage in a Spring Boot environment, add the x-file-storage-spring dependency to your pom.xml. You must also include the specific SDK for the storage platform you intend to use (e.g., aliyun-sdk-oss for Alibaba Cloud OSS).

    <!-- 引入本项目 -->
    <dependency>
        <groupId>org.dromara.x-file-storage</groupId>
        <artifactId>x-file-storage-spring</artifactId>
        <version>2.3.0</version>
    </dependency>
    <!-- 引入 阿里云 OSS SDK,如果使用其它存储平台,就引入对应的 SDK  -->
    <dependency>
        <groupId>com.aliyun.oss</groupId>
        <artifactId>aliyun-sdk-oss</artifactId>
        <version>3.16.1</version>
    </dependency>
  12. Integrate x-file-storage with Solon

    main

    To use x-file-storage in a Solon project (version 2.2.0+), you must use the Solon-specific starter instead of the core module. After adding the starter, follow the standard quickstart guide to add dependencies for your specific storage platforms (e.g., MinIO, Local, etc.).

    If you are using local storage, you must also include the solon.web.staticfiles dependency to enable static file serving.

    <!-- 1. Add the Solon starter -->
    <dependency>
        <groupId>org.dromara.x-file-storage</groupId>
        <artifactId>x-file-storage-solon</artifactId>
        <version>2.3.0</version>
    </dependency>
    
    <!-- 2. If using local storage, add static files support -->
    <dependency>
        <groupId>org.noear</groupId>
        <artifactId>solon.web.staticfiles</artifactId>
        <version>2.7.1</version>
    </dependency>