x-file-storage Documentation
repository·main·Indexed 25 days ago
https://github.com/dromara/x-file-storageA unified file storage abstraction for Java applications that allows developers to manage files across various storage providers with minimal code changes. It supports local storage, FTP, SFTP, WebDAV, and cloud providers including Alibaba Cloud OSS, Huawei Cloud OBS, Tencent Cloud COS, Amazon S3, Google Cloud Storage, Azure Blob Storage, and MinIO. Key features include file migration, image processing (resizing and thumbnails), ACL management, and file hash calculation (MD5, SHA256) during upload and download.
What's inside x-file-storage
- x-file-storage is a unified file storage abstraction that allows you to upload, download, and manage files across a wide variety of storage platforms with minimal code changes. It supports local storage, FTP, SFTP, WebDAV, and numerous cloud providers including Alibaba Cloud OSS, Huawei Cloud OBS, Tencent Cloud COS, Amazon S3, Google Cloud Storage, Azure Blob Storage, MinIO, and many others. It also supports file migration between different platforms and can be used to connect to services like Baidu Netdisk or Aliyun Pan via WebDAV (using Alist).
Supported storage platforms and capabilities
mainx-file-storage supports a wide variety of storage platforms including local file systems, FTP/SFTP, WebDAV, Amazon S3 (and S3-compatible providers), various cloud providers (Aliyun OSS, Huawei OBS, Tencent COS, etc.), and more.
Key behaviors for unsupported features:
- Copying: If a platform does not support native same-platform copying, the system automatically performs a cross-platform copy by downloading the file and then uploading it.
- Moving/Renaming: If a platform does not support native same-platform moving/renaming, the system performs a cross-platform move by copying the file and then deleting the source.
Move (Rename) files
mainMoving files works similarly to copying, with two modes:
- Same-platform move: Uses native provider methods (fast, no local bandwidth/disk usage). Supported by:
Local,FTP,SFTP,WebDAV,Qiniu Kodo,Upyos USS,Mongo GridFS, andVolcengine TOS. - Cross-platform move: Implemented as a Copy followed by a Delete. This is used for different platforms or unsupported providers.
Important Notes on ACL and Metadata:
- ACL and Metadata must be present in the
FileInfoobject for them to be moved. If the target platform doesn't support them, an exception will be thrown unless configured to ignore them.
- Same-platform move: Uses native provider methods (fast, no local bandwidth/disk usage). Supported by:
Copy files (Same-platform vs Cross-platform)
mainThe service supports two modes of copying:
- Same-platform copy: Uses the storage provider's native copy method. It is fast and doesn't use local network/disk space. Supported by most major providers.
- Cross-platform copy: Downloads the file and then uploads it to the destination. This is used when moving between different storage providers (e.g., Aliyun to Local) or when the provider doesn't support native copying (e.g., FTP, SFTP, FastDFS). It uses network bandwidth but avoids local disk usage.
Important Notes on ACL and Metadata:
- ACL and Metadata are copied if they exist in the
FileInfoobject. - If the target platform doesn't support the source's ACL/Metadata, an exception is thrown. You can configure parameters to ignore these errors.
Configure Mongo GridFS for directory simulation
mainMongo GridFS does not natively support directory hierarchies.
x-file-storagesimulates this by using the entire file path as the filename.Important Considerations:
- File Management: Because filenames contain
/characters, tools like Navicat may fail to preview or download files directly. You may need to rename them to remove slashes first. - Performance: Listing files can be slow in large collections because the library must fetch all files with a matching prefix to simulate the directory structure.
- Duplicate Filenames: While GridFS supports duplicate filenames,
x-file-storagewill overwrite existing files with the same name to maintain consistency with other storage platforms.
- File Management: Because filenames contain
What are Presigned URLs and when to use them
mainPresigned URLs allow temporary access to files stored in private buckets or files with private ACLs that cannot be accessed directly. They are useful for:
- Providing temporary access for downloading or viewing files.
- Enabling client-side uploads without exposing permanent credentials.
Supported storage platforms include:
- Huawei Cloud OBS
- Alibaba Cloud OSS
- Qiniu Kodo
- Tencent Cloud COS
- Baidu Cloud BOS
- MinIO
- Amazon S3 (including S3 V2)
- Google Cloud Storage
- Azure Blob Storage
- Volcengine TOS
Use Aspects to intercept file operations
mainInx-file-storage, Aspects allow you to intercept and intervene in file operations such as uploading and deleting files. This mechanism provides a way to inject custom logic (e.g., logging, security checks, or metadata processing) during the lifecycle of a file operation.Understand the difference between various paths in configuration and FileInfo
mainWhen using
x-file-storage, it is important to distinguish between the logical paths used for access and the physical paths used for storage.Path Definitions in
FileInfourl: The full access URL, constructed asdomain + basePath + path + filename.domain: The access domain (e.g.,https://file.abc.com/). If not needed, leave it empty.basePath: A prefix used to distinguish environments or projects sharing the same storage platform (e.g.,dev/ortest/).path: A sub-directory used to categorize files (e.g.,cover/for article covers,avatar/for user avatars).filename: The actual name of the file on the storage medium. Can be customized using.setSaveFilename().originalFilename: The original name of the uploaded file. If uploading viaInputStreamorbyte[], you must manually set this using.setOriginalFilename().storagePath: The physical location on the disk/server (e.g.,/www/wwwroot/file.abc.com/). This is used by platforms like SFTP, FTP, or Local Storage to map the logical structure to the physical disk. It is used for Nginx configuration and is not exposed in theFileInfoobject.
Local Storage Access via
path-patternsFor
local-plus(Local Storage Upgrade), you can use SpringWeb to serve files directly without Nginx by usingpath-patterns.Requirement: The
domainin your configuration must end with a/and match thepath-patternsstructure.Example configuration for direct SpringWeb access:
dromara: x-file-storage: local-plus: - platform: local-plus-1 enable-storage: true enable-access: true domain: http://127.0.0.1:8030/file/ base-path: local-plus/ path-patterns: /file/** storage-path: D:/Temp/Manage manual multipart upload chunk information
mainWhen performing manual multipart uploads, you need to track individual chunks (parts). This is handled via
FilePartInfoand can be persisted using a service that managesFilePartDetailentities.Required operations for chunk management:
saveFilePart(FilePartInfo info): Stores details likeplatform,uploadId,eTag,partNumber,partSize, andhashInfo.deleteFilePartByUploadId(String uploadId): Removes all chunk records associated with a specificuploadId, useful for cleaning up failed or abandoned uploads.
@Service public class FilePartDetailService extends ServiceImpl<FilePartDetailMapper, FilePartDetail> { public void saveFilePart(FilePartInfo info) { // Logic to save FilePartInfo } public void deleteFilePartByUploadId(String uploadId) { // Logic to remove parts by uploadId } }Directly upload HttpServletRequest for high performance
mainTo achieve high-performance uploads where files are streamed directly from the request without being saved to the local disk (non-disk-landing), you can pass the
HttpServletRequestdirectly tofileStorageService.of().Critical Configuration: You MUST enable lazy multipart resolution in your Spring configuration, otherwise the input stream will be consumed before it reaches the service:
spring.servlet.multipart.resolve-lazily: trueAccessing Request Parameters: To access non-file parameters from the request, wrap the request using
fileStorageService.wrapper(request)to get aHttpServletRequestFileWrapper. This allows you to retrieve multipart form data and parameters without prematurely reading the input stream.Warning: Do not define individual request parameters (e.g.,
@RequestParam String aaa) in your Controller method signature, as this will trigger the input stream to be read early and cause the upload to fail.@PostMapping("/upload-request") public FileInfo uploadRequest(HttpServletRequest request) { return fileStorageService.of(request).upload(); } @PostMapping("/upload-request2") public FileInfo uploadRequest2(HttpServletRequest request) { HttpServletRequestFileWrapper wrapper = (HttpServletRequestFileWrapper) fileStorageService.wrapper(request); // Access parameters via wrapper String aaa = wrapper.getParameter("aaa"); // Access multipart form data MultipartFormDataReader.MultipartFormData formData = wrapper.getMultipartFormData(); Map<String, String[]> parameterMap = formData.getParameterMap(); return fileStorageService.of(wrapper).upload(); }Install x-file-storage in a Spring Boot project
mainTo use x-file-storage in a Spring Boot environment, add the
x-file-storage-springdependency to yourpom.xml. You must also include the specific SDK for the storage platform you intend to use (e.g.,aliyun-sdk-ossfor Alibaba Cloud OSS).<!-- 引入本项目 --> <dependency> <groupId>org.dromara.x-file-storage</groupId> <artifactId>x-file-storage-spring</artifactId> <version>2.3.0</version> </dependency> <!-- 引入 阿里云 OSS SDK,如果使用其它存储平台,就引入对应的 SDK --> <dependency> <groupId>com.aliyun.oss</groupId> <artifactId>aliyun-sdk-oss</artifactId> <version>3.16.1</version> </dependency>Integrate x-file-storage with Solon
mainTo use
x-file-storagein a Solon project (version 2.2.0+), you must use the Solon-specific starter instead of the core module. After adding the starter, follow the standard quickstart guide to add dependencies for your specific storage platforms (e.g., MinIO, Local, etc.).If you are using local storage, you must also include the
solon.web.staticfilesdependency to enable static file serving.<!-- 1. Add the Solon starter --> <dependency> <groupId>org.dromara.x-file-storage</groupId> <artifactId>x-file-storage-solon</artifactId> <version>2.3.0</version> </dependency> <!-- 2. If using local storage, add static files support --> <dependency> <groupId>org.noear</groupId> <artifactId>solon.web.staticfiles</artifactId> <version>2.7.1</version> </dependency>