DOCSEARCHbyAlgolia
Submit Docs

S3Proxy Documentation

repository·master·Indexed 25 days ago

https://github.com/gaul/s3proxy

S3Proxy is an S3 API implementation that acts as a proxy to various storage backends, including Azure, Google Cloud, local filesystems, and SFTP. It allows users to translate S3 requests to other cloud providers, mock S3 for testing, or embed S3 functionality into Java applications. The tool supports transparent AES encryption, CORS configuration, and custom middlewares for request processing. It can be deployed on Kubernetes using provided manifests or run directly on hosts requiring Java 17 or newer.

Tokens
4.2K
Snippets
7
Records
21
Agent score
79%

What's inside S3Proxy

  1. How S3Proxy encryption works

    master

    S3Proxy provides transparent encryption for S3 clients, allowing secure data writes to various cloud backends. It uses the AES/CFB/NoPadding cipher mode, which enables random access (reading from an offset) by allowing the decryption process to account for the previous 16-byte AES block.

    Key and IV Management

    • Key: A 128-bit key is derived from a user-provided password and salt.
    • IV (Initialization Vector): A random IV is generated for each part and stored within the part's padding to ensure security.

    Storage Format

    Every uploaded part is appended with a 64-byte padding containing metadata required for decryption. This padding is appended to the encrypted stream before it is passed to the BlobStore.

    For a single blob, the structure is: [ENCRYPTED BYTES] [PADDING]

    For multipart uploads, the structure repeats for each part: [ENCRYPTED BYTES] [PADDING] [ENCRYPTED BYTES] [PADDING] ...

  2. Metadata behavior in the SFTP backend

    master

    The SFTP backend uses the NIO.2 blobstore implementation. Metadata handling depends on the underlying filesystem:

    • User Metadata: S3 user metadata is not portable across SFTP servers. The backend attempts to store content and user metadata in user extended attributes if the filesystem supports them, but since most SFTP servers do not provide portable extended attributes, this is treated as optional.
    • Object Attributes: Object bytes are stored as regular SFTP files. The object size and last-modified values are derived directly from the SFTP file attributes.

  3. Configure S3Proxy Kubernetes Manifests

    master

    When customizing S3Proxy Kubernetes manifests, be aware of the following architectural decisions and requirements:

    Health Checks and Lifecycle

    • Probes: Readiness and liveness probes use GET /healthz, which is unauthenticated. During shutdown, the proxy fails readiness to allow the Service to drain endpoints promptly.
    • Graceful Shutdown: To ensure requests are drained, the manifests include a preStop sleep of 5 seconds (to account for endpoint propagation delay) and a terminationGracePeriodSeconds of 40 (covering the sleep plus the proxy's 30-second request drain).

    Security and Configuration

    • Credentials: Credentials should be mounted as Secret files and consumed via the image's *_FILE environment variables. This keeps sensitive data out of the pod spec and environment.
    • Root Filesystem: The root filesystem is read-only. emptyDir volumes are used for /tmp and /data.
    • Capabilities: All capabilities are dropped except NET_BIND_SERVICE, which is required to bind port 80 as root.

    Resource Management

    • JVM Memory: The JVM is configured to size its heap from the container memory limit using -XX:MaxRAMPercentage=75. Always set a memory limit in your container spec.

    Version Requirements

    • The *_FILE environment variable pattern and the shutdown drain mechanism require an image version newer than 3.3.0.

  4. What are S3Proxy middlewares?

    master

    Middlewares allow you to modify S3Proxy's behavior during request processing. They can be used for tasks such as:

    • Routing/Mapping: bucket aliasing, bucket prefix scoping, bucket locator.
    • Behavior Emulation: eventual consistency modeling, large object mocking.
    • Security/Access: read-only, user metadata replacer.
    • Performance/Infrastructure: latency, sharded backend containers, no cache override, storage class override.
    • Data Manipulation: regex rename blobs.

  5. Identify encrypted blobs via the .s3enc suffix

    master

    S3Proxy automatically appends a .s3enc suffix to every stored blob to identify it as encrypted.

    Note for S3 Clients:

    • The .s3enc suffix is not visible to the S3 client.
    • The reported blob size will always reflect the unencrypted size, not the size including padding and encryption overhead.

  6. Understand SFTP storage mapping and bucket structure

    master

    In the SFTP backend, S3 buckets are mapped as first-level directories under the jclouds.sftp.basedir. Object keys are mapped as files within those bucket directories.

    The effective path mapping follows this pattern: <jclouds.sftp.basedir>/<bucket>/<object-key>

    Example Mapping:

    • jclouds.sftp.basedir: /data/backups
    • S3 Bucket: example-bucket
    • S3 Object Key: backups/app/1000/db_dump.zip
    • Resulting SFTP Path: /data/backups/example-bucket/backups/app/1000/db_dump.zip

    Note for Backup Clients: Because the bucket name is part of the SFTP path, ensure your S3-compatible client is configured to use the directory name you intend to exist under jclouds.sftp.basedir as the bucket name.

  7. Configure the SFTP storage backend

    master

    The SFTP backend exposes an SFTP server as an S3Proxy storage backend using the jclouds BlobStore interface. It uses Apache MINA SSHD's SFTP filesystem provider.

    To use the SFTP backend, you must configure both S3Proxy credentials (for clients connecting to S3Proxy) and jclouds credentials (for S3Proxy connecting to the SFTP server). You must also provide a pinned host-key fingerprint for security.

    Key Configuration Parameters:

    • jclouds.provider: Set to sftp.
    • jclouds.endpoint: The SFTP URL (e.g., sftp://127.0.0.1:2222/). If the port is omitted, it defaults to 22.
    • jclouds.identity: The username for the SFTP server.
    • jclouds.credential: The password for the SFTP server.
    • jclouds.sftp.basedir: The root directory on the SFTP server where buckets will be created. Defaults to /s3proxy.
    • jclouds.sftp.host-key: The expected SHA256 host-key fingerprint of the SFTP server.
    s3proxy.authorization=aws-v2-or-v4
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.identity=local-identity
s3proxy.credential=local-credential

jclouds.provider=sftp
jclouds.endpoint=sftp://127.0.0.1:2222/
jclouds.identity=sftp-user
jclouds.credential=sftp-password
jclouds.sftp.basedir=/s3proxy
jclouds.sftp.host-key=SHA256:...

  8. Secure the SFTP backend with a host-key fingerprint

    master

    The SFTP backend requires a pinned SFTP server host-key fingerprint to prevent connection hijacking. S3Proxy will reject the connection if the server presents a different host key than the one provided in jclouds.sftp.host-key.

    You can obtain an OpenSSH-format SHA256 host-key fingerprint using the ssh-keyscan and ssh-keygen commands:

    ssh-keyscan -p 2222 127.0.0.1 | ssh-keygen -lf -

  9. Deploy S3Proxy on Kubernetes

    master

    To run S3Proxy on Kubernetes, use the provided reference manifests. There are two primary deployment variants depending on your storage requirements:

    1. Cloud Backend (cloud-backend/): Proxies to a cloud blobstore (any jclouds provider). In this mode, S3Proxy is stateless, allowing you to scale replicas freely.
    2. Filesystem Backend (filesystem-backend/): Stores blobs on a PersistentVolumeClaim. You must run a single replica and use the Recreate deployment strategy to prevent rolling updates from deadlocking on ReadWriteOnce volumes.

    Deployment Steps:

    1. Edit the credentials in secret.yaml.
    2. Apply the manifests using kubectl:
    kubectl apply -f cloud-backend/

    Note: For Helm users, the third-party s3proxy-chart is an alternative.

  10. Run S3Proxy without Docker

    master

    To run S3Proxy directly on your host machine, you need Java 17 or newer.

    1. Configuration

    Configure S3Proxy using a properties file. For example, to use the local filesystem with anonymous access, create a configuration file (e.g., s3proxy.conf) with the following content:

    s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

    Before running, ensure the storage directory exists:

    mkdir /tmp/s3proxy

    2. Execution

    • Linux and macOS: Run the executable jar:
      chmod +x s3proxy
s3proxy --properties s3proxy.conf
    • Windows: Invoke java explicitly:
      java -jar s3proxy --properties s3proxy.conf

    3. Verification

    Test the setup by creating a bucket and listing all buckets using curl:

    # Create a bucket
curl --request PUT http://localhost:8080/testbucket

# List buckets
curl http://localhost:8080/
    mkdir /tmp/s3proxy
chmod +x s3proxy
s3proxy --properties s3proxy.conf

  11. Configure S3Proxy logging via environment variables

    master

    S3Proxy logging behavior is controlled through environment variables. You can adjust the verbosity of the logs and the output format (text vs. JSON).

    Use LOG_LEVEL to set the logging threshold and LOG_APPENDER to choose between human-readable string logs or machine-readable JSON logs.

  12. Configure CORS in S3Proxy

    master

    S3Proxy provides basic CORS preflight and request/response handling. CORS is configured globally via the properties file (or environment variables in Docker) and cannot be configured per bucket.

    Available configuration keys:

    • s3proxy.cors-allow-origins: Allowed origins (supports regex).
    • s3proxy.cors-allow-methods: Allowed HTTP methods (e.g., GET, PUT).
    • s3proxy.cors-allow-headers: Allowed headers.
    • s3proxy.cors-allow-credential: Boolean to allow credentials.
    • s3proxy.cors-allow-all: If set to true, accepts any origin and header.

    Supported methods for CORS: GET, PUT, POST, HEAD, and DELETE.

    s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud
s3proxy.cors-allow-methods=GET PUT
s3proxy.cors-allow-headers=Accept Content-Type
s3proxy.cors-allow-credential=true