DOCSEARCHbyAlgolia
Submit Docs

GoFakeS3 Documentation

repository·master·Indexed 21 days ago

https://github.com/johannesboyne/gofakes3

An AWS S3 fake server and testing library for local development and S3 integration testing. It supports running as a standalone service or an integrated in-memory server, facilitating the testing of AWS Lambda functions and browser-based uploads. Compatible with AWS SDK v2 for Go and AWS SDK v3 for JavaScript, it provides support for path-style and virtual-hosted addressing, as well as customizable backends for storage, versioning, and multipart uploads.

Tokens
12.3K
Snippets
44
Records
63
Agent score
72%

What's inside GoFakeS3

  1. Configure S3 connection addressing modes

    master

    GoFakeS3 supports different addressing styles. Choosing the right one depends on your environment and client configuration.

    1. Path-Style Addressing (Recommended)

    In this mode, the bucket name is part of the URL path. This is the most flexible approach for local testing.

    • URL Format: http://localhost:9000/mybucket/myobject
    • AWS SDK v2 Configuration: Set o.UsePathStyle = true in the client options.
    • AWS SDK v3 (JS) Configuration: Set forcePathStyle: true.
    • AWS SDK v1 (JS) Configuration: Set s3ForcePathStyle: true.

    2. Virtual-Hosted Style Addressing

    In this mode, the bucket name is part of the hostname.

    • URL Format: http://mybucket.localhost:9000/myobject
    • Requirement: Requires DNS configuration. If using localhost, you must add entries to your /etc/hosts file for every bucket used (e.g., 127.0.0.1 mybucket.localhost).
    • AWS SDK v2 Configuration: This is the default mode if UsePathStyle is not set.

    3. Environment Variable (AWS SDK v2)

    For applications using the AWS SDK v2 that do not explicitly configure the endpoint in code, you can set the following environment variable:

    • AWS_ENDPOINT_URL_S3=http://localhost:9000

  2. Integrate GoFakeS3 with your own applications

    master

    To use GoFakeS3 in your own development or testing environment, follow these steps:

    1. Start the server: Run the GoFakeS3 standalone server.
    2. Configure AWS SDK clients: Point your AWS SDK clients to the server URL (e.g., http://localhost:9000).
    3. Enable Path-Style Addressing: Ensure your SDK client is configured to use path-style addressing to communicate with the server.
    4. Set Credentials: Use the credentials provided in the examples or configure GoFakeS3 to accept your own specific credentials.

  3. Install GoFakeS3 via Docker or Go

    master

    You can run GoFakeS3 as a containerized service or install it directly into your Go environment.

    Docker (Recommended) Use Docker to pull and run the image.

    Go install Use go get to add the library to your Go project.

    # Docker
docker pull johannesboyne/gofakes3

# Go install
go get github.com/johannesboyne/gofakes3

  4. Configure Virtual-Hosted Style Addressing

    master

    By default, GoFakeS3 uses path-style addressing (/bucket/object). You can configure it to support Virtual-Hosted style addressing (bucket.host/object) using the following options:

    1. WithHostBucket(): Enables standard virtual-host style addressing where the first part of the host is treated as the bucket name.
    2. WithHostBucketBase(bases []string): Allows specifying specific domain bases (e.g., s3.amazonaws.com) that should trigger virtual-host style parsing.

  5. Understand S3 bucket listing (ListBucketResult)

    master

    GoFakeS3 implements two versions of bucket listing to support different S3 API behaviors:

    1. ListBucketResult (Legacy/V1): Uses a Marker and NextMarker for pagination.
    2. ListBucketResultV2 (Recommended): Uses ContinuationToken and NextContinuationToken for pagination. It also includes KeyCount, StartAfter, and NextContinuationToken.

    Both types embed ListBucketResultBase, which provides common fields like Name, IsTruncated, Delimiter, Prefix, MaxKeys, and Contents (a slice of *Content).

  6. Configure S3 addressing modes

    master

    GoFakeS3 supports both Path-Style and Virtual-Hosted Style addressing.

    1. Path-Style (Default): The bucket name is part of the URL path (e.g., http://localhost:9000/mybucket/file.txt).
    2. Virtual-Hosted Style: The bucket name is a subdomain (e.g., http://mybucket.localhost:9000/file.txt).

    To enable Virtual-Hosted style, use the -hostbucket flag. If you need to support multiple hostnames or a specific domain structure, use -hostbucketbase with a comma-separated list of domains.

    # Enable virtual-hosted style addressing
gofakes3 -backend mem -hostbucket

# Enable virtual-hosted style with a specific domain base
gofakes3 -backend mem -hostbucketbase example.com

  7. Handle PutObject conditional headers

    master

    When implementing PutObject, you can use the CheckPutConditions helper to validate S3 conditional headers against the current state of an object.

    Supported Conditions:

    • IfNoneMatch: If set to *, the write only succeeds if the object does not already exist. Returns ErrPreconditionFailed if it exists.
    • IfMatch: The write only succeeds if the object exists and its ETag matches the provided value. Returns ErrPreconditionFailed if the object is missing or the ETag differs.
    // Example of how a backend might use the helper
err := CheckPutConditions(conditions, &ConditionalObjectInfo{
    Exists: true,
    Hash:  existingHash,
})

  8. Enable the Debug Server

    master

    You can run a separate HTTP server to inspect the internal state of GoFakeS3 using expvar and pprof. This is useful for monitoring and profiling the server performance.

    Use the -debug.host flag to specify the address for the debug server. Once running, you can access endpoints like /debug/pprof/ for profiling or /debug/vars for exported variables.

    # Start the main server and a debug server on a different port
gofakes3 -backend mem -host :9000 -debug.host :8080

  9. Run an integrated S3 server and client

    master

    Use the integrated example to create an in-memory S3 server and a client within the same Go process. This is useful for testing S3 logic without managing external processes. It demonstrates setting up a temporary server, configuring the AWS SDK v2 to connect to it, and performing basic operations like creating buckets, uploading, downloading, listing, and deleting objects.

    go run main.go

  10. Use GoFakeS3 with AWS SDK v3 for JavaScript (Lambda)

    master

    When using the AWS SDK v3 for JavaScript (e.g., in a Lambda function), configure the S3Client with the local endpoint, credentials, and forcePathStyle: true to ensure compatibility with GoFakeS3.

    // Using AWS SDK v3 for JavaScript
import { S3Client, CreateBucketCommand } from "@aws-sdk/client-s3";

// Create an S3 client with custom endpoint
const s3Client = new S3Client({
  region: "us-east-1",
  endpoint: "http://localhost:9000",
  forcePathStyle: true, // Required for GoFakeS3
  credentials: {
    accessKeyId: "ACCESS_KEY",
    secretAccessKey: "SECRET_KEY",
  },
});

// Lambda handler using async/await
export const handler = async (event, context) => {
  try {
    const command = new CreateBucketCommand({
      Bucket: "my-bucket",
    });

    const response = await s3Client.send(command);
    return response;
  } catch (error) {
    console.error("Error:", error);
    throw error;
  }
};

  11. Perform direct browser-based S3 uploads

    master

    You can simulate a direct browser upload to GoFakeS3 using a standard HTML form. The form must use method="post" and enctype="multipart/form-data". The action URL should follow the pattern http://<host>:<port>/<bucket-name>/.

    <html
  >
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  </head>
  <body

    >

  <form action="http://localhost:9000/<bucket-name>/" method="post" enctype="multipart/form-data">
    Key to upload:
    <input type="input"  name="key" value="user/user1/test/<filename>" /><br />
    <input type="hidden" name="acl" value="public-read" />
    <input type="hidden" name="x-amz-meta-uuid" value="14365123651274" />
    <input type="hidden" name="x-amz-server-side-encryption" value="AES256" />
    <input type="text"   name="X-Amz-Credential" value="AKIAIOSFODNN7EXAMPLE/20151229/us-east-1/s3/aws4_request" />
    <input type="text"   name="X-Amz-Algorithm" value="AWS4-HMAC-SHA256" />
    <input type="text"   name="X-Amz-Date" value="20151229T000000Z" />

    Tags for File:
    <input type="input"  name="x-amz-meta-tag" value="" /><br />
    <input type="hidden" name="Policy" value='<Base64-encoded policy string>' />
    <input type="hidden" name="X-Amz-Signature" value="<signature-value>" />
    File:
    <input type="file"   name="file" /> <br />
    <!-- The following elements will be ignored -->
    <input type="submit" name="submit" value="Upload to Amazon S3" />
  </form>
</html>

  12. Integrate GoFakeS3 with AWS SDK v2 for Go

    master

    To use GoFakeS3 in Go tests, you can instantiate a backend (like s3mem), wrap it with gofakes3.New, and start an httptest.Server. When configuring the AWS SDK v2 client, ensure you:

    1. Provide static credentials.
    2. Use an insecure HTTP client if testing locally without TLS.
    3. Set the EndpointResolver to the URL of your test server.
    4. Crucially, enable UsePathStyle = true in the S3 client options.
    import (
	"context"
	"crypto/tls"
	"net/http"
	"net/http/httptest"
	"strings"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/johannesboyne/gofakes3"
	"github.com/johannesboyne/gofakes3/backend/s3mem"
)

// Set up gofakes3 server
backend := s3mem.New()
faker := gofakes3.New(backend)
ts := httptest.NewServer(faker.Server())
defer ts.Close()

// Setup AWS SDK v2 config
cfg, err := config.LoadDefaultConfig(
	context.TODO(),
	config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("ACCESS_KEY", "SECRET_KEY", "")),
	config.WithHTTPClient(&http.Client{
		Transport: &http.Transport{
			TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
		},
	}),
	config.WithEndpointResolverWithOptions(
		aws.EndpointResolverWithOptionsFunc(func(_, _ string, _ ...interface{}) (aws.Endpoint, error) {
			return aws.Endpoint{URL: ts.URL}, nil
		}),
	),
)
if err != nil {
	panic(err)
}

// Create an Amazon S3 v2 client, important to use o.UsePathStyle
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
	o.UsePathStyle = true
})

// Create a new bucket
_, err = client.CreateBucket(context.TODO(), &s3.CreateBucketInput{
	Bucket: aws.String("newbucket"),
})
if err != nil {
	panic(err)
}

// Upload an object
_, err = client.PutObject(context.TODO(), &s3.PutObjectInput{
	Body:   strings.NewReader(`{"configuration": {"main_color": "#333"}, "screens": []}`),
	Bucket: aws.String("newbucket"),
	Key:    aws.String("test.txt"),
})
if err != nil {
	panic(err)
}