Install Evaporate via npm
masterTo use Evaporate in your project, install it using npm:
$ npm install evaporaterepository·master·Indexed 23 days ago
https://github.com/ttlabs/evaporatejsA JavaScript library for resumable multipart uploads from browsers, Node.js, or Electron to AWS S3. It supports parallel uploads, MD5 checksums, S3 Transfer Acceleration, and CloudFront, providing controls to pause, resume, and monitor upload progress. Version 2.1.4 includes support for Node FileSystem (fs) Stream and customizable authentication methods.
To use Evaporate in your project, install it using npm:
$ npm install evaporateEvaporate is used by first creating an instance with a global configuration via Evaporate.create(config), and then adding specific files to upload using the .add(addConfig, overrides) method.
Key configuration options include:
signerUrl: The URL used to obtain upload signatures.aws_key: Your AWS access key.bucket: The target S3 bucket.cloudfront: Boolean to enable CloudFront usage.computeContentMd5: Boolean to enable MD5 checksum calculations.cryptoMd5Method: A function to handle MD5 hashing (e.g., using Node's crypto module).When adding a file, you can provide addConfig which includes:
name: The name of the file.file: The file object to upload.progress: A callback receiving upload stats (transfer rate, time remaining).complete: A callback invoked upon successful upload.You can also provide overrides to change properties like the bucket for a specific upload instance.
const Evaporate = require('EvaporateJS');
const Crypto = require('crypto');
const config = {
signerUrl: SIGNER_URL,
aws_key: AWS_KEY,
bucket: AWS_BUCKET,
cloudfront: true,
computeContentMd5: true,
cryptoMd5Method: data => Crypto
.createHash('md5')
.update(data)
.digest('base64');
};
const uploadFile = evaporate => {
const file = new File([""], "file_object_to_upload");
const addConfig = {
name: file.name,
file: file,
progress: progressValue => console.log('Progress', progressValue),
complete: (_xhr, awsKey) => console.log('Complete!'),
}
const overrides = {
bucket: AWS_BUCKET_2
};
evaporate.add(addConfig, overrides)
.then(
awsObjectKey =>
console.log('File successfully uploaded to:', awsObjectKey),
reason =>
console.log('File did not upload sucessfully:', reason);
)
}
return Evaporate.create(config).then(uploadFile);To ensure AWS signatures are valid, Evaporate needs to synchronize the client's clock with the server's clock. This is handled via Evaporate.getLocalTimeOffset(config).
If config.timeUrl is provided, Evaporate performs an XMLHttpRequest to that URL. It expects the response to be a date string. The offset is calculated as server_date - local_date.
If config.localTimeOffset is provided as a number in the configuration, that value is used directly instead of fetching it.
Evaporate uses a unique key to identify and resume specific file uploads. This key acts as a signature for a file even when its path is unknown. The key is constructed by joining the following file properties with a hyphen (-):
file.name (Filename)file.type (MIME type)lastModified (ISO date string of the last modified date)sizeBytes (File size in bytes)Format: <filename>-<mimetype>-<modifieddate>-<filesize>
Evaporate uses a multi-step process to ensure secure S3 uploads:
authorize(). This may involve fetching a signature from a signerUrl using the stringToSign and canonicalRequest generated by the library.AwsSignatureV2 and AwsSignatureV4. The version is determined by the awsSignatureVersion setting in your configuration.sendRequestToAWS() is called. This uses XMLHttpRequest to send the payload to the calculated awsUrl.backOffWait based on maxRetryBackoffSecs and retryBackoffPower from your configuration, then attempts to re-send the request unless the error is considered terminal (like a 404 on a part PUT) or the errorHandler returns true.Evaporate provides several configuration keys to control upload behavior:
maxConcurrentParts: Controls the number of parallel uploads for each part.computeContentMd5: Enables/disables MD5 checksum calculations for each part.cryptoMd5Method: Allows providing a custom function for MD5 digest calculation.awsSignatureVersion: Specifies the AWS Signature version (e.g., Version 2 or 4).customAuthMethod: A pluggable signing method to support AWS Lambda or async functions.s3Acceleration: Enables S3 Transfer Acceleration.s3FileCacheHoursAgo: Controls recovery for huge files by determining how long to cache parts.allowS3ExistenceOptimization: Enables optimization by checking if the file already exists on S3.nameChanged: A callback invoked if the requested object name was not used (e.g., when reusing an interrupted upload).When calling Evaporate.create(config) or new Evaporate(config), you can provide several configuration options.
Core Configuration:
bucket: (Required) The AWS S3 bucket name.aws_key: The AWS access key.signerUrl: (Required if customAuthMethod is not provided) The URL used to obtain signed requests.awsRegion: The AWS region (defaults to 'us-east-1').awsSignatureVersion: The AWS signature version (defaults to '4').Upload Behavior:
maxConcurrentParts: Maximum number of parts to upload simultaneously (defaults to 5).partSize: Size of each part in bytes (defaults to 6 * 1024 * 1024, i.e., 6MB).maxFileSize: Maximum allowed file size. If exceeded, add() will reject.s3Acceleration: Enables S3 Transfer Acceleration.cloudfront: Enables CloudFront usage.Resilience & Optimization:
computeContentMd5: If true, computes MD5 for parts (requires cryptoMd5Method).allowS3ExistenceOptimization: If true and computeContentMd5 is enabled, attempts to reuse existing S3 objects if the MD5 matches.onlyRetryForSameFileName: If true, retries are only attempted if the filename hasn't changed.s3FileCacheHoursAgo: Number of hours to look back when checking for cached/interrupted uploads (must be a whole number).Advanced:
customAuthMethod: A function to implement custom authentication logic.signParams: Object containing parameters to be sent to the signer.signHeaders: Object containing headers to be sent to the signer.evaporateChanged: A callback function invoked whenever the number of evaporating parts changes.PutPart), you can provide an onProgress callback within the request configuration to monitor the upload progress of that specific chunk. This is typically used to update UI progress bars for individual parts of a multipart upload.errorHandler function in your configuration. This is useful for custom logging or deciding whether an error should trigger a retry or an immediate abort. If the errorHandler returns true, the library assumes the error has been handled and will not attempt further automatic retries for that specific request.The Evaporate instance provides methods to control all active or queued uploads:
cancel(id): Cancels a specific upload by its ID, or all uploads if no ID is provided. Returns a Promise.pause(id, options): Pauses a specific upload or all uploads. If options.force is true, it aborts current parts immediately. Returns a Promise.resume(id): Resumes a specific paused upload or all paused uploads. Returns a Promise.Note: id refers to the unique identifier for the file upload (typically bucket/name).
Use the add(file, pConfig) method to queue a file for multipart upload to S3.
file: An object containing the actual File object (e.g., { file: myFile, name: 'my-filename.txt' }).pConfig: (Optional) An object to override or extend the global Evaporate configuration for this specific file.The method returns a Promise that resolves with the decoded S3 object name when the upload is complete, or rejects if the upload fails or is aborted.
You can also provide callback functions within the file object to hook into the upload lifecycle:
started, uploadInitiated, progress, complete, cancelled, paused, resumed, pausing, nameChanged, info, warn, error.signResponseHandler function in your configuration object. This function is called when the signature fetch is successful and allows you to process the response before the upload proceeds. It must return a Promise that resolves with the payload.