UpChunk Documentation

repository·master·Indexed 19 days ago

https://github.com/muxinc/upchunk

A JavaScript module for handling large file uploads via chunking using Fetch. UpChunk manages splitting files into chunks, making resumable PUT requests with correct range headers, and handling retries and network fluctuations. It supports pausing, resuming, and aborting uploads, and is compatible with any server supporting resumable upload patterns.

Tokens
3.8K
Snippets
8
Records
14
Agent score
15%

What's inside @mux/upchunk

  1. How UpChunk works

    master
    UpChunk is a JavaScript module for handling large file uploads via chunking. It splits a file into chunks (in multiples of 256KB) and makes a PUT request for each chunk with the correct Content-Length and Content-Range headers. It is fault-tolerant, supporting retries on failures, and allows for pausing and resuming uploads. While designed for Mux direct uploads, it works with any server supporting similar resumable upload patterns.
  2. How to cancel an upload

    master

    To cancel an upload, use the abort() method. To ensure the instance is cleaned up, you can manually delete the reference to the instance to allow for garbage collection.

    // upload is an UpChunk instance currently in-flight
    upload.abort();
    
    // To be sure, you can manually delete the instance.
    delete upload;
  3. Install @mux/upchunk

    master

    You can install UpChunk using NPM, Yarn, or by including it via a script tag in your HTML.

    # NPM
    npm install --save @mux/upchunk
    
    # Yarn
    yarn add @mux/upchunk
    <!-- Script Tag -->
    <script src="https://unpkg.com/@mux/upchunk@3"></script>
  4. Validate chunk size requirements

    master

    UpChunk requires chunk sizes to follow specific rules to ensure compatibility with storage providers like Google Cloud Storage (GCS).

    Rules:

    1. The chunkSize must be a positive number.
    2. The chunkSize must be a multiple of 256 (in KiB).
    3. The chunkSize must be between minChunkSize and maxChunkSize.

    If you are manually setting the chunkSize on an UpChunk instance, it will throw a TypeError if these conditions are not met.

  5. Use UpChunk with plain JavaScript

    master

    To use UpChunk in a standard JavaScript environment, import the module and call createUpload. You can provide the endpoint as a string or a function that returns a promise resolving to the URL. Subscribe to events like progress, error, and success to manage the UI state.

    import * as UpChunk from '@mux/upchunk';
    
    const picker = document.getElementById('picker');
    
    picker.onchange = () => {
      const getUploadUrl = () =>
        fetch('/the-endpoint-above').then((res) =>
          res.ok ? res.text() : throw new Error('Error getting an upload URL :(')
        );
    
      const upload = UpChunk.createUpload({
        endpoint: getUploadUrl,
        file: picker.files[0],
        chunkSize: 30720, // Uploads the file in ~30 MB chunks
      });
    
      // subscribe to events
      upload.on('error', (err) => {
        console.error('💥 🙀', err.detail);
      });
    
      upload.on('progress', (progress) => {
        console.log(`So far we've uploaded ${progress.detail}% of this file.`);
      });
    
      upload.on('success', () => {
        console.log("Wrap it up, we're done here. 👋");
      });
    };
  6. Use UpChunk with React

    master

    In a React application, you can manage upload progress and status messages using useState. Call createUpload within an event handler (like onChange of a file input) and attach event listeners to update the component state.

    import React, { useState } from 'react';
    import * as UpChunk from '@mux/upchunk';
    
    function Page() {
      const [progress, setProgress] = useState(0);
      const [statusMessage, setStatusMessage] = useState(null);
    
      const handleUpload = async (inputRef) => {
        try {
          const response = await fetch('/your-server-endpoint', { method: 'POST' });
          const url = await response.text();
    
          const upload = UpChunk.createUpload({
            endpoint: url, // Authenticated url
            file: inputRef.files[0], // File object with your video file’s properties
            chunkSize: 30720, // Uploads the file in ~30 MB chunks
          });
    
          // Subscribe to events
          upload.on('error', (error) => {
            setStatusMessage(error.detail);
          });
    
          upload.on('progress', (progress) => {
            setProgress(progress.detail);
          });
    
          upload.on('success', () => {
            setStatusMessage("Wrap it up, we're done here. 👋");
          });
        } catch (error) {
          setErrorMessage(error);
        }
      };
    
      return (
        <div className="page-container">
          <h1 className="page-container">File upload button</h1>
          <label htmlFor="file-picker">Select a video file:</label>
          <input
            type="file"
            onChange={(e) => handleUpload(e.target)}
            id="file-picker"
            name="file-picker"
          />
    
          <label htmlFor="upload-progress">Downloading progress:</label>
          <progress value={progress} max="100" />
    
          <em>{statusMessage}</em>
        </div>
      );
    }
    
    export default Page;
  7. createUpload(options)

    master

    Returns an instance of UpChunk and begins uploading the specified File.

    Options:

    • endpoint (string | function, required): URL to upload to. If a function, it receives the file as a parameter and must return a promise resolving to the URL string.
    • file (File, required): The file to upload.
    • headers (Object | function): Headers to include with each PUT request. Can be an object, a function returning an object, or a function returning a promise of an object.
    • chunkSize (integer, default: 30720): Size in kB of chunks. Must be a multiple of 256.
    • maxFileSize (integer): Maximum size of the input file in kb.
    • attempts (integer, default: 5): Number of retries for retriable failures.
    • delayBeforeAttempt (number, default: 1.0): Seconds to wait before retrying a chunk.
    • retryCodes (number[], default: [408, 502, 503, 504]): HTTP Status codes that trigger a retry.
    • method (string, default: PUT): HTTP method (PUT, PATCH, or POST).
    • dynamicChunkSize (boolean, default: false): Whether to scale chunkSize based on network conditions.
    • maxChunkSize (integer, default: 512000): Max size in kB when dynamicChunkSize is true.
    • minChunkSize (integer, default: 256): Min size in kB when dynamicChunkSize is true.
    • useLargeFileWorkaround (boolean, default: false): Fallback to reading entire file into memory for unreliable stream support.
  8. Configure UpChunkOptions

    master

    When calling createUpload(options), you can provide the following configuration keys:

    KeyTypeDescription
    endpointstring or (file?: File) => Promise<string>The destination URL for the upload.
    fileFileThe file object to upload.
    method'PUT' | 'POST' | 'PATCH'The HTTP method to use (defaults to 'PUT').
    headersXhrHeaders | (() => XhrHeaders) | (() => Promise<XhrHeaders>)Custom headers. Can be a static object or a function returning headers.
    maxFileSizenumberMaximum allowed file size in KiB.
    chunkSizenumberThe size of each chunk in KiB. Must be a multiple of 256.
    attemptsnumberMaximum number of retry attempts (defaults to 5).
    delayBeforeAttemptnumberSeconds to wait between retries (defaults to 1).
    retryCodesnumber[]List of HTTP status codes that trigger a retry.
    dynamicChunkSizebooleanIf true, adjusts chunkSize based on upload speed (defaults to false).
    maxChunkSizenumberThe upper limit for chunkSize in KiB.
    minChunkSizenumberThe lower limit for chunkSize in KiB.
    useLargeFileWorkaroundbooleanIf true, falls back to FileReader if ReadableStream fails to read the file.
    const options: UpChunkOptions = {
      endpoint: async (file) => {
        const response = await fetch('/get-url');
        const { url } = await response.json();
        return url;
      },
      file: myFile,
      method: 'POST',
      headers: () => ({ 'Authorization': 'Bearer token' }),
      chunkSize: 512,
      dynamicChunkSize: true,
    };
  9. Initialize an upload with createUpload()

    master

    To start a chunked upload, use the createUpload function. It returns an UpChunk instance that manages the file chunking, HTTP requests, retries, and event dispatching. You must provide an endpoint (a URL string or a function that returns a URL promise) and the file to be uploaded.

    import { createUpload } from '@mux/upchunk';
    
    const upload = createUpload({
      endpoint: 'https://your-upload-endpoint.com',
      file: myFile,
    });
    
    upload.on('success', () => {
      console.log('Upload complete!');
    });
  10. Handle UpChunk events

    master

    The UpChunk instance is an EventTarget. You can subscribe to various lifecycle events using .on(eventName, callback) or .once(eventName, callback). Use .off(eventName, callback) to unsubscribe.

    Available Events:

    • attempt: Dispatched when a chunk upload attempt begins. Detail includes chunkNumber, totalChunks, and chunkSize.
    • attemptFailure: Dispatched when a chunk upload fails but retries are still available. Detail includes message, chunkNumber, attemptsLeft, and response.
    • chunkSuccess: Dispatched when a single chunk is successfully uploaded. Detail includes chunk, chunkSize, attempts, timeInterval, and response.
    • error: Dispatched when the upload fails permanently. Detail includes message, chunk, attempts, and response.
    • offline: Dispatched when the browser detects the user has gone offline.
    • online: Dispatched when the browser detects the user is back online.
    • progress: Dispatched during upload. Detail is a number representing the percentage (0-100).
    • success: Dispatched when the entire file has been successfully uploaded.
    const upload = createUpload(options);
    
    upload.on('progress', (event) => {
      console.log(`Upload progress: ${event.detail}%`);
    });
    
    upload.on('success', () => {
      console.log('Upload finished successfully!');
    });
    
    upload.on('error', (event) => {
      console.error('Upload failed:', event.detail.message);
    });
  11. Control the upload lifecycle (pause, resume, abort)

    master

    You can manually control the state of an active upload using the following methods on the UpChunk instance:

    • pause(): Pauses the upload process. The current chunk may be completed, but no new chunks will be sent.
    • resume(): Resumes a paused upload.
    • abort(): Stops the upload immediately by pausing and aborting the current active XHR request.

    You can also check the current state via the paused and offline getters.

    ```ts
    const upload = createUpload(options);
    
    // To pause
    upload.pause();
    
    // To resume
    upload.resume();
    
    // To stop everything
    upload.abort();
    
    console.log('Is paused:', upload.paused);
    console.log('Is offline:', upload.offline);
  12. UpChunk Instance Events

    master

    Events are emitted as CustomEvent objects. Access event data via the detail key.

    • attempt: Fired before a chunk upload attempt. detail: { chunkNumber: Integer, chunkSize: Integer }
    • attemptFailure: Fired when a chunk attempt fails. detail: { message: String, chunkNumber: Integer, attemptsLeft: Integer }
    • chunkSuccess: Fired when a chunk is successfully uploaded. detail: { chunk: Integer, attempts: Integer, response: XhrResponse }
    • error: Fired when max retries are reached or a fatal error occurs. detail: { message: String, chunkNumber: Integer, attempts: Integer }
    • offline: Fired when the client goes offline.
    • online: Fired when the client goes online.
    • progress: Fired continuously. detail: [0..100] (percentage).
    • success: Fired when the upload completes successfully.