Meteor-Files

Meteor-Files is a library for Meteor.js designed for robust file upload and management. Key capabilities include:

• Advanced Uploads: Supports HTTP or DDP transports, optimized RAM usage for large files, pause/resume functionality, auto-pause on connection loss, and parallel multi-stream async uploads. It also supports non-Latin file names.
• Storage Flexibility: Integrates with third-party storage like AWS S3, Dropbox, Google Cloud Storage, and Google Drive. It also supports GridFS (both via GridFSBucket and the legacy gridfs-stream).
• File Serving: Provides custom download routes with support for progressive (chunked) downloads, correct mime-type and Content-Range headers, and proper 206 and 416 HTTP responses following RFC 2616.
• File System Integration: Automatically writes files to the file system and a special Collection. Users can control path, collection name, schema, chunk size, and naming functions.
• Cordova Support: Works in Cordova apps using FileReader for uploading and reading files.
• Post-processing: Supports file subversions like thumbnails, audio/video format conversions, and revisions.

You can use lifecycle hooks within the FilesCollection configuration to implement fine-grained authorization logic.

onBeforeUpload

Used to validate a file before it is written to the filesystem. If the function returns false or a string, the upload is rejected. This is useful for checking file size, extension, or user roles.

onBeforeRemove

Used to prevent unauthorized file deletions. If the function returns false, the removal is aborted.

onAfterUpload

Triggered immediately after a file is written to the filesystem. This is an isomorphic event. It is highly recommended to use this hook on the server to verify the actual file content (e.g., using mmmagic to check the real MIME type) to prevent extension/MIME-type spoofing.

// Example: Authorization via onBeforeUpload
const imagesCollection = new FilesCollection({
  collectionName: 'images',
  allowClientCode: true,
  async onBeforeUpload() {
    if (this.userId) {
      const user = await this.userAsync();
      if (user.profile.role === 'admin') {
        return true;
      }
    }
    return 'Not enough rights to upload a file!';
  }
});

// Example: Security via onAfterUpload (Server-side MIME check)
const imagesCollection = new FilesCollection({
  collectionName: 'images',
  onAfterUpload(file) {
    if (Meteor.isServer) {
      const { Magic, MAGIC_MIME_TYPE } = require('mmmagic');
      const magic = new Magic(MAGIC_MIME_TYPE);
      magic.detectFile(file.path, Meteor.bindEnvironment((err, mimeType) => {
        if (err || !mimeType.includes('image')) {
          this.remove(file._id);
        }
      }));
    }
  }
});

If you call insert() with the second argument set to false (disabling autoStart), you can use the .pipe() method to transform file data before it is uploaded.

Each .pipe(fn) call passes the current chunk (as a Base64 string/DataURL) to the function fn. The function must return a Base64 string. This is useful for client-side encryption or compression.

Note: If you modify the data, you may need to manually update the file name, extension, or mime-type if the transformation changes them.

import { imagesCollection } from '/imports/collections/images.js';

const encrypt = function encrypt(data) {
  // data is a Base64 string
  return encryptAndReturnAsBase64(data);
};

const zip = function zip(data) {
  return zipAndReturnAsBase64(data);
};

// Usage: pipe through multiple functions
imagesCollection.insert({
  file: fileElement,
  chunkSize: 'dynamic'
}, false) // false = do not auto-start
  .pipe(encrypt)
  .pipe(zip)
  .start();

The ostrio-meteor-files package supports different transport mechanisms for uploading files, each with different performance characteristics and use cases:

DDP (WebSockets)

Uses the Meteor Distributed Data Protocol via SockJS. It is the default way Meteor communicates with the server.

• Pros: Persistent connection; standard Meteor communication.
• Cons: Synchronous; high overhead due to EJSON; blocks other DDP requests (methods, subscriptions) during transfer.

HTTP (TCP/IP)

Standard web protocol. Recommended to use with HTTP/2, SSL/TLS, and SSL session caching for optimal performance.

• Pros: Asynchronous and simultaneous requests; no data-processing/encoding overhead (sends data as-is); high speed.
• Cons: Originally designed for text-based data (hypertext) rather than binary files.

RTC Data Channel (UDP)

Note: This is currently only available in the webrtc-data-channel branch and is in testing mode.

• Pros: Single socket; direct client-to-server tunneled connection; pure binary support; native mobile support; uses UDP.
• Cons: No support in mobile browsers; chunk size is limited to 64KB.

To fully offload storage to GCS, implement the following hooks in your FilesCollection configuration:

1. Uploading to GCS (onAfterUpload)

Use the onAfterUpload callback to upload the local file to your bucket. It is recommended to store the GCS path in the file version's metadata (e.g., versions[version].meta.pipePath) so it can be retrieved later. After a successful upload, use this.unlink() to remove the original file from the local filesystem.

2. Streaming from GCS (interceptDownload)

Use the interceptDownload hook to check if a file has a pipePath (indicating it is stored in GCS). If it does, create a readable stream from the GCS bucket using bucket.file(path).createReadStream() and pass it to this.serve(). Return true to signal that the download was handled.

3. Deleting from GCS

To ensure files are removed from GCS when deleted from the collection, intercept the remove method of your FilesCollection to call bucket.file(path).delete() for each version's pipePath before calling the original removal method.

Collections.files = new FilesCollection({
  onAfterUpload(fileRef) {
    _.each(fileRef.versions, (vRef, version) => {
      const filePath = 'files/' + (Random.id()) + '-' + version + '.' + fileRef.extension;
      const options = { destination: filePath, resumable: true };

      bucket.upload(fileRef.path, options, (error, file) => {
        if (!error) {
          // Update metadata with the GCS path
          this.collection.update(
            { _id: fileRef._id },
            { $set: { [`versions.${version}.meta.pipePath`]: filePath } },
            () => this.unlink(fileRef, version)
          );
        }
      });
    });
  },
  interceptDownload(http, fileRef, version) {
    const path = fileRef.versions[version]?.meta?.pipePath;
    if (path) {
      const remoteReadStream = getReadableStream(http, path, fileRef.versions[version]);
      this.serve(http, fileRef, fileRef.versions[version], version, remoteReadStream);
      return true;
    }
    return false;
  }
});

The Meteor-Files API is built around three primary classes:

1. FilesCollection: The main entry point used to initialize a file collection. It can be configured with schemas (e.g., SimpleSchema), access rules (allow/deny), and lifecycle hooks (onBeforeUpload, onBeforeRemove).
2. FileCursor: Returned by .findOne(). It represents a single file document and provides methods to interact with it:
   • link(): Returns a downloadable URL.
   • get(property): Returns the document as a plain object.
   • removeAsync(): Removes the document (returns a Promise<number>).
   • fetchAsync(): Resolves to the document as a plain object in an Array.
   • with(): Returns a reactive version of the cursor.
3. FilesCursor: Returned by .find(). It represents a set of matching documents and provides methods for bulk operations:
   • fetchAsync(): Returns all matching documents as an Array.
   • countAsync(): Returns the number of matching documents.
   • removeAsync(): Removes all matching documents.
   • forEachAsync(callback, context): Iterates over matching documents.
   • eachAsync(): Resolves to an Array of FileCursor instances for each document.

To upload a file, call insert() on your FilesCollection instance. You can pass an options object to configure the upload behavior.

Basic Usage

imagesCollection.insert({
  file: fileElement, // The File object from an input
  chunkSize: 'dynamic'
});

Uploading Base64 strings

If you are uploading a Base64 string instead of a File object, you must set isBase64: true and provide a fileName.

As DataURL:

imagesCollection.insert({
  file: 'data:image/png;base64,ivbor...', 
  isBase64: true, 
  fileName: 'pic.png'
});

As plain Base64:

imagesCollection.insert({
  file: 'base64str...', 
  isBase64: true, 
  fileName: 'pic.png', 
  type: 'image/png'
});

import { FilesCollection } from 'meteor/ostrio:files';
const imagesCollection = new FilesCollection({collectionName: 'images'});

// Standard file upload
imagesCollection.insert({
  file: e.currentTarget.files[0],
  chunkSize: 'dynamic'
});

// Base64 upload
imagesCollection.insert({
  file: 'base64str...', 
  isBase64: true, 
  fileName: 'pic.png', 
  type: 'image/png'
});

To generate thumbnails after file uploads, you must install a CLI image processing tool and its corresponding NPM wrapper.

1. Install CLI Tools

Install either GraphicsMagick (recommended for performance) or ImageMagick on your host machine.

macOS (Homebrew):

brew install graphicsmagick
# or
brew install imagemagick

2. Install NPM Packages

Install the ostrio:files Meteor package, along with gm (GraphicsMagick wrapper) and fs-extra.

meteor add ostrio:files
meteor npm install --save gm fs-extra

3. Configure ImageMagick usage (Optional)

If you are using ImageMagick instead of GraphicsMagick, you must explicitly tell the gm package to use the ImageMagick subclass:

const gm = require('gm');
const im = gm.subClass({ imageMagick: true });

brew install graphicsmagick
# or for ImageMagick:
# brew install imagemagick

meteor add ostrio:files
meteor npm install --save gm fs-extra