DOCSEARCHbyAlgolia
Submit Docs

workerpool

repository·master·Indexed 25 days ago

https://github.com/josdejong/workerpool

A library for Node.js and the browser that implements the thread pool pattern to offload CPU-intensive tasks. It allows for dynamic offloading of functions and the management of dedicated workers via external scripts, Data URLs, or integration with build tools like Vite, Webpack 5, and esbuild. Version 10.0.3.

Tokens
4.9K
Snippets
13
Records
28
Agent score
81%

What's inside workerpool

  1. How workerpool works: The Thread Pool Pattern

    master

    workerpool implements a thread pool pattern to handle concurrency.

    1. The Pool: A collection of workers is created to execute tasks.
    2. The Queue: When new tasks are submitted, they are placed in a queue.
    3. Execution: A worker executes one task at a time. Once a task is finished, the worker picks the next task from the queue.
    4. Concurrency: This allows CPU-intensive tasks to be offloaded from the main event loop (preventing the UI from hanging in browsers or the server from becoming unresponsive in Node.js) by running them in isolated processes or threads via Web Workers, child_processes, or worker_threads.

  2. How esbuild inlines workers via Data URLs

    master

    This example demonstrates a build-time pattern to convert worker files into Data URLs using esbuild and an inline-worker import pattern:

    1. Worker Transformation: The worker file (e.g., src/fib.js) is processed so that its default export is a string containing the Data URL: export default 'data:application/javascript;base64,...';.
    2. Main Script Import: In the main application (src/main.js), you import the worker using a special prefix:
      import workerDataUrl from 'inline-worker:./fib.js';
    3. Bundling: During the build process, esbuild replaces the import with the actual Data URL string, resulting in a single bundled file (dist/main.js) that contains both the main logic and the worker code.

  3. Emit events from workers using workerEmit()

    master

    Workers can send data back to the main thread while a task is still running using workerEmit(payload). This is useful for progress updates or status changes.

    Requirements:

    • workerEmit only works inside a worker.
    • It only works during the execution of a task.
    • On the main thread, listen for these events using the on option in Pool.exec().

  4. Configure workerpool for Webpack 5

    master

    Because Webpack 5 no longer provides built-in polyfills for Node.js dependencies, you must adapt your configuration to use workerpool correctly.

    1. Update Webpack Aliases: In your webpack.config.cjs, set os, child_process, and worker_threads to false in the resolve.alias section to prevent Webpack from attempting to polyfill these Node.js modules.
    2. Use WorkerUrlPlugin: Add WorkerUrlPlugin from the worker-url/plugin package to your plugins array. This allows you to obtain the URL of a worker instead of a direct worker instance.
    // webpack.config.cjs
const path = require("path");
const WorkerUrlPlugin = require('worker-url/plugin');

module.exports = {
  mode: "development",
  entry: path.resolve(__dirname, "./src/index.tsx"),
  output: {
    filename: "[name].[hash:8].js",
    path: path.resolve(__dirname, "./dist"),
  },
  resolve: {
    extensions: ['.js', '.jsx', '.ts', '.tsx'],
    // ! webpack5 no longer provides built-in polyfills for Node.js dependencies.
    alias: {
       "os": false,
       "child_process": false,
       "worker_threads": false
     }
  },
  plugins: [
    // add this
    new WorkerUrlPlugin(),
  ],
};

  5. Embed a worker using a Data URL

    master

    In browser environments, you can embed the worker code directly into your main application by passing a Data URL to workerpool.pool(). This avoids the need to host a separate worker script file.

    To achieve this, you must provide a Data URL string (e.g., 'data:application/javascript;base64,...') as the script argument.

    Note: This technique is only compatible with browser environments and will not work in Node.js.

  6. Initialize a workerpool using worker-url in Webpack 5

    master

    In a Webpack 5 environment, you cannot pass a worker instance directly to workerpool.pool(). Instead, use the worker-url package to resolve the worker's URL.

    Import WorkerUrl and instantiate it with the URL of your worker file (using import.meta.url). Then, pass the string representation of that URL to workerpool.pool().

    // App.tsx
import { WorkerUrl } from 'worker-url';
import workerpool from 'workerpool';

// worker-url is a webpack plugin that is used to obtain the URL of a worker instead of a worker instance.
const WorkerURL = new WorkerUrl(new URL('./worker/worker.ts', import.meta.url));

const pool = workerpool.pool(WorkerURL.toString(), {
  maxWorkers: 3,
});

  7. Use workerpool with Vite

    master

    When using workerpool in a Vite project, you must adapt how Web Workers are loaded and configured to ensure compatibility between development and production modes.

    1. Import the Worker URL: Use Vite's special query suffixes ?url&worker to import the worker file as a URL.
    2. Configure Worker Type: Vite uses module workers in development, which can cause issues if not explicitly handled. To ensure stability, set the workerOpts.type to "module" in development and leave it undefined (to allow Vite's default behavior) in production.
    3. Initialize the Pool: Pass the imported URL to workerpool.pool() and provide the workerOpts configuration.
    import WorkerURL from './worker/worker?url&worker'
const pool = workerpool.pool(WorkerURL, {
    maxWorkers: 3,
    workerOpts: {
        // By default, Vite uses a module worker in dev mode, which can cause your application to fail. 
        // Therefore, we need to use a module worker in dev mode and a classic worker in prod mode.
        type: import.meta.env.PROD ? undefined : "module"
    }
});

  8. Use dedicated workers with external scripts

    master

    For more complex logic, create a dedicated worker script using workerpool.worker(). This script registers functions that can then be called by a pool in your main application.

    1. Create the worker script

    In your worker file (e.g., myWorker.js), register the functions you want to expose:

    const workerpool = require('workerpool');

function fibonacci(n) {
  if (n < 2) return n;
  return fibonacci(n - 2) + fibonacci(n - 1);
}

workerpool.worker({
  fibonacci: fibonacci,
});

    2. Use the worker in your application

    You can interact with the dedicated worker in two ways:

    Via pool.exec()

    Pass the function name as a string and the arguments in an array:

    const pool = workerpool.pool(__dirname + '/myWorker.js');

pool.exec('fibonacci', [10])
  .then(result => console.log(result));

    Via pool.proxy()

    Create a promise-based proxy that allows you to call the registered functions as if they were local methods:

    const pool = workerpool.pool(__dirname + '/myWorker.js');

pool.proxy()
  .then(worker => {
    return worker.fibonacci(10);
  })
  .then(result => console.log(result));
    const workerpool = require('workerpool');

// create a worker and register public functions
workerpool.worker({
  fibonacci: fibonacci,
});

  9. Run the embedded worker example

    master

    To run the embedded worker demonstration, follow these steps to install dependencies, build the bundles, and launch the application:

    1. Install dependencies:
      npm install
    2. Build the embedded worker (dist/worker.embedded.js) and the application bundle (dist/app.bundle.js):
      npm run build
    3. Open app.html in your web browser.
    npm install
npm run build

  11. Embed worker code using Data URLs in the browser

    master

    In browser environments, you can embed worker code directly into your main application by passing a Data URL to the script argument of workerpool.pool(script). This avoids the need to host separate worker files and allows you to bundle the worker code alongside your main application bundle.

    Note: This technique is specific to the browser and will not work in Node.js.

  12. Load workerpool in different environments

    master

    Depending on your environment, load workerpool using the following methods:

    Node.js

    Use require in both your main application and your worker scripts:

    const workerpool = require('workerpool');

    Browser (Main Thread)

    Include the script tag in your HTML:

    <script src="workerpool.js"></script>

    Browser (Web Worker)

    Use importScripts inside your web worker script:

    importScripts('workerpool.js');

    Note: If you are using React or Webpack 5, additional configuration is required.