Turbowatch

repository·main·Indexed 21 days ago

https://github.com/gajus/turbowatch

A high-performance file change detector and task orchestrator for Node.js environments, specifically designed for complex monorepos. It provides advanced orchestration features including debouncing, retries, interruptible workflows, and graceful teardown. Turbowatch supports both CLI and programmatic modes, utilizes zx for shell command execution, and allows for custom file-watching backends.

Tokens
8.6K
Snippets
29
Records
34
Agent score
76%

What's inside turbowatch

  1. How persistent tasks work

    main

    In Turbowatch, tasks can be categorized as either Persistent or Non-Persistent.

    • Persistent Tasks: These are long-running processes that do not exit on their own, such as a development server (next dev) or a process in --watch mode. You must set persistent: true for these.
    • Non-Persistent Tasks: These are tasks that run to completion (e.g., a build script like tsc).

    Behavioral Difference: If a task is marked as persistent: false (the default), Turbowatch will ignore new FileChangeEvents if the current onChange routine is still executing and interruptible is set to false. This prevents overlapping executions of build scripts.

  2. Integrate Turbowatch with Turborepo

    main

    Turbowatch is designed to work alongside Turborepo. Since Turborepo does not currently support watch mode, you should use Turbowatch to handle the watching logic within a Turborepo task.

    To integrate them:

    1. Define a task in your package.json as persistent and disable cache.
    2. Run the task using the --parallel flag in Turborepo.

    Note on Dependencies: Avoid using Turborepo's dependsOn with Turbowatch, as it can produce undesirable effects. Instead, use Turbowatch expressions to identify when dependencies update. If your builds fail on the first attempt because Turbowatch is not aware of the Turborepo dependency graph, configure Turbowatch to watch node_modules; it will automatically retry failing builds once the dependencies are updated.

    // package.json
    "dev": {
      "cache": false,
      "persistent": true
    }
    
    // CLI
    turbo run dev --parallel
  3. Enable and pretty-print logs

    main

    Turbowatch uses the Roarr logger. To see logs in your terminal:

    1. Set the ROARR_LOG=true environment variable.
    2. (Optional) Pipe the output to @roarr/cli for pretty-printing.
    ROARR_LOG=true turbowatch | roarr
  4. Gracefully terminate Turbowatch

    main

    If you are not using the turbowatch CLI to run your script, you can manually trigger a graceful shutdown.

    1. Using shutdown: The watch function returns a TurbowatchController which contains a shutdown method. Calling this propagates an abort signal to all onChange handlers and sends a SIGTERM to processes initiated via spawn.
    2. Using AbortController: You can pass an AbortController into the watch configuration. Calling abortController.abort() will initiate the shutdown process.
    // Using shutdown()
    const { shutdown } = await watch({
      project: __dirname,
      triggers: [
        {
          name: 'test',
          expression: ['match', '*', 'basename'],
          onChange: async ({ spawn }) => {
            await spawn`sleep 60`;
          },
        }
      ],
    });
    
    process.once('SIGINT', () => {
      void shutdown();
    });
    
    // Using AbortController
    const abortController = new AbortController();
    
    void watch({
      abortController,
      project: __dirname,
      triggers: [
        {
          name: 'test',
          expression: ['match', '*', 'basename'],
          onChange: async ({ spawn }) => {
            await spawn`sleep 60`;
          },
        }
      ],
    });
    
    void abortController.abort();
  5. Handle `AbortSignal` in interruptible workflows

    main

    When a trigger is marked as interruptible: true, a new file change will trigger an abort signal. To ensure your spawned processes are actually killed when this happens, you must handle the abortSignal provided in the onChange handler.

    If you are using zx, Turbowatch already binds the AbortSignal to spawn. If you are using other tools, you may need to manually listen for the abort event on the signal and call .kill() on your process.

    import { type ProcessPromise } from 'zx';
    
    const interrupt = async (
      processPromise: ProcessPromise,
      abortSignal: AbortSignal,
    ) => {
      let aborted = false;
    
      const kill = () => {
        aborted = true;
        processPromise.kill();
      };
    
      abortSignal.addEventListener('abort', kill, { once: true });
    
      try {
        await processPromise;
      } catch (error) {
        if (!aborted) {
          console.log(error);
        }
      }
    
      abortSignal.removeEventListener('abort', kill);
    };
    
    export default watch({
      project: __dirname,
      triggers: [
        {
          expression: ['match', '*.ts', 'basename'],
          interruptible: true,
          name: 'sleep',
          onChange: async ({ abortSignal }) => {
            await interrupt($`sleep 30`, abortSignal);
          },
        },
      ],
    });
  6. Install and run Turbowatch

    main

    Turbowatch is a fast file change detector and task orchestrator for Node.js. You can install it via npm and run it by providing a configuration file (e.g., turbowatch.ts) to the turbowatch executable.

    Note: This project is deprecated. Consider using turbo watch or Tilt for new projects.

    npm install turbowatch
    
    # Create a configuration file
    cat > turbowatch.ts <<'EOD'
    import { defineConfig } from 'turbowatch';
    
    export default defineConfig({
      project: __dirname,
      triggers: [
        {
          expression: ['match', '*.ts', 'basename'],
          name: 'build',
          onChange: async ({ spawn }) => {
            await spawn`tsc`;
          },
        },
      ],
    });
    EOD
    
    # Run turbowatch
    npm exec turbowatch ./turbowatch.ts
  7. Wrap HMR services like Next.js in Turbowatch

    main

    While Turbowatch is not a replacement for Hot Module Replacement (HMR) services (like Next.js), you can wrap them in a Turbowatch trigger to maintain a consistent watch workflow.

    To prevent Turbowatch from interrupting the HMR process when files change, set interruptible: false. Additionally, set persistent: true to ensure Turbowatch correctly handles the long-running process and provides appropriate logging/warnings regarding configuration compatibility.

    void watch({
      project: __dirname,
      triggers: [
        {
          expression: ['dirname', __dirname],
          // Marking this routine as non-interruptible will ensure that
          // next dev is not restarted when file changes are detected.
          interruptible: false,
          name: 'start-server',
          onChange: async ({ spawn }) => {
            await spawn`next dev`;
          },
          // Enabling this option modifies what Turbowatch logs and warns
          // you if your configuration is incompatible with persistent tasks.
          persistent: true,
        },
      ],
    });
  8. Configure retries for failing triggers

    main

    You can configure automatic retries for a trigger by providing a retry object. This is useful for flaky tasks or tasks that depend on external resources.

    Retry Configuration Options:

    • factor (number): The exponential factor to use. Default is 2.
    • maxTimeout (number): The maximum number of milliseconds between two retries. Default is 30000.
    • minTimeout (number): The number of milliseconds before starting the first retry. Default is 1000.
    • retries (number): The maximum amount of times to retry the operation. Default is 3.
  9. Configure Turbowatch scripts

    main

    Turbowatch configuration scripts must export a Watcher function. This function is used by the CLI to initialize the watch process.

    When creating a script, ensure you export a configuration object that includes a Watcher property which is a function. This function is expected to return a TurbowatchController (or be part of the configuration input used by the watch API).

    // Example structure of a turbowatch script
    export default {
      Watcher: async () => {
        // ... implementation
      }
    };
  10. Use Watchman expressions for file matching

    main

    Turbowatch uses an Expression type based on Watchman expressions to filter file changes. Supported operators include:

    • ['allof', ...Expression[]]: True if all grouped expressions are true.
    • ['anyof', ...Expression[]]: True if any grouped expression is true.
    • ['not', Expression]: Inverts the sub-expression.
    • ['match' | 'imatch', string]: Matches against the file basename.
    • ['match' | 'imatch', string, 'basename' | 'wholename']: Matches against either the basename or the full path.
    • ['dirname' | 'idirname', string]: True if the file has a matching parent directory.
    // Example expressions
    const matchAllTs = ['match', '*.ts'];
    const matchAnyJsOrTs = ['anyof', ['match', '*.js'], ['match', '*.ts']];
    const notMatchTxt = ['not', ['match', '*.txt']];
  11. Configure log output throttling

    main

    By default, Turbowatch throttles log output to at most once per second per task to prevent interleaved logs from multiple processes from becoming unreadable.

    To disable this throttling and see all logs immediately (which may result in interleaved output), set the throttleOutput option with a delay of 0 in the watch configuration.

    // Example of disabling throttling
    void watch({
      throttleOutput: { delay: 0 },
      // ... other config
    });
  12. Watch `node_modules` and source directories

    main

    When working in monorepos or complex workspaces, you can watch both node_modules (for dependency changes) and src (for source changes). This example assumes source is in src and build output is in dist.

    import { watch } from 'turbowatch';
    
    void watch({
      project: path.resolve(__dirname, '../..'),
      triggers: [
        {
          expression: [
            'anyof',
            [
              'allof',
              ['dirname', 'node_modules'],
              ['dirname', 'dist'],
              ['match', '*', 'basename'],
            ],
            [
              'allof',
              ['not', ['dirname', 'node_modules']],
              ['dirname', 'src'],
              ['match', '*', 'basename'],
            ],
          ],
          name: 'build',
          onChange: async ({ spawn }) => {
            return spawn`pnpm run build`;
          },
        },
      ],
    });