DOCSEARCHbyAlgolia
Submit Docs

Watchman

repository·main·Indexed 12 days ago

https://github.com/facebook/watchman

A file watching service that monitors files and records changes, capable of triggering actions like asset rebuilding. It supports Windows, macOS, and Linux, and provides client language bindings for Python, Rust, and JavaScript (via fb-watchman), as well as Ruby (via RubyWatchman). The service uses a binary protocol (BSER) for high-performance communication and includes a C99 implementation of the Adaptive Radix Tree (libart) for efficient key-value storage.

Tokens
49.6K
Snippets
197
Records
268
Agent score
95%

What's inside Watchman

  1. What is libart and its core features

    main

    libart is a C99 implementation of the Adaptive Radix Tree (ART). It functions similarly to a traditional radix tree but optimizes space by dynamically changing node sizes (using 4, 16, 48, and 256 node sizes). This approach ensures overhead is no more than 52 bytes per key, though it is typically much lower.

    Key capabilities include:

    • O(k) operations: Often faster than hash tables due to better cache locality and avoiding O(k) hash function overhead.
    • Range queries: Supports Minimum and Maximum value lookups.
    • Prefix support: Provides prefix compression and prefix-based iteration.
    • Ordered iteration: Supports iterating through keys in order.

  3. Supported platforms and client languages for Watchman

    main

    Watchman provides official support for the following:

    Operating Systems:

    • Windows
    • macOS
    • Linux (recent Ubuntu and Fedora releases)

    Client Language Bindings:

    • Python
    • Rust
    • JavaScript

    Note that community-maintained support is available for Homebrew, FreeBSD, and Solaris.

  4. How Watchman locates a project root

    main

    Watchman determines the project root using a specific search hierarchy:

    1. Search for .watchmanconfig: It first looks for a .watchmanconfig file in the requested directory or its parents. If found, that directory is the project root.
    2. Search for root_files: If no .watchmanconfig is found, Watchman uses the list of files defined in the global root_files configuration (which defaults to common version control markers like .git, .hg, etc.). It searches upwards for any of these files.
    3. Fallback: If no root_files are found, Watchman will watch the requested directory itself, unless the enforce_root_files configuration is set to true.

    Configuration Keys:

    • root_files: A list of filenames that identify a project root.
    • enforce_root_files: If true, the watch-project command will fail if a valid project root (via .watchmanconfig or root_files) cannot be found.

  5. Understand the Watchman Protocol and PDU encoding

    main

    The Watchman service uses a request-response protocol over the Unix domain socket. Protocol Data Units (PDUs) can be encoded using either JSON or BSER.

    • JSON Encoding: Requests and responses are represented as single-line, compact JSON. A newline (\n) character is used to signal the end of a PDU.
      • Client Requests: Must be formatted as a JSON array.
      • Server Responses: Are always formatted as a JSON object.
    • BSER Encoding: A compact binary serialization format. It supports the same data types as JSON but is more efficient and supports non-UTF-8 strings. If a client sends a request using BSER, the server will respond using BSER.

  6. How BSER strings and encoding work

    main

    Unlike JSON, BSER strings are transmitted as raw binary strings without a defined encoding. This allows Watchman to handle filenames that may not have a valid encoding.

    Important details:

    • Keys: Keys in objects defined by Watchman commands are always ASCII. In general, keys in objects are expected to be UTF-8.
    • Values: Filenames and other data are treated as binary strings to ensure compatibility with filesystem APIs.

  7. Configure filesystem settling and VCS awareness

    main

    Watchman uses 'settling' to avoid sending notifications during transient filesystem activity.

    Version Control System (VCS) Awareness: By default, if a root is a VCS directory (detected by .git/index.lock or .hg/wlock), Watchman holds notifications until outstanding VCS operations complete. This prevents unnecessary work during operations like a rebase.

    Disabling VCS Deferral: If you need to observe the creation of control files (like locks) at the start of a VCS operation, set defer_vcs to false in your subscription object.

    $ watchman -j -p <<-EOT
["subscribe", "/path/to/root", "mysubscriptionname", {
  "expression": ["allof",
    ["type", "f"],
    ["not", "empty"],
    ["suffix", "php"]
  ],
  "defer_vcs": false,
  "fields": ["name"]
}]
EOT

  8. Use defer and drop to manage notification streams during application states

    main

    For complex integrations, you can signal the beginning and end of high-activity periods (like a Git update) using state-enter and state-leave. The subscribe command allows you to control how notifications are handled during these states using defer or drop.

    defer

    Specifies a list of state names to pause notifications. When a matching state is entered via state-enter, Watchman sends a notification containing the state-enter metadata and then holds all files payloads. Once the state is vacated via state-leave (or the client disconnects), Watchman sends a state-leave notification and flushes all accumulated notifications that occurred during the state.

    drop

    Specifies a list of state names to discard notifications. It behaves like defer, but instead of buffering notifications, it 'fast-forwards' the stream to the clock of the state-leave command. This effectively suppresses all notifications generated between the state-enter and state-leave events.

    Best Practice: If using multiple overlapping states, it is recommended to use drop for all states and then issue manual queries with since terms bounded by the clock fields from the subscription state PDUs to ensure no data is missed.

    // Example: Deferring notifications for a specific state
["subscribe", "/path/to/root", "mysubscriptionname", {
  "defer": ["mystatename"],
  "fields": ["name"]
}]

// Example: Dropping notifications for a specific state
["subscribe", "/path/to/root", "mysubscriptionname", {
  "drop": ["mystatename"],
  "fields": ["name"]
}]

  9. Use the 'empty' expression term

    main

    The empty expression term evaluates to true if a file meets all of the following criteria:

    • The file exists.
    • The file has a size of 0.
    • The file is either a regular file or a directory.

    In Watchman query expressions, you can use it as a string literal "empty" or within brackets ["empty"] depending on the specific expression syntax being used.

    "empty"

  10. Constraints and limitations of flush-subscriptions

    main

    When using flush-subscriptions, keep the following constraints in mind:

    • Session Scope: The command can only flush subscriptions associated with the current session.
    • Project Scope: While a single session can subscribe to multiple projects, a single flush-subscriptions call can only flush updates for one project at a time.

  11. How Watchman achieves query synchronization

    main

    Watchman provides a model of query consistency where queries sequenced-after a filesystem modification reflect those modifications in their results.

    To handle the asynchronous and deferred nature of filesystem notification APIs, Watchman generally uses cookie files. The process works as follows:

    1. Watchman generates a file with a unique filename in the watched directory.
    2. Watchman waits to receive the corresponding notification for that cookie file.
    3. Once the notification is received, Watchman assumes it has observed every prior change event and unblocks the query.

    Note on Platform Reliability:

    • Linux (inotify): Generally works well, but historical kernel regressions (e.g., Linux kernel 5.3) have caused stale data by reporting events before caches were flushed.
    • macOS (FSEvents): Synchronization is less reliable. FSEvents may coalesce events, and even after calling FSEventStreamFlushSync, there is no guarantee that all events pending in the kernel have been processed. This can lead to queries unblocking too early and returning stale data, though subsequent queries typically catch up.