GlusterFS Developer Documentation

cliutils is a Python library designed to create cluster-aware CLI tools for Gluster. It acts as a wrapper around the gluster system:: execute command, allowing a single executable to function as both a node component (running on all peers) and a user-facing CLI (run from a single node).

Key Advantages:

• Node Awareness: The execute_in_peers function merges output with gluster peer status to identify offline nodes, whereas the standard system:: execute simply skips them.
• Error Handling: Provides utility functions like node_output_ok and node_output_notok that return JSON (including success/failure status and node UUID) instead of relying on shell exit codes, which can be unreliable when aggregating output from multiple nodes.
• Simplified CLI: Provides easy handling of CLI arguments and allows for easy iteration over node-specific outputs.

To prevent FUSE inode collisions (where a directory inode number might match a previous file inode number), Gluster distributes the 64-bit inode space among all children of a replicate volume.

Each child receives an equal portion of the available space. For example, if there are 3 children, each gets 1/3 of the space. A lookup on a child returning inode 2 would be scaled to the corresponding value in that child's specific portion of the global space. Even with 1024 children, each child maintains a ~54-bit inode space, which is sufficient for real-world requirements.

GlusterFS 3.7 uses two specific Resource Agents (RAs) to manage High Availability (HA) for the ganesha.nfsd daemon:

1. ganesha_mon: Monitors the ganesha.nfsd daemon. It manages two attributes:

   • ganesha-active: Its presence indicates the daemon is running. Deleting this attribute triggers the failover of the Virtual IP (IPaddr RA) to another node.
   • grace-active: Its presence indicates the system is healthy. Deleting this attribute triggers the transition of surviving nodes into NFS-GRACE mode.

2. ganesha_grace: Monitors the grace-active attribute. If grace-active is deleted, this RA stops and will not restart. It uses a notify action to send DBUS messages to ganesha.nfsd on other cluster nodes, forcing them into NFS-GRACE mode.

GlusterFS uses a mem_pool data structure to reduce the latency caused by frequent system calls for memory allocation and de-allocation (e.g., during stack winding/unwinding in xlators). Instead of calling malloc/free repeatedly, you can allocate a pool of elements once and reuse them.

When the pool is exhausted, the system falls back to standard heap allocations (calloc), but it tracks these as pool_misses to help you tune the pool size.

In Distributed Hash Table (DHT) mode, GlusterFS requires atomicity for two critical design elements to prevent filesystem inconsistencies:

1. Atomicity during namespace operations: Every subvolume must maintain a consistent mapping of (path of directory, gfid). Specifically:

   • Each subvolume must have the same gfid associated with a directory path.
   • A single gfid must not be associated with more than one path in any subvolume.
   • Operations like mkdir, renamedir, rmdir, and directory creation during self-heal must be atomic across all subvolumes.

2. Atomicity during layout modification and reading: Each directory has an independent layout (partially stored on each subvolume) cached in the client's memory. Atomicity ensures that clients read or modify a single, complete layout even when parallel modifications are occurring.

The syncop framework is a coroutines-based, cooperative multi-tasking framework used within Gluster. It relies on two primary components:

• syncenv: An object that provides access to a pool of worker threads. All synctasks execute within a syncenv.
• synctask: A unit of work defined by a pair of function pointers: synctask_fn_t (the 'call') and synctask_cbk_t (the 'callback').

synctasks operate in two modes:

1. Synchronous: The calling thread waits for the synctask to complete.
2. Asynchronous: The calling thread schedules the synctask and continues execution immediately.

The framework guarantees that the callback is executed after the call completes.

The shd (self-heal daemon) is a GlusterFS process responsible for healing files in replicate or disperse volumes.

Key behaviors:

• Deployment: One shd instance runs on every server (brick) node. A single shd can handle multiple volumes if they share the same node.
• Heal Types:
  • Metadata heal: Repairs extended attributes, mode, and permissions.
  • Data heal: Repairs file contents.
  • Entry self-heal: Repairs directory entries.
• Split-brain handling:
  • Files in data/metadata split-brain are not healed automatically.
  • Directories in entry split-brain undergo a conservative merge (the union of entries from the replica pairs).