Schema Sync for Directus

To install the extension in a Docker-based Directus setup, follow these steps:

1. Create a schema-sync folder at your project root.
2. Update your Dockerfile to install the package and copy the folder:

   RUN pnpm install directus-extension-schema-sync
   COPY ./schema-sync ./schema-sync

3. Update your docker-compose.yaml to mount the folder as a volume so you can edit config files locally:

   volumes:
     - ./schema-sync:/directus/schema-sync

4. Rebuild and run your containers.
5. Run the installation command from your host to set up database columns and the config folder. Use --force to ensure the volume is correctly initialized:

   docker compose exec -it <service_name> npx directus schema-sync install --force

   (Replace <service_name> with your actual Directus service name, e.g., my-directus).

docker compose exec -it my-directus npx directus schema-sync install --force

The ExportManager class is the central orchestrator for the schema sync process. It follows a three-step lifecycle:

1. Add Exporters: Register schema exporters or collection-specific exporters using addExporter or addCollectionExporter.
2. Load/Import: Execute the import process using loadAll(merge?) to synchronize the current Directus state with the stored schema/data.
3. Attach Watchers: Enable automatic exports by calling attachAllWatchers. This hooks into Directus actions (create, update, delete) for specified collections to trigger an export and metadata update whenever changes occur.

Note: addCollectionExporter requires an IGetItemsService to fetch data for the collections being watched.

// Conceptual workflow
const manager = new ExportManager(logger, clearSchemaCache);

// 1. Add exporters
manager.addCollectionExporter(collectionConfig, getItemsService);

// 2. Load/Import
await manager.loadAll(true);

// 3. Start watching
manager.attachAllWatchers(actionHandler, updateMeta);

When the groupBy option is provided in CollectionExporterOptions, the exporter switches from a single-file mode to a directory-based mode.

Directory Structure

Instead of a single file at ${ExportHelper.dataDir}/${collectionName}.json, the exporter creates:

1. A directory: ${ExportHelper.dataDir}/${collectionName}/
2. A manifest file: ${ExportHelper.dataDir}/${collectionName}/${prefix || '_'}_${collectionName}.json (contains metadata like count and groupedBy).
3. Group files: ${ExportHelper.dataDir}/${collectionName}/${prefix || '_'}_${group_key}.json (contains the actual items for that group).

Group Key Generation

Group filenames are generated by joining the values of the groupBy fields with -- and replacing spaces with underscores: item[field1]--item[field2] (with spaces replaced by _).

Loading Grouped Data

When calling .load(), the exporter detects the partial: true flag in the manifest file and automatically invokes loadGroupedItems. This method iterates through all files in the collection directory, stitches the items back together, and then proceeds with the standard loadItems logic.

Schema Sync works as a Directus extension hook that monitors changes and synchronizes the database state with filesystem-based schema and data files.

The Synchronization Lifecycle

1. Initialization: On startup, the extension checks the SCHEMA_SYNC environment variable.
2. Import Phase (if enabled):
   • It retrieves the current export metadata (hash and timestamp).
   • It checks if the database is currently locked or if the hash matches the existing state. If no changes are detected, it skips the update.
   • If changes are detected, it loads the schema and data from the files, applying them to the database.
   • It triggers a reload of Directus modules (like Flows) to ensure the system recognizes the new schema.
3. Export/Watch Phase (if enabled):
   • The extension attaches watchers to specific Directus entities: collections, fields, and relations.
   • When a change is detected in these entities, the ExportManager triggers an export to update the files on disk.
   • The export metadata (hash/timestamp) is updated to reflect the new state.

Key Abstractions

• ExportManager: Orchestrates the process of adding exporters (like SchemaExporter) and collection exporters to manage what gets written to disk.
• SchemaExporter: Specifically handles the translation of Directus schema structures into files.
• UpdateManager: Manages database locks and ensures that schema updates are committed atomically to prevent corruption during concurrent operations.

The load(merge?: boolean) method determines how data from the JSON files is synchronized with the existing database records.

Merge Mode (load(true))

• Updates: If an item in the JSON matches an existing item in the database (based on the primary key or getKey), the exporter calculates a diff and updates only the changed fields.
• Inserts: New items found in the JSON that do not exist in the database are created.
• No Deletions: Existing items in the database that are not present in the JSON files are left untouched.

Replace Mode (load(false) or default)

• Updates: Same as merge mode.
• Inserts: Same as merge mode.
• Deletions: After processing imports, the exporter identifies all items currently in the database that were not part of the JSON import set and deletes them.

Note: When using Replace mode, the load() method returns a finishUp function (a Promise) that must be awaited to complete the deletion phase.

By default, Directus masks passwords during export. To sync users with their hashed passwords, add onExport and onImport hooks to your schema-sync/directus_config.js file within the directus_users object. This uses the underlying knex instance to fetch the raw hashed password from the database during export and update it during import.

Warning: Be extremely careful when committing hashed passwords to your repository.

// Inside directus_config.js
directus_users: {
	onExport: async (item, itemsSrv) => {
		if (item.password && item.password === '**********') {
			const user = await itemsSrv.knex.select('password').from('directus_users').where('id', item.id).first();
			if (user) {
				item.password = user.password;
			}
		}
		return item;
	},
	onImport: async (item, itemsSrv) => {
		if (item.password && item.password.startsWith('$argon')) {
			await itemsSrv.knex('directus_users').update('password', item.password).where('id', item.id);
			item.password = '**********';
		}
		return item;
	},
}

The behavior of the directus-extension-schema-sync extension is controlled by several environment variables. These determine whether the extension performs imports, exports, or both, and how data is handled.

Core Modes

• SCHEMA_SYNC: Determines the operational mode:
  • EXPORT: Only exports schema and data from the database to files.
  • IMPORT: Only imports schema and data from files to the database.
  • BOTH: Performs both export and import operations.
• SCHEMA_SYNC_DATA_ONLY: If set to true, the extension focuses on data operations. When used in the context of an import, it can trigger a data-only mode.
• SCHEMA_SYNC_MERGE: When performing an import, if this is set, the process will only upsert data rather than deleting existing records that are not in the source.

Schema Options

• SCHEMA_SYNC_SPLIT: A boolean that determines if the schema file should be split into multiple files per collection. Defaults to true if not explicitly set.
• SCHEMA_SYNC_SAFE: A boolean flag to enable safe mode operations.

Custom Configuration

• SCHEMA_SYNC_CONFIG: Path to an additional configuration file. This file must export a syncCustomCollections object of type ExportCollectionConfig to define custom collection exporters.