monstache

Monstache Version 6 defaults to using MongoDB change streams instead of tailing the oplog.

• Default behavior: Without specific configuration, Monstache watches the entire MongoDB deployment.
• Targeted watching: To watch specific namespaces instead of the whole deployment, use the change-stream-namespaces option, which accepts an array of strings.

To use the build.sh script, you must have the following installed on your host machine:

• Docker
• Bash shell

You can build a Go plugin for Monstache using the build.sh script, which utilizes Docker to ensure a consistent build environment.

Workflow:

1. Place your plugin source code (a .go file with a main package) into the docker/plugin directory.
2. Ensure the .plugin file contains the name of your plugin file (excluding the .go extension).
3. Execute ./build.sh from the docker/plugin directory.
4. The resulting .so file will be generated in the docker-build folder.

To activate the plugin in Monstache, provide the path to the generated .so file using the mapper-plugin-path argument.

You can use the build.sh script to build all Monstache binaries for linux, win, and mac targets within a Docker container. The script exports the resulting binaries to a local directory named docker-build.

Note: Running the script will remove any existing docker-build folder before exporting the new one.

./build.sh

Monstache Version 6 requires the following compatible versions of its data sources:

• MongoDB: 3.6 or higher
• Elasticsearch: 7.0 or higher

It utilizes the official MongoDB Go driver and the community-supported Elasticsearch driver from olivere.

To use the Monstache plugin builder script, you must have the following installed on your host machine:

• Docker
• Bash shell

Monstache supports running JavaScript (via the Otto engine) to transform or filter data. You can provide scripts via the --script configuration option (either as an inline string or a file path).

Scripts are scoped to a namespace. Within the script, you must export a function via module.exports.

• Filters: A function that returns a boolean. If true, the operation is kept; if false, it is dropped.
• Scripts (Mapping): A function used to transform the document data before indexing.
• Pipelines: Functions used for complex transformations.

Available helper: stringFromBinData(binData) can be used to convert MongoDB Binary data to a string within your script.

Monstache supports different strategies for handling document deletions in Elasticsearch, controlled via the DeleteStrategy configuration:

1. ignoreDeleteStrategy: Deletions in MongoDB are ignored and nothing is removed from Elasticsearch.
2. statefulDeleteStrategy: Monstache uses stored metadata (in a MongoDB meta collection) to find the exact index, routing, and parent ID required to perform a precise deletion. This is the most reliable method for complex routing setups.
3. statelessDeleteStrategy: Monstache attempts to find the document in Elasticsearch using the _id and the DeleteIndexPattern. If DisableDeleteProtection is enabled, it uses DeleteByQuery. Otherwise, it performs a search to find the correct index and routing before deleting.

When EnablePatches is enabled, Monstache can maintain a history of changes using JSON merge patches.

For namespaces defined in PatchNamespaces, Monstache performs the following during an update:

1. Fetches the existing document from Elasticsearch.
2. Calculates a JSON merge patch between the existing document and the new data.
3. Appends the patch to a list stored in the document under the key defined by MergePatchAttr (defaults to json-merge-patches).
4. Each patch entry includes a timestamp (ts) and a version (v).

This allows for reconstructing document states or auditing changes over time.