MSNoise: Monitoring Seismic Velocity Changes using Ambient Seismic Noise

MSNoise is a Python package designed for monitoring seismic velocity changes using ambient seismic noise. It automates the complete processing pipeline, including:

• Cross-correlations and stacking of raw seismic archives.
• Reference computation.
• Relative velocity estimation using methods such as MWCS, stretching, or wavelet methods to produce $dv/v$ curves.

MSNoise 2.x Architecture: The 2.x version is a rewrite built on xarray. It utilizes a config-set / lineage architecture, allowing users to run the same data through multiple parameter branches without needing to reprocess from scratch. The system is also extensible via a plugin system that can extend or replace any step in the pipeline.

The project is organized into several functional modules within msnoise/core/:

• db.py: Database connectivity (connect, get_engine).
• config.py: Configuration management (get_params, get_merged_params_for_lineage).
• stations.py: Station and data availability management (get_stations, get_waveform_path).
• workflow.py: DAG and job management (run_workflow_plan, get_next_lineage_batch).
• signal.py: Digital Signal Processing (DSP) (stack, winsorizing).
• io.py: Xarray-based I/O for various processing results (xr_save_ccf, xr_get_mwcs).
• fdsn.py: FDSN data fetching (fetch_and_preprocess).
• compute.py: Heavy mathematical computations (mwcs, whiten).
• preprocessing.py: Waveform preprocessing pipeline.

Entry Points:

• CLI: msnoise (via scripts/msnoise.py).
• Web UI: msnoise admin (via msnoise_admin.py).

MSNoise uses a DataSource abstraction to handle different data storage locations. Every Station can optionally be linked to a DataSource via data_source_id. If this is NULL, the project default (usually DataSource.ref=1) is used.

Supported URI schemes (parsed by fdsn.parse_datasource_scheme):

• Local/SDS: Bare paths or sds:///path (e.g., local SDS, BUD, IDDS, or PDF archives).
• FDSN: fdsn://http://... (FDSN web services).
• EIDA: eida://http://... (EIDA routing client).

Station-to-source assignment methods:

• set_station_source(db, net, sta, data_source_id): Per-station override.
• set_network_source(db, net, data_source_id): All stations in a specific network.
• set_all_stations_source(db, data_source_id): Project-wide default.

MSNoise 1.6 introduces an hpc configuration parameter to optimize performance when running on High Performance Computing (HPC) infrastructure with a remote central MySQL database.

When hpc = True, MSNoise reduces database overhead by marking jobs as Done without automatically creating the next job type in the workflow. This minimizes the number of SELECT, UPDATE, and INSERT actions on the database.

To use this mode, you must manually trigger the next step in the workflow using the --hpc flag with new_jobs.

## Workflow with hpc = Y

* `msnoise new_jobs` (creates CC jobs)
* `msnoise compute_cc` (processes CC jobs)
* `msnoise new_jobs --hpc CC:STACK` (creates STACK jobs based on CC jobs marked "Done")
* `msnoise stack -m` (processes STACK jobs)
* `msnoise new_jobs --hpc STACK:MWCS` (creates MWCS jobs based on STACK jobs marked "Done")

MSNoise uses a dispatch mechanism (propagate_downstream) to automatically create new jobs when a step completes. The behavior depends on the completed category:

The Mwcs Join Gate: A mwcs, stretching, or wavelet job is only created when BOTH a refstack_M REF sentinel AND a stack_N day for that pair are marked as Done.

MSNoise 2.0 uses a workflow DAG (Directed Acyclic Graph) driven by config sets instead of a flat-file approach.

• Config Sets: Every processing category (e.g., cc, filter, mwcs, stack) can have multiple parameterized instances running in parallel. For example, you can create two different filter config sets to process different frequency bands simultaneously.
• Lineages: A lineage is a unique path through the DAG, represented as a /-separated string of step names.
• Output Structure: The lineage determines the folder hierarchy under OUTPUT/. For example, a path might look like: OUTPUT/preprocess_1/cc_1/filter_1/stack_1/mwcs_1/_output/….

This architecture allows for complex, branching processing workflows where different parameters are tested in parallel branches.

MSNoise plugins are external Python packages that extend the default processing workflow (preprocess → CC → stack → mwcs/stretching/wavelet → dv/v). They allow developers to add new processing steps, custom job types, or new web-admin pages.

To ensure stability, plugins should always import from the msnoise.plugins module rather than msnoise.core.* directly. This protects your plugin from internal changes to the MSNoise core.

from msnoise.plugins import (
    connect,
    get_config, update_config,
    get_params,
    get_workflow_chains, get_workflow_order,
    get_next_lineage_batch, is_next_job_for_step,
    massive_update_job, propagate_downstream,
    xr_get_ccf, xr_save_ccf,
    get_stations, get_station_pairs,
)

MSNoise supports a plugin system where the default workflow can be branched at different steps. A plugin is a structured Python package that must be installed in your environment.

To use a plugin:

1. Install the plugin as a standard Python package.
2. Declare the plugin's package name in the plugins parameter in your project configuration. This must be done per project.

Plugins use entry points to define new commands (accessible via msnoise plugin <plugin_name> <command>) and custom Job Types at three specific workflow moments:

• When a new file is detected in the archive.
• When a new pair-job has to be done.
• When a new REF stack has been computed.

# Example of interacting with a plugin named 'amazing' that has a command 'sayhi'
msnoise plugin amazing sayhi