DOCSEARCHbyAlgolia
Submit Docs

ExMeX Core Framework

website·Indexed 32 days ago

https://exmex.tedamoh.com

A framework for data management featuring core modules for archiving, orchestration, monitoring, and testing. It includes tools for managing data lifecycles, retention policies, and module registration via stored procedures such as uspRegisterModule and uspSetArchivingDataObject. Available in versions 3.3.0 (Legacy), 3.4.0 (Latest Stable), and 3.5.0 (Development).

Tokens
63.3K
Snippets
231
Records
352
Agent score
45%

What's inside ExMeX

  1. Overview of ExMeX Framework Guides

    Guides in the ExMeX Framework are procedural, task-oriented documents designed to walk users through specific processes from start to finish. They are characterized by being task-focused, prescriptive, linear, and result-oriented. Use guides when performing an operation for the first time, following a proven procedure, or ensuring the correct sequence of steps is followed.

  2. Overview of the ExMeX Archiving Framework

    The archiving framework provides a structured approach to data lifecycle management based on retention policies. It manages the transition of data from active storage to archives and eventually to permanent deletion to optimize storage costs and system performance. This pattern is applicable to any data solution built with the ExMeX core framework, such as logging and monitoring frameworks.

  3. Identify available ExMeX Framework guide topics

    Guides are organized into the following functional areas:

    • Framework Installation & Maintenance: Installation, upgrading, rolling back releases, and backup/restore procedures.
    • Module Management: Registering modules, configuring settings, managing load windows, and orchestration.
    • Archiving Framework: Data object archiving setup, execution, and retention policy management.
    • Data Operations: Metadata work, data flow configuration, and data object management.

  4. Manage modules using the uspRegisterModule stored procedure

    The MetadataZoneCore.uspRegisterModule stored procedure is used to manage modules within the ExMeX Framework Core. It allows developers to activate/inactivate modules, configure load window usage, and control the RowsRead functionality. The procedure supports both direct ID-based updates and flexible updates using natural keys (source/target model and entity codes).

  7. Understand the role of Best Practices in ExMeX Framework

    Best practices in the ExMeX Framework are advisory, experience-based recommendations designed to help developers make informed strategic decisions. Unlike Guides, which provide step-by-step instructions for immediate execution, Best Practices focus on the 'why' and offer multiple approaches with associated trade-offs. They should be consulted during implementation strategy design, architectural decision-making, scalability planning, and complex troubleshooting.

  8. Register or update modules using uspRegisterModule

    The MetadataZoneCore.uspRegisterModule stored procedure is used to register, update, or delete modules in the ExMeX Framework Core. It enables orchestration by allowing modules to be activated, deactivated, and configured for load window usage. Currently, only the Update ('U') operation is fully implemented for production use; Add ('A') and Delete ('D') are placeholders.

  9. Retrieve latest module execution statistics using ufnModuleStats

    The ufnModuleStats function provides a consolidated, ordered overview of the latest execution steps performed by a specific module. It aggregates data from ModuleSemaphoreArchive (semaphore state changes), ModuleLogMessage (lifecycle logging), and ProcessStepVapourTrail (critical process steps) to create a timeline of events including log messages, semaphore activity, and data operations.

    To visualize the execution as a timeline, results should be sorted by [Timestamp - Step] and [Record Id - Step] in ascending order.

    SELECT *
FROM MetadataZoneCore.ufnModuleStats(1679)
ORDER BY [Timestamp - Step] ASC, [Record Id - Step] ASC;

  10. Use the Enhanced Archiving Framework in ExMeX 3.3

    ExMeX Core Framework 3.3 introduces enhancements to the Archiving Framework to streamline the archiving and deletion of outdated data based on customizable object and retention policies. Key improvements include:

    • Validation to ensure the target schema exists before archiving.
    • Ordered execution of archiving objects to maintain data integrity.
    • Support for multiple timestamp formats as archiving columns.
    • Updated column codes for archiving modules.

  11. Retrieve module execution statistics using ufnModuleStats

    The ufnModuleStats function provides a consolidated, chronological overview of the latest execution steps for a specific module. It aggregates data from ModuleSemaphoreArchive (semaphore state changes), ModuleLogMessage (lifecycle logging), and ProcessStepVapourTrail (critical process steps) to aid in operational troubleshooting, auditing, and root cause analysis in data pipelines.

    The function returns a list of steps containing the Module Id, Batch Id, step description, timestamps, record identifiers, log messages, and the source data object.

    -- Retrieve and order execution steps for module ID 1679
SELECT *
FROM MetadataZoneCore.ufnModuleStats(1679)
ORDER BY [Timestamp - Step] ASC, [Record Id - Step] ASC;

  12. Understand the ExMeX Archiving Framework retention concept

    The archiving framework manages the data lifecycle using two primary configurable variables to determine when data is moved from active storage to archives and when it is permanently deleted.

    • Retention Period: The duration data remains in the active (source) table before being moved to the archive table.
    • Archive Retention Period: The duration data remains in the archive table before being permanently deleted.

    These are globally configured in the ExMeX Core Framework but can be overridden for specific data objects. The total time data exists in the system is the sum of both periods: Total Retention Period = Retention Period + Archive Retention Period.