MSTICpy Documentation

MSTICPy is a collection of Python tools designed for security investigations and threat hunting. It is optimized for three primary workflows:

1. Acquiring and enriching data: Tools for gathering security telemetry and adding context.
2. Analyzing data: Tools for performing security analysis, including anomalous sequence detection and time series analysis.
3. Visualizing data: Tools for presenting findings, including specialized Jupyter widgets and visualizations.

While many features are designed for interactive use in Jupyter notebooks, the tools can also be used via the Python command line or imported directly into your own Python scripts.

MSTICPy includes several functions for security analysis and data transformation:

Transformations:

• Process Tree building
• Decoding encoded/compressed data
• IoC/observable extraction
• Syslog parsing

Analysis:

• Anomaly patterns in Office activity
• Time series decomposition
• Event clustering

MSTICPy uses Entities (such as IpAddress, Host, Url) to encapsulate attributes and methods for real-world objects. You can run context queries directly from these classes.

Context Lookups

Use methods like .ip_type() or .whois() on an IpAddress object to get enrichment data.

Threat Intelligence (TI) Lookups

If TI providers are configured in msticpyconfig.yaml, you can perform lookups using the .ti attribute on an Entity: IpAddress.ti.lookup_ip(list_of_iocs)

Alternatively, use the mp.TILookup() class directly.

MSTICPy includes custom widgets built on ipywidgets to streamline common security analysis tasks within Jupyter Notebooks. These include:

• List pickers
• Query time boundary settings
• Event displays
• Alert browsers

The QueryProvider is an extensible library used to query various log data sources, including:

• Microsoft Sentinel / Log Analytics
• Microsoft XDR
• Splunk
• OData
• Mordor data sets
• Local data

It supports built-in parameterized queries for complex operations and allows you to define custom queries using a simple YAML schema. Most components in msticpy use Pandas DataFrames as the standard input and output format.

MSTICPy is designed to be data-source agnostic, though it includes specific components for several major platforms:

• Supported Data Sources: Azure Sentinel, Splunk, Microsoft 365 Defender Advanced, and Microsoft Graph.
• Data Interchange Format: The APIs typically follow a pattern where they accept a pandas.DataFrame as input and return a pandas.DataFrame as output.

MSTICpy includes modules for advanced security analysis:

• Anomalous Sequence Detection: Detect unusual sequences of events (e.g., in Office or Active Directory logs) by extracting sessions and identifying unusual activity patterns, such as unauthorized mail forwarding rules.
• Time Series Analysis: Identifies unusual patterns in log data by accounting for normal seasonal variations (e.g., daily or weekly cycles), helping to highlight unusual traffic flows or event spikes.

Event clustering is used to manage large sets of repetitive log events that can obscure anomalous activity. While simple sorting and grouping work for exact duplicates, clustering is effective for identifying 'near-duplicates'—events that are subtly different due to timestamps, GUIDs, or remote hostnames in command lines.

This technique helps security investigators differentiate between:

• Common system processes with slightly varying command lines.
• Crafted User Agent strings designed to mimic common browsers.
• Common system commands used in unusual ways.
• Processes running from paths designed to look like common system processes.

Because machine learning typically requires numeric features and log data is primarily text, clustering involves selecting appropriate features and creating a numeric representation that captures the specific anomaly patterns you are searching for.

MSTICpy provides interactive visualizations using the Bokeh library:

• Event Timelines: Display log events on an interactive timeline. This allows you to zoom into specific time slots and view detailed information for plotted events.
• Process Trees:
  • Creation: Builds parent-child relationships between processes from host process creation logs.
  • Visualization: Displays the processed relationships as an interactive process tree.
  • Utilities: Includes functions to extract individual or partial trees from the dataset.

Query providers allow you to query data from diverse data sources using built-in templated queries or ad-hoc queries. Data is typically returned as a pandas.DataFrame.

To use a provider, you follow a three-step lifecycle:

1. Instantiate: Create a QueryProvider instance by specifying a data_environment.
2. Connect: Call .connect() with a connection string or authentication parameters specific to that environment.
3. Query: Use the loaded queries (which are callable functions) to retrieve data.

Each provider (e.g., Sentinel, Splunk) has its own specific documentation for connection details and available parameters.