DOCSEARCHbyAlgolia
Submit Docs

PyMISP Documentation

repository·main·Indexed 19 days ago

https://github.com/misp/pymisp

A Python library providing programmatic access to the Malware Information Sharing Platform (MISP) instance. Version 2.5.34.1 includes tools for PDF export with international font support, dummy event generation for testing, and various feed generators including a Redis-based real-time generator. It also provides utility scripts for importing IOCs via ioc2misp.py and creating attribute distribution visualizations using treemap.py.

Tokens
26.7K
Snippets
79
Records
124
Agent score
67%

What's inside PyMISP

  1. Work with MISP data entities using specialized classes

    main

    PyMISP provides a rich set of classes that map to MISP's data model. Use these classes to manipulate specific types of data:

    • Events and Attributes: Use MISPEvent for event management and MISPAttribute for individual data points.
    • Objects: Use MISPObject for structured data, MISPObjectAttribute for attributes within objects, and MISPObjectReference for linking objects.
    • Users and Organizations: Use MISPUser and MISPOrganisation to manage identity and access.
    • Taxonomies and Tags: Use MISPTag and MISPTaxonomy for categorization.
    • System Components: Use MISPServer, MISPFeed, and MISPSharingGroup to manage MISP infrastructure and sharing settings.

  2. Create custom MISP Object generators

    main

    To create a new MISP object generator, you should inherit from AbstractMISPObjectGenerator and use a pre-defined template.

    Your generator must:

    1. Generate attributes.
    2. Add them as class properties using the add_attribute method.

    When the object is sent to MISP, all defined class properties will be exported to the JSON export.

  3. Understand PyMISP's Mutable Mapping architecture

    main

    PyMISP is designed so that its core entities behave like Python dictionaries. The master class AbstractMISP inherits from collections.MutableMapping.

    This means that MISPEvent, MISPAttribute, MISPObjectReference, MISPObjectAttribute, and MISPObject can all be treated as dictionaries and can be easily imported/exported to/from JSON blobs.

    Key details:

    • Properties that should not be visible in the dictionary representation are either prepended with an underscore (_) or added to a private list __not_jsonable.
    • You can manage the private list using update_not_jsonable and set_not_jsonable.
    • The class provides helpers to load and export data to/from JSON strings.

  4. Run the Redis consumer and the Flask server

    main

    Once the generator and Redis are configured, use the following commands to process the data and serve it to MISP:

    1. Consume data from Redis: Run fromredis.py to move items from Redis into the generated feed files.
    2. Serve data to MISP: Run server.py (after activating the serv-env virtual environment) to host the files for MISP to consume.
    # Consume stored data in redis
python3 fromredis.py

# Serve data to MISP
. ./serv-env/bin/activate
python3 server.py

  5. Import data from Excel or CSV using CSVLoader

    main

    The CSVLoader tool allows you to import data from CSV files into MISP events.

    There are two ways to use it depending on your file structure:

    1. Automatic Mapping: If your CSV header contains valid object relations defined in your MISP template, you can initialize CSVLoader with the template_name and csv_path. The load() method will yield dictionaries that can be passed directly to event.add_object(**o).

    2. Manual Mapping: If your CSV header does not match MISP template relations, you must provide a list of fieldnames and set has_fieldnames=True to map the columns manually.

    Key parameters for CSVLoader:

    • template_name: The name of the MISP template to use for mapping.
    • csv_path: A pathlib.Path object pointing to the CSV file.
    • fieldnames: (Optional) A list of strings representing the column names to use when the CSV header is not standard.
    • has_fieldnames: (Optional) A boolean indicating if the CSV has a header row that should be used for mapping.
    from pymisp.tools import CSVLoader
from pymisp import MISPEvent
from pathlib import Path

# Scenario 1: CSV header matches template
csv1 = CSVLoader(template_name='file', csv_path=Path('tests/csv_testfiles/valid_fieldnames.csv'))
event = MISPEvent()
event.info = 'Test event from CSV loader'
for o in csv1.load():
    event.add_object(**o)

# Scenario 2: Manual fieldname mapping
csv2 = CSVLoader(template_name='file', csv_path=Path('tests/csv_testfiles/invalid_fieldnames.csv'),
                 fieldnames=['SHA1', 'fileName', 'size-in-bytes'], has_fieldnames=True)

for o in csv2.load():
    event.add_object(**o)

  6. Install PyMISP via pip

    main

    To install the basic version of PyMISP, use pip. It is strongly recommended to use a virtual environment.

    pip3 install pymisp

    You can also install optional dependencies for specific functionality:

    • fileobjects: Create PE/ELF/Mach-o objects
    • openioc: Import files in OpenIOC format
    • virustotal: Query VirusTotal and generate objects
    • docs: Generate documentation
    • pdfexport: Generate PDF reports from MISP events
    • url: Generate URL objects
    • email: Generate MISP Email objects
    • brotli: Use brotli compression when interacting with MISP

    Example installation with virustotal and email extras:

    pip3 install pymisp[virustotal,email]
    pip3 install pymisp[virustotal,email]

  7. Debug PyMISP operations

    main

    There are two ways to enable debugging in PyMISP:

    1. Enable via PyMISP constructor

    Pass debug=True to the PyMISP class instance. This enables logging.DEBUG to stderr for the entire module.

    2. Use the Python logging module

    You can configure logging manually to control output level or destination.

    To log to stderr:

    import logging
logger = logging.getLogger('pymisp')
logger.setLevel(logging.DEBUG)

    To log to a file:

    import pymisp
import logging

logger = logging.getLogger('pymisp')
logging.basicConfig(level=logging.DEBUG, filename="debug.log", filemode='w', format=pymisp.FORMAT)
    import pymisp
import logging

logger = logging.getLogger('pymisp')
logging.basicConfig(level=logging.DEBUG, filename="debug.log", filemode='w', format=pymisp.FORMAT)

  9. Create dummy events for MISP testing

    main

    The scripts in the examples/events/ directory are designed to populate a MISP instance with dummy data for testing purposes. You can use these scripts to simulate different scales of event creation, from single events with attachments to massive events with thousands of attributes.

    Available Scripts

    • create_dummy_event.py: Creates a specified number of events (defaults to 1). Each event includes a randomly generated Network activity/domain|ip attribute and a Payload delivery/attachment containing a file named dummy.
    • create_massive_dummy_events.py: Creates a specified number of events (defaults to 1). Each event contains a specified number of randomly generated attributes (defaults to 3000).

  10. Install and run the MISP feed generator

    main

    The feed generator is a Python script that creates a MISP feed from an existing MISP instance. To use it, clone the PyMISP repository, navigate to the example directory, configure your settings, and execute the generator script.

    Follow these steps:

    1. Clone the repository.
    2. Navigate to examples/feed-generator.
    3. Copy the default settings file to settings.py.
    4. Edit settings.py to match your MISP instance configuration.
    5. Run generate.py using Python 3.
    git clone https://github.com/MISP/PyMISP.git
cd examples/feed-generator
cp settings.default.py settings.py
# adjust your settings in settings.py
python3 generate.py

  11. Install PyMISP from source for development

    main

    If you are developing PyMISP, you must use poetry. First, clone the repository and initialize submodules, then use poetry install with the desired extras.

    git clone https://github.com/MISP/PyMISP.git && cd PyMISP
git submodule update --init
poetry install -E fileobjects -E openioc -E virustotal -E docs -E pdfexport -E email
    git clone https://github.com/MISP/PyMISP.git && cd PyMISP
git submodule update --init
poetry install -E fileobjects -E openioc -E virustotal -E docs -E pdfexport -E email

  12. Install and setup the Generic MISP feed generator

    main

    The Generic MISP feed generator allows for real-time MISP feed generation using Redis as a buffer. It consists of a generator, a Redis consumer, and a Flask-based server to serve the data to MISP.

    Prerequisites

    Install and verify the Redis server:

    # Install redis-server
sudo apt install redis-server

# Check if redis is running
redis-cli ping

    Setup Steps

    1. Clone the PyMISP repository and navigate to the example directory.
    2. Copy the default settings file and configure your specific settings (e.g., Redis connection details).
    3. Run the Redis consumer script.
    4. Install the server environment and run the Flask server to serve the generated data to MISP.
    # redis-server
sudo apt install redis-server

# Check if redis is running
redis-cli ping

#  Feed generator
git clone https://github.com/MISP/PyMISP
cd PyMISP/examples/feed-generator-from-redis
cp settings.default.py settings.py
vi settings.py  # adjust your settings

python3 fromredis.py

# Serving file to MISP
bash install.sh
. ./serv-env/bin/activate
python3 server.py