Melusine

repository·master·Indexed 18 days ago

https://github.com/maif/melusine

A high-level Python library for email processing designed to automate qualification workflows. Melusine combines deep learning frameworks (HuggingFace, PyTorch, TensorFlow) with deterministic rules (regex, keywords, heuristics) for tasks such as email routing, smart prioritization, snippet summarization, and precision filtering. It includes tools for email segmentation, message tagging, and a GmailConnector for interacting with the Gmail API.

Tokens
17.9K
Snippets
58
Records
78
Agent score
63%

What's inside melusine

  1. Overview of Melusine email processing

    master

    Melusine is a comprehensive email processing library designed to optimize email workflows. It allows developers to integrate deep learning frameworks (such as HuggingFace, PyTorch, and TensorFlow) and deterministic rules (regex, keywords, heuristics) into a full email qualification workflow.

    Key capabilities include:

    • Email Routing: High-accuracy routing of emails to intended destinations.
    • Prioritization: Identifying and prioritizing urgent emails.
    • Snippet Summarization: Extracting relevant information from long email bodies.
    • Filtering: Removing unwanted emails to reduce clutter.
  2. Overview of Melusine

    master

    Melusine is a comprehensive email processing library designed to optimize email workflows. It allows developers to integrate deep learning frameworks (HuggingFace, PyTorch, TensorFlow, etc.) and deterministic rules (regex, keywords, heuristics) into a full email qualification workflow.

    Key capabilities include:

    • Email Routing: High-accuracy routing of emails.
    • Smart Prioritization: Identifying and prioritizing urgent emails.
    • Snippet Summaries: Extracting relevant information from long emails.
    • Precision Filtering: Eliminating unwanted emails.

    Melusine provides pre-packaged tools for segmenting email conversations into individual messages, tagging message parts (body, signatures, footers), and handling transferred emails.

  3. Core features of Melusine

    master

    Melusine provides several out-of-the-box features and architectural advantages for email processing:

    Out-of-the-box features

    • Email Segmentation: Dividing an email conversation into individual messages.
    • Message Tagging: Identifying specific parts of a message, such as the Email body, signatures, footers, etc.
    • Transferred Email Handling: Managing emails that have been forwarded or transferred.

    Architectural Advantages

    • Streamlined Execution: Handles boilerplate code, including debug mode, pipeline execution, and code parallelization.
    • Flexible Integrations: Modular architecture for seamless integration with various AI frameworks.
    • Production Ready: Designed for robustness and stability in production environments.
  4. Implement a custom detector using MelusineDetector

    master

    To integrate a machine learning model (like a Zero-Shot Classifier) into a Melusine pipeline, you should subclass the MelusineDetector template class. This standardizes how models interact with the pipeline through three key lifecycle methods:

    1. pre_detect: Responsible for assembling or cleaning the text that will be passed to the classifier.
    2. detect: Executes the actual model inference on the prepared text.
    3. post_detect: Processes the model's raw output (e.g., applying a probability threshold) to determine the final detection result (e.g., True or False).

    Subclassing MelusineDetector also ensures that the detector automatically builds debug data, making the model's decisions explicable.

  5. How MelusineDetector works

    master

    The MelusineDetector class standardizes detection tasks within a MelusinePipeline using a Template pattern. It splits the detection process into three distinct lifecycle stages managed by the transform method:

    1. pre_detect: Prepares the input. This stage is used to select or combine specific columns needed for detection (e.g., merging BODY text with email headers).
    2. detect: Performs the core logic. This is where you apply regular expressions, machine learning models, or heuristics to the processed input.
    3. post_detect: Refines the results. This stage is used for thresholding, combining results from multiple models, or applying final business rules.

    By using MelusineDetector, you benefit from input column validation, support for multiple backends (Pandas, Python dictionaries), debug mode, and multiprocessing.

    # The transform method calls pre_detect, detect, and post_detect in order
    data_with_detection = detector.transform(data)
  6. Use Row-wise vs. Dataframe-wise methods in MelusineDetector

    master

    When implementing pre_detect, detect, or post_detect methods, you can choose between two execution modes:

    1. Row-wise methods: The method operates on a single row. To implement this, ensure the first parameter of your method is named row.

      • Benefit: Makes your code backend-independent (works with both PandasBackend and DictBackend).
      • Performance: PandasBackend supports multiprocessing for row-wise methods.
    2. Dataframe-wise methods: The method operates on the entire DataFrame at once. To implement this, do not name the first parameter row.

      • Constraint: These methods are typically tied to specific backends (e.g., PandasBackend).
  7. Create a custom MelusineRegex

    master

    To implement custom regex logic, subclass MelusineRegex and implement the required properties.

    • positive: A str or Dict[str, str] defining the patterns you want to match. If using a dictionary, keys are identifiers and values are the regex patterns.
    • match_list: A List[str] of example strings that must match your regex. This is used for automated testing during instantiation.
    • no_match_list: A List[str] of example strings that must not match your regex. This is also used for automated testing during instantiation.
    • negative (optional): A str or Dict[str, str] defining patterns that, if matched, will cancel out any positive matches.
    • neutral (optional): A str or Dict[str, str] defining patterns that, if matched, will "blur" the text, preventing subsequent positive matches from triggering on that specific content.
    from melusine.base import MelusineRegex
    from typing import Union, Dict, List
    
    class AnnoyingEmailsRegex(MelusineRegex):
        @property
        def positive(self) -> Union[str, Dict[str, str]]:
            return dict(
                VOLDY_BEING_VOLDY="Avada Kedavra",
                GANDALF_BEING_GANDALF="You shall not pass",
            )
    
        @property
        def match_list(self) -> List[str]:
            return [
                "Avada Kedavra is a spell used by Lord Voldemort",
                "And then, you know me, I was not gonna let it pass so I told them : You shall not pass and obviously everyone clapped",
            ]
    
        @property
        def no_match_list(self) -> List[str]:
            return ["Abracadabra, here I am", "I told them not to pass"]
  8. How Melusine pipelines process data

    master

    Melusine pipelines operate as a sequence of steps that transform input data into qualified output. A typical flow involves:

    1. Cleaner: Performs cleaning transformations, such as uniformizing line breaks (e.g., converting \r\n to \n).
    2. Normalizer: Performs text normalization, such as replacing or deleting non-UTF8 characters (e.g., converting éöà to eoa).
    3. Detector: Analyzes the processed text to identify specific patterns or states (e.g., an EmergencyDetector identifying urgent emails).

    More complex pipelines may include Email Segmentation (splitting conversations into unitary messages), ContentTagging (identifying parts like SIGNATURE or FOOTER), or specialized detectors like Appointment detection.

  9. How email segmentation and tagging work

    master

    Email segmentation is the process of dividing a single email thread into distinct messages using transition patterns (e.g., headers like From:, To:, or Subject:). Once segmented, individual messages can be processed line-by-line to apply tags to specific components.

    Common tags used in this process include:

    • HELLO / GREETINGS
    • BODY
    • SIGNATURE
    • FOOTER
    • TRANSITION (used to identify the boundaries between messages)

    This segmentation can be used to improve the performance of downstream machine learning models by providing cleaner, structured input.

  10. Understand the output of a Melusine pipeline

    master

    When running a pipeline via MelusinePipeline.transform(df), the resulting dataset (typically a DataFrame) includes qualified columns. Common outputs include:

    • messages: A list of individual messages extracted from each email.
    • emergency_result: A flag used to identify urgent emails.
  11. Use neutral matches to blur text

    master

    A neutral regex pattern is used to "neutralize" or "blur" parts of the text. When a neutral pattern matches, that specific segment of text is effectively removed from consideration for subsequent positive matches. This is useful for differentiating between intentional patterns and accidental matches (e.g., distinguishing between a serious threat and a joke using specific contractions).

    class IfritAlertRegex(MelusineRegex):
        @property
        def positive(self) -> Union[str, Dict[str, str]]:
            return dict(WORLD_BURN=r"see (the world|everything) (burn|in flames)")
    
        @property
        def neutral(self) -> Union[str, Dict[str, str]]:
            # Matching this will 'blur' the text so the positive pattern won't trigger
            return dict(JOKE=r"I wanna see (the world|everything) (burn|in flames)")
    
        @property
        def match_list(self) -> List[str]:
            return ["I want to see the world burn"]
    
        @property
        def no_match_list(self) -> List[str]:
            return ["I wanna see the world burn"]
  12. Use the MelusinePipeline class

    master

    The MelusinePipeline is the central orchestration component of Melusine. It extends the standard sklearn.Pipeline class by adding specialized features for email processing workflows:

    • Configuration-based Instantiation: Ability to initialize the pipeline directly from configuration files.
    • Input/Output Coherence Checks: Automated validation to ensure data flows correctly between pipeline steps.
    • Debug Mode: Enhanced visibility into the pipeline execution for troubleshooting.