repository·master·Indexed 22 days ago
https://github.com/domainaware/parsedmarcA Python package and CLI for parsing aggregate, failure, and SMTP TLS DMARC reports. It ingests email reports from sources such as IMAP and APIs, and exports them to data analysis platforms including Elasticsearch, Splunk, and Kafka.
parsedmarc is a Python module and CLI utility designed for parsing DMARC (Domain-based Message Authentication, Reporting, and Conformance) reports. It serves as a self-hosted, open-source alternative to commercial DMARC processing services. It is commonly used in conjunction with Elasticsearch and Kibana (or Splunk) to visualize email authentication data through premade dashboards.parsedmarc is a Python module and CLI utility designed to parse DMARC reports. It serves as a self-hosted, open-source alternative to commercial DMARC processing services. When integrated with visualization tools like Kibana, Grafana, or Splunk, it provides a complete pipeline for analyzing email authentication data.
.gz or .zip compressed reports.Since the version fixing issue #169, parsedmarc automatically performs backfilling of dkim_results_combined and spf_results_combined fields on startup. This ensures that older reports (which lack these fields) appear correctly in dashboard alignment-detail tables.
How it works:
parsedmarc runs a count query against configured aggregate index patterns to find documents that have DKIM/SPF results but are missing the _combined fields._update_by_query task with wait_for_completion=false. This prevents startup from being blocked.index_prefix_domain_map is configured (and no index_prefix is set): Every tenant prefix in the map gets its own index pattern, plus the unprefixed pattern. A SIGHUP reload re-reads the map to cover new tenants without a restart.index_prefix is set (in [elasticsearch] or [opensearch]): Only that specific prefix is targeted to avoid touching data from other deployments on a shared cluster.index_suffix is set: Both the suffixed and unsuffixed patterns are targeted.The DMARC failure reports dashboard (formerly DMARC Forensic Samples) provides information from DMARC failure reports (also known as forensic or ruf reports). These reports contain actual samples of emails that failed to pass DMARC.
Note on Data Availability:
To prevent breaking DMARC signatures (DKIM) and causing message failures, mailing lists should follow these best practices:
From, Date, or Subject.List-Unsubscribe headers instead of adding unsubscribe links to the message body.List-Unsubscribe: <https://list.example.com/unsubscribe-link>List-Id headers instead of modifying the subject line.List-Id: Example Mailing List <list.example.com>Multi-tenant support (available since 8.19.0) allows placing data into separate OpenSearch or Elasticsearch index prefixes based on a domain mapping.
Create a YAML file where each key is a tenant name and each value is a list of domain names (strings) related to that tenant. Note: A domain cannot be in multiple tenant lists; only the first matching list is used.
example:
- example.com
- example.net
whalensolutions:
- whalensolutions.comIn your [general] configuration section, set index_prefix_domain_map to the path of this YAML file.
Do not set an index_prefix option in the [elasticsearch] or [opensearch] sections.
When configured, reports for domains in the map are saved to an index prefixed with {tenant_name}_.
example:
- example.com
- example.net
- example.org
whalensolutions:
- whalensolutions.comparsedmarc implements a daily indexing strategy. Instead of a single large index, data is stored in a separate index for each day. This design facilitates compliance with data retention regulations (like GDPR) by allowing users to easily manage or delete time-based data. For implementation details on managing these indexes, refer to the official Elastic guide on managing time-based indexes efficiently.You can improve performance when processing large volumes of files or messages by increasing the number of parallel worker processes using the n_procs option under the [log_file] section (or globally in some contexts).
Important Notes:
1.Parsedmarc processes mailbox messages in batches of batch_size. To prevent data loss, a batch is only archived or deleted from the reports folder after it has been successfully written to all configured output destinations.
If any destination fails (e.g., Elasticsearch outage, full disk, expired Splunk token), the entire batch remains in the reports folder to be retried. This is an all-or-nothing mechanism.
max_unsaved_retries + 1 (default is 3).<archive_folder>/Unsaved. Messages in this folder are never deleted, regardless of your delete settings.Unsaved, move them back to the reports folder or run parsedmarc with reports_folder = Archive/Unsaved.Unsaved immediately in one-shot mode, set max_unsaved_retries = 0.parsedmarc implements a daily indexing strategy. It stores data in a separate index for each individual day. This design is intended to simplify compliance with data retention regulations, such as GDPR, by allowing users to easily delete or manage specific days of data.DMARC ensures that SPF and DKIM authentication mechanisms are actually authenticating against the same domain that the end user sees in the message's From header. For a message to pass a DMARC check, it must pass either DKIM or SPF, and the relevant indicators must be in alignment.
From header.Mail From header.Mail From header aligns with the domain in the message's From header.The mapping system in parsedmarc is used to identify sending sources (organizations, services, or industries) based on their reverse DNS base domains. This makes it easier to aggregate metrics, such as calculating the total volume of emails sent by a specific service like 'Intuit Mailchimp' even if they use multiple domains.
base_reverse_dns_map.csv: The primary mapping file. It contains three fields: base_reverse_dns, name, and type.known_unknown_base_reverse_dns.txt: A list of domains that have been identified but cannot yet be categorized into a specific organization or industry.psl_overrides.txt: A list of suffixes used to fold noisy subdomains into a single base domain (e.g., folding host01.foo.com to foo.com).When assigning a type in base_reverse_dns_map.csv, follow this order of precedence:
Finance, Healthcare, Retail).Note: Do not use Excel to edit CSV files, as it may corrupt Unicode characters. Use LibreOffice Calc instead.