Maigret OSINT Tool

repository·main·Indexed 12 days ago

https://github.com/soxoj/maigret

An Open Source Intelligence (OSINT) tool for collecting comprehensive dossiers on individuals by scanning thousands of websites for a specific username. Maigret version 0.6.4 supports CLI and Web UI modes, AI-assisted investigations via OpenAI-compatible models, and multiple export formats including HTML, PDF, JSON, CSV, and Neo4j. It features advanced filtering by tags and keywords, Cloudflare bypass via FlareSolverr, and support for Tor and I2P proxies.

Tokens
40.3K
Snippets
157
Records
205
Agent score
98%

What's inside Maigret

  1. Core Features of Maigret

    main

    Maigret is a powerful OSINT tool with the following capabilities:

    • Massive Site Support: Supports 3000+ sites. By default, it checks the top 500 most visited sites. Use the -a flag for a full scan or --tags to filter by category/country.
    • Recursive Search: Performs searches based on discovered usernames and other IDs.
    • Data Extraction: Uses socid_extractor to pull information from profiles and APIs, including links to other accounts.
    • Evasion & Access: Can detect and partially bypass blocks, censorship, and CAPTCHAs. Supports Tor and I2P sites.
    • Automated Updates: Automatically pulls an updated site database from GitHub every 24 hours.
    • Multiple Interfaces: Available as a CLI, a Python library, and a Web UI.
    • AI Analysis: Optional --ai mode uses OpenAI-compatible APIs to summarize search results into an investigation summary.
  2. Use Maigret as a command line tool

    main

    Maigret can be used in three primary ways:

    1. Command Line Tool: Run searches directly from your terminal using various modes (default or all sites).
    2. Web Interface: View results in a graph format and download all report formats on a single page.
    3. Library: Integrate Maigret's functionality directly into your own Python projects.
  3. Configure Cloudflare bypass with FlareSolverr

    main

    For sites requiring JavaScript challenges (tagged cf_js_challenge or cf_firewall), you can use an experimental feature to route checks through a FlareSolverr instance. This is optional and only required for specific protected sites.

    1. Run FlareSolverr via Docker:
    docker run -d -p 8191:8191 --name flaresolverr ghcr.io/flaresolverr/flaresolverr:latest
    1. Run Maigret with the --cloudflare-bypass flag.
    docker run -d -p 8191:8191 --name flaresolverr ghcr.io/flaresolverr/flaresolverr:latest
    # Then use Maigret with:
    maigret USERNAME --cloudflare-bypass
  4. How the activation mechanism works

    main

    The activation mechanism allows Maigret to handle sites that require additional authentication, such as cookies, JWT tokens, or custom headers.

    When a site check is performed using an invalid token (e.g., an expired JWT), the following lifecycle occurs:

    1. The site check makes an HTTP request to the urlProbe endpoint with the invalid token.
    2. The response contains an error message that matches one of the strings defined in the activation/marks field of the site's record.
    3. Upon detecting this error, Maigret triggers the specific activation function associated with that site (defined by the method key).
    4. The activation function performs the necessary steps (e.g., fetching a new token via a specific URL) and updates the site check record with the new credentials.
    5. The next attempt to check the site (via retry or a new run) uses the updated, valid credentials.
    "Vimeo": {
        "tags": ["us", "video"],
        "headers": {
            "Authorization": "jwt eyJ0..."
        },
        "activation": {
            "url": "https://vimeo.com/_rv/viewer",
            "marks": [
                "Something strange occurred. Please get in touch with the app's creator."
            ],
            "method": "vimeo"
        },
        "urlProbe": "https://api.vimeo.com/users/{username}?fields=name...",
        "checkType": "status_code",
        "url": "https://vimeo.com/{username}"
    }
  5. Understand account presence check methods

    main

    Maigret determines if an account exists on a site using one of three checkType methods defined in data.json:

    • message: The most reliable method. It checks if any string from presenceStrs is present in the HTML response and ensures none of the strings from absenceStrs are present.
    • status_code: Checks if the HTTP response status code is in the 2XX range.
    • response_url: Checks that there is no redirect and the response status code is 2XX.

    Note on Anti-bot handling: Maigret treats specific anti-bot HTTP status codes (e.g., LinkedIn's HTTP 999) as a standard "Not Found/Available" signal rather than a server error to prevent false positives.

  6. Perform AI-assisted investigations

    main

    Maigret can use OpenAI-compatible models to analyze search results and generate a neutral investigation summary (covering details like real name, location, occupation, interests, and confidence levels).

    Configuration

    • API Key: Set via the OPENAI_API_KEY environment variable or the openai_api_key key in settings.json.
    • Endpoint: Defaults to https://api.openai.com/v1. You can change this using openai_api_base_url in settings.json to support Azure OpenAI, OpenRouter, or local servers.
    • Model: Specify the model using the --ai-model flag.
    export OPENAI_API_KEY=sk-...
    maigret user --ai
    
    # Use a specific model
    maigret user --ai --ai-model gpt-4o-mini
  7. Use urlProbe for API-based presence checks

    main

    By default, Maigret uses the url pattern for both the request and the reported link. If a site requires an API or JSON endpoint to check for account existence, use the urlProbe key in data.json.

    • url: Must always be a human-openable profile page (e.g., https://github.com/{username}).
    • urlProbe: The actual endpoint used for the presence check (e.g., https://api.github.com/users/{username}).

    This ensures that Maigret performs an efficient check while still providing the user with a clickable, browsable link in the final report.

  8. Understand site protection tags and bypasses

    main

    Maigret uses the protection field in data.json to handle anti-bot mechanisms. These tags are categorized into Load-bearing (Maigret changes behavior) and Documentation-only (Maigret records the reason for failure).

    Load-bearing Tags

    • tls_fingerprint: The site blocks non-browser TLS handshakes. Maigret automatically switches to curl_cffi with Chrome-class TLS to bypass this.

    Documentation-only Tags

    These tags often result in the site being marked as "disabled": true because they require a real browser or external fixes:

    • ip_reputation: Site blocks datacenter/cloud IPs. Site remains enabled; users should use a proxy (--proxy) or residential IP.
    • cf_js_challenge: Cloudflare JS challenge. Requires a real browser; site is disabled.
    • cf_firewall: Cloudflare WAF block. Requires per-site bypass; site is disabled.
    • aws_waf_js_challenge: AWS WAF JS challenge. Requires a real browser or official SDK; site is disabled.
    • ddos_guard_challenge: DDoS-Guard anti-bot page. Requires a JS-capable bypass; site is disabled.
    • js_challenge: Fallback for unidentified JS-challenge systems.
    • custom_bot_protection: Fallback for custom/in-house bot protection systems.
  9. Pivot from ORCID to mainstream social identities

    main

    Once an ORCID is identified, Maigret can use the data within the academic record to find mainstream social media profiles through several pivot points:

    • ORCID researcher-urls: Self-declared links to lab homepages, Twitter, Bluesky, or Mastodon.
    • ORCID external-identifiers: Links to Scopus, ResearcherID, or Loop profiles.
    • OpenAlex last_known_institutions: Provides ROR IDs and institution IDs to pivot into institutional directories.
    • DBLP <url> tags: Often contains direct links to Google Scholar and ResearchGate.
    • Scholia wikidata_qid: Provides a Wikidata QID which can be used via SPARQL to find identifiers for VIAF, LCCN, GND, IMDb, GitHub, and hundreds of others.
  10. Configure Personal Info Extraction and Recursive Search

    main

    Maigret automatically parses account webpages to extract personal information (IDs, images, locations, etc.) and links to other profiles. It can also use discovered identifiers to initiate a recursive search, automatically starting new searches for usernames found in links.

    • Personal Info Extraction: Enabled by default. Disable with --no extracting.
    • Recursive Search: Enabled by default. Disable with --no-recursion.
  11. How tag states work in the web interface

    main

    In the Maigret web interface, the tag cloud allows you to toggle the inclusion or exclusion of tags through three distinct states:

    1. Include (Green): The tag is active and will be used to filter results.
    2. Exclude (Dark/Strikethrough): The tag is active as an exclusion filter.
    3. Neutral (Red): The tag has no effect on the search.
  12. Extract email addresses from academic profiles

    main

    Within the academic chain (ORCID, OpenAlex, arXiv, DBLP, Scholia), ORCID is the primary source for email addresses.

    • Verification: Look for the history.verified-primary-email flag in the response; this indicates the address was validated at registration.
    • Usage: Use these discovered emails as pivots for other OSINT tools like Holehe, HIBP, or Gravatar.
    • Alternative: For scientists using GitLab (common in research institutes like CERN), the public_email field may leak an email address, whereas GitHub's REST API typically does not expose it by default.