mwparserfromhell Documentation

Unlike the MediaWiki engine which generates HTML and resolves template contents, mwparserfromhell acts as a direct interface to the raw source code only. This leads to the following behaviors:

• No Template Transclusion Awareness: The parser cannot see the contents of a template. If a template contains a closing tag (e.g., {{end-bold}} containing </b>), the parser will not recognize it and may treat the preceding tag as unfinished.
• Link Adjacency: Templates placed immediately after external links (e.g., http://example.com{{foo}}) are treated as part of the link itself.
• XML Tag Treatment: Any text that resembles an XML tag is treated as a tag, regardless of whether it is a recognized name, because valid tag names depend on specific MediaWiki extensions.
• Namespace and Link Limitations:
  • Localized namespace names are not recognized; for example, [[File:...]] is treated as a standard wikilink.
  • Word-ending links (linktrail) are not supported due to their language-specific nature.

A mwparserfromhell.Wikicode object acts as a container for the parsed MediaWiki syntax. It inherits string-like behavior (e.g., str(wikicode) returns the original text) but allows for structural manipulation of templates, links, and formatting.

Key capabilities include:

• Filtering: Use .filter_templates() to find all template nodes.
• Template Access: Templates have .name and .params attributes.
• Parameter Retrieval: Use .get(key_or_index) to access specific template parameters.
• Modification: Wikicode objects can be treated like lists, supporting methods like .append(), .insert(), .remove(), and .replace() to alter the content.

import mwparserfromhell

text = "I has a template! {{foo|bar|baz|eggs=spam}} See it?"
wikicode = mwparserfromhell.parse(text)

templates = wikicode.filter_templates()
foo = templates[0]
print(foo.name)          # 'foo'
print(foo.params)       # ['bar', 'baz', 'eggs=spam']
print(foo.get("eggs").value) # 'spam'

mwparserfromhell handles nested structures automatically. When calling filter_templates(), the method returns all templates found at all levels of nesting.

If you want to avoid automatic recursion and instead traverse the tree manually, pass recursive=False to filter_templates(). This allows you to inspect the contents of a node (which are themselves Wikicode objects) and call filter_templates() on them individually.

# Recursive behavior (default)
text = "{{foo|{{bar}}={{baz|{{spam}}}}}}"
print(mwparserfromhell.parse(text).filter_templates())
# Output: ['{{foo|{{bar}}={{baz|{{spam}}}}}}', '{{bar}}', '{{baz|{{spam}}}}', '{{spam}}']

# Manual traversal
code = mwparserfromhell.parse("{{foo|this {{includes a|template}}}}")
# Get only the top-level template
top_level = code.filter_templates(recursive=False)[0]
# Access the nested content inside the parameter
inner_content = top_level.get(1).value
# Parse the inner content manually
print(inner_content.filter_templates()[0])

Because mwparserfromhell acts as a direct interface to the source code rather than the rendered HTML, users should be aware of several limitations:

1. Transclusion Blindness: It cannot see syntax elements produced inside a template. If a template contains a closing tag like `</b

`, the parser may treat the preceding opening tag as unclosed.

2. Link Adjacency: Templates immediately following an external link (e.g., http://example.com{{foo}}) may be incorrectly parsed as part of the link.
3. Overlapping Syntax: When syntax elements cross over (e.g., {{echo|''Hello}}), the parser may treat the first construct as plain text.
   • Workaround: Use mwparserfromhell.parse(text, skip_style_tags=True) to treat text formatting tags (like '' and ''') as plain text to avoid confusion.
4. Wiki-specific settings: It does not support Word-ending links or recognize localized namespace names (treating [[File:...]] as a standard wikilink).

Wikicode objects are mutable and support several methods for altering markup:

• Adding/Removing Parameters: Use .add(name, value) to add a parameter to a template, or .remove() to delete nodes.
• Replacing Content: Use .replace(old_text, new_text) to swap specific strings within the Wikicode object.
• List-like operations: Wikicode objects support .append(), .insert(), and .remove().
• Matching names: Use the .matches(string) method on a template's name to perform case-insensitive and whitespace-agnostic comparisons.

text = "{{cleanup}} '''Foo''' is a [[bar]]. {{uncategorized}}"
code = mwparserfromhell.parse(text)

for template in code.filter_templates():
    # Use .matches() for robust name comparison
    if template.name.matches("Cleanup") and not template.has("date"):
        template.add("date", "July 2012")

# Replace a specific template string
code.replace("{{uncategorized}}", "{{bar-stub}}")

print(str(code))
# {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}

If you are not using a specialized library, you can parse MediaWiki content by fetching it via the MediaWiki API using the requests library. Ensure you provide a User-Agent header and target the revisions property with rvprop=content to extract the raw wikitext. Pass the extracted content string to mwparserfromhell.parse().

import mwparserfromhell
import requests

API_URL = "https://en.wikipedia.org/w/api.php"

def parse(title):
    params = {
        "action": "query",
        "prop": "revisions",
        "rvprop": "content",
        "rvslots": "main",
        "rvlimit": 1,
        "titles": title,
        "format": "json",
        "formatversion": "2",
    }
    headers = {"User-Agent": "My-Bot-Name/1.0"}
    req = requests.get(API_URL, headers=headers, params=params)
    res = req.json()
    revision = res["query"]["pages"][0]["revisions"][0]
    text = revision["slots"]["main"]["content"]
    return mwparserfromhell.parse(text)