DOCSEARCHbyAlgolia
Submit Docs

feedparser

repository·main·Indexed 25 days ago

https://github.com/kurtmckee/feedparser

A Python library for parsing Atom, RSS, and JSON feeds. Version 6.0.14 supports Python 3.12 and 3.13. It provides tools to map XML elements from Atom 0.3, Atom 1.0, RSS 1.0, and RSS 2.0 (including Dublin Core and Content namespaces) into Python dictionaries. Recent updates include optimistic encoding detection to reduce memory usage, the removal of built-in custom HTTP request parameters and User-Agent headers in favor of external clients like requests, and the removal of certain module-level metadata attributes.

Tokens
34.9K
Snippets
69
Records
248
Agent score
78%

What's inside feedparser

  1. Understand how Universal Feed Parser sanitizes markup

    main

    To prevent security risks in feed aggregators, Universal Feed Parser automatically sanitizes embedded markup in several key elements.

    Sanitized elements by default:

    • entry.content
    • entry.summary
    • entry.title
    • feed.info
    • feed.rights
    • feed.subtitle
    • feed.title

    Important: Handling text/plain content If content is declared as (or determined to be) text/plain, it is not sanitized to avoid data loss. You should check the content type using entries[i].summary_detail.type. If it is text/plain, you must perform your own HTML escaping before rendering the content to ensure safety.

  2. Understand the `feed.docs` element

    main

    The feed.docs element is a URL pointing to the specification that the feed conforms to.

    Note that this element is rare in practice; most publishers omit it, and most clients ignore it. If the provided URL is a relative URI, it is resolved according to the project's base URI resolution rules.

  3. How Universal Feed Parser handles character encoding detection

    main

    Universal Feed Parser follows RFC 3023 to determine character encoding by checking the interaction between HTTP Content-Type headers and XML declarations.

    For application/xml subtypes (e.g., application/atom+xml, application/rss+xml), the precedence order is:

    1. The charset parameter in the HTTP Content-Type header.
    2. The encoding attribute in the XML declaration.
    3. utf-8.

    For text/xml subtypes, the XML declaration's encoding attribute is ignored. The precedence order is:

    1. The charset parameter in the HTTP Content-Type header.
    2. us-ascii.

  4. Understanding the security model: Whitelist vs Blacklist sanitization

    main

    The Universal Feed Parser uses a whitelist approach for sanitizing HTML, SVG, MathML, and CSS. Instead of trying to identify and block dangerous tags (a blacklist), it only allows known-safe elements and attributes.

    This approach is necessary because blacklisting is often ineffective against advanced attacks, such as:

    • Malicious CSS: Using style attributes or <style> tags to embed JavaScript (e.g., via expression() in older Internet Explorer versions).
    • Encoded Payloads: Using HTML entity encoding (decimal or hexadecimal) to hide executable strings within attribute values, which browsers may decode and execute.
    • Dangerous Tags: Tags like <script>, <applet>, <embed>, <object>, and <meta> that can trigger automatic code execution or redirects.
    • Event Handlers: Attributes starting with on (e.g., onload, onerror) that execute JavaScript.

  5. Use parsed creation dates with entries[i].created_parsed

    main
    If you need to perform date arithmetic or comparisons rather than working with raw strings, use entries[i].created_parsed. While entries[i].created returns the date as a string in the original feed format, entries[i].created_parsed provides the date as a parsed object (see advanced.date for details on the parsed format).

  6. Parse feed publication dates using feed.published_parsed

    main
    While feed.published returns the raw string, the feedparser library also parses this date into a structured format stored in feed.published_parsed. Use published_parsed when you need to perform date arithmetic or programmatic comparisons rather than just displaying the raw string.

  7. Understand the feed.ttl element

    main
    The feed.ttl element is part of the RSS (Rich Site Summary) specification. Its purpose is poorly defined in the official spec, but some clients interpret it as an inline caching mechanism. Note that this element ignores standard HTTP caching mechanisms and infrastructure. In practice, it is rarely used and its behavior is inconsistent across different feed readers.

  8. Understand the `etag` field in feed metadata

    main

    The etag field represents the ETag of the feed as specified in the HTTP headers. It is used to identify specific versions of a resource on a web server.

    Important constraints:

    • etag is only present if the feed was retrieved from a web server.
    • etag is only present if that web server explicitly provided an ETag HTTP header.
    • If you parse a feed from a local file or a string in memory, the etag field will not be present.

  9. How content normalization works in Universal Feed Parser

    main
    Universal Feed Parser normalizes data from various feed formats (Atom, CDF, and nine versions of RSS) so that they can be treated uniformly. This allows you to access elements of one format using the terminology and structure of another. For example, you can access an Atom feed using RSS terminology, or an RSS feed using Atom terminology.

  10. How namespace handling works in Universal Feed Parser

    main

    Universal Feed Parser attempts to expose all data in feeds, including elements within extension namespaces.

    Namespaced elements are handled in two ways:

    1. Core Mappings: Some common namespaced elements are automatically mapped to core elements.
    2. Prefix Mapping: Other namespaced elements are available using the pattern prefixelement.

    The available namespaces are stored in a namespaces dictionary in the parsed result, mapping {prefix: namespaceURI}. If a default namespace is defined in the feed, it is accessible via namespaces[''].

    Important Security Note: Data from namespaced elements is not sanitized, even if it contains HTML markup.

  11. Access detailed metadata for Atom elements

    main

    In Atom feeds, several elements (including title, subtitle, rights, summary, and content) share a common content model. Universal Feed Parser captures detailed metadata for these elements via _detail attributes or specific content structures.

    For title, subtitle, rights, and summary, you can access a dictionary containing the type, base URL, language, and the actual value.

    For the content element, the parser provides a list of content objects, where each object contains the MIME type, base URL, language, and the value (the actual content string).

  12. Understand the `modified` field in feed metadata

    main

    The modified field represents the last-modified date of a feed, extracted from the Last-Modified HTTP header of the web server that served the feed.

    Important constraints:

    • modified is only available if the feed was retrieved from a web server.
    • modified is only available if that web server explicitly provided a Last-Modified HTTP header.
    • If you parse a feed from a local file or a string in memory, the modified field will not be present.