repository·master·Indexed 21 days ago
https://github.com/scrapinghub/extructA specialized Python library for extracting embedded metadata from HTML documents. It supports multiple formats including JSON-LD, W3C HTML Microdata, Open Graph, Microformats (via mf2py), RDFa (experimental via rdflib), and Dublin Core Metadata (DC-HTML-2003). The library provides a high-level extract() method for all-in-one extraction, individual extractor classes for specific syntaxes, and a command-line tool for fetching and processing URLs.
extruct provides extraction capabilities for the following formats:
mf2py)rdflib)The extruct command line tool requires the requests library, which is not included in the base installation. To install extruct with the necessary dependencies for the CLI, use the cli extra requirement:
pip install 'extruct[cli]'You can install the extruct library using pip.
pip install extructThe extruct command line tool fetches a URL and outputs extracted metadata (Microdata, JSON-LD, RDFa, Open Graph, and Microformat) directly to stdout.
To extract all supported metadata formats from a page:
extruct "http://example.com"Use extruct.dublincore.DublinCoreExtractor to extract Dublin Core metadata from HTML <meta> and <link> elements. The extract(html) method returns a list containing a dictionary with elements (individual metadata items), namespaces, and terms. Each element in elements includes the URI, content, name, and optional lang or scheme.
from extruct.dublincore import DublinCoreExtractor
html = '<meta name="DC.title" content="Example Title" />'<br>
dublinlde = DublinCoreExtractor()
data = dublinlde.extract(html)Use extruct.microformat.MicroformatExtractor to extract Microformats from HTML. The extract(html) method returns a list of dictionaries containing the type (e.g., h-entry) and a properties dictionary. Properties can contain simple string values or nested dictionaries for complex types (like h-card).
from extruct.microformat import MicroformatExtractor
html = '<article class="h-entry"><h1 class="p-name">Title</h1></article>'<br>
microformate = MicroformatExtractor()
data = microformate.extract(html)Use extruct.jsonld.JsonLdExtractor to extract JSON-LD structured data from HTML. The extract(html) method returns a list of expanded JSON-LD nodes containing keys like @context, @type, and other schema properties.
from extruct.jsonld import JsonLdExtractor
html = '...<script type="application/ld+json">{"@type": "Person", "name": "John Doe"}</script>...' # truncated
jslde = JsonLdExtractor()
data = jslde.extract(html)
# data returns a list of JSON-LD dictionariesBy default, different syntaxes return data in their native formats. You can use the uniform=True option to transform the output of microformat, opengraph, microdata, dublincore, and json-ld into a consistent structure:
{
'@context': '...',
'@type': '...',
# All other properties as keys here
}Note: The rdfa structure is not currently uniformed.
# Set uniform=True to get a standardized dictionary structure
data = extruct.extract(r.text, base_url, syntaxes=['microdata', 'opengraph', 'rdfa'], uniform=True)Use extruct.w3cmicrodata.MicrodataExtractor to extract W3C Microdata from HTML. The extract(html) method returns a list of dictionaries, where each dictionary contains the type (itemtype) and a properties dictionary mapping property names to their values.
from extruct.w3cmicrodata import MicrodataExtractor
html = '...'<html><body itemscope itemtype="http://n.whatwg.org/work"><img itemprop="work" src="images/house.jpeg"></body></html>...' # truncated for brevity
mde = MicrodataExtractor()
data = mde.extract(html)
# data returns a list of items with 'type' and 'properties'Use extruct.opengraph.OpenGraphExtractor to extract Open Graph Protocol (OGP) metadata. The extract(html) method returns a list containing a dictionary with a namespace mapping and a properties list. The properties list contains pairs of [property_name, value] (e.g., ['og:title', 'Title Value']).
from extruct.opengraph import OpenGraphExtractor
html = '<meta property="og:title" content="Himanshu\'s Open Graph Protocol"/>...'<br>
opengraphe = OpenGraphExtractor()
data = opengraphe.extract(html)Use extruct.rdfa.RDFaExtractor to extract RDFa from HTML. Note that this is marked as experimental and may require html5lib to be installed to function correctly. The extract(html, base_url=...) method returns a list of expanded JSON-LD nodes. Providing a base_url is useful for resolving relative resources.
from extruct.rdfa import RDFaExtractor
html = '<body prefix="dc: http://purl.org/dc/terms/">...</body>'<br>
rdfae = RDFaExtractor()
data = rdfae.extract(html, base_url='http://www.example.com/index.html')