url-metadata

The urlMetadata function returns a Promise that resolves to a JSON object containing extracted metadata.

Key behaviors:

• Redirects: The url field in the returned object represents the final destination in the request chain. The redirects field provides a count and a chain of RedirectHop objects.
• Raw Content: Extracted content is returned raw and unsanitized. You are responsible for sanitizing all values (e.g., using DOMPurify for HTML or parameterized queries for SQL) before use.
• Dynamic Fields: Any meta tags found on the page that are not part of the standard set are appended to the object as new fields.
• Special Field Formats:
  • canonical: A string of the first <link rel="canonical"> found. canonicalUrls is an array of all canonical URLs found.
  • hreflang, favicons, and responseHeaders: Arrays of objects.
  • jsonld: An array of objects.
  • citation_* tags: These return as an array of strings to support the Google Scholar spec (e.g., citation_author: ['Author A', 'Author B']).
  • Other meta tags: Usually returned as strings with comma-separated values.

To bypass 403 errors or access restricted pages, use proxyUrl. Providing a proxyUrl triggers proxy mode, where the request is routed through a third-party service.

• Timeout: In proxy mode, the default timeout automatically increases from 10s to 60s.
• Parameters: Use proxyParams to pass vendor-specific query parameters (e.g., API keys) verbatim. If a vendor requires authentication via headers, use requestHeaders instead.
• Limitations: This package only parses HTML. If the proxy returns JSON or CSV, it will throw unsupported content type.
• Metadata: redirects, responseHeaders, and performance timing reflect the proxy's interaction, not the target server's.

// Example with ScraperAPI
const metadata = await urlMetadata('https://hardto.get', {
  proxyUrl: 'https://api.scraperapi.com/',
  proxyParams: {
    api_key: '<YOUR_API_KEY>'
  }
});

// Example with ScrapingAnt
const metadata = await urlMetadata('https://hardto.get', {
  proxyUrl: 'https://api.scrapingant.com/v2/general',
  proxyParams: {
    'x-api-key': '<YOUR_API_KEY>'
  }
});

403, 404, or CAPTCHA errors

If the target server blocks your request, use the built-in Proxy Mode to bypass bot detection.

Proxy mode 'unsupported content type' errors

This occurs if your proxy URL or parameters return something other than HTML. Ensure your proxy is configured to return HTML content.

Proxy mode 'unsupported content type: text/x-component' errors

This happens when using render: true (headless browser) on sites like Next.js App Router that return React Server Component payloads instead of HTML. This is a target site behavior. Try retrying without render: true or using a different proxy.

'No fetch implementation found'

This occurs in environments without a native fetch API (e.g., older browsers or Node.js < 18.17).

• Solution: Upgrade your environment or downgrade to url-metadata version 2.5.0 which uses the request module.

'DNS Lookup' errors

The package includes an SSRF filtering agent that prevents calls to private, link-local, or reserved IP addresses.

• Solution: Pass custom requestFilteringAgentOptions to modify or disable this behavior.

'Response status code 0' or 'CORS' errors

This indicates a network or protocol-level failure. Possible causes:

• CORS: Change the mode option (e.g., cors, same-origin) or ensure the target server allows your origin.
• Certificates/Protocol: Accessing https with invalid certificates or mixing http and https origins.
• Browser Extensions: Ad-blockers or privacy protectors may be interfering.

By default, the return type Result is loose (any) to ensure compatibility. To get autocompletion and type safety, you should cast the result using one of the following interfaces:

1. urlMetadata.KnownFields: Includes all standard known fields. Any extra meta tags found on the page will be accessible via any (e.g., metadata.foobar).
2. urlMetadata.KnownFieldsStrict: A closed shape containing only known fields. This will cause compile errors if you attempt to access arbitrary page-specific meta tags.
3. Partial<urlMetadata.KnownFields>: When using the fields or omitEmpty: true options, the return type automatically narrows to this partial type, making known fields optional.

Refer to index.d.ts for the full list of exported interfaces including Options, HreflangTag, FaviconTag, Heading, ImgTag, RedirectHop, and UrlMetadataError.

import urlMetadata from 'url-metadata';

// 1. Known fields + arbitrary meta tags (loose)
const metadata = await urlMetadata(url) as urlMetadata.KnownFields;
metadata.title;      // string
metadata.foobar;     // any

// 2. Known fields ONLY (strict)
const strict = await urlMetadata(url) as urlMetadata.KnownFieldsStrict;

// 3. Automatic narrowing with options
const partial = await urlMetadata(url, { fields: ['og', 'network'] });
partial['og:title'];        // string | undefined
partial.responseStatusCode; // number | undefined