open-graph-scraper

repository·master·Indexed 20 days ago

https://github.com/jshemas/opengraphscraper

A Node.js module for scraping Open Graph, Twitter Card, and other metadata from websites. It supports server-side use, custom meta tag extraction, JSON-LD parsing, and the ability to scrape from either a URL or a raw HTML string. Version 6.12.0 utilizes the Fetch API for requests and provides TypeScript type definitions for results and configuration options.

Tokens
4.5K
Snippets
15
Records
18
Agent score
71%

What's inside open-graph-scraper

  1. Understand the scraper result formats

    master

    The scraper returns one of two result types depending on whether the request succeeded or failed:

    SuccessResult

    Returned when the fetch and scraping process completes successfully.

    • error: false
    • html: The raw HTML string of the page.
    • response: The response object from the fetch request.
    • result: An OgObject containing the scraped metadata.

    ErrorResult

    Returned when an error occurs during the process.

    • error: true
    • html: undefined
    • response: undefined
    • result: An OgObject (may contain error details).
    export interface SuccessResult {
      error: false;
      html: string;
      response: object;
      result: OgObject;
    }
    
    export interface ErrorResult {
      error: true;
      html: undefined;
      response: undefined;
      result: OgObject;
    }
  2. Handle Success and Error results

    master

    The run function returns a Promise that resolves to a SuccessResult or rejects with an ErrorResult.

    SuccessResult Structure

    When scraping is successful, the returned object follows this shape:

    • error: false
    • result: The scraped Open Graph object (ogObject).
    • response: The HTTP response object from the fetch request.
    • html: The raw HTML string of the page.

    ErrorResult Structure

    If the scraper encounters an error, it throws an object with this shape:

    • error: true
    • result: An object containing:
      • success: false
      • requestUrl: The URL that was attempted.
      • error: The error message string.
      • errorDetails: The original error object.
    • response: undefined
    • html: undefined
  3. Understand the Scraper Result types

    master

    The scraper returns one of two result types depending on whether the operation succeeded or failed:

    SuccessResult

    Returned when the scrape is successful.

    • error: false
    • html: The raw HTML string of the page.
    • response: The response object from the fetch request.
    • result: An OgObject containing the scraped metadata.

    ErrorResult

    Returned when the scrape fails (e.g., network error, invalid URL).

    • error: true
    • html: undefined
    • response: undefined
    • result: An OgObject (usually containing error details).
    type ScrapeResult = SuccessResult | ErrorResult;
  4. Basic usage of open-graph-scraper

    master

    Import open-graph-scraper and call it with an options object containing a url. The function returns a Promise that resolves to an object containing error (boolean), html (string), result (metadata object), and response (Fetch API response).

    const ogs = require('open-graph-scraper');
    const options = { url: 'http://ogp.me/' };
    ogs(options)
      .then((data) => {
        const { error, html, result, response } = data;
        console.log('error:', error);  // This returns true or false. True if there was an error. The error itself is inside the result object.
        console.log('html:', html); // This contains the HTML of page
        console.log('result:', result); // This contains all of the Open Graph results
        console.log('response:', response); // This contains response from the Fetch API
      })
  5. Set a custom User Agent

    master

    If a site blocks the default undici user agent, you can provide a custom user agent via fetchOptions.headers.

    const ogs = require("open-graph-scraper");
    const userAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36';
    ogs({ url: 'https://www.wikipedia.org/', fetchOptions: { headers: { 'user-agent': userAgent } } })
      .then((data) => {
        const { error, html, result, response } = data;
        console.log('error:', error);
        console.log('html:', html);
        console.log('result:', result);
        console.log('response:', response);
      })
  6. Define custom meta tags to scrape

    master

    Use the customMetaTags option to extract specific meta tags that are not part of the standard Open Graph or Twitter Card sets. Each object in the array requires a property (the tag name/attribute) and a fieldName (the key in the resulting result.customMetaTags object). The multiple boolean indicates if the tag appears more than once on the page.

    const ogs = require('open-graph-scraper');
    const options = {
      url: 'https://github.com/jshemas/openGraphScraper',
      customMetaTags: [{
        multiple: false, // is there more than one of these tags on a page (normally this is false)
        property: 'hostname', // meta tag name/property attribute
        fieldName: 'hostnameMetaTag', // name of the result variable
      }],
    };
    ogs(options)
      .then((data) => {
        const { result } = data;
        console.log('hostnameMetaTag:', result.customMetaTags.hostnameMetaTag); // hostnameMetaTag: github.com
      })
  7. Configure JSON-LD parsing behavior

    master

    Use jsonLDOptions to control how errors are handled during JSON-LD parsing.

    • throwOnJSONParseError: If true, the error will be thrown.
    • logOnJSONParseError: If true, the error will be logged to the console.
    const ogs = require("open-graph-scraper");
    const userAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36';
    ogs({ url: 'https://www.wikipedia.org/', jsonLDOptions: { throwOnJSONParseError: true } })
      .then((data) => {
        const { error, html, result, response } = data;
        console.log('error:', error);
        console.log('html:', html);
        console.log('result:', result);
        console.log('response:', response);
      })
  8. Scrape metadata from an HTML string

    master

    Instead of providing a URL, you can pass a raw HTML string via the html option. This is useful if you have already fetched the content or are using open-graph-scraper-lite in a browser environment.

    const ogs = require('open-graph-scraper');
    const options = {
      html: `<html><head>
      <link rel="icon" type="image/png" href="https://bar.com/foo.png" />
      <meta charset="utf-8" />
      <meta property="og:description" name="og:description" content="html description example" />
      <meta property="og:image" name="og:image" content="https://www.foo.com/bar.jpg" />
      <meta property="og:title" name="og:title" content="foobar" />
      <meta property="og:type" name="og:type" content="website" />
      </head></html>`
    };
    ogs(options)
      .then((data) => {
        const { result } = data;
        console.log('result:', result);
      })
  9. Import types for TypeScript

    master

    If using TypeScript, you can import types like SuccessResult from open-graph-scraper/types to ensure type safety for the scraper's response.

    // example of how to get types
    import type { SuccessResult } from 'open-graph-scraper/types';
    const example: SuccessResult = {
      result: { ogTitle: 'this is a title' },
      error: false,
      response: {},
      html: '<html></html>'
    }
    
    // import example
    import ogs from 'open-graph-scraper';
    const options = { url: 'http://ogp.me/' };
    ogs(options)
      .then((data) => {
        const { error, html, result, response } = data;
        console.log('error:', error);
        console.log('html:', html);
        console.log('result:', result);
        console.log('response:', response);
      });
  10. Configure URL validation with ValidatorSettings

    master

    The ValidatorSettings interface allows you to customize the strictness of the URL validation performed by the scraper using validator.js settings.

    Common options include:

    • allow_protocol_relative_urls: Allows URLs starting with //.
    • protocols: A list of allowed protocols (e.g., ['http', 'https']).
    • require_host: If false, validation passes even if the host is missing.
    • require_protocol: If true, the URL must include a protocol.
    • validate_length: If false, skips string length validation.
    const urlValidatorSettings: ValidatorSettings = {
      allow_fragments: true,
      allow_protocol_relative_urls: true,
      allow_query_components: true,
      allow_trailing_dot: true,
      allow_underscores: true,
      protocols: ['http', 'https'],
      require_host: true,
      require_port: false,
      require_protocol: true,
      require_tld: true,
      require_valid_protocol: true,
      validate_length: true
    };
  11. Configure JSON-LD parsing options

    master

    The JSONLDOptions interface allows you to control how the scraper handles JSON-LD data found in the HTML.

    • throwOnJSONParseError: If true, the scraper will throw an error if JSON-LD parsing fails.
    • logOnJSONParseError: If true, the scraper will log errors to the console instead of throwing.
    const jsonLDOptions: JSONLDOptions = {
      throwOnJSONParseError: false,
      logOnJSONParseError: true
    };