MetaInspector Ruby Gem

repository·master·Indexed 21 days ago

https://github.com/jaimeiniesta/metainspector

A Ruby gem for web scraping that extracts metadata from URLs, including titles, descriptions, links, images, and meta tags. It provides features for handling HTTP response status, custom request configurations (timeouts, retries, headers), and support for raw HTML strings. MetaInspector includes specialized parsers for head links, stylesheets, and canonicals, and allows for the export of scraped data as a hash.

Tokens: 7.6K
Snippets: 31
Records: 33
Agent score: 77%

What's inside MetaInspector

Install MetaInspector
master
You can install MetaInspector via RubyGems or by adding it to your Gemfile for Rails applications.
Via CLI:
```
gem install metainspector
```
Via Gemfile:
```
gem 'metainspector'
```
```
gem install metainspector
```
https://github.com/jaimeiniesta/metainspector/blob/master/README.md
Configure Faraday options and Caching
master
You can pass low-level configuration to the underlying Faraday library using the :faraday_options key.
SSL Configuration
To disable SSL verification (e.g., to bypass Faraday::SSLError):
```
MetaInspector.new('https://example.com', faraday_options: { ssl: { verify: false } })
```
HTTP Caching
To enable response caching, pass a :store key within the :faraday_http_cache option. This requires a compatible cache store (e.g., from ActiveSupport::Cache):
```
cache = ActiveSupport::Cache.lookup_store(:file_store, '/tmp/cache')
page = MetaInspector.new('http://example.com', faraday_http_cache: { store: cache })
```
https://github.com/jaimeiniesta/metainspector/blob/master/README.md
How to implement a custom parser in MetaInspector
master
To create a specialized parser, inherit from MetaInspector::Parsers::Base.
When implementing a custom parser, you must initialize it with a main_parser instance. This main_parser acts as a central hub, allowing your specialized parser to request the parsed document or communicate with other parsers (e.g., requesting a base_url from the LinksParser via the main parser).
Use the private cleanup method to process Nokogiri search results. This method transforms a collection of results into a cleaned array of unique, non-empty, stripped strings.
```
module MetaInspector
  module Parsers
    class MyCustomParser < Base
      def parse
        # Access the document via the main_parser
        doc = @main_parser.document
        
        # Perform searches and use cleanup to sanitize results
        results = doc.css('.some-class').map { |el| el['data-value'] }
        cleanup(results)
      end
    end
  end
end
```
https://github.com/jaimeiniesta/metainspector/blob/master/lib/meta_inspector/parsers/base.rb
Configure MetaInspector request options
master
When initializing MetaInspector.new(url, options), you can pass several options to control the scraping behavior:
Encoding
If you encounter MetaInspector::RequestError, "invalid byte sequence in UTF-8", force the encoding:
```
page = MetaInspector.new(url, :encoding => 'UTF-8')
```
Timeouts and Retries
Control how long to wait for connections and reading, and how many times to retry:
- connection_timeout: Max seconds to wait for a connection (default: 20).
- read_timeout: Max seconds to wait to read the page once connected (default: 20).
- retries: Number of retry attempts (default: 3).
```
page = MetaInspector.new('www.google', :connection_timeout => 10, :read_timeout => 5, :retries => 4)
```
Redirections
By default, MetaInspector follows up to 10 redirects.
- To disable: :allow_redirections => false.
- To limit count: Use :faraday_options => { redirect: { limit: 5 } }.
- To add logic between redirects, use a callback via :faraday_options.
Headers
Override default headers (User-Agent and Accept-Encoding) using the :headers key:
```
page = MetaInspector.new('example.com', :headers => {'User-Agent' => 'My custom User-Agent'})
```
Non-HTML Content
By default, MetaInspector raises an error for non-HTML content types. To allow them:
```
page = MetaInspector.new('http://example.com/image.png', :allow_non_html_content => true)
```
URL Normalization
MetaInspector uses the Addressable gem to normalize URLs (adding schemes, trailing slashes, etc.). To disable this, use:
```
page = MetaInspector.new(url, :normalize_url => false)
```
Image Downloading
To prevent MetaInspector from downloading image headers (used to find the largest image), use:
```
page = MetaInspector.new('http://example.com', download_images: false)
```
https://github.com/jaimeiniesta/metainspector/blob/master/README.md
Handle MetaInspector exceptions
master
Scraping can fail due to network or parsing issues. MetaInspector wraps these in specific error classes that you should rescue:
- MetaInspector::TimeoutError: Raised when fetching a page exceeds the configured timeouts after all retries.
- MetaInspector::RequestError: Raised during the request phase (e.g., 404 Not Found, SSL failure, invalid URI).
- MetaInspector::ParserError: Raised when there is an error parsing the page contents.
- MetaInspector::NonHtmlError: Raised when the content type is not text/html (unless :allow_non_html_content is set to true).
Example Rescue Pattern:
```
begin
  page = MetaInspector.new(url)
rescue MetaInspector::TimeoutError
  # Handle timeout (e.g., retry later)
  enqueue_for_future_fetch_attempt(url)
rescue MetaInspector::RequestError => e
  # Handle request failures
  puts "Request failed: #{e.message}"
else
  # Success
  render_rich(page)
end
```
```
begin
  page = MetaInspector.new(url)
rescue MetaInspector::TimeoutError
  enqueue_for_future_fetch_attempt(url)
  render_simple(url)
else
  render_rich(page)
end
```
https://github.com/jaimeiniesta/metainspector/blob/master/README.md
Basic usage of MetaInspector
master
To use MetaInspector, require the gem, initialize a new instance with a URL, and access the scraped properties.
```
require 'metainspector'

page = MetaInspector.new('http://github.com')
puts page.title
puts page.meta['description']
```
https://github.com/jaimeiniesta/metainspector/blob/master/README.md
Access meta tags
master
MetaInspector provides three ways to access meta tags, ranging from highly structured to flattened for convenience. Note that all keys are converted to lowercase.
1. meta_tags: Returns a nested hash grouped by tag type (name, http-equiv, property, charset). Values are always arrays to handle duplicates.
2. meta_tag: Returns a hash grouped by type, but values are singular (not arrays).
3. meta: A flattened, simplified hash containing all meta tags across all types for easy access.
```
# 1. Grouped by type (values are arrays)
page.meta_tags['property']['og:title'] # => ['An OG title']

# 2. Grouped by type (values are singular)
page.meta_tag['name']['description']   # => 'the description'

# 3. Flattened (easiest for single values)
page.meta['og:title']                  # => 'An OG title'
page.meta['author']                    # => 'Joe Sample'
```
https://github.com/jaimeiniesta/metainspector/blob/master/README.md

Scrape text content (titles, descriptions, and headings)

master

Retrieve text-based metadata. Many methods have a "best" variant that uses heuristics to pick the most relevant content (e.g., checking og:description if the standard meta description is missing).

page.title               # Title from head section
page.best_title          # Heuristic best title
page.author              # Author from meta tag
page.best_author         # Heuristic best author
page.description         # Meta description
page.best_description    # Heuristic best description (checks meta, og:description, twitter:description, etc.)
page.h1                  # Array of h1 text
page.h2                  # Array of h2 text
page.h3                  # Array of h3 text
page.h4                  # Array of h4 text
page.h5                  # Array of h5 text
page.h6                  # Array of h6 text

Scrape links

master

Access various collections of links found on the page, categorized by type or protocol.

page.links.raw           # Every link found, unprocessed
page.links.all           # Every link as an absolute URL
page.links.http          # Every HTTP link
page.links.non_http      # Every non-HTTP link
page.links.internal      # Every internal link as an absolute URL
page.links.external      # Every external link

Initialize a MetaInspector instance
master
To scrape a page, create a new instance of MetaInspector by providing a URL.
- Standard URL: Provide the full URL including the scheme.
- Implicit HTTP: If the scheme is omitted, http:// is used by default.
- Custom HTML: You can bypass network requests by providing a raw HTML string via the :document option.
```
# Standard usage
page = MetaInspector.new('https://github.com')

# Implicit http://
page = MetaInspector.new('github.com')

# Using provided HTML document
page = MetaInspector.new('https://github.com', :document => "<html>...</html>")
```
https://github.com/jaimeiniesta/metainspector/blob/master/README.md
Access HTTP response status and headers
master
You can inspect the raw HTTP response metadata using the response method on the instance.
```
page.response.status  # Returns integer status code (e.g., 200)
page.response.headers # Returns a hash of response headers
```
https://github.com/jaimeiniesta/metainspector/blob/master/README.md

Scrape URL and host information

master

MetaInspector provides methods to extract and manipulate URL components, including support for removing tracking parameters.

page.url                 # URL of the page
page.tracked?            # true if the url contains known tracking parameters
page.untracked_url       # url with known tracking parameters removed
page.untrack!            # removes known tracking parameters from the url
page.scheme              # Scheme (http, https)
page.host                # Hostname (e.g., github.com)
page.root_url            # Root url (scheme + host, e.g., https://github.com/)

MetaInspector Ruby Gem

What's inside MetaInspector

SSL Configuration

HTTP Caching

Encoding

Timeouts and Retries

Redirections

Headers

Non-HTML Content

URL Normalization

Image Downloading