DOCSEARCHbyAlgolia
Submit Docs

Muffet

repository·main·Indexed 25 days ago

https://github.com/raviqqe/muffet

A high-speed website link checker that recursively scrapes and inspects pages for broken links. It supports multiple output formats including text, JSON, and JUnit XML, and can be installed via Go, Homebrew, or Docker. Muffet inspects tags such as <a>, <img>, <link>, and <script>, and provides extensive CLI configuration for rate limiting, connection management, and URL filtering via regular expressions.

Tokens
5.1K
Snippets
15
Records
52
Agent score
82%

What's inside Muffet

  3. Run Muffet using Docker

    main

    If your website is hosted remotely or your Docker installation allows you to connect to the host network, you can run Muffet using a Docker image to test your websites. Pass the target URL as an argument to the docker run command.

    docker run raviqqe/muffet https://shady.bakery.hotland

  8. Configure the docs content collection

    main

    In an Astro project using Starlight, define the docs content collection by using defineCollection from astro:content. The collection must use the docsLoader() from @astrojs/starlight/loaders and the docsSchema() from @astrojs/starlight/schema to correctly process documentation files.

    import { defineCollection } from "astro:content";
import { docsLoader } from "@astrojs/starlight/loaders";
import { docsSchema } from "@astrojs/starlight/schema";

export const collections = {
  docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }),
};

  9. Run Muffet to check website links

    main

    To perform a recursive scrape and inspection of all pages in a website, run the muffet command followed by the target URL. Muffet inspects various tags including a, img, link, and script.

    muffet https://shady.bakery.hotland

  10. Find links within HTML nodes using linkFinder

    main

    The linkFinder type provides functionality to traverse an HTML tree and extract URLs from specific attributes (like href, src, srcset, and content). It resolves relative URLs against a provided base URL and uses a linkFilterer to validate the discovered links.

    To use linkFinder, you must provide an implementation of the linkFilterer interface (which must include an IsValid(*url.URL) bool method) via newLinkFinder.

  11. Format page results with pageResultFormatter

    main

    The pageResultFormatter is used to convert pageResult data into a human-readable string format. It supports colorized output via the aurora library and can include detailed success link information if verbose mode is enabled.

    To use it, initialize a formatter using newPageResultFormatter(verbose bool, color bool) and then call the Format(r *pageResult) method.

  12. Execute Muffet commands via the command struct

    main

    The command struct is the primary entry point for executing Muffet's crawling and checking logic. You can instantiate it using newCommand and trigger the execution by calling Run(args []string).

    Run returns true if the operation was successful (all checked pages passed) and false otherwise. It handles error printing to stderr automatically.