repository·master·Indexed 42 days ago
https://github.com/janreges/siteone-crawler-guiA desktop application providing a graphical interface for the SiteOne Crawler command-line tool. It enables users to perform in-depth website crawls, analyze errors, generate HTML, JSON, or TXT reports, create sitemaps, and produce offline versions of websites without using a terminal. Built with Electron, Svelte, and TypeScript, it includes advanced configuration for resource filtering, markdown export, and real-time progress monitoring.
The application is built using the following technologies:
Follow this workflow to perform a website crawl:
SiteOne Crawler application.Result tab to:Note: All generated content (reports, offline websites, cache, etc.) is automatically stored in a folder named SiteOne-Crawler on your Desktop.
Run the development server to launch the Electron app with hot-reloading:
npm run devImportant: Builds must be run on the target platform.
Note on Type Checking: Due to existing svelte-check issues, you may need to remove npm run typecheck && from the build script in package.json to successfully create a build.
Available build commands:
# For Windows (x64)
npm run build:win-x64
# For macOS (x64)
npm run build:mac-x64
# For macOS (arm64)
npm run build:mac-arm64
# For Linux (x64)
npm run build:linux-x64
# For Linux (arm64)
npm run build:linux-arm64To build or modify the application, you must first clone the repository and initialize the required submodules, which include the command-line SiteOne Crawler tool.
# checkout repository
git clone https://github.com/janreges/siteone-crawler-gui.git
cd siteone-crawler-gui
# initialize and download submodule with command-line SiteOne Crawler into src/siteone-crawler
git submodule update --init --recursive
# install dependencies
npm installTo use the application as an end-user, follow these steps:
The MiniStatsData class is used to aggregate and summarize crawler statistics. It tracks metrics related to HTML performance, HTTP status code distributions, and content type sizes (HTML, Images, JS, CSS, Fonts, etc.).
htmlNumber (count), htmlTimeTotal, htmlTimeAvg, htmlTimeMin, htmlTimeMax, and htmlTimeAvgDirection ('up' or 'down').code200x, code300x, code400x, code500x, and codeErr.Number) and total size (Size) for various file types:contentHtmlcontentImgcontentJscontentCsscontentFontscontentFiles (fallback for other types)The CrawlerForm component provides real-time feedback during a crawl through several mechanisms:
TimelineState tracks progress (e.g., 10/100 | 10%) extracted from the terminal output.MiniStatsData provides granular details about processed URLs (file type, duration, size).NetworkStats tracks network-level activity.CrawlerMessageType.STOPPED), the UI allows switching to a 'Result' tab to access generated files (HTML reports, JSON, sitemaps, or offline clones) via direct links.The CrawlerForm component manages the configuration and execution state of the SiteOne Crawler. It provides a structured interface to set parameters for crawling depth, device types, resource filtering, and output formats.
Key configuration categories include:
// Example of the shape of the configuration object used by the form
const formData = {
url: 'https://example.com',
maxDepth: 2,
device: 'desktop',
workers: 3,
disableJavascript: true,
resultStorage: 'file',
outputHtmlReport: '/path/to/report.html'
};Once crawling is complete, the GUI provides direct access to various report types:
sitemap.xml or sitemap.txt files.Manual Access:
All outputs are stored in a folder named SiteOne-Crawler on your desktop. You can navigate there via terminal using:
cd [path_to_output_folder]
The CrawlerFormContent class defines the configuration schema used to control the SiteOne Crawler. It covers everything from basic URL settings to advanced expert modes, resource filtering, and various export formats (Markdown, Offline, Sitemap, etc.).
url, maxDepth, device (desktop, tablet, mobile), userAgent, proxy, timeout, etc.output (text, json), htmlReportOptions, and display preferences like showInlineCriticals or noColor.disableJavascript, disableStyles, disableFonts, disableImages, etc.workers, maxReqsPerSec, memoryLimit, allowedDomainForCrawling, includeRegex, ignoreRegex, and maxVisitedUrls.debug, resultStorage (memory, file), httpCacheDir, and websocketServer.outputHtmlReport, outputJsonFile, and outputTextFile using placeholders like %domain% and %datetime%.mailTo, mailSmtpHost, etc.).sitemapXmlFile or sitemapTxtFile.uploadTo, uploadRetention, uploadPassword).The crawler can securely upload HTML reports to the crawler.siteone.io server for remote access:
1h, 4h, 12h, 24h, 3d, 7d, 30d, 365d, forever.crawler to view it. Passwords are stored securely using Bcrypt or Argon2id.The crawler includes an analyzer for SEO and OpenGraph metadata. You can control the depth of the analysis using: