repository·master·Indexed 24 days ago
https://github.com/goclone-dev/gocloneA utility for downloading entire websites to a local directory while preserving the original relative link structure for offline browsing. It provides a CLI for cloning sites and a programmatic Go API via the CloneSite function and CloneOptions struct. Features include support for custom User-Agents, proxies, cookies, and an integrated local server to host cloned content.
To build the binary manually from the source code, clone the repository, build the binary using go build, and optionally move it to your PATH.
# Clone the repository
git clone https://github.com/goclone-dev/goclone.git
cd goclone
# Build and run
go build -o goclone cmd/goclone/main.go
# Move binary to a directory in your PATH (optional)
mv goclone /usr/local/bin/If you have Go installed (version >= 1.20), you can install goclone directly using go install.
# Go version >= 1.20
go install github.com/goclone-dev/goclone/cmd/goclone@latestYou can install goclone using Homebrew by tapping the official repository and then installing the tool.
# tap
brew tap goclone-dev/goclone
# install tool
brew install gocloneTo download a website to your local computer, run goclone followed by the target URL. Goclone will download HTML, CSS, JS, images, and other files, arranging them in a relative link-structure so you can browse the site offline.
# goclone <url>
goclone https://configtree.coThe goclone CLI supports several flags to customize the cloning and serving process.
Usage:
goclone <url> [flags]
Flags:
-C, --cookie strings Pre-set these cookies
-h, --help help for goclone
-o, --open Automatically open project in default browser
-p, --proxy_string string Proxy connection string. Support http and socks5 https://pkg.go.dev/github.com/gocolly/colly#Collector.SetProxy
-s, --serve Serve the generated files using Echo.
-P, --servePort int Serve port number. (default 5000)
-u, --user_agent string Custom User AgentUse the CloneOptions struct to customize the behavior of the cloning process.
| Field | Type | Description |
|---|---|---|
Serve | bool | If true, starts a local server to host the cloned site after crawling completes. |
Open | bool | If true, automatically opens the cloned site (either the local server URL or the index.html file) in the default system browser. |
ServePort | int | The port number used by the local server when Serve is true. |
Cookies | []string | A list of cookie strings in the format key=value to be applied to the target URLs. |
Proxy | string | The proxy URL to use for the crawling process. |
UserAgent | string | A custom User-Agent string to use for the crawler requests. |
type CloneOptions struct {
Serve bool
Open bool
ServePort int
Cookies []string
Proxy string
UserAgent string
}Serve function starts a local web server using the Echo framework to host the files from a cloned website. It serves static files from the specified projectPath and is configured with CORS enabled for all origins and common HTTP methods. The server runs on the provided port and supports graceful shutdown via interrupt signals (e.g., Ctrl+C).The CloneSite function is the primary programmatic entrypoint for cloning websites. It handles cookie setup, crawling the target URLs, restructuring HTML links for local use, and performing post-clone actions like serving the site or opening it in a browser.
To use it, provide a context.Context, a slice of target URLs (args), and a CloneOptions struct.
You can customize the cloning process and how the resulting files are handled using the following flags:
| Flag | Shorthand | Type | Default | Description |
|---|---|---|---|---|
--open | -o | bool | false | Automatically open the project in your default browser after cloning. |
--serve | -s | bool | false | Serve the generated files using the Echo web framework. |
--servePort | -P | int | 5000 | The port number used when using the --serve flag. |
--proxy_string | -p | string | "" | Proxy connection string. Supports http and socks5. |
--user_agent | -u | string | "" | Custom User Agent string to use for the request. |
--cookie | -C | string slice | nil | Pre-set cookies to include in the request. |
Example: Clone and serve locally on port 8080
goclone https://example.com --serve --servePort 8080The goclone command is used to download a website from the internet to a local directory. It preserves the original site's relative link structure, allowing you to browse the mirrored site locally as if it were online.
Basic Usage:
goclone <url>If no URL is provided, the tool will display the usage information.
goclone <url>goclone, use the version subcommand.The goclone tool is a command-line interface used to clone web content. The entrypoint executes the command logic defined in the cmd package. To use the tool, you typically run the binary followed by a target URL.
Note: Detailed command flags and subcommands are managed by the underlying cmd package implementation.