OpenSERP Documentation

OpenSERP supports two primary execution modes for extracting search results from engines like Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia:

1. Browser mode (Default): Uses headless Chromium via go-rod. This is the most compatible path and is supported by all engines.
2. Raw HTTP mode: Uses direct HTTP requests and goquery for parsing. This is currently only supported by Google, Yandex, Baidu, and Ecosia.

Use Browser mode if you encounter parsing issues or need compatibility with engines that do not support Raw HTTP mode.

When using the /mega/search or /mega/image endpoints, you can specify a mode to control how engines are queried:

OpenSERP uses a resilience stack to protect requests in this order:

1. Engine rate limiter
2. Retry with backoff
3. Circuit breaker
4. Proxy policy and proxy health
5. Response cache

Critical Error Behavior:

• ErrCaptcha: This error is non-retryable. If a captcha is encountered, the request will fail immediately.
• Fallback: Dedicated endpoints are engine-pure by default. Fallback to other engines is only enabled if resilience.allow_endpoint_fallback is configured. Fallback responses are not cached on dedicated endpoints.

A complete OpenSERP engine package typically follows this structure:

• url.go: Pure URL builders (should reject empty queries).
• selectors.go: Stable selectors used by browser mode, raw mode, and parser tests. Tip: Prefer stable data attributes over generated CSS classes. If a selector is fragile, provide 2-3 explicit fallbacks here.
• parse_html.go: Implementation of ParseHTML(io.Reader) using goquery.
• search.go: Browser-mode implementation using core.Browser.
• search_raw.go: (Optional) Raw HTTP implementation.
• features.go: (Optional) Extraction of specific SERP features.
• *_test.go and testdata/: URL and parser fixtures for testing.

The /mega/search and /mega/image endpoints execute multiple engines in parallel to provide a unified view of search results.

Key behaviors:

• Engine Selection: Uses the engines query parameter if provided; otherwise, it queries all configured engines.
• Partial Success: If some engines fail, the request still succeeds. Failed engines are listed in meta.engines_failed.
• Deduplication: Results are deduplicated based on their normalized URL.
• Clustering: It builds clusters from enriched results before performing flat deduplication. Clusters are sorted by score (descending) and then by best rank (ascending).

Cluster Score Calculation: sum(1 / rank for each occurrence) / engines_queried (capped at 1.0 and rounded to two decimals).

GET /mega/search?engines=google,bing&text=query

To set up the local environment for the Python agent search tool example, install the necessary dependencies using pip and then execute the main script.

pip install -r requirements.txt
python main.py

You can run the OpenSERP API server using prebuilt Docker images from Docker Hub (karust/openserp).

To run the server via a single command:

docker run --rm -p 127.0.0.1:7000:7000 karust/openserp:latest serve -a 0.0.0.0 -p 7000

Alternatively, use Docker Compose if you have a configuration file ready:

docker compose up

To use the OpenSERP examples locally, you must run a server. By default, examples expect the server to be running on http://localhost:7000. You can start the server using Docker or by building from the Go source code.

Using Docker

Run the prebuilt image and map port 7000:

docker run -p 127.0.0.1:7000:7000 -it karust/openserp serve -a 0.0.0.0 -p 7000

Using Go Source

Build and run the binary directly:

go build -o openserp . && ./openserp serve

This example performs an image search and generates a self-contained gallery.html file that can be opened directly in a web browser to view the results.

To run the example:

1. Install dependencies using npm install.
2. Execute the script with node index.js.
3. Open the generated gallery.html in your browser.

To search for different images, edit the query variable inside index.js before running the script.

This example demonstrates how to perform a single Google search using the openserp Python SDK and print the resulting titles and URLs.

To run the example, install the required dependencies and execute the main script:

pip install -r requirements.txt
python main.py

Customizing the search

You can modify the search behavior by editing the following variables in main.py:

• text: The search query string.
• engine: The search engine to use (e.g., google).
• limit: The maximum number of results to return.

Using the hosted API

By default, the example is configured to connect to a local server. To switch to the hosted OpenSERP API, uncomment the relevant configuration line at the top of main.py.

New engines require several layers of testing to ensure stability.

Required Tests for Initial PR

• URL Builder Tests: Table-driven tests for your url.go logic.
• Parser Tests: Tests using small, sanitized HTML fixtures to verify ParseHTML logic.
• Integration Tests: Only include these if necessary, and ensure they are gated with testutil.RequireIntegration(t).

Running Tests

To run default tests (which must pass without browser or network access):

make test

To run live/browser-based integration checks:

make test-integration

Pre-PR Checklist

Before submitting a Pull Request, run the following to ensure code quality:

make fmt
make lint
make test

OpenSERP provides official SDKs and integrations for various environments: