yagooglesearch

You can install yagooglesearch via pip or by cloning the repository and installing from source using pyproject.toml.

# Using pip
pip install yagooglesearch

# From source
git clone https://github.com/opsdisk/yagooglesearch
cd yagooglesearch
virtualenv -p python3 .venv
source .venv/bin/activate
pip install .

Google may block your IP if too many requests are made. SearchClient provides two ways to handle this:

1. Automatic Management (Default): Set yagooglesearch_manages_http_429s=True. When a 429 is detected, the client will sleep for http_429_cool_off_time_in_minutes and then retry. Each subsequent detection increases the sleep time by the http_429_cool_off_factor.
2. Manual Management: Set yagooglesearch_manages_http_429s=False. When a 429 is detected, the search() method will immediately return a list containing the single string "HTTP_429_DETECTED". You can then decide how to pause or rotate proxies in your own code.

To use multiple proxies, you must manually instantiate a new SearchClient for each proxy in your rotation loop.

import yagooglesearch

proxies = [
    "socks5h://127.0.0.1:9050",
    "socks5h://127.0.0.1:9051",
    "http://127.0.0.1:9052",
]

search_queries = ["python", "site:github.com pagodo"]
proxy_rotation_index = 0

for search_query in search_queries:
    proxy_index = proxy_rotation_index % len(proxies)
    client = yagooglesearch.SearchClient(
        search_query,
        proxy=proxies[proxy_index],
    )

    if proxies[proxy_index].startswith("http://"):
        client.verify_ssl = False

    urls_list = client.search()
    print(urls_list)
    proxy_rotation_index += 1

To perform a Google search, instantiate a yagooglesearch.SearchClient with your query and desired parameters, then call .search(). It is highly recommended to call .assign_random_user_agent() to help avoid rate limiting.

import yagooglesearch

query = "site:github.com"

client = yagooglesearch.SearchClient(
    query,
    tbs="li:1",
    max_search_result_urls_to_return=100,
    http_429_cool_off_time_in_minutes=45,
    http_429_cool_off_factor=1.5,
    # proxy="socks5h://127.0.0.1:9050",
    verbosity=5,
    verbose_output=True,  # False (only URLs) or True (rank, title, description, and URL)
)
client.assign_random_user_agent()

urls = client.search()

for url in urls:
    print(url)

You can provide a proxy string during SearchClient initialization. The library supports http, https, socks5, and socks5h schemes.

Recommendation: Use socks5h if you are using SOCKS to ensure DNS resolution happens on the proxy server rather than locally.

client = yagooglesearch.SearchClient(
    "site:github.com",
    proxy="socks5h://127.0.0.1:9050",
)

If you are using an HTTPS proxy with a self-signed certificate, you must disable SSL verification. This can be done during instantiation via verify_ssl=False or by setting the attribute on the client instance after creation.

# Option 1: During instantiation
client = yagooglesearch.SearchClient(
    query="site:github.com",
    proxy="http://127.0.0.1:8080",
    verify_ssl=False,
    verbosity=5,
)

# Option 2: After instantiation
client = yagooglesearch.SearchClient(query="site:github.com", proxy="http://127.0.0.1:8080")
client.verify_ssl = False

When yagooglesearch_manages_http_429s is enabled, you can tune the recovery behavior using these parameters in the SearchClient constructor:

• http_429_cool_off_time_in_minutes: The initial sleep duration in minutes (e.g., 60).
• http_429_cool_off_factor: The multiplier applied to the sleep time for every consecutive 429 error (e.g., 1.1).

The tbs parameter allows you to apply verbatim or time-based filters to your search.

Verbatim search: Use tbs="li:1".

Time-based filters:

The get_tbs helper function converts Python datetime.date objects into the specific tbs string format required by Google for custom date range searches (cdr:1,cd_min:MM/DD/YYYY,cd_max:MM/DD/YYYY).

Use this function to populate the tbs parameter in the SearchClient constructor.

from datetime import date
from yagooglesearch import SearchClient, get_tbs

from_date = date(2021, 1, 1)
to_date = date(2021, 6, 1)
tbs_param = get_tbs(from_date, to_date)

client = SearchClient(query="search term", tbs=tbs_param)
results = client.search()