googlesearch Python Library

repository·master·Indexed 21 days ago

https://github.com/nv7-github/googlesearch

A Python library using requests and BeautifulSoup4 to scrape Google search results. It provides a programmatic interface to perform searches with customizable parameters for result count, language, region, and safe search, as well as advanced options to retrieve SearchResult objects containing titles, URLs, and descriptions.

Tokens
839
Snippets
6
Records
6
Agent score
24%

What's inside googlesearch

  1. Configure search parameters like num_results, unique, lang, and region

    master

    The search() function accepts several arguments to customize the query:

    • num_results: The number of results to return (default is 10).
    • unique: Set to True to ensure unique links in the results.
    • lang: The language code for the search (e.g., "fr" for French).
    • region: The country code for the search results (e.g., "us" for the US).
    • safe: Set to None to turn off the safe search function.
    from googlesearch import search
    
    # Example: 100 unique results in French from the US with safe search off
    search("Google", num_results=100, unique=True, lang="fr", region="us", safe=None)
  2. Manage pagination and request throttling

    master

    When requesting large numbers of results (more than 100), the library sends multiple requests. Use these options to manage the process:

    • sleep_interval: The number of seconds to wait between requests to avoid being blocked.
    • start_result: Specifies the starting index for the results if you want to manage batching manually.
    from googlesearch import search
    
    # Get 200 results, starting from the 10th result, with a 5-second delay between pages
    search("Google", num_results=200, sleep_interval=5, start_result=10)
  3. Use proxies and disable SSL verification

    master

    You can route searches through an HTTP or SOCKS5 proxy. If your proxy requires a custom CA certificate and you wish to bypass verification, set ssl_verify=False.

    from googlesearch import search
    
    proxy = 'http://username:password@proxy.host.com:8080/'
    # or for socks5
    # proxy = 'socks5://username:password@proxy.host.com:1080/'
    
    results = search("proxy test", num_results=100, lang="en", proxy=proxy, ssl_verify=False)
    for i in results:
        print(i)
  4. Use advanced search to get SearchResult objects

    master

    By default, search() returns simple results. To extract more metadata, set advanced=True. This returns a list of SearchResult objects which have the following properties:

    • title
    • url
    • description
    from googlesearch import search
    
    results = search("Google", advanced=True)
    for result in results:
        print(result.title, result.url, result.description)