DOCSEARCHbyAlgolia
Submit Docs

scholarly Python Package

repository·main·Indexed 23 days ago

https://github.com/scholarly-python-package/scholarly

A Python module for retrieving author and publication information from Google Scholar. It provides functionality to search for authors by name, ID, organization, or keyword, and to search for publications. Key features include the fill() method for detailed data retrieval, citedby() for finding citing articles, and a ProxyGenerator to manage free or premium proxies (such as ScraperAPI and Luminati) to avoid IP blocking.

Tokens
3.8K
Snippets
19
Records
25
Agent score
34%

What's inside scholarly

  1. Use ProxyGenerator to avoid Google Scholar blocking

    main

    To avoid being blocked by Google Scholar, use the ProxyGenerator class to manage connections. You must initialize a ProxyGenerator object, select a connection method, and then pass that object to scholarly.use_proxy().

    Important: Create a new ProxyGenerator object whenever you change the proxy method to avoid unexpected behavior.

    Proxy Selection Methods:

    • ScraperAPI(api_key, ...)
    • Luminati(usr, passwd, port, ...)
    • FreeProxies()
    • SingleProxy(http, https)
    • Tor_Internal(tor_cmd, ...) (Deprecated since v1.5)
    • Tor_External(tor_sock_port, tor_control_port, tor_password) (Deprecated since v1.5)

    Proxy Usage Modes:

    • Smart Mode (Default): scholarly.use_proxy(pg) — scholarly uses the proxy only for requests that might be blocked, otherwise it uses FreeProxies.
    • Full Proxy Mode: scholarly.use_proxy(pg, pg) — All requests are routed through your specified proxy.
    from scholarly import ProxyGenerator
import scholarly

pg = ProxyGenerator()
# Select a method (e.g., SingleProxy)
success = pg.SingleProxy(http='<your http proxy>', https='<your https proxy>')

if success:
    scholarly.use_proxy(pg)
    # Perform scholarly actions
    author = next(scholarly.search_author('Steven A Cholewiak'))
    scholarly.pprint(author)

  2. Set up environment variables for testing

    main

    When running test_module.py, you can configure the connection method and credentials using a .env file in the working directory.

    Available CONNECTION_METHOD values:

    • luminati
    • scraperapi
    • freeproxy
    • tor
    • tor_internal
    • none (default)

    Luminati Credentials: If using luminati, add the following to your .env:

    • USERNAME=<LUMINATI_USERNAME>
    • PASSWORD=<LUMINATI_PASSWORD>
    • PORT=<PORT_FOR_LUMINATI>
    touch .env
nano .env

# Inside .env:
CONNECTION_METHOD = luminati
USERNAME = <LUMINATI_USERNAME>
PASSWORD = <LUMINATI_PASSWORD>
PORT = <PORT_FOR_LUMINATI>

  3. Use proxies to avoid Google Scholar blocking

    main

    Certain queries like scholarly.citedby or scholarly.search_pubs can lead to your IP being blocked by Google Scholar. It is highly recommended to use a ProxyGenerator at the start of your session.

    scholarly supports free proxies via pg.FreeProxies() and various premium (paid) services. Use scholarly.use_proxy(pg) to apply the configuration.

    from scholarly import scholarly, ProxyGenerator

# Set up a ProxyGenerator object to use free proxies
# This needs to be done only once per session
pg = ProxyGenerator()
pg.FreeProxies()
scholarly.use_proxy(pg)

# Now search Google Scholar from behind a proxy
search_query = scholarly.search_pubs('Perception of physical stability and center of mass of 3D objects')
scholarly.pprint(next(search_query))

  4. Install Tor optional dependency

    main

    If you need to use Tor methods (note: Tor methods are deprecated since v1.5 and are not actively supported), you can install the tor extra via pip.

    Note: This option is unavailable via conda installation.

    For standard shells:

    pip3 install scholarly[tor]

    For zsh (default on macOS):

    pip3 install scholarly'[tor]'
    pip3 install scholarly[tor]

  10. Example: Inspect a publication and its citations

    main

    You can use scholarly.fill() on a specific publication object to get more details, and then use scholarly.citedby() to find the papers that cited that specific publication.

    from scholarly import scholarly

# ... (assuming author is already retrieved as in previous example)

# Take a closer look at the first publication
pub = scholarly.fill(author['publications'][0])
print(pub)

# Which papers cited that publication?
print([citation['bib']['title'] for citation in scholarly.citedby(pub)])

  11. Search for authors and retrieve publication details

    main

    Use scholarly.search_author(name) to get an iterator of author results. You can then use scholarly.fill(result) to retrieve the full details for a specific author or a specific publication. To find citations for a publication, use scholarly.citedby(publication_object).

    from scholarly import scholarly

# Get an iterator for author results
search_query = scholarly.search_author('Steven A Cholewiak')

# Retrieve the first result
first_author_result = next(search_query)

# Retrieve all the details for the author
author = scholarly.fill(first_author_result)

# Take a closer look at the first publication
first_publication = author['publications'][0]
first_publication_filled = scholarly.fill(first_publication)

# Print the titles of the author's publications
publication_titles = [pub['bib']['title'] for pub in author['publications']]
print(publication_titles)

# Which papers cited that publication?
citations = [citation['bib']['title'] for citation in scholarly.citedby(first_publication_filled)]
print(citations)

  12. Example: Retrieve author data and publication titles

    main

    This example demonstrates how to search for an author, use scholarly.fill() to retrieve their full profile data, and then extract the titles of their publications.

    from scholarly import scholarly

# Retrieve the author's data, fill-in, and print
search_query = scholarly.search_author('Steven A Cholewiak')
author = scholarly.fill(next(search_query))
print(author)

# Print the titles of the author's publications
print([pub['bib']['title'] for pub in author['publications']])