DOCSEARCHbyAlgolia
Submit Docs

hrequests

repository·main·Indexed 21 days ago

https://github.com/daijro/hrequests

A feature-rich replacement for the Python requests library designed to mimic human-like browsing behavior. Version 0.9.2 supports TLS fingerprint spoofing, seamless transitions between HTTP requests and headless browser automation (Chrome and Firefox), and high-performance concurrency via gevent and nohup. It includes built-in HTML parsing using CSS selectors via selectolax and provides specialized Session objects for emulating specific browsers and operating systems.

Tokens
20.1K
Snippets
68
Records
77
Agent score
76%

What's inside hrequests

  1. Geo-target proxies with Evomi

    main

    When configuring Evomi proxies, you can specify geographic locations to target specific traffic origins.

    • Continent: Use names like Africa, Asia, Europe, Oceania, North America, or South America.
    • Country: Use two-letter country codes (e.g., country='US' for United States, country='CA' for Canada).
    • Region: Target a specific state, province, or territory. Available for Residential and Mobile proxies.
    • City: Target a specific city. Available for Residential proxies only.
    from hrequests.proxies import evomi

# Target a specific city in North America
proxy = evomi.ResidentialProxy(continent='North America', city='New York', username='...', key='...')

# Target a specific region with Mobile proxies
mobile_proxy = evomi.MobileProxy(region='Southern Cape', username='...', key='...')

  2. Reuse a BrowserEngine across multiple sessions

    main

    To save time on startup, you can create a single BrowserEngine and pass it to multiple BrowserSession instances. The engine is completely thread-safe.

    engine = hrequests.BrowserEngine()

# Use the same engine for multiple sessions
page1 = hrequests.BrowserSession(engine=engine)
page2 = hrequests.BrowserSession(engine=engine)

  3. Use Grequests-style concurrency with async methods

    main

    For high-performance concurrency, use the async_ prefix methods (e.g., async_get, async_post). These methods create unsent request objects that leverage gevent for speed. To actually execute these requests, you must pass the collection of request objects to hrequests.map, hrequests.imap, or hrequests.imap_enum.

    # 1. Create unsent requests
reqs = [
    hrequests.async_get('https://www.google.com/', browser='firefox'),
    hrequests.async_get('https://www.duckduckgo.com/'),
    hrequests.async_get('https://www.yahoo.com/')
]

# 2. Execute them using map
responses = hrequests.map(reqs, size=3)

  4. Send background requests using nohup

    main

    You can send requests in the background by passing nohup=True as a keyword argument. This returns a LazyTLSRequest object. The request is sent immediately, but the thread will not pause until you access an attribute of the response (e.g., .reason, .text, or .status_code). This is useful for initiating multiple requests concurrently without waiting for them to finish before moving to the next line of code.

    resp1 = hrequests.get('https://www.google.com/', nohup=True)
resp2 = hrequests.get('https://www.google.com/', nohup=True)

# Requests are running in the background. Accessing an attribute waits for completion:
print('Resp 1:', resp1.reason)
print('Resp 2:', resp2.reason)

  5. Manage Session lifecycle and properties

    main

    Making Requests

    Use the session to perform HTTP requests. The session automatically updates its cookies property with each response, making it ideal for multi-step workflows on the same domain.

    resp = session.get('https://www.google.com/')
# session.cookies is automatically updated

    Updating Session Properties

    You can dynamically update session properties like os to regenerate headers for a different operating system:

    session.os = 'win'
# session.headers is automatically regenerated based on the new OS

    Closing Sessions

    To free up memory, you should close your sessions. You can do this manually or by using a context manager.

    Manual close:

    session.close()

    Context Manager (Recommended):

    with hrequests.Session() as session:
    resp = session.get('https://www.google.com/')
    print(resp)

  6. Install hrequests

    main

    To install hrequests with full support, including headless browsing capabilities, use the [all] extra. After installing the package, you must run the installation command to set up necessary dependencies.

    Full installation (includes headless browsing):

    pip install -U hrequests[all]
python -m hrequests install

    Minimal installation (excludes headless browsing):

    pip install -U hrequests

  7. Add Firefox extensions to a BrowserSession

    main

    Hrequests supports adding unpacked Firefox extensions to a session. This is useful for tools like captcha solvers or ad blockers.

    Note: Only Firefox extensions are supported.

    Pass a list of absolute paths to the extensions parameter in render():

    # Example: Using a captcha solver extension
resp = hrequests.get('https://accounts.hcaptcha.com/demo', browser='firefox')
with resp.render(extensions=['C:\extensions\hektcaptcha']) as page:
    page.awaitSelector('.hcaptcha-success')
    page.click('input[type=submit]')
    with resp.render(extensions=['C:\extensions\hektcaptcha', 'C:\extensions\fastforward']):
    # ... interaction ...

  8. Render an existing Response in a browser

    main

    The .render() method on a Response object allows you to render the response's content in a browser page. This is useful for interacting with dynamic content or submitting forms.

    Once the page is closed (either via a context manager or calling .close()), the original Response content and session cookies are updated with the results of the browser session.

    Usage Patterns

    Using a context manager (Recommended):

    session = hrequests.Session()
resp = session.get('https://www.somewebsite.com/')
with resp.render(mock_human=True) as page:
    page.type('.input#username', 'myuser')
    page.type('.input#password', 'p4ssw0rd')
    page.click('#submit')
# session & resp are now updated

    Without a context manager:

    session = hrequests.Session()
resp = session.get('https://www.somewebsite.com/')
page = resp.render(mock_human=True)
page.type('.input#username', 'myuser')
page.type('.input#password', 'p4ssw0rd')
page.click('#submit')
page.close()  # MUST close the page manually!
    session = hrequests.Session()
resp = session.get('https://www.somewebsite.com/')
with resp.render(mock_human=True) as page:
    page.type('.input#username', 'myuser')
    page.type('.input#password', 'p4ssw0rd')
    page.click('#submit')

  9. Use Evomi residential proxy rotation

    main

    Hrequests includes built-in support for Evomi residential proxy rotation. You can create a proxy object using the evomi module and pass it to individual requests, a Session, or a BrowserSession.

    To use it with a single request:

    import hrequests
from hrequests.proxies import evomi

proxy = evomi.ResidentialProxy(username='your_username', key='your_key')
resp = hrequests.get('https://example.com', proxy=proxy)

    To use it with a Session (all requests in the session will use the proxy):

    session = hrequests.Session(proxy=proxy)
resp = session.get('https://example.com')

    To use it with a BrowserSession:

    page = hrequests.BrowserSession(proxy=proxy)
page.goto('https://example.com')
    from hrequests.proxies import evomi
proxy = evomi.ResidentialProxy(username='daijro', key='password')
resp = hrequests.get('https://example.com', proxy=proxy)

  10. Integrate Camoufox with Firefox BrowserSession

    main

    When using a Firefox BrowserSession, you can pass Camoufox-specific parameters via **kwargs. This allows for advanced fingerprinting and browser customization.

    Example parameters include window (tuple), block_images (bool), and addons (list of paths).

    page = hrequests.BrowserSession(window=(1024, 768), block_images=True, addons=['/path/to/addon'], ...)

  11. Simple Usage with hrequests

    main

    You can perform HTTP requests using simple methods like get, post, put, delete, head, options, and patch.

    Requests automatically use tls-client to spoof the TLS client fingerprint, making them appear more like a real browser. The Response object returned is a near 1:1 replica of the standard requests.Response object.

    import hrequests

resp = hrequests.get('https://www.google.com/')

  12. Use the HTML and Element classes for parsing

    main

    The hrequests parsing API is built around three main classes: HTML, Element, and BaseParser.

    • HTML: Represents an entire HTML document. It is the entry point for parsing a full page and supports pagination via the .next() method.
    • Element: Represents a specific HTML tag or node within a document. You can derive elements from an HTML object using CSS selectors.
    • BaseParser: The base class providing common functionality like text extraction, link retrieval, and CSS searching.

    To use them, you typically start with an HTML object created from a response or raw HTML string, then use .find_all() or .find() to navigate to specific Element objects.

    from hrequests import HTML

# Create an HTML object from a string
html_doc = HTML(url='https://example.com', html='<html><body><div id="target">Hello</div></body></html>')

# Find an element using a CSS selector
target_element = html_doc.find('#target')
print(target_element.text)  # Output: Hello