stealth-requests

Install the core package using pip:

$ pip install stealth_requests

If you want to use advanced parsing features like Lxml or BeautifulSoup4, install the parsers extra:

$ pip install 'stealth_requests[parsers]'

Stealth-Requests mimics the requests API. You can perform one-off requests using the top-level module or use a StealthSession to maintain state (like the Referer header) across multiple requests.

To enable automatic retries for failed requests (e.g., status codes 429, 503, 522), pass the retry argument with the number of attempts.

import stealth_requests as requests

# One-off request
resp = requests.get('https://link-here.com')

# Request with retries
resp = requests.get('https://link-here.com', retry=3)

# Using a session to track headers like Referer
from stealth_requests import StealthSession

with StealthSession() as session:
    resp = session.get('https://link-here.com')

The StealthResponse object provides convenience properties to quickly extract common data types from a page:

• resp.emails: Returns a tuple of email addresses.
• resp.phone_numbers: Returns a tuple of phone numbers.
• resp.images: Returns a tuple of image URLs.
• resp.links: Returns a tuple of link URLs.

import stealth_requests as requests

resp = requests.get('https://link-here.com')

print(resp.emails)
print(resp.phone_numbers)
print(resp.images)
print(resp.links)

You can use proxies by passing a proxies dictionary to the request method, supporting both http and https protocols.

import stealth_requests as requests

proxies = {
    "http": "http://username:password@proxyhost:port",
    "https": "http://username:password@proxyhost:port",
}

resp = requests.get('https://link-here.com', proxies=proxies)

If you have installed the parsers extra, you can convert a StealthResponse into standard parsing objects:

• resp.tree(): Returns an Lxml tree.
• resp.soup(): Returns a BeautifulSoup object.

Additionally, StealthResponse includes built-in convenience methods from Lxml:

• text_content(): Returns all text content in the response.
• xpath(expression): Executes an XPath expression directly on the response.

# Requires: pip install 'stealth_requests[parsers]'
import stealth_requests as requests

resp = requests.get('https://link-here.com')

# Get Lxml tree
tree = resp.tree()

# Get BeautifulSoup object
soup = resp.soup()

# Use built-in Lxml convenience methods
text = resp.text_content()
results = resp.xpath('//div[@class="example"]')

The StealthResponse object automatically parses HTML metadata. You can access these via the .meta property. Available fields include:

• title: str | None
• author: str | None
• description: str | None
• thumbnail: str | None
• canonical: str | None
• twitter_handle: str | None
• keywords: tuple[str] | None
• robots: tuple[str] | None

import stealth_requests as requests

resp = requests.get('https://link-here.com')
print(resp.meta.title)

Use the resp.markdown() method to convert an HTML response into a Markdown string. This is useful for creating simplified, readable versions of web pages.

Parameters:

• content_xpath (str, optional): An XPath expression to narrow down which part of the HTML is converted (e.g., to exclude headers/footers).
• ignore_links (bool, optional): If True, links will be excluded from the Markdown output.

import stealth_requests as requests

resp = requests.get('https://link-here.com')

# Convert specific section to markdown
md_content = resp.markdown(content_xpath='//article')

# Convert without links
md_no_links = resp.markdown(ignore_links=True)

Use AsyncStealthSession to perform asynchronous requests using async/await syntax.

from stealth_requests import AsyncStealthSession

async with AsyncStealthSession() as session:
    resp = await session.get('https://link-here.com')

The StealthResponse.tables property returns a list of dictionaries. Each dictionary represents a table where keys are column headers and values are lists of cell contents. Tables without recognizable headers are automatically skipped.

import stealth_requests as requests

resp = requests.get('https://link-here.com')

# Each table becomes a dict: {column_name: [values]}
for table in resp.tables:
    print(table)

For scenarios requiring multiple requests to the same host or maintaining state (like cookies), use StealthSession (synchronous) or AsyncStealthSession (asynchronous) directly. This avoids the overhead of creating and destroying a session for every individual request.

from stealth_requests import StealthSession

with StealthSession() as s:
    response = s.get('https://example.com')
    response2 = s.get('https://example.com/next-page')

The package provides pre-configured partial functions for common HTTP methods to simplify syntax. These functions behave identically to calling request(method, url, ...).

from stealth_requests import get, post, put, patch, delete, head, options

get('https://example.com')
post('https://example.com', data={'key': 'value'})
put('https://example.com', data={'key': 'value'})
patch('https://example.com', data={'key': 'value'})
delete('https://example.com')
head('https://example.com')
options('https://example.com')