twscrape Documentation

Proxies can be applied at different levels. The priority order is:

1. api.proxy (highest)
2. TWS_PROXY environment variable
3. Account-specific proxy (lowest)

Python API usage:

Set a proxy for the entire API instance:

api = API(proxy="http://login:pass@example.com:8080")

Set a proxy for a specific account:

await api.pool.add_account(
    "user1",
    "pass1",
    "user1@example.com",
    "email_pass1",
    proxy="http://login:pass@example.com:8080",
)

CLI usage:

TWS_PROXY=socks5://user:pass@127.0.0.1:1080 twscrape user_by_login xdevelopers

await api.pool.add_account(
    "user1",
    "pass1",
    "user1@example.com",
    "email_pass1",
    proxy="http://login:pass@example.com:8080",
)

api = API(proxy="http://login:pass@example.com:8080")
doc = await api.user_by_login("xdevelopers")

The most stable way to add accounts is using browser cookies containing auth_token and ct0. Cookie accounts with ct0 are activated immediately and do not require a login_accounts step.

To get cookies:

1. Open x.com in a browser.
2. Open DevTools (F12).
3. Go to Application -> Cookies.
4. Copy the auth_token and ct0 values.

CLI Usage:

Add a cookie-based account:

twscrape add_cookie my_account "auth_token=xxx; ct0=yyy"

Or prompt for the cookie value interactively:

twscrape add_cookie my_account

When breaking out of an async generator (like api.search) early, use contextlib.aclosing to ensure the account lock is released promptly.

from contextlib import aclosing

async with aclosing(api.search("elon musk")) as gen:
    async for tweet in gen:
        if tweet.id < 200:
            break

Install the core library using pip:

pip install twscrape

For browser-like TLS fingerprinting, install the optional curl-cffi backend. You can then use the curl backend by setting the TWS_HTTP_BACKEND environment variable:

pip install "twscrape[curl]"

TWS_HTTP_BACKEND=curl twscrape user_by_login xdevelopers

When using QueueClient, certain exceptions are raised to the caller that indicate the request cannot proceed:

• GqlFeaturesOutdatedError: Raised when the internal GQL features no longer match the X API. This indicates the twscrape library needs an update and retrying will not help.
• AbortReqError: Raised when a request is aborted (e.g., due to Cloudflare blocking or internal dependency errors).
• ConnectError: Raised if the proxy is misconfigured or the host is unreachable after 3 attempts.

Other errors like HandledError (rate limits) or XClIdAccountError are caught internally by QueueClient to trigger account rotation or locking.

The AccountsPool uses a locking mechanism to manage concurrency. When an account is retrieved via get_for_queue, a lock is placed on that account for the specified queue name in the database. This lock is stored as a JSON object in the locks column.

Key behaviors:

• Automatic Expiry: Locks are typically set to expire after 15 minutes. An account is considered available for a queue if its lock for that queue is NULL or if the lock timestamp is in the past.
• Concurrency Control: By using different queue names (e.g., "search" vs "profile_scraping"), you can allow an account to be used for one task type while it is locked for another.
• Wait Logic: get_for_queue_or_wait uses next_available_at to determine when the next account in the queue will become free, allowing it to sleep efficiently between polls rather than busy-waiting.

When initializing the API class, you can control how it handles account availability:

• raise_when_no_account=True: Raises NoAccountError instead of waiting indefinitely.
• wait_timeout: Limits how long to wait for a locked account (in seconds).
• wait_interval: Controls how often the pool checks for available accounts (in seconds).

api = API(raise_when_no_account=True, wait_timeout=30, wait_interval=1)

The LoginConfig dataclass allows you to customize the authentication flow for an account:

• email_first (bool): If set to True, the system attempts to handle email-based authentication steps earlier in the process. Defaults to False.
• manual (bool): If set to True, the system will prompt for manual user input (e.g., via input()) when an email confirmation code is required, rather than attempting to fetch it automatically via IMAP. Defaults to False.

You can control which HTTP client twscrape uses by setting the TWS_HTTP_BACKEND environment variable.

• TWS_HTTP_BACKEND=curl: Forces the use of CurlClient (requires curl-cffi).
• TWS_HTTP_BACKEND=httpx: Forces the use of HttpxClient (requires httpx).

If curl is selected but curl-cffi is not installed, an ImportError will be raised with instructions to install it.