@the-convocation/twitter-scraper

To run the CycleTLS bypass example, you must provide your Twitter credentials in a .env file located in the example directory using the following keys:

• TWITTER_USERNAME
• TWITTER_PASSWORD
• TWITTER_EMAIL

TWITTER_USERNAME=your_username
TWITTER_PASSWORD=your_password
TWITTER_EMAIL=your_email

To run the React integration example, you must first set up a CORS proxy because Twitter's CORS headers prevent direct API calls from external websites.

1. Configure Environment Variables: Copy .env.example to .env.local and update the environment variables with your account credentials if necessary.
2. Start the CORS Proxy: Navigate to the cors-proxy example folder and run:

   yarn start

3. Start the React App: In the react-integration folder, start the Vite development server by running:

   yarn dev

# In cors-proxy folder
yarn start

# In react-integration folder
yarn dev

When authenticating with Twitter, standard Node.js TLS handshakes may trigger 403 Forbidden errors because Cloudflare detects non-browser clients via TLS fingerprinting. To bypass this, you can use the @the-convocation/twitter-scraper/cycletls entrypoint, which provides a cycleTLSFetch implementation that mimics Chrome browser TLS fingerprints.

import { Scraper } from '@the-convocation/twitter-scraper';
import { cycleTLSFetch, cycleTLSExit } from '@the-convocation/twitter-scraper/cycletls';

const scraper = new Scraper({
  fetch: cycleTLSFetch,
});

Because the Twitter API does not have permissive CORS headers, browser-based applications must use a CORS proxy. You can configure this by providing a transform.request function in the Scraper options to intercept and mutate requests.

corsproxy.io is a recommended public proxy. Note that corsproxy.org is currently reported as not working with this package.

const scraper = new Scraper({
  transform: {
    request(input: RequestInfo | URL, init?: RequestInit) {
      if (input instanceof URL) {
        const proxy = 'https://corsproxy.io/?' + encodeURIComponent(input.toString());
        return [proxy, init];
      } else if (typeof input === 'string') {
        const proxy = 'https://corsproxy.io/?' + encodeURIComponent(input);
        return [proxy, init];
      } else {
        throw new Error('Unexpected request input type');
      }
    },
  },
});

Install the package using NPM or Yarn. This package requires Node.js v16.0.0 or greater. TypeScript types are included in the distribution.

npm install @the-convocation/twitter-scraper

# or

yarn add @the-convocation/twitter-scraper

Edge runtimes (like Cloudflare Workers) may have fetch implementations that differ from the web standard. You can override the scraper's fetch by passing a custom function in the Scraper options. If the custom fetch has incompatible types, you may need to wrap it in a shim to ensure it returns a web-compliant Response.

// Basic override
const scraper = new Scraper({
  fetch: fetch,
});

// Using a shim for type compatibility
const scraper = new Scraper({
  fetch: (input, init) => {
    // Transform input and init into your function's expected types...
    return fetch(input, init).then((res) => {
      // Transform res into a web-compliant response...
      return res;
    });
  },
});

Follow these steps to set up and run the CycleTLS demonstration:

1. Install dependencies: yarn install
2. Configure your .env file with Twitter credentials.
3. Start the example: yarn start

The FlowSubtaskHandlerApi is the interface provided to custom subtask handlers to allow them to progress the authentication flow. It contains two primary methods:

1. sendFlowRequest(request: TwitterUserAuthFlowRequest): Sends a request to the Twitter API to move to the next step. The request must include the current flow_token and an array of subtask_inputs.
2. getFlowToken(): Returns the current flow_token required for subsequent requests in the sequence.

By default, the scraper uses WaitingRateLimitStrategy, which waits for the rate-limiting period to expire. This can take a long time. You can implement the RateLimitStrategy interface to define custom behavior when a rate limit event occurs.

Available built-in strategies:

• WaitingRateLimitStrategy: The default (waits for expiry).
• ErrorRateLimitStrategy: Throws an error immediately upon a rate-limit event.

import { Scraper, RateLimitStrategy } from '@the-convocation/twitter-scraper';

class CustomRateLimitStrategy implements RateLimitStrategy {
  async onRateLimit(event: RateLimitEvent): Promise<void> {
    // your own logic...
  }
}

const scraper = new Scraper({
  rateLimitStrategy: new CustomRateLimitStrategy(),
});