twitter-scraper Go Library

Some methods (like GetTweets) return Go channels. These are designed to abstract away cursor management.

Key behaviors:

• They use the same underlying Fetch endpoints and are subject to the same rate limits.
• If an underlying Fetch method encounters an error, the error is passed into the result object (e.g., twitterscraper.TweetResult).
• Important: When using methods that return a result object containing an error, you must check if result.Error != nil before attempting to access the data to avoid issues when the scraper stops.

The Twitter frontend API has global and endpoint-specific rate limits:

• Global Limit: Do not make more than one request per 1.5 seconds from a single account.
• Endpoint Limit: Most endpoints are limited to approximately 150 requests per 15 minutes.

To scale, you can use multiple accounts and proxies. As of February 2024, managing ~20 accounts per IP address was reported to be stable without bans.

You can log in using a username and password. Note that you must use the username, not the email address.

• Standard login: scraper.Login("username", "password")
• With email confirmation: scraper.Login("username", "password", "email")
• With 2FA: scraper.Login("username", "password", "code")

Warning: Logging in with password may require email confirmation and is a common reason for account bans.

The preferred method for authentication is using SetCookies. You can deserialize cookies from a JSON file to maintain sessions.

To load cookies from a JSON file:

// Deserialize from JSON
var cookies []*http.Cookie
f, _ := os.Open("cookies.json")
json.NewDecoder(f).Decode(&cookies)

scraper.SetCookies(cookies)
if !scraper.IsLoggedIn() {
    panic("Invalid cookies")
}

To save current cookies to a file:

cookies := scraper.GetCookies()

data, _ := json.Marshal(cookies)
f, _ = os.Create("cookies.json")
f.Write(data)

Install the library using go get to add it to your Go project:

go get -u github.com/imperatrona/twitter-scraper

The scraper manages guest tokens automatically during the prepareRequest phase of RequestAPI.

If the scraper is not logged in (!s.isLogged), it checks if a valid guest token exists. A token is considered invalid if:

1. IsGuestToken() returns false.
2. The token was created more than 3 hours ago (s.guestCreatedAt.Before(time.Now().Add(-time.Hour*3))).

If the token is invalid, GetGuestToken() is called automatically before the request is sent. Additionally, if a response returns a X-Rate-Limit-Remaining: 0 header, the scraper clears the guest token to ensure a fresh one is requested for subsequent attempts.

The Scraper uses a fluent interface (method chaining) to configure various settings before performing scraping tasks. You can chain methods like WithDelay, WithReplies, and SetSearchMode to customize the scraper's behavior.

scraper := twitterscraper.New().
    WithDelay(2).
    WithReplies(true).
    SetSearchMode(twitterscraper.SearchLatest)

The TweetSchedule struct is used as the input parameter for CreateScheduledTweet. It defines the payload for the new scheduled post:

• Text (string): The tweet content.
• Date (time.Time): The scheduled execution time (must be in the future).
• Medias ([]*Media): An optional slice of media objects to attach to the tweet.

The getSearchTimeline method (used internally by search functions) adjusts the product variable based on the s.searchMode property of the Scraper instance. You can influence the search results by setting the searchMode before calling search methods:

• SearchLatest: Targets the latest tweets.
• SearchPhotos: Targets image-based results.
• SearchVideos: Targets video-based results.
• SearchUsers: Targets user/profile results.

The ScheduledTweet struct represents a tweet that has been scheduled. It contains the following fields:

• ID (string): The unique rest ID of the tweet.
• Text (string): The content of the tweet.
• Date (time.Time): The timestamp when the tweet is scheduled to be posted.
• Photos ([]*Photo): A list of associated images.
• GIFs ([]*GIF): A list of associated GIFs.
• Videos ([]*Video): A list of associated videos.

Retrieve details for a specific tweet using its ID. Note that GetTweet uses the TweetDetail endpoint which requires authentication. If authentication is not provided, the scraper automatically uses the TweetResultByRestId endpoint instead, which does not return InReplyToStatus or Thread tweets.

Rate Limit: 150 requests / 15 minutes

tweet, err := scraper.GetTweet("1328684389388185600")