TikTokPy

repository·main·Indexed 42 days ago

https://github.com/russell-newton/tiktokpy

An asyncio-based Python library for scraping TikTok data without requiring login credentials or API keys. It leverages Playwright for browser automation and provides both synchronous (TikTokAPI) and asynchronous (AsyncTikTokAPI) interfaces to extract video, user, and challenge information, including engagement statistics and media downloads via yt-dlp.

Tokens
10.4K
Snippets
40
Records
50
Agent score
52%

What's inside TikTokPy

  1. Getting Started with TikTokPy

    main

    TikTokPy allows you to extract data from TikTok without requiring login credentials or API keys. To begin using the library, you should follow these steps:

    1. Install the library: Follow the installation guide to add TikTokPy to your project.
    2. Understand the mechanics: Review the explanation of how the library works and its limitations (what it cannot do).
    3. Learn usage patterns: Consult the usage guide to learn how to grab specific data from TikTok.
    4. Download media: If your primary goal is downloading videos or slideshows, refer to the specific 'Download Videos and Slideshows' example.
    5. Consult the API: For detailed technical specifications of the library's functions and classes, use the API Reference.
  2. How TikTokPy retrieves data

    main

    TikTokPy operates without login credentials or cookies by using a two-step retrieval process:

    1. Grabbing Preloaded Content: It extracts data embedded directly in the HTML within a <script id="SIGI_STATE" type="application/json"> tag. This includes initial data for videos, comments, and users.
    2. Grabbing Additional API Data: It intercepts subsequent API calls made by the page's JavaScript. TikTokPy uses the same cookies and headers that the page scripts would use, allowing it to mimic a normal browsing session without being logged in.

    Note: This explanation applies to library versions prior to 0.2.0.

  3. Emulate a mobile device to access slideshows

    main

    TikTok restricts slideshows to mobile viewing. To retrieve data from slideshows, you must initialize the API with emulate_mobile=True.

    Warning: You cannot use emulate_mobile=True and scroll_down_time simultaneously. If you need both slideshow data and additional data via scrolling, you must create two separate API instances: one for mobile emulation (for slideshows) and one for standard scraping (for scrolling).

    # Using TikTokAPI
    from tiktokapipy.api import TikTokAPI
    
    def get_challenge_videos_scroll_down():
        with TikTokAPI(emulate_mobile=True) as api:
            slideshow = api.video(link_to_slideshow)
            ...
    
    # Using AsyncTikTokAPI
    from tiktokapipy.async_api import AsyncTikTokAPI
    
    async def get_challenge_videos_scroll_down():
        async with AsyncTikTokAPI(emulate_mobile=True) as api:
            slideshow = await api.video(link_to_slideshow)
            ...
  4. Initialize TikTokPy using Context Managers

    main

    TikTokPy provides both synchronous (TikTokAPI) and asynchronous (AsyncTikTokAPI) interfaces. You should always use the API as a context manager to ensure proper lifecycle management of the underlying Playwright BrowserContext. The initialization arguments allow you to configure proxies, custom executable locations, and other browser settings.

    # Synchronous usage
    from tiktokapipy.api import TikTokAPI
    
    def do_something():
        with TikTokAPI() as api:
            ...
    
    # Asynchronous usage
    from tiktokapipy.async_api import AsyncTikTokAPI
    
    async def do_something():
        async with AsyncTikTokAPI() as api:
            ...
  5. Configure navigation retries and timeouts

    main

    To handle slow navigation or timeouts, you can configure retry logic in the API constructor using navigation_retries and navigation_timeout.

    • navigation_retries: The number of times to attempt navigation again (e.g., setting this to 2 allows for up to 3 total attempts).
    • navigation_timeout: The duration in seconds after which a navigation attempt is considered failed and a retry is forced.

    If all retries are exhausted, a TikTokAPIError is raised. This is common when using wait_until="networkidle".

    # Using TikTokAPI
    from tiktokapipy.api import TikTokAPI
    
    def do_something():
        # retry twice (up to 3 navigation attempts), force-retry navigation after 10 seconds
        with TikTokAPI(navigation_retries=2, navigation_timeout=10) as api:
            ...
    
    # Using AsyncTikTokAPI
    from tiktokapipy.async_api import AsyncTikTokAPI
    
    async def do_something():
        # retry twice (up to 3 navigation attempts), force-retry navigation after 10 seconds
        async with AsyncTikTokAPI(navigation_retries=2, navigation_timeout=10) as api:
            ...
  6. Download Videos and Slideshows

    main

    To download content, you can use the download() method on a Video object.

    Requirements:

    • For download() to work, you must have yt-dlp installed (pip install yt-dlp or pip install tiktokapipy[download]).
    • download() works for standard videos but not for slideshows.

    Handling Slideshows: Slideshows consist of multiple images and an audio track. To download them as a single video, you must manually download the images and audio, then use a tool like ffmpeg to stitch them together.

    # Basic video download using yt-dlp
    with TikTokAPI() as api:
        video = api.video(video_url)
        video.download()
  7. Trigger additional data collection by scrolling

    main

    By default, TikTokPy only captures preloaded content and API calls made during the initial page load. To collect more data, you must trigger additional API calls by scrolling down the page. This is controlled by the scroll_down_time parameter, which determines how long the library scrolls.

    You can set scroll_down_time in the API constructor or pass it directly to specific API calls to override the default.

    Note: This feature is incompatible with emulate_mobile=True.

    # Using TikTokAPI
    from tiktokapipy.api import TikTokAPI
    
    def get_challenge_videos_scroll_down():
        # scroll down for 2.5 seconds when making requests
        with TikTokAPI(scroll_down_time=2.5) as api:
            challenge = api.challenge(tag_name)
            for video in challenge.videos:
                ...
    
    # Using AsyncTikTokAPI
    from tiktokapipy.async_api import AsyncTikTokAPI
    
    async def get_challenge_videos_scroll_down():
        # scroll down for 2.5 seconds when making requests
        async with AsyncTikTokAPI(scroll_down_time=2.5) as api:
            challenge = await api.challenge(tag_name)
            async for video in challenge.videos:
                ...
  8. Use DeferredIterators for efficient data collection

    main

    The library uses DeferredIterator subclasses to fetch data lazily. Instead of loading all results into memory at once, the iterator fetches chunks of data from the TikTok API only when needed during iteration.

    Key implementations include:

    • DeferredCommentIterator: Iterates through comments for a specific video_id.
    • DeferredItemListIterator: Iterates through videos from a specific from_type ('post' or 'challenge') and target_id.
    • DeferredChallengeIterator: Iterates through a list of provided challenge names.

    Important Context Rule:

    • If you are using an asynchronous API context, you must use async for with these iterators.
    • If you are using a synchronous API context, you must use a standard for loop. Attempting to use the wrong loop type will raise a TikTokAPIError.
  9. Understand the structure of TikTok API responses

    main

    TikTokPy uses Pydantic models to represent the raw, unprocessed data retrieved from TikTok. These models are organized by the type of content being accessed. When working with raw responses, you will encounter several primary response types that wrap the actual data (like videos, users, or challenges) along with metadata like status codes and pagination cursors.

    Key response types include:

    • VideoResponse: Contains video data (item_module), comments (comment_item), and status information (video_page).
    • UserResponse: Contains user data (user_module), video lists (item_module), and status information (user_page).
    • ChallengeResponse: Contains challenge details (challenge_page) and associated videos (item_module).
    • SentToLoginResponse: Triggered when a request requires authentication, providing a redirect_url to the login page.
  10. Initialize the TikTokAPI synchronous client

    main

    The TikTokAPI class is the primary entry point for scraping TikTok data synchronously. It must be used as a context manager (using the with statement) to ensure the underlying Playwright browser and context are properly initialized and closed.

    Configuration Options

    ParameterTypeDefaultDescription
    headlessboolNoneWhether to use headless browsing.
    data_dump_filestrNoneIf specified, the raw JSON scraped from TikTok will be saved to a file named {data_dump_file}.{ModelName}.json before parsing.
    navigation_timeoutfloat30Timeout in seconds for page navigation. Set to 0 to disable.
    navigation_retriesint0Number of times to retry navigation if a network timeout occurs.
    context_kwargsdictNoneExtra arguments passed to Playwright's browser.new_context().
    **kwargsdictNoneExtra arguments passed to Playwright's browser.launch() (e.g., proxy).

    Note: The navigator_type parameter is deprecated. Chromium is always used.

    from tiktokapipy import TikTokAPI
    
    with TikTokAPI(headless=True, data_dump_file="my_dump") as api:
        # Your scraping logic here
        pass