pyktok

repository·main·Indexed 19 days ago

https://github.com/dfreelon/pyktok

A Python module for collecting TikTok videos, metadata, comments, and raw JSON data by scraping embedded data and undocumented APIs. Version 0.0.31. Features include functions for downloading single or multiple videos via URLs, scraping by user or hashtag, and extracting comments. It utilizes Playwright for scraping and supports browser cookie initialization via specify_browser() to access protected content. A Streamlit-based GUI is also provided.

Tokens
2.8K
Snippets
13
Records
13
Agent score
16%

What's inside pyktok

  1. Run Pyktok via Streamlit GUI

    main

    Pyktok includes a Streamlit-based graphical interface. To use it:

    1. Navigate to the pyktok directory in your terminal.
    2. Run the following command:
    streamlit run app.py

    This will open a browser window with graphical controls for the library.

  2. Install pyktok

    main

    Install the pyktok package via pip, and ensure you install the Playwright binaries and their system dependencies to allow the scraper to function correctly.

    pip install pyktok
    playwright install
    # optionally:
    playwright install-deps
  3. Initialize pyktok with browser cookies

    main

    To ensure Pyktok can access content that requires being logged in, use specify_browser() to initialize the module with cookies from your local browser. This is often necessary for functions that scrape protected data. Common values include 'chrome', 'firefox', or 'edge'. Note that recent updates may make this step optional for some users.

    import pyktok as pyk
    
    # Initialize with cookies from your browser
    pyk.specify_browser('chrome')
  4. Download metadata from users, hashtags, or related videos

    main

    Use save_tiktok_multi_page() to scrape approximately 30 metadata lines from specific TikTok entity pages.

    Parameters:

    • query: The username, hashtag, or video URL depending on ent_type.
    • ent_type: One of 'user', 'hashtag', or 'video_related'.
    • save_video: Boolean. If True, downloads the videos.
    • metadata_fn: The filename for the resulting CSV.

    Troubleshooting: If you encounter an EmptyResponseException, try setting headless=False (though this parameter is not explicitly shown in the usage examples, it is mentioned as a solution).

    import pyktok as pyk
    
    # Scrape a user page
    pyk.save_tiktok_multi_page('tiktok', ent_type='user', save_video=False, metadata_fn='tiktok.csv')
    
    # Scrape a hashtag page
    pyk.save_tiktok_multi_page('datascience', ent_type='hashtag', save_video=False, metadata_fn='datascience.csv')
    
    # Scrape related videos from a specific video page
    pyk.save_tiktok_multi_page('https://www.tiktok.com/@tiktok/video/7106594312292453675', ent_type='video_related', save_video=False, metadata_fn='7106594312292453675.csv')
  5. Download metadata from multiple URLs

    main

    Use save_tiktok_multi_urls() to process a list of TikTok URLs. This is useful for batch processing metadata or videos.

    Parameters:

    • urls: A list of TikTok video URLs.
    • save_video: Boolean. If True, downloads the video files.
    • filename: The name of the CSV file for metadata.
    • sleep: Integer. The number of seconds to wait between executions to avoid being autobanned by TikTok.
    import pyktok as pyk
    
    tiktok_videos = [
        'https://www.tiktok.com/@tiktok/video/7106594312292453675?is_copy_url=1&is_from_webapp=v1',
        'https://www.tiktok.com/@tiktok/video/7011536772089924869?is_copy_url=1&is_from_webapp=v1'
    ]
    
    # Download metadata only, with a 1-second sleep between requests
    pyk.save_tiktok_multi_urls(tiktok_videos, False, 'tiktok_data.csv', 1)
  6. Download TikTok comments

    main

    Use save_tiktok_comments() to extract comments from a video.

    Parameters:

    • url: The TikTok video URL.
    • comment_count: The approximate number of comments to download.
    • save_comments: Boolean. If True, saves comments to disk.
    • return_comments: Boolean. If True, returns the comments to working memory.
    import pyktok as pyk
    
    # Download 30 comments from a video and save to disk
    pyk.save_tiktok_comments('https://www.tiktok.com/@tiktok/video/7106594312292453675', comment_count=30, save_comments=True, return_comments=False)
  7. Get raw TikTok JSON data

    main

    Use alt_get_tiktok_json() to retrieve the full, raw JSON object embedded in a TikTok page. This is useful if you need to extract specific data fields that are not covered by the standard metadata functions.

    import pyktok as pyk
    
    tt_json = pyk.alt_get_tiktok_json('https://www.tiktok.com/@tiktok/video/7011536772089924869?is_copy_url=1&is_from_webapp=v1')
  8. Download single TikTok videos and metadata

    main

    Use save_tiktok() to download a single video and its metadata directly to a CSV file.

    Parameters:

    • url: The TikTok video URL.
    • save_video: Boolean. If True, the video file is downloaded. If False, only metadata is saved.
    • filename: The name of the CSV file to save the metadata to.
    import pyktok as pyk
    
    # Download video and metadata to 'video_data.csv'
    pyk.save_tiktok('https://www.tiktok.com/@tiktok/video/7106594312292453675?is_copy_url=1&is_from_webapp=v1', True, 'video_data.csv')
  9. Save multiple TikTok videos from a list of URLs

    main

    Use save_tiktok_multi_urls to iterate through a list of video URLs and perform the same save operations as save_tiktok for each.

    Parameters:

    • video_urls (list or str): A list of URLs, or a string representing a path to a text file containing one URL per line.
    • save_video (bool): Whether to download the video files.
    • metadata_fn (str): Filename for the combined metadata CSV.
    • sleep (int): Maximum random sleep time (in seconds) between requests to avoid rate limiting.
    • browser_name (str, optional): Browser name for cookie extraction.
    import pyktok
    
    urls = ['https://www.tiktok.com/@user/video/1', 'https://www.tiktok.com/@user/video/2']
    
    pyktok.save_tiktok_multi_urls(
        video_urls=urls,
        save_video=True,
        metadata_fn='batch_metadata.csv',
        sleep=4,
        browser_name='firefox'
    )
  10. Scrape multiple videos from a user, hashtag, or related videos

    main

    The save_tiktok_multi_page function automates the process of finding video URLs for a specific entity and then downloading them.

    Parameters:

    • tt_ent (str): The username (for user), hashtag name (for hashtag), or URL (for video_related).
    • ent_type (str): One of 'user', 'hashtag', or 'video_related'.
    • video_ct (int): The number of videos to attempt to fetch.
    • headless (bool): Whether to run the underlying browser in headless mode.
    • save_video (bool): Whether to download the video files.
    • metadata_fn (str): Filename for the metadata CSV.
    • sleep (int): Max sleep time between requests.
    • browser_name (str, optional): Browser name for cookie extraction.
    import pyktok
    
    # Scrape up to 30 videos from a specific user
    pyktok.save_tiktok_multi_page(
        tt_ent='username',
        ent_type='user',
        video_ct=30,
        save_video=True,
        metadata_fn='user_videos.csv',
        browser_name='chrome'
    )
  11. Extract and save TikTok comments

    main

    The save_tiktok_comments function retrieves comments for a specific video and can save them directly to a CSV.

    Parameters:

    • video_url (str): The URL of the TikTok video.
    • filename (str): The CSV filename. If empty, it defaults to [video_id]_comments.csv.
    • comment_count (int): The number of comments to attempt to fetch.
    • headless (bool): Whether to run the underlying browser in headless mode.
    • save_comments (bool): If True, writes the comments to the CSV.
    • return_comments (bool): If True, returns the comments as a pandas DataFrame.
    import pyktok
    
    # Save 50 comments from a video to a CSV
    comments_df = pyktok.save_tiktok_comments(
        video_url='https://www.tiktok.com/@user/video/123456789',
        filename='video_comments.csv',
        comment_count=50,
        browser_name='chrome'
    )
  12. Configure browser cookies for TikTok extraction

    main

    To bypass restrictions and extract data, pyktok often requires cookies from a logged-in TikTok session. You can use specify_browser(browser) to extract cookies from a browser installed on your system.

    Supported browser names (passed to browser_cookie3) include strings like "chrome", "firefox", or "edge".

    import pyktok
    
    # Specify the browser to extract cookies from
    pyktok.specify_browser('chrome')