DOCSEARCHbyAlgolia
Submit Docs

Learn Python 3 Web Spider Curriculum

repository·master·Indexed 30 days ago

https://github.com/wistbean/learn_python3_spider

A structured learning environment and curriculum for mastering Python web scraping. Covers foundations, libraries like Scrapy, BeautifulSoup, and Selenium, data handling with MySQL and MongoDB, concurrency, and advanced anti-scraping techniques including CAPTCHA solving, JS obfuscation, and mobile app automation via Appium.

Tokens
19.1K
Snippets
12
Records
124
Agent score
98%

What's inside wistbean-learn_python3_spider

  1. Overview of Python Web Scraping Curriculum

    master

    The learn_python3_spider repository provides a structured learning path for mastering Python web scraping, ranging from absolute basics to advanced anti-scraping techniques. The curriculum is organized into several key stages:

    1. Foundations: Introduction to scraping concepts and packet sniffing (using Chrome and Fiddler).
    2. Library Usage: Practical guides for urllib, requests, BeautifulSoup, selenium, Appium, and scrapy.
    3. Data Handling: Techniques for parsing JSON, saving data to CSV, MySQL, and MongoDB, and performing data visualization.
    4. Concurrency & Performance: Using multi-threading, multi-processing, and coroutines to speed up scraping.
    5. Advanced Anti-Scraping: Dealing with CSS encryption, JS obfuscation, CAPTCHAs, IP proxy pools, and reverse engineering JS/APP data.
    6. Specialized Scraping: WebSocket scraping and distributed crawling architectures.
    7. Real-world Projects: Practical examples like scraping WeChat, Douban, Bilibili, and StackOverflow.

  2. Set up Incremental for project versioning

    master

    To use Incremental for managing your Python project's version, modify your setup.py by adding use_incremental=True to the setup() call. You must also include incremental in both setup_requires and install_requires.

    After configuring setup.py, initialize the version file by running the update command with the --create flag. This generates a _version.py file in your package. Finally, expose the version in your package's __init__.py so it is accessible to users.

    setup(
       use_incremental=True,
       setup_requires=['incremental'],
       install_requires=['incremental'],
       ...
   )

  3. Store scraped items in Redis for post-processing

    master

    To push scraped items into a Redis queue for asynchronous post-processing, configure the RedisPipeline in your ITEM_PIPELINES setting:

    1. Add 'scrapy_redis.pipelines.RedisPipeline': 300 to ITEM_PIPELINES.
    2. (Optional) Customize the Redis key using REDIS_ITEMS_KEY (defaults to %(spider)s:items).
    3. (Optional) Customize the serializer using REDIS_ITEMS_SERIALIZER (defaults to ScrapyJSONEncoder).

  4. Write custom matchers by extending BaseMatcher

    master

    To create a custom matcher, inherit from hamcrest.core.base_matcher.BaseMatcher and implement two methods:

    1. _matches(self, item): Returns True if the item matches the criteria, False otherwise.
    2. describe_to(self, description): Appends text to the description object to produce a human-readable failure message.

    It is recommended to make matchers stateless so they can be reused. A common pattern is to use a factory function to instantiate the matcher.

    from hamcrest.core.base_matcher import BaseMatcher
from hamcrest.core.helpers.hasmethod import hasmethod

class IsGivenDayOfWeek(BaseMatcher):
    def __init__(self, day):
        self.day = day  # Monday is 0, Sunday is 6

    def _matches(self, item):
        if not hasmethod(item, 'weekday'):
            return False
        return item.weekday() == self.day

    def describe_to(self, description):
        day_as_string = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']
        description.append_text('calendar date falling on ') \
                   .append_text(day_as_string[self.day])

def on_a_saturday():
    return IsGivenDayOfWeek(5)

# Usage in a test:
# assert_that(d, is_(on_a_saturday()))

  6. Process Schematron schemas using the XSLT1 pipeline

    master

    To validate a document using the ISO Schematron 2010 XSLT1 implementation, follow a four-stage pipeline. This process converts a Schematron schema into an XSLT script which is then used to validate XML documents.

    1. Preprocess inclusions: Use iso_dsdl_include.xsl to assemble the schema from various parts. Skip this if your schema is not split into multiple files.
    2. Expand abstract patterns: Use iso_abstract_expand.xsl to convert abstract patterns to real patterns. Skip this if your schema does not use abstract patterns.
    3. Compile to XSLT: Use iso_svrl_for_xslt1.xsl to compile the schema into an XSLT script. If your schema uses Schematron phases, provide them as invocation parameters.
    4. Validate document: Run the generated XSLT script against your target XML document. If using the SVRL script, the output will be an XML document containing validation results.

  8. Initialize and update versions with incremental.update

    master

    Incremental provides a CLI tool incremental.update (requires click to be installed) to manage and automate version bumps.

    Initialization: Run python -m incremental.update <projectname> --create to generate the initial _version.py file.

    Updating Versions: Run python -m incremental.update <projectname> followed by one of these flags:

    • --newversion=<version>: Set to a specific version (e.g., 1.2.3 or 17.1.0dev1).
    • --rc: Sets the version to a release candidate (e.g., <year-2000>.<month>.0rc1) or increments the existing RC number.
    • --dev: Sets the development release number to 0 or increments it.
    • --patch: Increments the patch number. If used with --rc, it increments the patch and makes it a release candidate.
    • No flags: Strips the release candidate number to create a "full release".
    python -m incremental.update <projectname> --create

  9. Configure Scrapy-Redis for distributed crawling

    master

    To enable distributed crawling and ensure multiple spider instances share the same request queue and duplicate filter, add the following settings to your Scrapy project configuration:

    • SCHEDULER: Set to "scrapy_redis.scheduler.Scheduler" to store the request queue in Redis.
    • DUPEFILTER_CLASS: Set to "scrapy_redis.dupefilter.RFPDupeFilter" to share the duplicate filter across spiders.

    To allow pausing and resuming crawls, set SCHEDULER_PERSIST = True to prevent the cleanup of Redis queues.

    SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_PERSIST = True

  10. Access Python Learning Resources and Tutorials

    master

    Beyond code examples, the project provides links to broader Python learning resources, including:

    • Python Tutorials: A curated list of resources available at wistbean.github.io/categories/python/.
    • Video Tutorials: Recommended Bilibili videos covering undervalued Python techniques.
    • Chrome Extensions: Recommendations for powerful Chrome plugins used in web scraping.
    • Community/Books: You can find more Python-related books by searching for the WeChat ID fxxkpython (Name: 帅彬老仙) and sending the keyword 帅书.

  11. Learn Python Scraping Libraries and Techniques

    master

    The curriculum covers a wide array of tools and techniques for different scraping scenarios:

    HTTP & Parsing

    • urllib: Making Python act like a browser.
    • requests: A more user-friendly alternative to urllib.
    • BeautifulSoup: Parsing HTML/XML to avoid complex regular expressions.
    • JSON: Parsing data from API responses.

    Browser Automation & Mobile

    • selenium: Automating browser interactions (including integration with phantomJS).
    • Appium: Automating mobile applications (e.g., scraping WeChat).

    Frameworks & Performance

    • scrapy: A powerful framework for large-scale crawling and database integration.
    • Concurrency: Implementing threading, multiprocessing, and coroutines for high-speed scraping.

    Data Storage & Analysis

    • Storage: Saving scraped data to CSV, MySQL, and MongoDB.
    • Analysis: Using Python for data visualization.

  12. Bypass Anti-Scraping Mechanisms

    master

    For advanced users, the repository provides guidance on overcoming common web protections:

    • Header Mimicry: Using custom headers to disguise scraping requests.
    • IP Proxy Pools: Using proxy services (like Bright Data) to rotate IP addresses and avoid bans.
    • CAPTCHA Solving: Identifying and automatically solving image and slider CAPTCHAs.
    • Encryption Handling:
      • CSS Encryption: Dealing with websites that use CSS to hide data.
      • JS Obfuscation: Reverse engineering JavaScript to understand data generation.
      • JS Reverse Engineering: Techniques for code extraction and debugging.
    • Mobile App Scraping: Reverse engineering Android apps and low-level encryption to extract data.