website·Indexed 29 days ago
https://vuepress-notes-delta.vercel.appA comprehensive technical guide covering Python3 fundamentals, network programming, and web scraping. It includes detailed sections on the Scrapy and Feapder frameworks, data extraction using BeautifulSoup and lxml, and advanced JavaScript reverse engineering, including symmetric/asymmetric encryption (AES, DES, SM series), webpack analysis, OB obfuscation, and environment emulation (JSVMP).
Babel provides a suite of libraries to handle JavaScript AST transformations:
@babel/core: The core compiler providing the compilation API.@babel/parser: Parses JavaScript code into an AST (JSON structure).@babel/traverse: Used to traverse and modify nodes within the AST.@babel/generator: Converts an AST back into JavaScript source code.@babel/types: Used to validate node types and build new AST nodes.npm install @babel/core --save-devSymmetric encryption uses the same key for both encryption and decryption. Both the sender and receiver must possess the identical key and algorithm. Common symmetric algorithms include:
The SM (ShangMi) series consists of commercial cryptography standards used in China. The most common algorithms encountered in web development and reverse engineering are:
debugger statement to inspect the call stack. This is particularly useful for locating data encryption logic when standard DOM or XHR breakpoints are too broad or deep in the call stack. For a hook to work, the hook script must be executed in the same scope as the function being hooked.__proto__) that points to the explicit prototype (prototype) of the class that created it. When accessing a property or method, if it is not found on the object itself, JavaScript searches up the prototype chain. This mechanism is critical when mimicking browser environments (like navigator or document) because detection scripts often check if a method exists on the object or its prototype chain using hasOwnProperty or Object.getOwnPropertyDescriptor.Queue for managing URLs and data, retrying for handling request failures, and pymysql for bulk data storage. The scraper follows a three-step pipeline: extracting category links, discovering product detail URLs through pagination, and parsing final product details (title and price). It includes commented-out logic for integrating a paid proxy API to avoid IP blocking.import pymysql
import requests
import retrying
from lxml import etree
from queue import Queue
from feapder.network.user_agent import get
class AmazonShop:
def __init__(self):
# Database connection
self.db = pymysql.connect(host='localhost', user='root', password='root', db='py_spider')
self.cursor = self.db.cursor()
# Target API and Headers
self.amazon_api = 'https://www.amazon.cn/nav/ajax/hamburgerMainContent...'
self.headers = {
"Referer": "https://www.amazon.cn",
"User-Agent": "Mozilla/5.0...",
"X-Requested-With": "XMLHttpRequest",
}
# Queues for pipeline management
self.classify_url_queue = Queue() # Category URLs
self.detail_url_queue = Queue() # Product detail URLs
self.shop_info_queue = Queue() # Final product data
def create_table(self):
sql = """
create table if not exists amazon_shop(
id int primary key auto_increment,
price varchar(255) not null,
title varchar(255) not null,
goods_url varchar(255) not null
);
"""
self.cursor.execute(sql)
@retrying.retry(stop_max_attempt_number=3)
def get_shop_data(self, url):
# Dynamic User-Agent rotation
self.headers['User-Agent'] = get()
# To use proxies, uncomment:
# proxies = {'http': 'http://' + ip, 'https': 'https://' + ip}
# response = requests.get(url, headers=self.headers, proxies=proxies, timeout=2)
response = requests.get(url, headers=self.headers)
return response
def parse_shop_info(self):
while True:
goods_data = self.detail_url_queue.get()
goods_url = 'https://www.amazon.cn' + goods_data['detail_href']
try:
response = self.get_shop_data(goods_url)
tree = etree.HTML(response.text)
# Extract Title
title = tree.xpath('//div[@id="centerCol"]//h1/span/text()')[0] if tree.xpath('//div[@id="centerCol"]//h1/span/text()') else tree.xpath('//title/text()')[0]
# Extract Price
if tree.xpath('//div[@id="centerCol"]//div[@id="apex_desktop"]//span[@class="a-price-whole"]/text()'):
price = "￥" + tree.xpath('//div[@id="centerCol"]//div[@id="apex_desktop"]//span[@class="a-price-whole"]/text()')[0]
else:
price = '-'.join(tree.xpath('//td[@class="a-span12"]//span[@class="a-offscreen"]/text()'))
self.shop_info_queue.put((title.strip(), price, goods_url))
except Exception as e:
print('Error:', e)
finally:
self.detail_url_queue.task_done()
def save_shop_info(self):
while True:
info_list = []
for _ in range(30): # Batch insert 30 records
info = self.shop_info_queue.get()
info_list.append((0,) + info)
self.shop_info_queue.task_done()
sql = "insert into amazon_shop(id, price, title, goods_url) values (%s, %s, %s, %s);"
self.cursor.executemany(sql, info_list)The V8 engine processes JavaScript code through several distinct stages to execute it. The AST (Abstract Syntax Tree) serves as the intermediate representation of the source code, allowing the engine to perform semantic interpretation and execution. The pipeline is as follows:
The scrapy-redis library enables a distributed architecture where multiple Scrapy processes (Slaver nodes) share a single Redis database (Master node). The Master node handles URL fingerprint deduplication, request distribution, and data storage, while Slaver nodes execute the crawling logic and submit new requests back to the Master.
To implement a distributed spider, you must inherit from RedisSpider and define a redis_key which specifies the Redis key containing the initial start URLs.
from scrapy_redis.spiders import RedisSpider
class MyDistributedSpider(RedisSpider):
name = "my_spider"
# The key in Redis where start URLs are stored
redis_key = 'my_spider:start_urls'
def parse(self, response, **kwargs):
# Crawling logic here
passThe Sekiro RPC framework allows a client (like a browser) to register functions that can be invoked remotely via a server. Key concepts include: