simonwillisonblog Documentation

Guide Sections are purely organizational groupings of chapters within a guide. They are used to structure the Table of Contents (TOC) and sidebar display but do not have their own pages, URLs, or content (like descriptions or draft statuses).

Ordering Logic:

• Top-level ordering: Standalone chapters and sections share a single ordering namespace at the guide level. A standalone chapter with order=0 appears before a section with order=1.
• Nested ordering: Chapters assigned to a section use their order field to determine their position within that specific section.
• Navigation: While sections organize the UI, they are invisible to the 'Previous/Next' navigation logic, which follows a flat, linear walk of all chapters.

NAME OF GUIDE
- Getting started          (standalone chapter, order=0)
- Basics                   (section, order=1)
  - Glossary               (chapter in section, order=0)
  - Installation           (chapter in section, order=1)
- Intermediate             (section, order=2)
  - Chapter A              (chapter in section, order=0)

This notebook demonstrates how to scrape comment data from MetaFilter's 'Ask' section. The process involves two main steps: first, identifying the URLs of individual comments by parsing the activity pages, and second, visiting each comment URL to extract structured data like the title, date, time, HTML content, and tags.

import requests
from BeautifulSoup import BeautifulSoup as Soup

# 1. Find comment URLs from an activity page
url = 'http://www.metafilter.com/activity/18146/comments/ask/'
page = Soup(requests.get(url).content)
divs = page.findAll('div', {'class': 'copy'})

urls = []
for div in divs:
    for blockquote in div.findAll('blockquote'):
        urls.append(blockquote.findAll('a')[-1]['href'])

# 2. Extract structured data from each URL
def get_comment(url):
    # Implementation details for parsing individual comment pages
    ...

fetched = [get_comment(u) for u in urls]

This guide demonstrates how to extract and reconstruct blog content (Entries, Blogmarks, or Quotations) from a Wayback Machine archive (tar.gz format) and convert it into a structured JSON format compatible with a Django-based blog system.

Workflow Overview

1. Extract Paths: Open the .tar.gz archive and identify relevant HTML file paths using regex.
2. Identify Missing Content: Compare the paths found in the archive against existing records in the Django database (Entry, Blogmark, or Quotation models) to find what is missing.
3. Parse HTML: Use BeautifulSoup to scrape the HTML content from the archive, identifying the content type (entry, quotation, or blogmark) based on specific DOM structures.
4. Export to JSON: Save the reconstructed items into a JSON file for import.

# Example of the core logic: parsing an item from soup
import json
from BeautifulSoup import BeautifulSoup as Soup

# Assuming 'missing' contains paths to files in the tar archive
items = []
for path in missing:
    html = tar.extractfile('simonwillison.net%sindex.html' % path).read()
    soup = Soup(html)
    item = soup_to_item(soup)
    item['slug'] = [b for b in path.split('/') if b][-1]
    items.append(item)

# Export to JSON
with open('/tmp/missing-content.json', 'w') as f:
    json.dump(items, f, indent=2)

The get_comment(url) function takes a specific comment URL (including the fragment/hash) and returns a dictionary containing the following fields:

• title: The post title (extracted from h1.posttitle).
• date: The date of the comment.
• time: The time of the comment.
• html: The raw HTML content of the comment.
• url: The original URL.
• comment_id: The hash extracted from the URL.
• tags: A list of strings representing the tags associated with the post.

def get_comment(url):
    url, hash = url.split('#')
    soup = Soup(requests.get(url).content.decode('utf8'))
    div = soup.find('div', {'id': 'c' + hash})
    span = div.find('span', {'class': 'smallcopy'})
    time = span.findAll('a', {'target': '_self'})[-1].text
    date = span.renderContents().split(' on ')[-1].strip().split(' [')[0]
    span.extract()
    html = div.renderContents().decode('utf8')
    if html.endswith('<br />'):
        html = html[:-(len('<br />'))]
    return {
        'title': soup.find('h1', {'class': 'posttitle'}).renderContents().split('<br')[0],
        'date': date,
        'time': time,
        'html': html,
        'url': url,
        'comment_id': hash,
        'tags': [a.text for a in soup.findAll('a', {'rel': 'tag'})],
    }