newspaper4k Documentation

To extract data from an Article, you must follow a specific sequence of method calls:

1. download(): Fetches the HTML. A freshly initialized article has no content until this is called. This method can be run in a multi-threaded fashion.
2. parse(): Extracts meaningful text, title, authors, images, etc., from the downloaded HTML. Note: Calling parse() before download() will raise an ArticleException.
3. nlp(): Performs Natural Language Processing to populate summary and keywords. This is computationally expensive and should be used sparingly. Note: You must call both download() and parse() before calling nlp(). Currently, nlp() only works on western languages.

Available properties after parse():

• text: The main article text.
• title: The article title.
• authors: A list of authors.
• top_image: The primary image URL.
• images: A list of all image URLs.
• movies: A list of video URLs (YouTube, Vimeo, etc.).

# The standard workflow
article = newspaper.article(url='http://example.com/article')
article.download()
article.parse()

print(article.text)
print(article.title)

# Optional NLP step
article.nlp()
print(article.summary)
print(article.keywords)

Newspaper4k uses an extractive summarization method, meaning it selects the most relevant existing sentences from the article rather than generating new text. The process follows these steps:

1. Sentence splitting: Uses NLTK Punkt tokenizer; sentences shorter than 10 characters are discarded.
2. Keyword extraction: Extracts the top SUMMARIZE_KEYWORD_COUNT (default: 10) keywords.
3. Sentence scoring: Each sentence is scored based on a weighted sum of four features:
   • Title similarity (Weight 1.5): Fraction of non-stop title words in the sentence.
   • Keyword frequency (Weight 2.0): An average of the Simple Bigram Score (SBS) and Density-Based Score (DBS).
   • Sentence length (Weight 1.0): Peaks near MEAN_SENTENCE_LEN (default: 20 words).
   • Sentence position (Weight 1.0): Heuristic favoring the beginning and end of the article.
4. Selection: The highest-scoring sentences (up to max_summary_sent, default: 5) are selected and re-ordered by their original position to maintain coherence.

Use newspaper.build(url) to create a source object for an entire website. This parses the front page, detects category links, and identifies RSS feeds to build a list of article links.

Key Methods:

• category_urls(): Returns a list of detected category URLs.
• articles: A list of article objects found on the site.
• download_articles(): Downloads all discovered articles using a multi-threaded approach. This can be slow and may lead to IP blocking if used aggressively.

Concurrency: Downloading is multi-threaded. You can control the number of threads using the number_threads argument in build() or via Configuration.number_threads.

import newspaper

# Build the source with a specific thread count
cnn_paper = newspaper.build('http://cnn.com', number_threads=3)

# Get category URLs
print(cnn_paper.category_urls())

# Access individual articles
article_urls = [article.url for article in cnn_paper.articles]

# To download all articles in bulk:
articles = cnn_paper.download_articles()

You can extract metadata and content from a specific URL using newspaper.article(url).

Key attributes available on the article object:

• authors: A list of author names.
• publish_date: The publication date.
• text: The main body text of the article.
• top_image: The URL of the main image.
• movies: A list of movie URLs found in the article.
• keywords: A list of extracted keywords (requires calling .nlp()).
• summary: A generated summary (requires calling .nlp()).

Note: To populate keywords and summary, you must first call the .nlp() method.

import newspaper

article = newspaper.article('https://edition.cnn.com/2023/10/29/sport/nfl-week-8-how-to-watch-spt-intl/index.html')

print(article.authors)
print(article.publish_date)
print(article.text)
print(article.top_image)
print(article.movies)

article.nlp()
print(article.keywords)
print(article.summary)

Newspaper4k's default downloader may not execute JavaScript. To scrape modern websites, use Playwright to render the page, extract the HTML content, and then pass that content to Newspaper4k using the input_html parameter.

Required packages:

pip install newspaper4k playwright
playwright install

Workflow:

1. Use Playwright to navigate to the URL and wait for rendering.
2. Retrieve the rendered HTML via page.content().
3. Initialize an article with newspaper.article(url, input_html=content).

from playwright.sync_api import sync_playwright
import newspaper
import time

def scrape_with_playwright(url):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url)
        time.sleep(1)  # Allow JS to render
        content = page.content()
        browser.close()

    # Use input_html to parse the rendered content
    article = newspaper.article(url, input_html=content, language='en')
    return article

article = scrape_with_playwright('https://example.com')
article.nlp()
print(article.summary)