Newspaper3k Python Library

Newspaper supports automatic language detection, but you can explicitly specify a language using the language parameter in the Article constructor or the newspaper.build() function. This is useful for ensuring accurate NLP and parsing for specific languages.

from newspaper import Article

# Explicitly set language to Chinese
url = 'http://www.bbc.co.uk/zhongwen/simp/chinese_news/2012/12/121210_hongkong_politics.shtml'
a = Article(url, language='zh')
a.download()
a.parse()

print(a.title)

# Or when building from a source
import newspaper
sina_paper = newspaper.build('http://www.sina.com.cn/', language='zh')

To apply the krTheme Sphinx styles (derived from Mitsuhiko's Flask themes) to your project's documentation, follow these steps:

1. Install the theme: Place the krTheme folder into a directory named _themes inside your documentation folder. You can also use git submodules to manage this dependency.
2. Configure Sphinx: Update your conf.py file to include the _themes path and set the theme name.

Available themes:

• flask: The standard theme for large projects.
• kr_small: A small, one-page theme intended for very small addon libraries.

sys.path.append(os.path.abspath('_themes'))
html_theme_path = ['_themes']
html_theme = 'flask'

To contribute to the project, clone the repository and install requirements locally.

Local Installation

git clone git://github.com/codelucas/newspaper.git
cd newspaper
pip3 install -r requirements.txt
python3 setup.py install

Running Tests

Run the unit test suite (which uses mocks):

python3 tests/unit_tests.py

To test the full-text algorithm specifically, use the fulltext parameter:

python3 tests/unit_tests.py fulltext

To avoid rate limiting while speeding up downloads, use news_pool to allocate a specific number of threads per news source. This is more efficient than downloading articles one by one or using a single global thread pool for all sources.

Use news_pool.set(papers, threads_per_source=N) where papers is a list of newspaper.build() objects and N is the number of threads to allocate to each individual source.

import newspaper
from newspaper import news_pool

slate_paper = newspaper.build('http://slate.com')
tc_paper = newspaper.build('http://techcrunch.com')
espn_paper = newspaper.build('http://espn.com')

papers = [slate_paper, tc_paper, espn_paper]
# Allocates 2 threads per source, totaling 6 threads
news_pool.set(papers, threads_per_source=2)
news_pool.join()

# After join(), articles are downloaded and ready
print(slate_paper.articles[10].html)

To extract content from a single URL, use the Article class. The workflow involves initializing the object with a URL, calling .download() to fetch the HTML, .parse() to extract metadata and text, and optionally .nlp() to perform natural language processing for keywords and summaries.

from newspaper import Article

url = 'http://fox13now.com/2013/12/30/new-year-new-laws-obamacare-pot-guns-and-drones/'
article = Article(url)

article.download()
article.parse()

print(article.authors)
print(article.publish_date)
print(article.text)
print(article.top_image)

# For NLP features (keywords and summary)
article.nlp()
print(article.keywords)
print(article.summary)

On Debian or Ubuntu systems, you must install several system dependencies before installing the Python package via pip3.

# Install system dependencies
sudo apt-get install python3-pip
sudo apt-get install python-dev
sudo apt-get install libxml2-dev libxslt-dev
sudo apt-get install libjpeg-dev zlib1g-dev libpng12-dev

# Download and run NLP corpora downloader
curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python3

# Install the package
pip3 install newspaper3k

If a site requires JavaScript execution to show content, use an external service (like a Web Unblocker) to fetch the rendered HTML, then pass that HTML into article.download(input_html=...).

import requests
from newspaper import Article

url = 'https://example.com/some-news-story'

# Fetch rendered HTML from an external service
html = requests.post(
    'https://webunlocker.novada.com/request',
    headers={'Authorization': 'Bearer YOUR_NOVADA_KEY'},
    data={'target_url': url, 'response_format': 'html', 'js_render': 'True'},
).text

# Pass the rendered HTML directly to newspaper
article = Article(url)
article.download(input_html=html)
article.parse()
print(article.title)

For absolute control over the construction of a news source, use the Source class instead of the high-level newspaper.build() function. This allows you to manually trigger each step of the lifecycle.

The sequence of dependent methods is:

1. Source(url)
2. .download()
3. .parse()
4. .set_categories() -> .download_categories() -> .parse_categories()
5. .set_feeds() -> .download_feeds()
6. .generate_articles()

from newspaper import Source

cnn_paper = Source('http://cnn.com')
cnn_paper.download()
cnn_paper.parse()
cnn_paper.set_categories()
cnn_paper.download_categories()
cnn_paper.parse_categories()
cnn_paper.set_feeds()
cnn_paper.download_feeds()
cnn_paper.generate_articles()

print(cnn_paper.size())

On macOS, use Homebrew or MacPorts to install required libraries, then install the package and NLP corpora.

# Install system dependencies via brew
brew install libxml2 libxslt
brew install libtiff libjpeg webp little-cms2

# Install the package
pip3 install newspaper3k

# Download NLP corpora
curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python3