news-please Documentation

You can use news-please within your own Python code to extract semi-structured information from news articles. Note that Library mode is for extracting information from specific URLs or HTML; for full website crawling (starting from a root URL) or continuous RSS crawling, use the CLI mode.

Extracted attributes include: headline, lead paragraph, main text, main image, name(s) of author(s), publication date, and language.

from newsplease import NewsPlease

# Extract from a single URL
article = NewsPlease.from_url('https://www.nytimes.com/2017/02/23/us/politics/cpac-stephen-bannon-reince-priebus.html?hp')
print(article.title)

# Export to JSON
import json
with open("article.json", "w") as file:
    json.dump(article.get_serializable_dict(), file)

news-please requires Python 3.8 or higher. You can install it using pip:

$ pip install news-please

To export results to ElasticSearch (which also enables the versioning feature), update your config.cfg (or config_lib.cfg for library mode).

Default config location is ~/news-please/config. You can specify a custom location using the -c parameter with the CLI.

[Scrapy]
ITEM_PIPELINES = {
  'newsplease.pipeline.pipelines.ArticleMasterExtractor':100,
  'newsplease.pipeline.pipelines.ElasticsearchStorage':350
}

[Elasticsearch]
host = localhost
port = 9200
use_ca_certificates = False           # Set to True if authentication is required
ca_cert_path = '/path/to/cacert.pem'
client_cert_path = '/path/to/client_cert.pem'
client_key_path = '/path/to/client_key.pem'
username = 'root'
secret = 'password'

To export articles to a Redis database, add the RedisStorage module to your ITEM_PIPELINES and configure the connection parameters.

[Scrapy]
ITEM_PIPELINES = {
  'newsplease.pipeline.pipelines.ArticleMasterExtractor':100,
  'newsplease.pipeline.pipelines.RedisStorage':350
}

[Redis]
host = localhost
port = 6379
db = 0
ssl_check_hostname = True
username = "news-please"
max_connections = 24

To store articles in a PostgreSQL database, add the PostgresqlStorage module to your ITEM_PIPELINES and provide connection details.

[Scrapy]
ITEM_PIPELINES = {
  'newsplease.pipeline.pipelines.ArticleMasterExtractor':100,
  'newsplease.pipeline.pipelines.PostgresqlStorage':350
}

[Postgresql]
host = localhost
port = 5432
database = 'news-please'
user = 'user'
password = 'password'

To run the crawler in its full mode (which can follow internal links and RSS feeds), use the CLI command. By default, results are stored as JSON files in a data folder, and original HTML files are also saved.

$ news-please