GNews Python Library

The RSS backend has several constraints compared to other potential backends:

• Result Limit: A maximum of approximately 100 results per query.
• Date Format: Does not provide absolute ISO dates; instead, it uses relative strings (e.g., "2 hours ago").
• Media: No thumbnails or favicons are provided.
• Stability: Behavior can occasionally be affected by changes to Google's RSS feeds.

Full-text extraction is subject to the following constraints:

• Paywalls: Articles behind paywalls cannot be extracted.
• JS-heavy sites: Modern websites that rely heavily on JavaScript may block plain HTTP requests used by the extractor.
• Bot detection: Sites protected by services like Cloudflare may block extraction attempts.

If get_full_article() fails consistently for a specific domain, the site may be actively blocking automated requests.

GNews uses an exponential backoff formula with uniform jitter to prevent 'thundering herd' problems. The delay for each retry attempt is calculated as:

delay = min(retry_backoff_max, retry_backoff_base * 2 ** attempt) + uniform(0, retry_backoff_base)

Where attempt is the zero-indexed retry number.

When using GNews with a searchapi_key:

• Method Compatibility: The SearchApi backend is used by get_news, get_news_by_topic, get_news_by_location, and get_news_by_site.
• RSS Fallback: The get_top_news() method always uses the RSS backend, as SearchApi requires a specific search query to function.
• Key Benefits: Compared to RSS, SearchApi provides absolute ISO dates, thumbnails, favicons, result ranking, pagination, and higher rate limits.

If you are contributing to the project, clone the repository and install the dependencies from the requirements file:

git clone https://github.com/ranahaani/GNews.git
cd GNews
pip install -r requirements.txt

If you are contributing to the project, you can set up a development environment using Docker or a manual Git clone.

Option 1: Setup with Docker

1. Install docker and docker-compose.
2. Configure the .env file with your MongoDB credentials.
3. Run:

docker-compose up --build

Option 2: Install Using Git Clone

1. Clone the repository:

git clone https://github.com/ranahaani/GNews.git

2. Set up a virtual environment:

# MacOS/Linux
virtualenv venv
source venv/bin/activate

# Windows
virtualenv venv
.\venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

Install the core GNews package using pip. This requires Python 3.10 or higher.

pip install gnews

You can replace the default RSS backend with SearchApi to access more reliable Google News data. To use it, obtain an API key from SearchApi and pass it to the GNews constructor using the searchapi_key parameter. When a key is provided, methods like get_news, get_news_by_topic, get_news_by_location, and get_news_by_site will automatically use the SearchApi backend instead of RSS.

from gnews import GNews

g = GNews(searchapi_key="YOUR_SEARCHAPI_KEY", max_results=10)
articles = g.get_news("artificial intelligence")

By default, the Google News RSS feed returns redirect URLs (news.google.com/rss/articles/...). To automatically resolve these into real article URLs, install the playwright extra and the Chromium browser.

Setup:

pip install gnews[playwright]
playwright install chromium

Once installed, URL resolution is automatic. If resolution fails (e.g., due to a paywall or timeout), GNews silently falls back to the Google URL without crashing. For production environments where you want to avoid Playwright, using the SearchApi backend is recommended as it returns real URLs natively.

from gnews import GNews

g = GNews(max_results=5)
articles = g.get_news("AI")

# With gnews[playwright] installed, this returns the real URL:
print(articles[0]['url'])  # https://www.politico.com/news/...