Auto-News

The OperatorCollection class is a specialized operator designed to handle the end-to-end lifecycle of news collections. It manages pulling data from various sources (like YouTube, Twitter, or RSS), performing local storage, deduplication, summarization, ranking, and finally publishing the results.

It relies on several key stages:

1. Pulling: Fetching items from Notion databases.
2. Filtering: Pre-filtering by user rating and post-filtering by relevance scores.
3. Scoring: Using vector similarity (via Milvus) to rank content.
4. Pushing: Publishing the processed collection back to a target (e.g., Notion).

from src.ops_collection import OperatorCollection

# Example of the logical flow an end-user would implement
collection = OperatorCollection()
# 1. Pull
raw_pages = collection.pull(collection_type="weekly", sources=["Youtube", "RSS"])
# 2. Score
scored_pages = collection.score(raw_pages, top_k_similar=4)
# 3. Filter
final_pages = collection.post_filter(scored_pages, k=3, min_score=4.5)
# 4. Push
collection.push(final_pages, takeaway_pages=[], targets=["notion"])

OperatorRSS is a specialized operator designed to manage the full lifecycle of RSS feed aggregation. It handles pulling data from RSS feeds, deduplicating articles, scoring them based on relevance, summarizing content using LLMs, ranking them by category, and finally pushing them to targets like Notion.

Key capabilities include:

• Pulling: Fetches articles from RSS URLs stored in Notion databases.
• Deduplication: Prevents processing the same article multiple times using MD5 hashing of titles and publication dates.
• Scoring: Uses Milvus vector search to score articles against existing content.
• Summarization: Uses LLMs to generate summaries, with fallback to web loading if the RSS entry is empty.
• Ranking: Categorizes and rates articles using LLMs.
• Pushing: Exports processed articles to Notion 'ToRead' databases.

# Example of the logical flow an end-user would implement using OperatorRSS methods:
# 1. Pull articles
pages = operator_rss.pull()
# 2. Deduplicate
deduped_pages = operator_rss.dedup(pages)
# 3. Score
scored_pages = operator_rss.score(deduped_pages)
# 4. Filter
filtered_pages = operator_rss.filter(scored_pages, k=3, min_score=4)
# 5. Summarize
summarized_pages = operator_rss.summarize(filtered_pages)
# 6. Rank
ranked_pages = operator_rss.rank(summarized_pages)
# 7. Push to Notion
operator_rss.push(ranked_pages, targets=['notion'])

The OperatorBase class serves as the foundation for defining data processing pipelines in Auto-News. To create a new operator, inherit from OperatorBase and implement the core lifecycle methods. While the base class provides several utility methods for data handling, you are expected to provide implementations for the primary processing steps: pull, dedup, summarize, rank, score, and push.

class MyCustomOperator(OperatorBase):
    def pull(self):
        # Implement data retrieval logic
        return {}

    def dedup(self, data, target):
        # Implement deduplication logic
        return data

    def summarize(self, data):
        # Implement summarization logic
        return data

    def rank(self, data):
        # Implement ranking logic
        return data

    def score(self, data):
        # Implement scoring logic
        return data

    def push(self, ranked_data, targets, topk=3):
        # Implement pushing logic
        return