DOCSEARCHbyAlgolia
Submit Docs

gofeed

repository·master·Indexed 25 days ago

https://github.com/mmcdole/gofeed

A robust Golang library for parsing RSS, Atom, and JSON feeds. It provides a universal parser that converts various feed formats into a unified gofeed.Feed model, as well as specialized parsers for format-specific granularity. Features include support for Basic Authentication, custom translators, feed type detection, and built-in handling for Dublin Core and Apple iTunes extensions.

Tokens
4.3K
Snippets
7
Records
25
Agent score
83%

What's inside gofeed

  1. Implement a Custom Translator

    master

    To control how specific feed types are converted into the universal gofeed.Feed model, you can implement the gofeed.Translator interface. This is useful for prioritizing certain fields or handling custom logic during the translation process.

    1. Create a struct that embeds a default translator (e.g., *gofeed.DefaultRSSTranslator).
    2. Override the Translate(feed interface{}) (*gofeed.Feed, error) method.
    3. Assign your custom translator to the corresponding field on the gofeed.Parser (e.g., fp.RSSTranslator).
    type MyCustomTranslator struct {
  defaultTranslator *gofeed.DefaultRSSTranslator
}

func NewMyCustomTranslator() *MyCustomTranslator {
  t := &MyCustomTranslator{}
  t.defaultTranslator = &gofeed.DefaultRSSTranslator{}
  return t
}

func (ct *MyCustomTranslator) Translate(feed interface{}) (*gofeed.Feed, error) {
  rss, found := feed.(*rss.Feed)
  if !found {
    return nil, fmt.Errorf("Feed did not match expected type of *rss.Feed")
  }

  f, err := ct.defaultTranslator.Translate(rss)
  if err != nil {
    return nil, err
  }

  // Custom logic
  if rss.ITunesExt != nil && rss.ITunesExt.Author != "" {
    f.Author = rss.ITunesExt.Author
  } else {
    f.Author = rss.ManagingEditor
  }
  
  return f, nil
}

// Usage
fp := gofeed.NewParser()
fp.RSSTranslator = NewMyCustomTranslator()

  2. Use the Universal Feed Parser

    master

    The gofeed.Parser is a universal parser that converts RSS, Atom, and JSON feeds into a unified gofeed.Feed model. This is the recommended approach when you want to handle multiple feed formats using a single interface.

    Common methods include:

    • ParseURL(url string): Parses a feed from a URL.
    • ParseString(data string): Parses a feed from a string.
    • Parse(reader io.Reader): Parses a feed from an io.Reader.
    • ParseURLWithContext(url string, ctx context.Context): Parses a feed from a URL using a provided context (useful for timeouts).
    fp := gofeed.NewParser()
feed, _ := fp.ParseURL("http://feeds.twit.tv/twit.xml")
fmt.Println(feed.Title)

  3. Use Specialized Feed Parsers (RSS, Atom, JSON)

    master

    If you require high performance or need to access fields specific to a single format that are not part of the universal model, use the specialized parsers directly. These parsers map fields to models that match the feed type exactly.

    • rss.Parser: For RSS feeds.
    • atom.Parser: For Atom feeds.
    • json.Parser: For JSON feeds.
    // RSS
fp := rss.Parser{}
rssFeed, _ := fp.Parse(strings.NewReader(feedData))

// Atom
fp := atom.Parser{}
atomFeed, _ := fp.Parse(strings.NewReader(feedData))

// JSON
fp := json.Parser{}
jsonFeed, _ := fp.Parse(strings.NewReader(feedData))

  4. Configure the Universal Parser

    master

    You can customize the behavior of gofeed.Parser by setting specific fields before calling a parse method:

    • User-Agent: Set fp.UserAgent to a custom string to identify your client.
    • Authentication: Set fp.AuthConfig to a *gofeed.Auth struct containing Username and Password for feeds requiring Basic Authentication.
    // Custom User-Agent
fp := gofeed.NewParser()
fp.UserAgent = "MyCustomAgent 1.0"
feed, _ := fp.ParseURL("http://feeds.twit.tv/twit.xml")

// Basic Authentication
fp.AuthConfig = &gofeed.Auth{
  Username: "foo",
  Password: "bar",
}

  5. Configure Parser limits and options

    master

    The Parser struct provides several configuration fields:

    • MaxByteSize: Limits the number of bytes read from a response body. If exceeded, ErrResponseTooLarge is returned. Set to 0 for no limit.
    • KeepOriginalFeed: If set to true, the original RSS/Atom/JSON source is retained in the resulting Feed and can be accessed via Feed.OriginalFeed().
    • UserAgent: Customizes the User-Agent header sent with HTTP requests.
    • Client: Allows providing a custom *http.Client.

  6. Access Feed Extensions

    master

    Elements outside the feed's default namespace are treated as extensions. These are stored in tree-like structures under Feed.Extensions and Item.Extensions.

    gofeed provides built-in support for common extensions, mapping them to dedicated structs:

    • Dublin Core: Accessible via Feed.DublinCoreExt and Item.DublinCoreExt.
    • Apple iTunes: Accessible via Feed.ITunesExt and Item.ITunesExt.

  7. Parse a feed from an io.Reader

    master
    The Parse method accepts an io.Reader containing the feed content (XML or JSON). It peeks at the beginning of the stream to detect the feed type and then parses it incrementally (for RSS/Atom) or fully (for JSON).

  10. Sort Feed items by publication date

    master
    The Feed type implements sort.Interface, allowing you to sort its Items slice using the standard library's sort.Sort() function. Sorting will order items from oldest to newest based on their PublishedParsed timestamp.

  11. Translate RSS feeds using DefaultRSSTranslator

    master

    The DefaultRSSTranslator converts an *rss.Feed into the universal Feed type. It includes mapping rules for RSS-specific fields like Dublin Core and iTunes extensions.

    By default, the translator performs an HTML scan of the feed's description or item content to find an image if no explicit image is provided. You can disable this behavior to improve performance on large feeds by setting DisableContentImageScan to true.

  12. Initialize a new Parser with NewParser

    master
    Use NewParser() to create a universal feed parser. This parser automatically detects whether a feed is RSS, Atom, or JSON and translates it into a unified Feed type. By default, it uses a standard User-Agent of Gofeed/1.0.