RSS Libraries¶
Legacy Wiki Page
This page was migrated from the old MoinMoin-based wiki. Information may be outdated or no longer applicable. For current documentation, see python.org.
Python RSS Code¶
Articles:
http://codeboje.de/blog/archives/Feeds-and-python-II.html - Short articles about using Universal Feed Parser
Libraries:
the Universal Feed Parser - Reads 9 RSS versions and Atom
RSS.py - reads most RSS versions, produces RSS 1.0
SpycyRoll - I donât know much about - this is an aggregator, and perhaps other stuff as well
PyRSS2Gen - produces RSS 2.0
Feed Parser¶
Feed Parser is an awesome RSS reader. It is now hosted on Google Code & Sourceforge - Universal Feed Parser on Google Code (Project Page on SourceForge).
Universal Feed Parser documentation.
Download it, and then start a Python prompt in the same directory.
1 import feedparser
2
3 python_wiki_rss_url = "http://www.python.org/cgi-bin/moinmoin/" \
4 "RecentChanges?action=rss_rc"
5
6 feed = feedparser.parse( python_wiki_rss_url )
You now have the RSS feed data for the PythonInfo wiki!
Take a look at it; Thereâs a lot of data there.
Of particular interest:
|
|
|
URL of the feedâs RSS feed |
|
version of the RSS feed |
|
|
|
|
|
Link to RecentChanges - Web page associated with the feed. |
|
|
|
A gigantic list of all of the RecentChanges items. |
For each item in feed["items"], we have:
|
|
|
|
|
title for item |
|
change summary |
|
URL to the page |
|
for wiki, a link to the diff for the page |
|
for wiki, a link to the page history |
Aggregating Feeds with Feed Parser¶
If youâre pulling down a lot of feeds, and aggregating them:
First, you may want to use Future threads to pull down your feeds. That way, you can send out 5 requests immediately, and wait for them all to come back at once, rather than sending out a request, waiting for it to come in, send out another request, wait for it to come back in, etc., etc.,.
1 from future import Future
2
3 hit_list = [ "http://...", "...", "..." ] # list of feeds to pull down
4
5 # pull down all feeds
6 future_calls = [Future(feedparser.parse,rss_url) for rss_url in hit_list]
7 # block until they are all in
8 feeds = [future_obj() for future_obj in future_calls]
Now that you have your feeds, extract all the entries.
1 entries = []
2 for feed in feeds:
3 entries.extend( feed[ "items" ] )
âŠand sort them, by SortingListsOfDictionaries:
1 sorted_entries = sorted(entries, key=lambda entry: entry["date_parsed"])
2 sorted_entries.reverse() # for most recent entries first
Congratulations! Youâve aggregated a bunch of changes!
Contributors¶
Links¶
RawDog is an RSS aggregator written in Python & using Feed Parser
Feedjack Planet like Feed aggregator using Universal Feed Parser and the django webframework
Discussion¶
Getting the âauthorâ/âcontributorâ out of most ModWiki RSS feeds with the feedparser module is a bit confusing as of now. Right now (feedparser 3.3), it goes into the ârdf_valueâ attribute of the entry.
Iâm moving the following out of the main text:
RawDog is a ready made aggregator if you donât want to write your own.
Are you concerned that Iâm encouraging people to reduplicate efforts, making aggregator after aggregator after aggregator?
Thatâs not the case; there are good reasons to write aggregators yet.
In particular: I wrote the code because I needed a MoinMoin macro that aggregated RSS feeds.
I imagine that there are other good reasons to write aggregating code.
That said, RawDog is Python, and it is using Feed Parser, so Iâve linked it at the bottom of the page.
â LionKimbro 2004-12-27 08:44:40
Next, I moved this out of the main text:
Usually, such performance is not necessary, unless you have thousands of feeds to retrieve every hour. If you have less than a few hundred feeds an hour to retrieve, one a time is probably better - why peak out your processor/bandwidth?
This makes sense if youâre just writing a client aggregator for reading blogs. But if youâre compiling parts of a web page, then you want to generate a response within 20 seconds, not 3 minutes.
Similarly, Iâm removing:
Other things to help you be polite:
try and retrieve things a few times a day or week. donât request hourly updates unless you need them.
avoid updates on the hour or half hour. Try a random time into the hour, like 27 or 33 or whatever, or poll at an interval like 693 minutes rather than 600, so that only rarely do you poll sites near the hour boundry. It is problematic for sites to get polled by hundreds of aggregators on the hour.
be sure to use HttpConditionalGetRequests and honour content-expires response flags
be sure to support HttpCompression in the responses.
Maybe thereâs some other page on some other wiki where this belongs. I donât think that space is here.
Iâm mainly concerned here with giving an example of how the RSS library works, the kinds of things you can do with it, how to combine itâs use with the Futures module.
This isnât really about writing aggregators.
â LionKimbro 2004-12-27 08:44:40
Can someone please give a sample of how FeedParser works along with Future Threads to make an RSS AGGREGATOR.
Manasa