Yaogang Lian

Muffin Tutorial: Build a Personal Full-Text RSS Reader (Part 1)

Disclaimer: This tutorial is for demonstration purpose only. Please don’t scrape website content without site owners’ permission.

In this tutorial we will use Muffin to build something really cool — a personal full-text RSS reader that works beautifully on all devices. All the project code can be found in this GitHub repository. The git commit history largely follows this tutorial.

For many years I have relied on RSS feeds to keep me informed of all the technology news, but things are getting harder these days. With the recent shutdown of Google Reader, I finally decided to build a RSS reader for my personal use.

What I want from a RSS reader is actually quite simple:

  1. It doesn’t crash.
  2. The interface should be minimalistic.
  3. It should work well on all devices, including desktop and mobile.
  4. It should show the full text of an article, not just its summary.
  5. It should only show the content, not the full web page.

I have tried numerous RSS readers out there but none of them meets all the five criteria. So let’s use Muffin to build one!

Design

Since a key requirement is for the RSS reader to work on all devices, it would be quickest to build it as a webapp. We might wrap the webapp into a native app for extra benefits — but that’s further down the road.

How are we going to get full texts from RSS feeds? Most RSS feeds only come with article summaries. To read the full text of an article, you have to click a link to open an external webpage, and if you are like me, you may then use Safari Reader or iReader or Readability to get rid of the clutter on the page. That’s a lot of steps before you even start reading.

How can we make this experience significantly better? We can set up a server that checks the RSS feeds periodically, fetches full texts of new articles, removes all the clutter, and saves them to a database. Then the webapp client can retrieve clutter-free, full-text articles directly from our server via JSON APIs.

This means we need to set up a server stack. Fortunately, Muffin makes this very easy to do. Muffin has two preferred server stacks: Google App Engine and Node.js/MongoDB. In this tutorial we will pick Google App Engine, but you can easily port the code to the Node.js/MongoDB stack.

As for the user interface, we want to keep it minimalistic. A simple design is to have a list of feeds on the left, and a list of articles in the selected feed shown on the right, as seen in the image below.



When the user clicks on an article, we would show its full text in the Safari Reader style.



On mobile devices, we can simply hide the list of feeds, which can still be revealed by tapping a button on the top bar. And the full-text view would cover the whole screen.





That’s all for the design. In the true agile style, let’s quickly create a prototype.

Prototype

Create the project

Muffin is designed with developers’ productivity in mind. It’s very easy to get a project up and running with Muffin.

First make sure you have Muffin installed. If not, check Muffin’s website for instructions. Since we are going to use Google App Engine as the server stack, you also need to install the Google App Engine SDK for Python.

Once you have all the prerequisites, creating a project with Muffin takes just one line:

1
$ muffin new RSSReader -s gae

We named our project RSSReader and specified the server stack (gae is a shorthand for Google App Engine).

Muffin will prompt you that the application RSSReader has been created, and you’ll need to change into the project directory, then install dependencies. Let’s do that.

1
2
$ cd RSSReader
$ muffin install

This invokes Muffin’s built-in package manager to install all the frontend dependencies. The default dependencies include jQuery, Underscore.js, Backbone.js, and a few essential Muffin packages such as muffin/Logger, muffin/I18n and muffin/forms.

It’s time to put everything under version control. We will use git.

1
2
3
$ git init
$ git add .
$ git commit -m"Initial commit."

Muffin also sets up proper .gitignore files so you don’t have to worry about committing extraneous files.

Let’s take a brief look at the project structure Muffin created.



Muffin sets up the project with a “multi-app architecture”, a feature borrowed from Django. For example, on the client side, two apps are created by default — apps/base takes care of basic needs such as logging, and apps/main is your main app. Segregating a site into several self-contained smaller apps makes the whole project more manageable, and more importantly, these apps can be reused.

Now we are ready to see the app in action. You can let Muffin start the server for you but I like to use Google App Engine’s launcher app, so that server logs are not mixed with Muffin logs. Open your GoogleAppEngineLauncher app, which is installed with the Google App Engine Python SDK. Choose “Add Existing Application” from the menu and navigate to the server directory under the project root. Press the green ‘Run’ button to start the server.

You also need to compile the client files. Because we are going to work on the client code for a while, we will run Muffin in the watch mode:

1
$ muffin watch

Muffin first compiles all the client files, then watches the client directory for future file changes. When you change a file in the client directory, Muffin automatically recompiles the file and even reloads the browser for you — the so-called “live reload” feature.

Now click the ‘Browse’ button in the GoogleAppEngineLauncher app and you will see the app running in the browser.



That’s a good start! With just a few commands, you have a brand new webapp up and running.

Application layout

Next, we are going to create the application layout. I always like to see things on screen as quickly as possible, so that I can have a visual feeling of the app.

The project boilerplate comes with Twitter Bootstrap 3.0, which has great support for creating responsive layouts. Let’s leverage that. Replace LayoutView.jade with the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#wrapper.row
aside.col-sm-4.col-xs-8.hidden-xs
ul.nav.nav-pills.nav-stacked
li.active
a(href='#') Ars Technica
li
a(href='#') TNW
li
a(href='#') Make
#content-area.col-sm-8
.topbar
.title Ars Technica
ul.article-list-view
.article-cell
.article-cell
.article-cell
.article-cell
.article-cell
.article-cell
.article-cell
.article-cell

This sets up 4 columns on the left and 8 columns on the right. Note the col-sm- prefix, which is Bootstrap’s way of specifying which grid we are working on. For the xs grid (think of iPhone), we simply hide the left sidebar and let the right side span the full screen width.

Now we are going to add some simple styles. Replace main.less with the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
body {
font-family: Helvetica, Arial, sans-serif;
background-image: url('/images/paper.jpg');
}
#wrapper {
margin: 0;
}
aside {
padding: 4% !important;
overflow-y: scroll;
-webkit-overflow-scrolling: touch;
}
#content-area {
padding: 0;
margin: 0;
}
.topbar {
position: relative;
width: 95%;
height: 44px;
line-height: 44px;
padding: 0 1em;
border-bottom: 1px solid #ccc;
.title {
font-size: 22px;
color: #777777;
}
}
.article-list-view {
margin: 0;
padding: 0 2em 0 0;
overflow-y: scroll;
-webkit-overflow-scrolling: touch;
}
.article-cell {
padding: 1em 0;
margin: 1em;
min-height: 100px;
background: #ddd;
}

Note here we have set up the left sidebar and the content area as scrolling divs, using overflow-y: scroll;. Setting -webkit-overflow-scrolling: touch; enables the native-style momentum scrolling on iOS. These two settings combined give the webapp a much better native feeling. I have also set up the content area to extend all the way to the right, so that its scrollbar shows up at the edge of the browser window.

For the scrolling divs to work we must set correct heights on those divs. This can be done with CSS, but it’s cleaner to do it with JavaScript. Open LayoutView.coffee, remove all the getStarted and setView code since they are no longer needed, and then add the following:

1
2
3
4
5
6
7
8
9
10
initialize: ->
@$el.html @template()
# Resize sidebar and content area to fit the screen
$(window).on 'resize orientationchange', @resize
@resize()
resize: =>
@$('aside').css {height: $(window).height()}
@$('.article-list-view').css {height: $(window).height()-44}

Note that the resize method is called on window resize as well as orientation changes (e.g., when you rotate an iPhone).

Now the app looks like this:



Try resizing your browser window and you’ll see the layout automatically adjusting to the window size. When the screen width is less than 768px, the left sidebar disappears. This is how it looks on iPod right now:



The app looks quite crude for now, but we have set up a solid application layout to build upon!

Data modeling and APIs

Now that the layout is in place, it would be really nice to show some real data in the app UI. But before we can do that, we need to figure out the data models.

For the RSS reader, data modeling is as simple as it can get. We only need two data models: Feed and Article. A feed has a name, a link, and a last-updated timestamp. An article has a title, a link, a last-updated timestamp, a summary, a full-text body, and an optional thumbnail. There is a “one-to-many” relationship between a feed and an article. Don’t worry if we missed a thing or two — Google App Engine’s datastore is schemaless, so we can always come back and change the data models.

Now add the following code to server/apps/models.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
class Feed(BaseModel):
name = db.StringProperty(required=True, indexed=True)
link = db.LinkProperty(required=True, indexed=False)
updated = db.DateTimeProperty(required=True, indexed=True)
def toJSON(self):
return {
'id': str(self.key().id_or_name()),
'name': self.name
}
class Article(BaseModel):
title = db.StringProperty(required=True, indexed=True)
link = db.LinkProperty(required=True, indexed=True)
updated = db.DateTimeProperty(required=True, indexed=True)
summary = db.TextProperty(indexed=False)
body = db.TextProperty()
imageUrl = db.LinkProperty(indexed=False)
# Relationships
feed = db.ReferenceProperty(Feed, collection_name="articles")
def toJSON(self):
return {
'id': str(self.key().id_or_name()),
'title': self.title,
'link': self.link,
'updated': self.updated.strftime("%A, %e %B %Y %r %Z"),
'summary': self.summary,
'body': self.body,
'imageUrl': self.imageUrl,
}

It takes just a few more lines to set up JSON APIs for these two models. In server/apps/api_v1.py, add the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# GET /api/v1/feeds
class FeedsHandler(webapp.RequestHandler):
@as_json
def get(self):
feeds = Feed.all()
return [f.toJSON() for f in feeds]
# Get /api/v1/feeds/#{feed-id}/articles
class ArticlesHandler(webapp.RequestHandler):
@as_json
def get(self, feedId):
feedId = long(feedId)
feed = Feed.get_by_id(feedId)
return [a.toJSON() for a in feed.articles.order('-updated').run(limit=20)]
routes = [
(r'^%s/feeds$' % baseUrl, FeedsHandler),
(r'^%s/feeds/(\d+)/articles$' % baseUrl, ArticlesHandler),
(r'^%s/$' % baseUrl, MainHandler)
]

Now navigate to http://localhost:#{port}/api/v1/feeds. You should be greeted with an empty list []. It’s a very humble JSON response, but it shows that our JSON APIs are working!

Load feeds, fetch feeds

We’ve made great progress so far. Now we only wish that we had some real data in the datastore so that our JSON responses weren’t empty. To do that, we need to load some RSS feeds. Since this app is for personal use, we can simply load the RSS feeds from a yaml file.

Set up a file server/data/feeds.yaml as below:

1
2
3
4
5
6
7
8
- name: "Ars Technica"
link: "http://feeds.arstechnica.com/arstechnica/index"
- name: "TNW"
link: "http://feeds2.feedburner.com/thenextweb"
- name: "Make"
link: "http://makezine.com/blog/feed/"

Now we create a file server/apps/aggregator.py to manage loading and updating RSS feeds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import fix_path
import webapp2 as webapp
from apps import DEBUG
from apps.models import *
import logging, yaml, re, datetime
class LoadFeedsHandler(webapp.RequestHandler):
def get(self):
""" Load feeds from data/feeds.yaml """
f = open('data/feeds.yaml', 'r')
array = yaml.load(f)
feeds = []
for item in array:
name = item['name']
link = item['link']
feed = Feed.all().filter('name =', name).get()
if feed is None:
feed = Feed(name=name, link=link, updated=datetime.datetime.now())
else:
feed.name = name
feed.link = link
feed.updated = datetime.datetime.now()
feeds.append(feed)
db.put(feeds)
self.response.out.write('load-feeds: done')
#
# Application
#
app = webapp.WSGIApplication([
(r'/aggregator/load-feeds', LoadFeedsHandler)
], debug=DEBUG)

Also set up the new route inside server/apps/include.yaml:

1
2
3
4
5
handlers:
- url: /aggregator/.*
script: apps.aggregator.app
- url: /api/v1/.*
script: apps.api_v1.app

Now navigate to http://localhost:#{port}/aggregator/load-feeds and you should see “load-feeds: done”. Awesome! The feeds are loaded. Now open the SDK console in the Google App Engine launcher app, click on “Datastore Viewer” on the left panel, and you should see three feeds in the datastore.

It’s time to try our API endpoint again. Navigate to http://localhost:#{port}/api/v1/feeds, and voila! This is what we get:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[
{
"id": "5066549580791808",
"name": "Ars Technica"
},
{
"id": "5629499534213120",
"name": "Make"
},
{
"id": "6192449487634432",
"name": "TNW"
}
]

This is great. But we still need to fetch and parse the feeds to get a list of articles. This is actually quite easy, thanks to the wonderful feedparser. Copy feedparser’s source files into server/vendor and add the following code to server/apps/aggregator.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
from google.appengine.api import urlfetch
import datetime
from feedparser import feedparser
# ...
class UpdateFeedsHandler(webapp.RequestHandler):
def get(self):
for feed in Feed.all():
res = urlfetch.fetch(feed.link, deadline=10)
data = feedparser.parse(res.content)
feedUpdated = data['feed']['updated_parsed']
feedUpdated = datetime.datetime(*feedUpdated[:6]) # convert `time.struct_time` into a `datetime.datetime` object
if feedUpdated != feed.updated:
toSave = []
for entry in data.entries:
articleUpdated = entry.updated_parsed
articleUpdated = datetime.datetime(*articleUpdated[:6]) # convert `time.struct_time` into a `datetime.datetime` object
summary = re.sub(r'<.*?>', '', entry.summary)
summary = re.sub(r'&nbsp;', '', summary)
a = Article.all().filter("link =", entry.link).get()
if a is None:
a = Article(title=entry.title, link=entry.link, feed=feed, updated=articleUpdated, summary=summary)
toSave.append(a)
elif a.updated != articleUpdated:
a.title = entry.title
a.updated = articleUpdated
a.summary = summary
toSave.append(a)
feed.updated = feedUpdated
toSave.append(feed)
db.put(toSave)
self.response.out.write('update-feeds: done')
app = webapp.WSGIApplication([
(r'/aggregator/load-feeds', LoadFeedsHandler),
(r'/aggregator/update-feeds', UpdateFeedsHandler)
], debug=DEBUG)

The code simply iterates over all the feeds, fetch and parse each of them, then saves articles into the datastore. Run the code by navigating to http://localhost:#{port}/aggregator/update-feeds. It may take a few seconds but when it’s done, you should have dozens of articles in the datastore. Let’s try out the other API endpoint, http://localhost:#{port}/api/v1/feeds/5066549580791808/articles, which returns a list of articles in a feed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[
{
"body": null,
"updated": "Tuesday, 17 September 2013 04:19:50 PM ",
"title": "Failed Pentagon fax machine blocks FOIA requests",
"imageUrl": null,
"summary": "Please hold all requests until at least October, when OSD might buy a new one.",
"link": "http:\/\/feeds.arstechnica.com\/~r\/arstechnica\/index\/~3\/5Jhc1spgn2Q\/story01.htm",
"id": "5312840185413632"
},
{
"body": null,
"updated": "Tuesday, 17 September 2013 04:00:37 PM ",
"title": "Startup wants your iPad to be a 3D scanner, surveying device, and more",
"imageUrl": null,
"summary": "Occipital's Structure Sensor launches on Kickstarter, snaps cleanly on an iPad.",
"link": "http:\/\/feeds.arstechnica.com\/~r\/arstechnica\/index\/~3\/4cAaLWDjkd4\/story01.htm",
"id": "5453577673768960"
},
...
]

Hooray! We have got real data and real APIs set up.

Wire it up

We are finally ready to show real data in the app UI. Now let’s switch gears and move back to the client side. First generate the data models:

1
2
3
4
5
6
7
$ muffin generate model feed
14:27:06 [INFO]: * Created client/apps/main/models/Feed.coffee
14:27:06 [INFO]: * Created client/apps/main/models/FeedList.coffee
$ muffin generate model article
14:27:21 [INFO]: * Created client/apps/main/models/Article.coffee
14:27:21 [INFO]: * Created client/apps/main/models/ArticleList.coffee

Then, in client/apps/main/models/ArticleList.coffee, change the url method to:

1
2
url: ->
"<?= settings.baseURL ?>/feeds/#{@feed.get('id')}/articles"

This sets up the API endpoint for getting a list of articles in a feed.

We will create the left sidebar from real feeds, so remove the li‘s under ul.nav from LayoutView.jade, and add the following to the initialize method in LayoutView.coffee:

1
2
3
4
5
6
initialize: ->
# ...
# Data binding
@feeds = new FeedList()
@feeds.on 'reset', @render
@feeds.fetch {reset: true}

And set up the render method:

1
2
3
4
5
6
7
render: =>
$ul = @$('aside ul').empty()
@feeds.each (feed) =>
$li = $("<li class='feed'><a href='#'>#{feed.toJSON().name}</a></li>")
$ul.append $li
@$('aside ul li:first').click()
@

To show the list of articles when selecting a feed, we need to implement the showArticles method and add it to the events array:

1
2
3
4
5
6
7
8
9
10
11
12
13
events:
'click li.feed': 'showArticles'
showArticles: (e) =>
$current = $(e.currentTarget)
$current.siblings().removeClass('active')
$current.addClass('active')
index = $current.index()
feed = @feeds.at(index)
articles = new ArticleList()
articles.feed = feed
p = new ArticleListView {collection: articles, feed, el: @$('.article-list-view')}

To keep code modular, we will create a separate view for the article list:

1
2
3
$ muffin generate view ArticleListView
14:57:24 [INFO]: * Created client/apps/main/views/ArticleListView.coffee
14:57:24 [INFO]: * Created client/apps/main/templates/ArticleListView.jade

Change ArticleListView.coffee to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Backbone = require 'Backbone'
ArticleList = require '../models/ArticleList'
class ArticleListView extends Backbone.View
articleCellTemplate: _.tpl(require '../templates/_article_cell.html')
events: {}
initialize: ->
# Set up data structures backing the view
@collection.on 'reset', @render
@collection.fetch {reset: true}
render: =>
$list = @$el.empty()
@collection.each (article) =>
$list.append @articleCellTemplate({article: article.toJSON()})
@
module.exports = ArticleListView

You can remove ArticleListView.jade since we are not using it. We also need to set up a template for the article cell, so create a file _article_cell.jade in client/apps/main/templates/:

1
2
3
4
5
6
7
8
9
10
11
li.article-cell
<% if (article.imageUrl != null) { %>
a(href='#')
img.article-image(src!="<%- article.imageUrl %>")
<% } %>
.content
.title
a(href='#') <%- article.title %>
.summary <%= article.summary %>
.dateline <%- article.updated %>
a.hidden-xs(href!='<%- article.link %>', target='_blank') Link

And add its styles in main.less:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
.article-list-view {
margin: 0;
padding: 0 2em 0 0;
list-style: none;
font-family: Georgia, Serif;
text-align: left;
overflow-y: scroll;
-webkit-overflow-scrolling: touch;
}
.article-cell {
position: relative;
padding: 1em 0;
min-height: 100px;
@media (max-width: 480px) {
border-bottom: 1px solid #ccc;
}
img.article-image {
position: absolute;
max-width: 20%;
max-height: 105px;
}
.content {
padding-left: 22%;
.title {
font-size: 18px;
font-weight: bold;
a {
color: black;
}
}
.summary {
margin-top: 4px;
}
.dateline {
font-size: 13px;
color: #999;
a {
color: #999;
text-decoration: underline;
}
}
}
}

The app now looks like this on desktop:



And it looks like this on my iPod Touch:



Our hard work has finally paid off. We have a functional prototype that works on all devices, and shows real data from real APIs!

In the next part of this tutorial, we will scrape full texts of articles and implement a nice Safari Reader style UI.

Yaogang Lian

An iOS, Mac and web developer. Focusing on building productivity and educational apps.