Yaogang Lian

Muffin Tutorial: Build a Personal Full-Text RSS Reader (Part 3)

In the first two parts we have built a fully functional RSS reader that has a minimalistic interface, shows full texts of articles in a nice full screen view, and works beautifully on all devices.

In the last part of this tutorial, we are going to optimize the app’s performance, tie some loose ends, and add a couple more nifty features.

Optimize App Performance

Make a production build

Although we have built a decent, fully functional app, it does feel sluggish at times. The initial loading seems to take a long time, and changing feeds feels slow too. Why is that?

There are a few contributing factors. The primary one is that we are running a development build, which doesn’t concatenate or compress or cache any script. So every time the app loads, it has to load dozens of scripts, inevitably slowing down the loading process. We can see the effect of this using Chrome’s developer console.



It shows that during the initial load, the app has to make 33 network requests, including 13 scripts.

Let’s stop muffin watch and make a production build instead:

1
$ muffin minify

Reload the browser and the developer console now shows:



The total number of network requests has dropped to 13, and the number of scripts has dropped to 2! This is because for the production build, Muffin automatically compresses and concatenates JavaScript files.

The rest of the 13 network requests includes images and JSON APIs. Let’s optimize them next.

Use memcache to cache JSON responses

We only have two API endpoints on the server, /feeds and /feeds/#{feed-id}/articles. Since the JSON responses of these two APIs are the same for all users, we can cache them on the server side using memcache.

This is absurdly easy to do with Google App Engine. Open the file server/apps/api_v1.py and replace @as_json with @memcached (24 * 60 * 60). Now the JSON responses are cached.

Note that we are setting the expiration period to a full day. This seems to be too long, but don’t fret, we will manually flush the memcache whenever we update the feeds.

Cache thumbnails

Another huge factor for the app’s sluggishness is that it has to load many “thumbnails”, which are actually full-size images! A typical such image is as below:



This image is 640px wide and 426px tall, with a file size of 243 KB!

Clearly loading these full-size images as thumbnails takes a toll on the app’s performance. We need to resize these images and cache the thumbnails on our server, so that they can load much faster.

First, add a couple new fields to the Article model:

1
2
3
imageBlob = blobstore.BlobReferenceProperty()
origImageUrl = db.LinkProperty(indexed=False)
imageUrl = db.LinkProperty(indexed=False)

Note that we now use origImageUrl to store the full-size image URL, and use imageUrl for the thumbnail resized on our server.

We are going to use another task queue for caching thumbnails, so add the following to queue.yaml:

1
2
3
4
- name: upload-images
rate: 1/s
retry_parameters:
task_retry_limit: 5

Next, open aggregator.py and add another request handler:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
class UploadArticleImageHandler(webapp.RequestHandler):
def get(self):
articleId = long(self.request.get('id'))
article = Article.get_by_id(articleId)
link = self.request.get('link')
res = urlfetch.fetch(link, deadline=20)
if res.status_code == 200:
img = images.Image(res.content)
if img.width < 100:
article.origImageUrl = None
article.imageUrl = None
else:
img.resize(width=150)
img.im_feeling_lucky()
imageData = img.execute_transforms(output_encoding=images.JPEG)
# Create the file
file_name = files.blobstore.create(mime_type='image/jpeg')
# Open the file and write to it
with files.open(file_name, 'a') as f:
f.write(imageData)
# Finalize the file. Do this before attempting to read it.
files.finalize(file_name)
# Get the file's blob key
blob_key = files.blobstore.get_blob_key(file_name)
# Save the blob key and image serving url in the Article
article.imageBlob = blob_key
article.imageUrl = images.get_serving_url(blob_key)
article.put()
self.response.out.write('OK')

Here we use Google App Engine’s Image API to resize the image then save it as a JPEG file. Also remember to add the route (r'/aggregator/upload-article-image', UploadArticleImageHandler), and push tasks to the queue in GetFullArticleHandler:

1
2
3
4
5
6
7
8
9
10
S = PyQuery(html)
article.origImageUrl = S("img:first").attr('src')
article.imageUrl = None
article.body = html
article.put()
if article.origImageUrl is not None and article.imageUrl is None:
taskqueue.add(queue_name='upload-images', url='/aggregator/upload-article-image', params={'link': article.origImageUrl, 'id': articleId}, method='GET')
self.response.out.write('OK')

After all these changes, reset the datastore and reload the feeds. Reload the browser and you’ll see the app loads much faster now. The full-size image above is now properly resized and cached on our server:



The thumbnail is 150px wide and 99px tall, with a file size of 3KB.

Tie Loose Ends

There are a few loose ends we need to tie on both client and server side. But hey, we are almost done!

Set up cron jobs

As we said at the beginning of this tutorial, our strategy is to let the server update the RSS feeds periodically. So we need to set up a cron job for that.

Add the following to cron.yaml:

1
2
3
4
cron:
- description: update RSS feeds
url: /aggregator/update-feeds
schedule: every 30 minutes from 00:00 to 23:59

This sets up a cron job that updates the RSS feeds every 30 minutes.

Flush memcache

As we said earlier, we are going to flush the memcache whenever we update the RSS feeds. So let’s add another cron job to cron.yaml:

1
2
3
- description: flush memcache
url: /aggregator/flush-memcache
schedule: every 30 minutes from 00:05 to 23:59

Note that flush-memcache is called 5 mins after updating the RSS feeds, so that the server has ample time to parse the feeds and fetch full-texts of articles.

We also need to set up the request handler in aggregator.py:

1
2
3
4
class FlushMemcacheHandler(webapp.RequestHandler):
def get(self):
memcache.flush_all()
self.response.out.write('flush-memcache: done')

Purge old articles

As the server updates the RSS feeds every 30 minutes, the number of articles in the datastore keeps growing. Eventually this will cause the datastore to outgrow its quota. The solution is simple — purge old articles that we no longer need.

Add a daily cron job to cron.yaml:

1
2
3
- description: purge old articles
url: /aggregator/purge-old-articles
schedule: every day 04:00

Then add this request handler in agggregator.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class PurgeOldArticlesHandler(webapp.RequestHandler):
def get(self):
oneDayAgo = datetime.datetime.now() - datetime.timedelta(days=1)
toDelete = []
blobsToDelete = []
for a in Article.all().filter("updated <", oneDayAgo):
toDelete.append(a)
if a.imageBlob is not None:
blobsToDelete.append(a.imageBlob.key())
db.delete(toDelete)
blobstore.delete(blobsToDelete)
self.response.out.write('purge-old-articles: done')

Note that we also deleted the thumbnails of old articles from the server.

Loading spinner

Now that we have tied all the loose ends on the server side, let’s move on to the client side. First we are going to add a loading spinner to give the user a better visual cue during the initial load.

Add the spinner to the top of LayoutView.jade:

1
2
#spinner
img(src='images/loading.gif')

Add the following styles to main.less:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#spinner {
position: fixed;
left: 50%;
top: 40%;
width: 80px;
height: 80px;
margin-top: -40px;
margin-left: -40px;
.border-radius(8px);
background-color: rgba(0,0,0,0.8);
z-index: 9999;
img {
position: absolute;
top: 50%;
left: 50%;
margin-top: -12px;
margin-left: -12px;
width: 24px;
height: 24px;
}
}

And add a method hideSpinner to LayoutView.coffee:

1
2
hideSpinner: ->
$('#spinner').fadeOut()

And finally, call hideSpinner at the end of showArticles. This is how it looks:



Swipe gestures

It would be nice if we can use swipe gestures in the full-text reader on mobile devices. So, instead of tapping the “prev” and “next” buttons on the top bar, the user can simply swipe to the left or right to change articles. Let’s make that happen.

We are going to use Hammer.js for handling the gestures. Since Hammer.js isn’t a Muffin package, we need to add a shim to client/config.json, so that Muffin knows how to install it. Add the following to the end of “dependencies” in client/config.json:

1
2
3
4
5
"EightMedia/hammer.js": {
"version": "*",
"type": "script",
"main": "dist/hammer.js"
}

Then run muffin install and you’ll have Hammer.js installed in client/components.

It takes just a couple of lines to enable the swipe gestures. Add the following to ReaderView.coffee:

1
2
3
4
5
6
7
8
9
require 'EightMedia/hammer.js'
initialize: ->
# ...
# Gestures via hammer.js
hammer = Hammer(@el, {stop_browser_behavior: false, drag: false, tap: false})
hammer.on 'swipeleft', @showNextArticle
hammer.on 'swiperight', @showPreviousArticle

You may need to restart muffin watch since we changed the configuration file. Also, you might find it hard to test the swipe gestures since the web view itself also moves around. We will fix this issue shortly.

Keyboard shortcuts

Now that we have added swipe gestures on mobile devices, it’s only fair to add keyboard shortcuts on the desktop.

We will use backbone-shortcuts to handle keyboard shortcuts. Again, we need to add a shim since it’s not a Muffin package. This shim is a bit more involved because “backbone-shortcuts” depends on keymaster.js. Let’s add the following to the end of “dependencies” in client/config.json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
"madrobby/keymaster": {
"version": "*",
"type": "script",
"main": "keymaster.js"
},
"bry4n/backbone-shortcuts": {
"version": "*",
"type": "script",
"main": "backbone.shortcuts.js",
"dependencies": {
"jashkenas/backbone": "*",
"madrobby/keymaster": "*"
}
}

Run muffin install again to install these two scripts. Then restart muffin watch.

We are going to add three keyboard shortcuts to the full-text reader: left arrow to go to the previous article, right arrow to go to the next article, and the escape key to dismiss the reader. Thanks to “backbone-shortcuts”, we only need to add a few lines to ReaderView.coffee:

1
2
3
4
5
6
7
8
9
10
11
12
require 'bry4n/backbone-shortcuts'
shortcuts:
'left': 'showPrevArticle'
'right': 'showNextArticle'
'esc': 'dismiss'
initialize: ->
# ...
# Set up keyboard shortcuts
_.extend(@, new Backbone.Shortcuts)
@delegateShortcuts()

Give it try on the desktop and the keyboard shortcuts should be working.

App name and app icon

It’s time to change the application name and icon. We need to change the application name in two places. First, open index.jade and change title Webapp to title RSS Reader. Second, open client/assets/strings/en/str.coffee and change title: "Webapp" to title: "RSS Reader".

A great app can’t be without a great app icon. So let’s make one. Since this is a RSS reader, let’s play with the idea of reading news sent through the mail. Here is the final design.



On iOS devices you can save the webapp to the home screen as a standalone app. This gives the webapp a better native feeling.




The app icon looks great on the home screen.

iOS wrapper

The standalone webapp works quite well on iOS devices. However, as you play more with the app, you may find two remaining issues that can get annoying. First, the web view can move around, especially when you scroll all the way to the top or the bottom. Second, the app reloads itself every time you open it.

Unfortunately these two issues can’t be solved with CSS or JavaScript. But don’t give up just yet. There is an easy solution to both issues — wrap the webapp into a native iOS app.

The iOS wrapper is the simplest iOS app you can write — it simply presents a full-screen webview. However, having access to this webview gives us the opportunity to fix the first issue. This is the line that does the magic:

1
webView.scrollView.bounces = NO;

Also, as a native app, the iOS wrapper automatically supports multitasking so it doesn’t reload when you switch back from other apps.

The code for the iOS wrapper is in the repo, under the “iOS” directory. Remember to change the webview’s URL otherwise the app won’t load at all.

One More Thing…

Before we call it a day, we are going to implement another really cool feature — the offline mode. You will appreciate this if you use an iPod Touch and wander off the wifi range.

First, we need to enable the offline mode on the webapp itself. This is done with HTML5’s application cache. Muffin has already created a cache manifest file in the project boilerplate: client/assets/index.appcache. We just need to enable it. Open index.jade and replace html(lang="en") with:

1
html(lang="en", manifest!="<?= settings.env === 'production' ? 'index.appcache' : '' ?>")

This tells the app to use index.appcache only for the production build.

We also need to add all the application files to index.appcache:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
CACHE MANIFEST
# <?= (new Date()).toString() ?>
CACHE:
index.html
javascripts/apps/main/start.js
strings/en/str.js
stylesheets/main.css
components/twbs/bootstrap/dist/css/bootstrap.css
images/paper.jpg
images/loading.gif
images/refresh-btn.png
images/menu-btn.png
images/previous-btn.png
images/next-btn.png
NETWORK:
*

On the server side, we need to add mime-type support for the cache manifest file. Add the following to server/app.yaml:

1
2
3
4
5
handlers:
- url: /(.*\.appcache)
static_files: public/\1
mime_type: text/cache-manifest
upload: public/(.*\.appcache)

Now let’s make a production build:

1
$ muffin minify

And reload the browser. Chrome’s developer console shows that Application Cache has been created:



If you reload the browser again, the developer console reports that everything is now loaded from Application Cache:



Now that all the application files are cached on the client, we need to prompt the user to reload the app when we push out an update. Add the following to index.jade:

1
2
3
4
5
6
7
8
9
10
11
12
script.
// Reload the app when the manifest file changes
if ('applicationCache' in window) {
window.applicationCache.addEventListener('updateready', function (e) {
// Swap it in and reload the page.
window.applicationCache.swapCache();
if (confirm('The application has been updated. Would you like to reload now?')) {
window.localStorage.clear();
window.location.reload(true);
}
}, false);
}

Let’s try it on the device. First reload the browser to make sure the latest application files are loaded. Then turn on the airplane mode and reopen the app. This is what I see on my iPod Touch:



Clearly the app is loaded, but there is no data to display, because the feeds and articles are not cached on the client side.

To cache the feeds and articles on the client, we will use the fantastic Backbone.dualStorage library. It takes exactly one line to make this work. First let’s add a shim to client/config.json for “Backbone.dualStorage”:

1
2
3
4
5
6
7
8
"nilbus/Backbone.dualStorage": {
"version": "*",
"type": "script",
"main": "backbone.dualstorage.coffee",
"dependencies": {
"jashkenas/backbone": "*"
}
}

Now run muffin install again, and you’ll have “Backbone.dualStorage” installed.

Now open client/apps/main/start.coffee, and add this line right after importing Backbone:

1
require 'nilbus/Backbone.dualStorage'

Rebuild the app and test on the device again. This time, in airplane mode, the iPod Touch loads the app just fine.




“Backbone.dualStorage” did the all magic behind the scenes — all the feeds and articles are cached in local storage so they can be loaded in the offline mode.

The End

Whew! We have come a long way. In the three parts of this tutorial, we started with an app idea and worked all the way to its completion. Let’s take a look at how much (or how little) code we have written:



There are 237 lines of CoffeeScript in client/apps.



And there are 221 lines of Python in server/apps. In total, it took us less than 500 lines of code to build this beautiful full-text RSS reader.

Now if you have a developer account on Google App Engine, you can make a final production build with muffin minify, and upload the entire server directory to Google App Engine. It’s time to kick back and enjoy reading your RSS feeds!

Yaogang Lian

An iOS, Mac and web developer. Focusing on building productivity and educational apps.