I started sending out ReadFeeder.net beta invitations this week! I won’t lie and say it’s been smooth sailing, but the bugs that have been caught were easily fixed, and I’ve personally fully switch over to it. Since I documented the entire journey of creating ReadFeeder, it seems fitting to document this next stage in it’s life. So on that note, this is the first of a continuation of a series involving the life and times of ReadFeeder.net
So it turns out there were still some lingering bugs from the Big Refactor, which caused things like the OPML import to not work properly, and displaying the feed’s original name instead of the name the user gave it. Fixing those caused a couple of other bugs to appear, specifically regarding the Queue Bookmarks. But those have all been fixed now, and I’m pleased to say that things are currently quiet on that front.
During the Big Refactor I put in place some extra fields to be used during the regularly scheduled feed updates that would help make the update process smarter. Amazon’s t1.micro instances don’t give you dedicated CPU time, but they do let you “burst” up to as many as two processing “units” for short periods of time. In order to stop any one person’s apps from hogging the CPU all the time, the t1.micro instance will penalize you if you use it too much or too often. Previously this meant that I had to slow down and space out my updates, which meant that it would take up to several hours to finish updating all the feeds, otherwise Amazon would stick me in the penalty box.
I initially started ReadFeeder.net on the same schedule, updating all the feeds approximately every two hours, which works up until it reaches about 400 feeds. Even after adding only a few people to the beta program, it quickly surpassed 500 feeds. Some of the Big Refactor helped here, since two or more people subscribed to the same feed wouldn’t cause it to update the feed multiple times, but because I selected beta users of different backgrounds and reading habits, the overlap wasn’t that high.
But, I was now collecting both the last time a feed was checked, and the last time that feed had a new article in it. Using these two fields, the update process will now calculate a “staleness” factor for a feed, which is just the time between it’s last article and the last time ReadFeeder checked it (how long since it was “fresh”). It also knows the “age” of a feed, basically the time since it’s last check. The current, and very simple, algorithm has ReadFeeder only checking for updates to a feed if it’s “age” is more than half of it’s “staleness”. That means that if a feed hasn’t been updated in an hour, it’ll be checked again in 30 minutes. However, if it wasn’t updated in the last day, it won’t be checked again for another 12 hours.
The effect was nothing short of astonishing. Rather than checking 500 feeds every 2 hours, ReadFeeder now checks between 10 and 20 feeds every 10 minutes, and the CPU load has been barely high enough to register on Amazon’s monitoring charts, and is far from hitting the penalty limits. Yet, at the same time, feeds that produce a large quantity of content are being updated more often than before. And because ReadFeeder always places recently imported articles at the top of your list, even if a stale site isn’t checked as often, those articles will still show up at the top even if they were published before others from a feed that gets checked more often. It’s a small tradeoff in latency on articles from slow feeds, but a huge win for latency on articles from heavy feeds, and makes an even bigger impact on the amount of users that can be supported on the same resources.
I plan on continuing to monitor and tweak the updating algorithm to squeeze as much performance out of ReadFeeder as I can, while improving the user experience, but this has already been such a massive improvement that I will be able to start adding beta testers at a much faster pace than I had expected. So whether you are a heavy news reader or just looking for a different way of managing a smaller number of feeds, please take a minute and signup for the Beta program.