Ubuntu API Website Teardown

Ubuntu API Website

For much of the past year I’ve been working on the Ubuntu API Website, a Django project for hosting all of the API documentation for the Ubuntu SDK, covering a variety of languages, toolkits and libraries.  It’s been a lot of work for just one person, to make it really awesome I’m going to need help from you guys and gals in the community.

To help smooth the onramp to getting started, here is a breakdown of the different components in the site and how they all fit together.  You should grab a copy of the branch from Launchpad so you can follow along by running: bzr branch lp:ubuntu-api-website

Django

First off, let’s talk about the framework.  The API website uses Django, a very popular Python webapp framework that’s also used by other community-run Ubuntu websites, such as Summit and the LoCo Team Portal, which makes it a good fit. A Django project consists of one or more Django “apps”, which I will cover below.  Each app consists of “models”, which use the Django ORM (Object-Relational Mapping) to handle all of the database interactions for us, so we can stick to just Python and not worry about SQL.  Apps also have “views”, which are classes or functions that are called when a URL is requested.  Finally, Django provides a default templating engine that views can use to produce HTML.

If you’re not familiar with Django already, you should take the online Tutorial.  It only takes about an hour to go through it all, and by the end you’ll have learned all of the fundamental things about building a Django site.

Branch Root

When you first get the branch you’ll see one folder and a handful of files.  The folder, developer_network, is the Django project root, inside there is all of the source code for the website.  Most of your time is going to be spent in there.

Also in the branch root you’ll find some files that are used for managing the project itself. Most important of these is the README file, which gives step by step instructions for getting it running on your machine. You will want to follow these instructions before you start changing code. Among the instructions is using the requirements.txt file, also in the branch root, to setup a virtualenv environment.  Virtualenv lets you create a Python runtime specifically for this project, without it conflicting with your system-wide Python installation.

The other files you can ignore for now, they’re used for packaging and deploying the site, you won’t need them during development.

./developer_network/

As I mentioned above, this folder is the Django project root.  It has sub-folders for each of the Django apps used by this project. I will go into more detail on each of these apps below.

This folder also contains three important files for Django: manage.py, urls.py and settings.py

manage.py is used for a number of commands you can give to Django.  In the README you’ll have seen it used to call syncdbmigrate and initdb.  These create the database tables, apply any table schema changes, and load them with initial data. These commands only need to be run once.  It also has you run collectstatic and runserver. The first collects static files (images, css, javascript, etc) from all of the apps and puts them all into a single ./static/ folder in the project root, you’ll need to run that whenever you change one of those files in an app.  The second, runserver, runs a local HTTP server for your app, this is very handy during development when you don’t want to be bothered with a full Apache server. You can run this anytime you want to see your site “live”.

settings.py contains all of the Django configuration for the project.  There’s too much to go into detail on here, and you’ll rarely need to touch it anyway.

urls.py is the file that maps URLs to an application’s views, it’s basically a list of regular-expressions that try to match the requested URL, and a python function or class to call for that match. If you took the Django project tutorial I recommended above, you should have a pretty good understanding of what it does. If you ever add a new view, you’ll need to add a corresponding line to this file in order for Django to know about it. If you want to know what view handles a given URL, you can just look it up here.

./developer_network/ubuntu_website/

If you followed the README in the branch root, the first thing it has you do is grab another bzr branch and put it in ./developer_network/ubuntu_website.  This is a Django app that does nothing more than provide a base template for all of your project’s pages. It’s generic enough to be used by other Django-powered websites, so it’s kept in a separate branch that each one can pull from.  It’s rare that you’ll need to make changes in here, but if you do just remember that you need to push you changes branch to the ubuntu-community-webthemes project on Launchpad.

./developer_network/rest_framework/

This is a 3rd party Django app that provides the RESTful JSON API for the site. You should not make changes to this app, since that would put us out of sync with the upstream code, and would make it difficult to pull in updates from them in the future.  All of the code specific to the Ubuntu API Website’s services are in the developer_network/service/ app.

./developer_network/search/

This app isn’t being used yet, but it is intended for giving better search functionality to the site. There are some models here already, but nothing that is being used.  So if searching is your thing, this is the app you’ll want to work in.

./developer_network/related/

This is another app that isn’t being used yet, but is intended to allow users to link additional content to the API documentation. This is one of the major goals of the site, and a relatively easy area to get started contributing. There are already models defined for code snippets, Images and links. Snippets and Links should be relatively straightforward to implement. Images will be a little harder, because the site runs on multiple instances in the cloud, and each instance will need access to the image, so we can’t just use the Django default of saving them to local files. This is the best place for you to make an impact on the site.

./developer_network/common/

The common app provides views for logging in and out of the app, as well as views for handling 404 and 500 errors when the arise.  It also provides some base models the site’s page hierarchy. This starts with a Topic at the top, which would be qml or html5 in our site, followed by a Version which lets us host different sets of docs for the different supported releases of Ubuntu. Finally each set of docs is placed within a Section, such as Graphical Interface or Platform Service to help the user browse them based on use.

./developer_network/apidocs/

This app provides models that correspond directly to pieces of documentation that are being imported.  Documentation can be imported either as an Element that represents a specific part of the API, such as a class or function, or as a Page that represents long-form text on how to use the Elements themselves.  Each one of these may also have a given Namespace attached to it, if the imported language supports it, to further categorize them.

./developer_network/web/

Finally we get into the app that is actually generates the pages.  This app has no models, but uses the ones defined in the common and apidocs apps.  This app defines all of the views and templates used by the website’s pages, so no matter what you are working on there’s a good chance you’ll need to make changes in here too. The templates defined here use the ones in ubuntu_website as a base, and then add site and page specific markup for each.

Getting Started

If you’re still reading this far down, congratulations! You have all the information you need to dive in and start turning a boring but functional website into a dynamic, collaborative information hub for Ubuntu app developers. But you don’t need to go it alone, I’m on IRC all the time, so come find me (mhall119) in #ubuntu-website or #ubuntu-app-devel on Freenode and let me know where you want to start. If you don’t do IRC, leave a comment below and I’ll respond to it. And of course you can find the project, file bugs (or pick bugs to fix) and get the code all from the Launchpad project.

Posted in OpenSource, Programming, Projects, Work | Tagged , , , , , , , , , | 1 Comment

10 minute Doctor Who app

It may surprise some of you (not really) to learn that in addition to being a software geek, I’m also a sci-fi nerd. One of my current guilty pleasures is the British Sci-Fi hit Doctor Who. I’m not alone in this, I know many of you reading this are fans of the show too.  Many of my friends from outside the floss-o-sphere are, and some of them record a weekly podcast on the subject.

Tonight one of them was over at my house for dinner, and I was reminded of Stuart Langridge’s post about making a Bad Voltage app and how he had a GenericPodcastApp component that provided common functionality with a clean separation from the rest of his app. So I decided to see how easy it would be to make a DWO Whocast app with it.  Turns out, it was incredibly easy.

Here are the steps I took:

  1. Create a new project in QtCreator
  2. Download Stuart’s GenericPodcastApp.qml into my project’s ./components/ folder
  3. Replace the template’s Page components with GenericPodcastApp
  4. Customize the necessary fields
  5. Add a nice icon and Suru-style gradients for good measure

That’s it! All told it took my less than 10 minutes to put the app together, test it, show it off, and submit my Click package to the store.  And the app doesn’t look half bad either.  Think about that, 10 minutes to get from an idea to the store.  It would have been available to download too if automatic reviews were working in the store (coming soon).

That’s the power of the Ubuntu SDK. What can you do with it in 10 minutes?

Update: Before this was even published this morning the app was reviewed, approved, and available in the store.  You can download it now on your Ubuntu phone or tablet.

Posted in OpenSource, Programming | Tagged , , , , , , , , | 2 Comments

Winning the 1%

Yesterday, in a conference call with the press followed immediately by a public Town Hall with the community, Canonical announced the first two hardware manufacturers who are going to ship Ubuntu on smartphones!

Now many have speculated on why we think we can succeed where so many giants have failed.  It’s a question we see quite a bit, actually.  If Microsoft, RIM/Blackberry and HP all failed, what makes us think we can succeed?  It’s simple math, really.  We’re small.  Yeah, that’s it, we’re just small.

Unlike those giants who tried and failed, we don’t need to dominate the market to be successful. Even just 1% of the market would be enough to sustain and continue the development of Ubuntu for phones, and probably help cover the cost of developing it for desktops too.  The server side is already paying for itself.  Because we’re small and diversified, we don’t need to win big in order to win at all.  And 1%, that’s a very reachable target.

 

Posted in OpenSource | Tagged , , , , , | 11 Comments

My first Debian package uploaded

Today I reached another milestone in my open source journey: I got my first package uploaded into Debian’s archives.  I’ve managed to get packages uploaded into Ubuntu before, and I’ve attempted to get one into Debian, but this is the first time I’ve actually gotten a contribution in that would benefit Debian users.

I couldn’t have done with without the the help and mentorship of Paul Tagliamonte, but I was also helped by a number of others in the Debian community, so a big thank you to everybody who answered my questions and walked me through getting setup with things like Alioth and re-learning how to use SVN.

One last bit of fun, I was invited to join the Linux Unplugged podcast today to talk about yesterday’s post, you can listen it it (and watch IRC comments scroll by) here: http://www.jupiterbroadcasting.com/51842/neckbeard-entitlement-factor-lup-28/

Posted in OpenSource, Programming | Tagged , , , , , , , | 1 Comment

A new 80/20 rule for open source

I didn’t have a post for yesterday, which means I haven’t managed to follow even the lenient rules I had set out for myself at the beginning.  I could claim that, since yesterday was a holiday in the USA it falls under the weekend exception, but that’s just cheating.  I fell short of my UbBloPoMo goals, I’ll own up to it.  It’s still the longest run of consistent postings this blog has ever seen, so I don’t consider this a failure. I hope you’ve all been enjoying it.

Part of the reason I didn’t post anything was because I spent the end of last week dealing with fallout from a couple of things I was involved in, my own post about Mozilla and the Community Council statement about Mint, and I was quite frankly not in the mood for either a lighthearted or a diplomatic post, so I made the decision not to post something I might later regret[1]. But during the controversy around both, one unifying meme started to emerge to me, which is the subject of this post, what I see as a new 80/20 rule.

Put simply, this rule says that people will tend to appreciate it more when you give them 20% of something, and resent you if you give them 80%.  It seems completely counter-intuitive, I know, but that’s what I was seeing in all of those conversations.  People by and large were saying that the reason Canonical and Mozilla were being judged so harshly was because they already did most of what those people wanted, which made them resented that they didn’t do everything.

When asked why they were so happy when somebody like Valve only gave them free (gratis) games, the response was almost always because they didn’t expect anything at all. But Canonical, because they gave almost everything away for free, is resented for something as minor as a CLA or closed-source Smart Scopes Service.  When I compare Canonical’s licensing approach to Mint with Red Hat’s approach to CentOS and Oracle, I was again told that they appreciated Red Hat requiring those derivatives to strip trademarks and re-build packages, but resented Canonical’s approach of explicitly letting Mint distribute Ubuntu packages with trademarks, because we didn’t go as far as they wanted. With Mozilla people took to closed-source, ad-filled websites or activity-tracking social media networks to berate Mozilla for daring to put commercial content in unused screen space.  Again, they were fine with ads on websites and Google tracking their conversations because it was expected, the fact that they existed at all was seen as a gift.  But because Mozilla has previously been so non-commercial and privacy focused , this small exception was seen as highly offensive to those same people.

To put this into a non-technical analogy, imagine you are hungry and somebody walks by with a sandwich.  If they stop and give you 20% of their sandwich, you will appreciate them sharing with you out of their bounty.  But if the same person gave you 80% of the sandwich, your first thought might be to question why they held back that 20%.  It’s a strange phenomenon, but when they keep 80% you still consider the whole sandwich as belonging to them, and thus your 20% is a gift.  But when they give you 80%, you perceive the whole thing as yours, and their remaining 20% as something taken from you.

This isn’t just a strange phenomenon, it’s a very troubling one.  It means that we are more likely to punish those people and projects that treat us well, and praise those that treat us poorly. That’s what I saw happening last week, and once I recognized what was happening I was able to see that it’s been happening for a very long time now.  And it’s not just Canonical and Mozilla, look at any open source project with a sizable user community and you’ll see the same thing.

I don’t know how to change this for the whole open source community, but the realization is certainly going to change the way I view those projects that I use.  I am going to make it a point to remember that none of the sandwich belongs to me, and to see everything somebody makes and gives me as a gift, and appreciate it even when it’s not everything that I wanted, because having 80% is always better than having 20%. I hope this post changes the way some of you view open source projects too.

[1] I agree with Jono’s On Accountability post, I don’t unpublish what I write or block comments that I don’t like.  But the other side of that coin is that I self-censor posts that are written in the heat of the moment, and have decided against publishing a number of things after letting them sit in draft status overnight and rereading them with a fresh perspective.

Posted in OpenSource, Politics | Tagged , , , | 31 Comments

Everything is Awesome

Since the kids were at my parent’s house for a sleepover, my wife and I got to go out and have a date night.  We saw the Lego Movie.  Without the kids (both of whom love Lego).  I don’t even feel bad about it.

It was, indeed, Awesome.

Posted in Uncategorized | Tagged , | 5 Comments

Apologies to razvi and iloveubuntu.net

Let me just start off by saying that I love iloveubuntu.net, I subscribe to their RSS feed and read if pretty regularly, as regularly as I read any other feed.  I have a great deal of respect for razvi and the writing he does for that site, and I would never mean any disrespect to him.

So what happened?  When I started my UbBloPoMo project, one of the things I said I would do was write a post in the evening, and schedule it to publish the following day.  That is what I’ve done for every post this month, and the last one was no exception.  I had written my article, and scheduled it to publish at 4am my local time (9am UTC), and at that point in time I was unaware of razvi’s editorial on the same subject (which may or may not have been published by the time I wrote mine).

My choice in headline was in no way a reference to his.  In fact, I chose the wording specifically because it wasn’t similar to any other headline for any other article I had read yesterday, so that nobody would feel like I was singling them out in particular. Of all the different ways a headline could have been worded, it was just a matter of random chance (or dump luck on my part) that mine and his were as similar as they were.

I still stand by what I said, about Mozilla and about us as a community.  But I wanted to apologize to razvi for the confusion and hurt feelings caused by my headline.  And to reiterate, I still love iloveubuntu.net, I still have a great deal of respect for razvi, and I will continue making his site part of my regular news reading.

Posted in OpenSource, Politics | Tagged , , | 2 Comments

Mozilla turns evil, sells your eyes to the highest bidder

Seriously?  If you find yourself writing a headline like this, you need to take a break from the internet.  If you clicked on this one because you wanted more blood-sport FOSS drama, you’re out of luck (and you should also take a break from the internet).  Come on guys, what’s happened to us that we keep jumping from one outrage to another?  For a group that loves to make allusions to 1984 when privacy is concerned, you’d think we would put up more resistance to these 2-minutes hate sessions when they are pushed on us.

Mozilla was the first open source project I ever took interest in.  I downloaded the source code for Netscape 5 (remember that?) and was *amazed* that they would just give it away to anybody like that.  I am involved in open source now, all these years later, because of that one experience.  Firefox is still one of the most well known and most used pieces of open software in the world.  They have been consistently open and doing good for us users. You don’t just disregard all of that because of a few ads.

So let’s quit this self-indulgent outrage, Firefox is still a great open source project.  Mozilla aren’t selling out, they aren’t turning evil, and they haven’t suddenly stopped caring about users. They have a nice feature that shows website thumbnails for previously viewed website, it’s very handy once you have previously viewed websites.  For new users, they’re empty, and empty tiles are useless.  So Mozilla wants to pre-populate them, until you’ve used the browser enough to fill that space.  That’s great, it turns something doing nothing, and makes it do something, that’s a good thing.  And they might make some money off it, which for everybody who’s concerned about Google’s dominance and infiltration into our privacy should be a good thing.

Are they ads?  Yes.  Or No, depending on who you ask.  But you’re asking the wrong question. The question should be: do they make Firefox worse for me? or you? or other people?  And the answer is we simply don’t know yet, because as of today those tiles are still empty for new users.  Mozilla thinks they can make Firefox better, both the web browser and the project behind it, and after 15 years of doing exactly that they’ve more than earned the right to try.

But let’s get back to the bigger problem, this constant stream of flamewars and hate fests.  We can’t go on tearing down free and open source projects like this.  No matter how much one of these might benefit your favored project at the moment, sooner or later the outrage machine is going to turn on it too. If we care at all about open source as a philosophy, we need to care about the people and projects that live by it.

As Russ Allbery saidrecently, after the Debian project had just weathered an outrage storm of it’s own:

But people should also get engaged and interested in understanding other *people* and finding ways to work with other people in difficult situations, since at the end of the day our communities are about people, not software.

Posted in OpenSource, Politics, Projects | Tagged , , , , | 13 Comments

Razing the Roof

Today was a distracting day for me.  My homeowner’s insurance is requiring that I get my house re-roofed[1], so I’ve had contractors coming and going all day to give me estimates. Beyond just the cost, we’ve been checking on state licensing, insurance, etc.  I’ve been most shocked at the differences in the level of professionalism from them, you can really tell the ones for whom it is a business, and not just a job.

But I still managed to get some work done today.  After a call with Francis Ginther about the API website importers, we should soon be getting regular updates to the current API docs as soon as their source branch is updated.  I will of course make a big announcement when that happens

I didn’t have much time to work on my Debian contributions today, though I did join the DPMT (Debian Python Modules Team) so that I could upload my new python-model-mommy package with the DPMT as the Maintainer, rather than trying to maintain this package on my own.  Big thanks to Paul Tagliamonte for walking me through all of these steps while I learn.

I’m now into my second week of UbBloPoMo posts, with 8 posts so far.  This is the point where the obligation of posting every day starts to overtake the excitement of it, but I’m going to persevere and try to make it to the end of the month.  I would love to hear what you readers, especially those coming from Planet Ubuntu, think of this effort.

[1] Re-roofing, for those who don’t know, involves removing and replacing the shingles and water-proofing paper, but leaving the plywood itself.  In my case, they’re also going to have to re-nail all of the plywood to the rafters and some other things to bring it up to date with new building codes.  Can’t be too safe in hurricane-prone Florida.

Posted in OpenSource, Programming, Upstream | Tagged , , , , , , , , | 1 Comment

Getting into Debian

Quick overview post today, because it’s late and I don’t have anything particular to talk about today.

First of all, the next vUDS was announced today, we’re a bit late in starting it off but we wanted to have another one early enough to still be useful to the Trusty release cycle.  Read the linked mailinglist post for details about where to find the schedule and how to propose sessions.

I pushed another update to the API website today that does a better job balancing the 2-column view of namespaces and fixes the sub-nav text to match the WordPress side of things. This was the first deployment in a while to go off without a problem, thanks to  having a new staging environment created last time.  I’m hoping my deployment problems on this are now far behind me.

I took a task during my weekly Core Apps update call to look more into the Terminal app’s problem with enter and backspace keys, so I may be pinging some of you in the coming week about it to get some help.  You have been warned.

Finally, I decided a few weeks ago to spread out my after-hours community a activity beyond Ubuntu, and I’ve settled on the Debian new maintainers Django website as somewhere I can easily start.  I’ve got a git repo where I’m starting writing the first unit tests for that website, and as part of that I’m also working on Debian packaging for the Python model-mommy library which we use extensively in Ubuntu’s Django website. I’m having to learn (or learn more) Debian packaging, Git workflows and Debian’s processes and community, all of which are going to be good for me, and I’m looking forward to the challenge.

Posted in OpenSource, Programming, Work | Tagged , , , , , , , , , , , , , | 2 Comments