Fun with Django, Meta-classes and dynamic models (Updated)

Posted by: Michael Hall 13 years, 10 months ago

(Comments)

I’ve started a new project at work that’s proven to be both fun and challenging.  The request was simple enough, our clients wanted something like MS Access, where they could define their own record types, run queries, edit data and get reports.

But they wanted it on the web. Backed by an SQL database.  Using normalized tables.

And soon.

Now we’ve been making pretty heavy use of Django here for a while, so it was a natural choice to base any new project on.  But how would we let our clients define Django models?  We couldn’t ask them to write Python objects, even if Python is an incredibly simple language.  No, this had to be point-and-click.  And on the web.

Metaclasses

One of the more esoteric features of Python is Metaclasses.  Basically these are classes who’s instances are other classes.    Confusing right?  It took me some experimenting to get used to it too, since most of my prior object-oriented programming experience was in Java.  But it does give you a really powerful tool for generating custom classes on demand.  And Django models are nothing more than Python classes.  You can see where this is going.

So it turns out there are a few examples of people talking about using Metaclasses to generate Django models here and here, which I used as a jumping off point.  They also pointed out some of the shortfalls of using dynamic models, mostly that many of Django’s features expect to find your models in INSTALLED_APPS, specifically syncdb and the Admin app.  But the idea was straight forward enough: use a static Django model to define dynamic ones.

Dynamic Models

The models themselves weren’t very complicated, I started off with DynamicApp, DynamicModel and DynamicModelField, each with fields to define the most common attributes of it’s namesake.  With the easy part out of the way, now it was time to turn these into actual django.db.models.Model subclasses.

It just so happens that django.db.models.Model is already using a metaclass: django.db.models.base.ModelBase, which takes in all the fields you define in your model, the inner Meta class, and anything else you put into your model definition, and turns it into the actual Model class that you use.  Using the magic of Python’s type() function, we can call this metaclass directly, and pass it all that information as parameters.  So now I can turn a DynamicModelField into an actual Field class, and I can turn a DynamicModel into an actual Model class.  And that’s all there is to it.

Ha! not quite.

Syncdb

Remember where I said that syncdb can’t find dynamic models?  Well now I have my Model class, but there isn’t a table in my database to back it up.  And unfortunately it wasn’t a matter of calling a simple Django function to get it.  I ended up copying a small but significant amount of code out of django.core.management.commands.syncdb, but the end result was magical, I could turn my dynamically defined Model into just the right SQL table.

Admin

One of the wonderful apps that comes with Django by default is the Admin.  For those not familiar with the Django admin, it gives you a very simple yet effective interface to manage all your new Django models and their associated data with very little configuration necessary.  You can add, edit and delete data from an web interface that is generated based on your model definition.

Once again, though, it expects you to have written some actual code, and for that code to be in admin.py under one of the entries in INSTALLED_APPS.  But it turns out there’s not a lot of magic going on here, all it does is execute the contents of admin.py on startup, and one of the things you need in there is a call to admin.site.register().  Well that works just as well if you call it from DynamicModel.save() as it does from admin.py.  So a couple lines of code later, and now my new Models are showing up in the Django Admin.  Now I was cooking with fire!

And got burned.

Model Cache

It probably wouldn’t be very good for performance if Django had to execute a Model’s metaclass every time you wanted to reference it.  So the Django devs were smart enough to cache the resulting Model class after the first time.  If you’re interested, this happens in django.db.models.loading.  It took me a good couple of hours walking back through the entire process a new Model goes through (after my newly created ones weren’t showing up properly) before I found this little tidbit.  The fix was simple enough, since the cache is keyed off the app and model name, I just had to delete my model’s entry every time it’s DynamicModel record was saved.

Changing Models

By now I was riding high on my success, I could define a model, define it’s fields, and with a click of a button in the Admin, it would generate my tables exactly as they should be.  But part of the requirements were that the client would be able to add or delete fields from their models.  Which meant that I would have to add or drop columns from the table to match.  Which isn’t exactly my idea of a fun or safe activity.

Thankfully we’ve been using South for our Django database migrations for about the last year, and it’s been very reliable against MySQL (Oracle is another story, unfortunately).  South will inspect your Django models, and keep a running history of changes you make to it.  It will then create a migration script for each set of changes.  These scripts are nothing more than python code files that make calls back into South when they run.  A little digging into South’s code revealed what would need to be called to add and drop columns, so I added them to DynamicModelField’s .save()  and .delete() methods, clear the model cache one more time, and I was rocking!

Dynamic Interface

Now that I had the models, I needed a dynamic interface for them.  The Django admin is great for admins, but is too inflexible for making end-user interfaces.  Over the last year I’ve been developing a Django app that would allow us to quickly built up user friendly interfaces in Django and, critically, I decided early on to do as much as I could just by inspecting the Django models definitions just like the Admin app does.  That decision payed off big time with this new project, because I could just pass my dynamically created models to that existing framework, and out came a standard, robust and friendly interface.

Unfortunately none of this work is open sourced yet (see below), but I’m hopeful that it will be sometime in the near future.  My current employer is very open-source friendly and has already allowed me to release several apps and libraries under a BSD license.

Update: The parts of this project that deal with dynamic models has been released under a BSD-style license.  Documentation will be coming to it soon: https://bitbucket.org/mhall119/dynamo/overview

• Help make Gnome Software beautiful →