Empty Thoughts

One of the things that has kept me very busy over the past several months is something called Django-SQLAlchemy. I started toying with the idea back in the fall and it just kind of rumbled around in my head for a while as I thought through different approaches to integrate SQLAlchemy into my Django projects. Yes, I’m aware you can just import sqlalchemy and you’re done. But no one is every really talking about that, are they? What people want is the ability to keep everything the same but still have the power and flexibility of SQLAlchemy available to them. That’s what this project aims to do.

Finally, some time in or around December 2007, Brian Rosner and I began discussing the project. I was glad to hear that he had been thinking about similar things, and we decided that we would put forth some effort on the project. Brian and I have been working for the past several months trying to do a proof of concept and develop a roadmap for the project. A lot of hackish code was developed, and we frequently went down one road only to discover there was a much easier way. In the end we figured out what we think is a plausible plan for implementing SQLAlchemy into Django.

At PyCon 2008 in Chicago Brian and I decided to set up a Birds of a Feather session in one of the Open Spaces to discuss django-sqlalchemy. Much to our surprise we had a great turnout. First of all Mike Bayer, creator of SQLAlchemy, and Jason Kirtland, core contributor to SQLAlchemy, both showed up and were extremely helpful. They indicated that they are really interested in seeing the project become a success, and they would help in any way possible. During the meeting they even discussed modifying some functionality within SQLAlchemy in order to make our job a bit easier. Adam Gomaa also showed up and expressed an interest in helping on the project. We added Adam as a contributor immediately and we all began hacking away.

Over the next several days, Adam, Brian, and I made really good progress. We solidified some of the hackish filter translation code, we created a test runner, We made a lot of progress on mapping the related fields, and most importantly we completely replace the declarative layer (previously Elixir based), with code based on the declarative layer that Mike created in SQLAlchemy. We also bugged Mike and Jason continuously, and every time they provided us with valuable guidance that saved us hours of grief. We can not thank them enough for their help.

Development Roadmap

The fundamental approach to the django-sqlalchemy project is to implement SQLAlchemy as a custom database backend. Actual SQLAlchemy backends are made available through custom project settings. By doing so we can “hijack” all calls to the QuerySet with our own custom QuerySet that interprets these calls and passes them on to SQLAlchemy. We also need to be able to provide SQLAlchemy hooks into our Django models and this is handled by the declarative layer of the project. Finally, things like management commands that have affect on the database backend need to be overridden to do the right thing in SQLAlchemy terms.

1. Management Commands – Management commands that affect the database backend in any way need to be overridden to pass on calls to the SQLAlchemy layer. This affects at least the following: syncdb, sql*, reset, dbshell, createcachetable, dbshell, dumpdata, flush, inspectdb, loaddata.

2. Declarative Mapper – mapping the Django model structure into SQLAlchemy Table and Mapper information so that access to SQLAlchemy is available while still providing access to all of Django’s expectations for functionality of a model. This includes mapping all of the fields, related fields, polymorphic relations, etc. Plus once we get it functionally compliant, there will be a lot of hooks for extensions that will provide greater access to SQLAlchemy power.

3. QuerySet Mapping – Map the backend django-sqlalchemy database so that we can override Django’s QuerySet into a SqlAlchemyQuerySet that interprets and passes on all backend query functionality to SQLAlchemy. This includes the handing of sessions, metadata, and transactions.

Benefits

So why are we doing all of this? Well mostly we are scratching our own itch. We also sense that there is a lot of interest in the community for this type of thing. Additionally we realize that there are a lot of benefits provided by a django-sqlalchemy integration, that will help make Django even more powerful:

• Addition database support: MS-SQL, Firebird, MaxDB, MS Access, Sybase and Informix. IBM has also released a DB2 driver

• Connection Pooling

• Multi-Database Support, including things like sharding

• Extremely powerful query language

Initially all of these things may not be available as we first aim to achieve functional equivalence to Django’s ORM, but over time we will provide hooks and ways to tie into the additional functionality provided by SQLAlchemy.

Interest

So if you’re still reading along at this point, you probably have some interest in the project. We welcome you to get involved in any way that works best for you. We have set up the following resources for the project:

Google Code for Ticket Tracking and Wiki

Gitorious for Source Control

Google Group for Discussion

IRC Channel #django-sqlalchemy

I will also try to be good about posting progress updates here.