Shattered Crystalline Matrix

Seems like everyone is suddenly talking about Twisted / Divmod / etc, so I thought I'd throw in some commentary on why I use Mantissa. I'll start at the bottom of the stack, with Twisted. (If I start discussing "why *Python*?" I'll probably never finish this post!) The core of Twisted is an asynchronous network / I/O framework; around this core is a variety of code built on top of this framework: Twisted includes implementations of many network protocols, and has a number of useful abstractions that form part of those implementations.

One of the key principles that people in the Twisted community seem to generally subscribe to, is that the primary goal of a piece of code is to do the "right" thing. Of course, despite the best efforts of the great coding wizards, mistakes are still made, and manpower is always short, so the result is not always a perfect glistening gem of coding perfection; but the focus is still on "do it right". There is a mantra (attributed to a variety of people) that goes "Make it work, make it right, make it fast"; in my opinion, when that ordering is violated, the results are typically a disaster. Any codebase that achieves widespread use must at least get the "make it work" part done first, but often the next step is "make it fast"; not necessarily in the sense of the performance of the codebase, but also in terms of development effort. The result is an environment and culture that optimizes for getting the wrong thing done quickly, and in the process, typically making it harder or impossible to get the right thing done.

In many contexts, this isn't such a big deal; it's okay to get a result that is only "50%" or even "25%" in many cases, either just ignoring the breakage, or dealing with it piecemeal as you go along. Unfortunately (or fortunately, perhaps), I'm an idealist at heart. I retain enough pragmatism to operate in the real, flawed, world, but idealistic compromise is mentally unpleasant to me, so I avoid it as much as possible. There are mostly only two contexts I write code in: the personal context, where I'm doing it mostly for enjoyment, and thus have the freedom to be as idealistic as I want; and then the business context, where I'm being paid to make things work in the long-term, and need to produce high-quality results. So, either way, I'm looking to spend more time writing code that does the right thing (which ultimately saves me time in the long run), rather than worrying about getting any old junk working as quickly as possible, and this aligns perfectly with the attitude of the Twisted community, at least as I perceive it. Thus, while others complain with the learning curve involved with this stack, I have no problems spending the time needed to become familiar with everything I need to unlock the true power of the stack. In most cases, I find that I'm not "struggling" to learn the technology, it's just that it does so much more for me, that I need to spend more time finding out about how to use it, and ultimately saving myself far more time in the long run. The time spent learning is not time lost, but time invested.

So, Twisted provides asynchronous networking and protocol implementations, such as a web server; the layers of the stack above Twisted start to tie me down more in terms of architectural approach, but this is still an acceptable cost for me (although I wouldn't necessarily choose this set of technologies for every single project I write). Axiom provides synchronous in-process database access, and more importantly, schema management. This frees me from the task of managing a separate database server process, while simultaneously providing a mapping from the database schema to data objects, and a framework ("upgraders") for managing changes in that schema (as opposed to the usual ad-hoc scripting approach that seems to be the norm).

Nevow… well, there are a lot of things wrong with Nevow; it contains large swathes of outdated code that should generally just be avoided, and some of the internals ("context", anyone?) are really awful, and need to be replaced. In addition, it ends up replacing half of twisted.web's webserver implementation; while this replacement is mostly for the better, that code really belongs back in twisted.web, instead of having the web server code schizophrenically split between the two codebases. Yet, despite all of that, Nevow still does a pretty good job of letting me do "the right thing", even if Nevow itself isn't always doing the right thing; the bad parts of Nevow are an eyesore, but it's not that hard to just ignore them, and use the good parts. The templating system has a heart of gold, allowing me to cleanly separate my templating markup, my rendering logic, and my data model. Add Athena to the mix, and I now have bi-directional communication between browser and server (COMET, if you like), and a JavaScript modules system that continues the theme of code separation, keeping my JavaScript separate from the rest; I also get a JavaScript class-based object model.

Mantissa sits atop the other elements of the stack, as the application server. It provides abstractions along application lines, allowing different applications (in the form of "offerings") to co-exist in the same Mantissa server, and along user lines, storing each user's data in their own Axiom store (database, and this is actually mostly an Axiom feature, although it has been argued that the code should be moved into Mantissa), and providing a "sharing" system to control access to the data in the users' stores. For the most part, Mantissa's functionality is currently web-based; but this is simply a matter of where the effort has been focused so far, and not a reflection of the underlying design — Mantissa's design is multi-protocol, allowing for an HTTP server sitting alongside anything else you might want (SMTP, IMAP, XMPP, SIP, whatever). This brings me to another important property that is pervasive within this stack; while Mantissa (and dependencies) may not have the popularity and thus the manpower that other projects enjoy, the components are typically designed in a fashion amenable to extension. If I need some functionality that nobody has had time to work on yet, I'll obviously have to write a fair amount of code to get the job done; but I typically won't have to overpower Mantissa in a brutal battle of wills just to get it to let me start working on what I want done. I simply provide the code for the parts that haven't been implemented yet, plugging them into the parts that *have* been implemented. This way I reap the benefits of the work that has been done by others without running into any nasty surprises later on once the need to go beyond that work arises; something that is essentially guaranteed to happen when working on any significantly large project.

In summary, I use Mantissa in a lot of my projects because it provides a lot of useful functionality that aids me in my quest to write good software, while at the same time getting out of my way when it comes time to go beyond the bounds of what it provides me with. Invariably, this choice means that certain architectural decisions have been made for me, decisions that can't easily be changed or reversed by building on top of the stack; for some projects, those decisions are inappropriate, and thus I'll c
hoose differently; but those decisions have usually been made by some pretty smart people, and are generally sound ones in the contexts in which they are appropriate.

Hopefully I've managed to convey the high-level attraction I have to Mantissa and related software; please note that I'm not trying to justify the low-level decisions involved here. Want to argue about how PostgreSQL is a superior database solution than SQLite? Why per-user database are a bad idea? How "COMET" and web applications that require JS are ruining the web? Sure, I'd be glad to have those discussions, but the point of this post was not to defend the specifics; think of it more as a window onto the development world I immerse myself in most of the time; and hopefully a little of my enthusiasm has been passed on to you!

tags:

• python
• web
• frameworks
• templating
• nevow
• django
• tal

So, Colin has been ranting and raving about Django and templating and such again; subsequently, I had a brief conversation on IRC with someone else about the subject, and figured I’d do a bit of a braindump into my blog.

Like Colin, I have a strong dislike of using plaintext templating mechanisms for generating anything other than text/plain data. This is primarily due to the mixing of levels that this kind of usage entails; it is a design flaw very similar to the kind of design flaw that enables SQL injection attacks in poorly written applications. Your data should be filtering through a higher level abstraction which is then serialized by code specifically written to implement the output format; dumping it straight into a plaintext template is just begging for trouble, in the form of random garbage not conforming to the expected format. One only needs to look at all the malformed RSS feeds out there to see an example of how bad this can get.

However, there’s more to it; for example, templating systems like TAL are not plaintext, but are still problematic. The second problem is an issue of programming languages; in a typical webdev project I’m already writing Python code, JavaScript code, and maybe SQL code; writing yet more code in some relatively half-baked and featureless language is not an appealing concept. In addition, even if the business logic is separated out from the template, you still have presentation logic mingled with the actual template content; this may seem superficially attractive, but ultimately I find it to be counterproductive.

At this point, Colin and I have a divergence of opinion; while he prefers to make heavy use of stan (a somewhat “hackish” Nevow feature that allows you to write XML using Python syntax), I prefer to almost completely avoid it. To put it simply, I want to write Python code in my Python source files, JavaScript code in my JS source files, SQL code in SQL source files, XML in XML files, and CSS in CSS files. Given that all of these formats/tools allow me to do just that, I would rather avoid embedding copious amounts of one language in another, as I find working with JS embedded in Python (or whatever) to be more awkward and confusing than having it separated out.

I do think that using plaintext templating for emitting more structured data has its place in things like wikiish systems (eg. JotSpot) or CMSish systems, as an enabling technology for people who are nominally non-programmers; but for a “real” programmer with a full array of libraries and language at their disposal, it seems to just be bad system design.

In closing, I’ll note that I use Nevow for (X)HTML templating, making heavy use of patterns and templates, with a few render specials. If you’re not familiar with Nevow, that probably sounds like complete Greek; I suggest you do some more reading to get an idea of how Nevow templates work, but in very broad terms a pattern is an XML element marked with a nevow:pattern="name" element; the element is then not directly present in the output, but is available to the presentation code, and is usually used for making N copies of a particular element structure. A slot, on the other hand, is a element, which is “filled” by the presentation code; that is, replaced with content supplied at render time.