Tag: mantissa

Divmod / PyPy status update

by on Aug.04, 2013

Just a quick status update:

  • Epsilon test suite passes on PyPy.
  • Nevow test suite passes on PyPy.
  • Axiom (trunk) test suite has two failures on PyPy, fixed by this branch (which just deletes the tests); I don’t expect this to affect any application code.
  • Mantissa test suite mostly fails. This is due to modules that indirectly import xmantissa.terminal which imports PyCrypto directly or indirectly (via twisted.conch) — PyCrypto does not build on PyPy.
  • I haven’t looked at Combinator; the test suite has a whole bunch of test failures on CPython due to a change in Subversion, and there’s no real reason to run Combinator with PyPy (I would expect it to be slower than on CPython).
  • I haven’t looked at Quotient or Imaginary yet, as they depend on Mantissa.
  • I haven’t looked at Hyperbola, Prime, Reverend, or Sine — I doubt anyone cares about these (I don’t even know what Prime does, and it doesn’t have a test suite).

The next thing I’m going to work on is making the Mantissa dependency on PyCrypto optional; while having an SSH server is nice, there is plenty of functionality in Mantissa that does not depend on or interact with the SSH server in any way, so it’ll still be useful for most applications. With any luck, once this is fixed, the entire test suite will pass; it’s hard to predict given how little of the test suite is currently runnable.

Leave a Comment :, , , more...

An introduction to Mantissa (part 3): Navigation powerups and other friends

by on Mar.04, 2012

This is the fourth post in a series of articles about Mantissa.

In the previous article I described how an offering can provide powerups to be included in a product, which will then be installed on a user store; in this installment, I will discuss what form these powerups can actually take, and how they allow you to expose your application’s functionality to the user.

One of the most commonly-implemented powerup interfaces in Mantissa is INavigableElement. Mantissa has a somewhat generalized idea of “navigation”, whereby a nested menu structure can be defined through INavigableElement powerups, and then displayed by different implementations for different protocols; for example, the web view has a visual dropdown menu system, whereas the SSH server presents a textual menu system. A typical INavigableElement powerup implementation will look something like this:

from zope.interface import implements
from axiom.item import Item
from axiom.attributes import integer
from xmantissa.ixmantissa import INavigableElement
from xmantissa.webnav import Tab

class PonyCreator(Item):
    """
    Powerup for creating and managing ponies.
    """
    implements(INavigableElement)
    powerupInterfaces = [INavigableElement]

    ponyQuota = integer(allowNone=False, default=10)

    def getTabs(self):
        return [Tab('ZOMG PONIES!', self.storeID, 1.0)]

INavigableElement only has one method, getTabs, which returns a list of “tabs” or menu items to be presented in the nav. The primary components of a tab are a title (which is how the item is displayed in the UI), the storeID of an item in the same store which the tab points to, and a float between 0.0 and 1.0 indicating the sort priority of the tab (higher values sort sooner). In this case, we have the tab pointing directly at the PonyCreator item itself; in order for this to work, we’ll need some extra code to allow PonyCreator to be exposed via the web.

In order for an item in a user’s store to be privately accessible via the web by that user, it needs to be adaptable to the (somewhat poorly-named) INavigableFragment interface. This is almost always done by defining an adapter from the item type to INavigableFragment:

from twisted.python.components import registerAdapter
from nevow.athena import LiveElement
from xmantissa.webtheme import ThemedDocumentFactory
from xmantissa.ixmantissa import INavigableFragment, ITemplateNameResolver

class PonyCreatorView(LiveElement):
    """
    Web view for Pony management.
    """
    implements(INavigableFragment)

    title = u'Pony management'
    docFactory = ThemedDocumentFactory('pony-creator', 'resolver')

    def __init__(self, ponyCreator):
        super(PonyCreatorView, self).__init__()
        self.ponyCreator = ponyCreator
        self.resolver = ITemplateNameResolver(self.ponyCreator.store.parent)

registerAdapter(PonyCreatorView, PonyCreator, INavigableFragment)

Our element will be wrapped in the Mantissa shell when it is rendered, so we cannot control the page title directly from the template, but the title attribute provides a way for our element to specify the page title. ThemedDocumentFactory is used to retrieve the template through the theme system; the arguments are the name of the template (‘pony-creator’) and the name of the attribute holding the ITemplateNameResolver implementation used to retrieve the template. This attribute is set in __init__ using a slightly awkward method; the template resolver should really be passed in by Mantissa somehow, but currently there is no mechanism for doing this, so instead we retrieve the default resolver ourselves from the site store.

This is all that is needed for hooking some code up to the web view; any further UI behaviour would be implemented in HTML / JavaScript in PonyCreatorView, usually by invoking additional methods defined on PonyCreator.

Next up: Sharing, or “How do I publish public / shared content?”

Leave a Comment :, , more...

An introduction to Mantissa (part 2): Offerings

by on Jul.10, 2010

This is the third post in a series of articles about Mantissa.

The primary code entry point in Mantissa is the offering. Your application can provide an IOffering plugin in xmantissa.plugins, which hands some chunks of your code over to be used by Mantissa. There is no real reason to implement IOffering yourself; instead, simply instantiate xmantissa.offering.Offering in your plugin module. For example, you might have an xmantissa/plugins/ponies.py module containing the following code:

from twisted.python.filepath import FilePath

from xmantissa.offering import Offering

import ponies
from ponies.ponies import PonyCreator
from ponies.rainbows import RainbowEditor
from ponies.theme import PoniesTheme

plugin = Offering(
    name=u'Ponies',
    description=u"""
    A web-based system for generating ponies.
    """,
    siteRequirements=[],
    appPowerups=[],
    installablePowerups=[(u'ponies', u'Pony creation and editing', PonyCreator),
                         (u'rainbows', u'Rainbow editing', RainbowEditor)],
    loginInterfaces=[],
    themes=[PoniesTheme('base', 0)],
    version=ponies.version,
    staticContentPath=FilePath(ponies.__file__).sibling('static'))

There are a number of elements here, some of which we will only return to later, but I’ll run through all of them here briefly.

  • name: A short identifier used as the name of this offering.
  • description: A long description explaining what this offering is.
  • siteRequirements: Interfaces that this offering requires from the site store.
  • appPowerups: Powerups to be installed on the app store.
  • installablePowerups: A list of 3-tuples, (name, description, item class). Each of these tuples describes a powerup that can potentially be installed on a user store.
  • loginInterfaces: This allows your offering to provide additional ways of logging into Mantissa. For example, a webmail offering might provide interfaces that are used when logging in via SMTP or POP3.
  • themes: The themes system provides a way to provide and override the templates used by your widgets.
  • staticContentPath: A path pointing at the root of your static content. The file hierarchy at this location will be served up by Mantissa at /static/YourOfferingName.
  • version: A Version object specifying the version number of your offering.

For more information, take a look at xmantissa.ixmantissa.IOffering, which documents each of these attributes in more detail. The main highlights here, though, are appPowerups, for the powerups you want installed on the app store, and installablePowerups, for powerups you want to be available for installation on the user store.

To set up a Mantissa site, the first step is to run axiomatic mantissa; this will prompt you for an admin password as part of installing the base Mantissa components. To start the site up, you run axiomatic start; usually you would pass the -n flag during development, which prevents the server from daemonizing, and directs log output to the console instead of to a log file. If you log in as the admin user (admin@localhost by default) via the web interface, you can install an Offering through the admin UI; alternatively, you can install offerings from the command line by running axiomatic offering install SomeOfferingName.

When your offering is installed, the powerups in appPowerups will be installed onto the app store for your offering. The installablePowerups you provided will not be installed anywhere immediately, instead they will be exposed in the admin UI for products. A product is an administratively-defined collection of powerups; the admin can select any of the installablePowerups from any of the installed offerings when creating a product. Products are used in conjunction with signup methods, which are also locally configured by the admin. A signup has a product associated with it; when a user signs up using this method, all of the powerups that comprise the product will be installed on their user store.

Whew! This may all seem like more indirection than is actually necessary, but if you view Mantissa as a layer that multiplexes between users and applications, it may make a little more sense.

Next up: Powerup interfaces, or “How does the user actually interact with my code?”

1 Comment :, more...

An introduction to Mantissa (part 1): The Data Model

by on Jun.07, 2010

This is the second post in a series of articles about Mantissa.

As you may have already surmised, Mantissa is designed to host more than one application in the same deployment. Due to the way Mantissa builds on Axiom, it is important to first understand the data model, before starting to understand how the components of your application plug into Mantissa.

A Mantissa deployment, or “node”, consists of a top-level Axiom store called the site store, and various substores contained therein. The substores fall into two categories; user stores, and application stores. The site store primarily contains IService powerups for the various services running in the node (think HTTP server, SMTP server, etc.), and the substore items for the user / app stores. It also contains the IRealm implementation for authentication (using twisted.cred, naturally), along with account / login method information for each user account, as well as other configuration data such as API keys (think PayPal API, or similar).

The user stores are where your actual application data will be stored in a typical Mantissa application. Each user has their own Axiom user store (SQLite database), where data for that user is stored. The account / login method items for that particular user are also duplicated in the user store, to allow user stores to be transferred between site stores. Items in different user stores should not interact with each other, except via the interstore messaging API. This forms the core of Mantissa’s scaling model; while each individual user’s data must all be located on the same node, different users may have their data stored on different nodes, allowing you to spread your users across as many nodes as you need to in order to achieve the desired performance.

Application stores are sort of a special kind of user store; internally, they’re treated as a user store with a username corresponding to the name of the Offering (offerings will be discussed later), and a domain of None. Each application installed on a Mantissa node will have an application store; exactly what to put in there isn’t entirely clear, the best I can come up with is “application-global data”.

Mantissa has a system called “sharing”, whereby access to items in one user’s store can be granted to other users, or even anonymous public access granted. In the case of “websharing” (which works over HTTP), the items are accessed using a URL that specifies which user’s data is being accessed, as well as the identifier of the particular item. For example, the URL might look like http://myapp.com/mithrandi/timeline or http://mithrandi.myapp.com/timeline (“mithrandi” is the username, “timeline” is the share ID).

For a user-centric application (like Google Docs or Dropbox), this works pretty well; if the data in your application is primarily of global scope, then this may not be a good fit at all. There are some ways around this: firstly, not every “user” needs to be an actual live user. Thus, you can divide your data across “artificial” user stores, and share it to the users that need to have access to it, retaining Mantissa’s scalability in the process. For “global” data access or operations, you can send a message to each user store to perform a certain query or operation, and then collate the results; this model of “send the computation to the data” is similar in some ways to MapReduce.

However, if your application truly does not fit into the model, then one possibility is to store the application data in a completely separate service, using Mantissa only for configuration and authorization. The fact that Mantissa stores certain data in Axiom does not prevent you from making queries to an external PostgreSQL database, or Cassandra database, or web service, or whatever else you can come up with; it just means that you derive no specific benefit from Mantissa’s database functionality.

Another serious caveat is that while the conceptual model and APIs for Mantissa scalability are mostly in place, the actual implementation of scaling is not there at all. There is currently no networked implementation of the interstore messaging API, there are no tools for managing a cluster of Mantissa nodes, distributing or migrating user stores, backup/replication of user stores, and so on. The foundation for building these tools and components exists, but at the moment, you’ll need to build them yourself (or pay someone to do it for you) if you need them. Thus, if you are looking for “Facebook size” scalability straight out of the box, Mantissa is probably not for you. On the other hand, depending on your needs, building the tools you need as you grow your application may not pose a significant challenge, so if you’re not afraid of writing a little code, don’t be scared off right away by this. You could also avoid relying on Mantissa for scalability, by storing your application data in another datastore that already meets your scalability needs

Next up: Offerings, or “How do I expose my application to Mantissa?”.

Leave a Comment :, more...

An introduction to Mantissa (part 0)

by on Jun.07, 2010

This is the first post in a series of articles about Mantissa; a Python application server built on Twisted, Nevow, and Axiom.

“Application server” is a somewhat fuzzy term that seems to mean different things to different people; but in essence, Mantissa serves as a layer that joins your actual application code to the various ways (different protocols, for example) that your application may be accessed. Or, to put it another way, when writing a Mantissa application, the deployment target is Mantissa, not “a web server”, even if your application is only going to be accessed from the web. That’s not to say that Mantissa completely abstracts away the details of the protocol, making it impossible to control how your application is presented; rather, it provides common ground to mediate that interaction. It provides mechanisms for the common portions of functionality, such as access control, data storage, authentication and authorization, and also makes provision for defining standard interfaces through which your application is exposed to the various protocols you may wish to make use of.

This series is not going to be of the “build your own wiki in 60 seconds” kind. While there are some quirks and rough patches in Mantissa that may make it not the best choice for hacking together an application really quickly, it is probably still something that is doable; but unfortunately, I’m not the right person to be writing that sort of article. What I hope to do, instead, is provide a more in-depth introduction that allows you to understand and embrace (or reject!) the design philosophy of Mantissa for longer-term projects, while noting incomplete areas and other potential pitfalls along the way that may need to be avoided.

This is also not going to be an Axiom, Nevow, Python, or Twisted tutorial; while I will touch on aspects of those components as they relate to Mantissa, you will need to look elsewhere for further detail.

That’s it for now, except to thank Jeremy in passing for nagging me about this until I got around to doing it (or starting it, at least).

Leave a Comment :, more...

Methanal

by on Dec.12, 2008

For those of you who didn't read about this elsewhere already, I've just released Methanal 0.1.2. The project was recently moved to Launchpad / bzr in the hopes of making it more accessible.

Methanal is a Mantissa widget library, primarily consisting of an Athena-based forms implementation (if you don't get the obscure pun, it might help if I tell you that methanal is the IUPAC name for what is more commonly known as formaldehyde), but we're also building a collection of miscellaneous widgets that you may find useful. Unfortunately everything depends on Mantissa at the moment, but a lot of the widgets only depend on the Mantissa themes system, so if that was moved into Nevow, a lot of them would be usable in any Athena application.

Leave a Comment :, , , more...

Why Mantissa?

by on Jul.03, 2008

Seems like everyone is suddenly talking about Twisted / Divmod / etc, so I thought I'd throw in some commentary on why I use Mantissa. I'll start at the bottom of the stack, with Twisted. (If I start discussing "why *Python*?" I'll probably never finish this post!) The core of Twisted is an asynchronous network / I/O framework; around this core is a variety of code built on top of this framework: Twisted includes implementations of many network protocols, and has a number of useful abstractions that form part of those implementations.

One of the key principles that people in the Twisted community seem to generally subscribe to, is that the primary goal of a piece of code is to do the "right" thing. Of course, despite the best efforts of the great coding wizards, mistakes are still made, and manpower is always short, so the result is not always a perfect glistening gem of coding perfection; but the focus is still on "do it right". There is a mantra (attributed to a variety of people) that goes "Make it work, make it right, make it fast"; in my opinion, when that ordering is violated, the results are typically a disaster. Any codebase that achieves widespread use must at least get the "make it work" part done first, but often the next step is "make it fast"; not necessarily in the sense of the performance of the codebase, but also in terms of development effort. The result is an environment and culture that optimizes for getting the wrong thing done quickly, and in the process, typically making it harder or impossible to get the right thing done.

In many contexts, this isn't such a big deal; it's okay to get a result that is only "50%" or even "25%" in many cases, either just ignoring the breakage, or dealing with it piecemeal as you go along. Unfortunately (or fortunately, perhaps), I'm an idealist at heart. I retain enough pragmatism to operate in the real, flawed, world, but idealistic compromise is mentally unpleasant to me, so I avoid it as much as possible. There are mostly only two contexts I write code in: the personal context, where I'm doing it mostly for enjoyment, and thus have the freedom to be as idealistic as I want; and then the business context, where I'm being paid to make things work in the long-term, and need to produce high-quality results. So, either way, I'm looking to spend more time writing code that does the right thing (which ultimately saves me time in the long run), rather than worrying about getting any old junk working as quickly as possible, and this aligns perfectly with the attitude of the Twisted community, at least as I perceive it. Thus, while others complain with the learning curve involved with this stack, I have no problems spending the time needed to become familiar with everything I need to unlock the true power of the stack. In most cases, I find that I'm not "struggling" to learn the technology, it's just that it does so much more for me, that I need to spend more time finding out about how to use it, and ultimately saving myself far more time in the long run. The time spent learning is not time lost, but time invested.

So, Twisted provides asynchronous networking and protocol implementations, such as a web server; the layers of the stack above Twisted start to tie me down more in terms of architectural approach, but this is still an acceptable cost for me (although I wouldn't necessarily choose this set of technologies for every single project I write). Axiom provides synchronous in-process database access, and more importantly, schema management. This frees me from the task of managing a separate database server process, while simultaneously providing a mapping from the database schema to data objects, and a framework ("upgraders") for managing changes in that schema (as opposed to the usual ad-hoc scripting approach that seems to be the norm).

Nevow… well, there are a lot of things wrong with Nevow; it contains large swathes of outdated code that should generally just be avoided, and some of the internals ("context", anyone?) are really awful, and need to be replaced. In addition, it ends up replacing half of twisted.web's webserver implementation; while this replacement is mostly for the better, that code really belongs back in twisted.web, instead of having the web server code schizophrenically split between the two codebases. Yet, despite all of that, Nevow still does a pretty good job of letting me do "the right thing", even if Nevow itself isn't always doing the right thing; the bad parts of Nevow are an eyesore, but it's not that hard to just ignore them, and use the good parts. The templating system has a heart of gold, allowing me to cleanly separate my templating markup, my rendering logic, and my data model. Add Athena to the mix, and I now have bi-directional communication between browser and server (COMET, if you like), and a JavaScript modules system that continues the theme of code separation, keeping my JavaScript separate from the rest; I also get a JavaScript class-based object model.

Mantissa sits atop the other elements of the stack, as the application server. It provides abstractions along application lines, allowing different applications (in the form of "offerings") to co-exist in the same Mantissa server, and along user lines, storing each user's data in their own Axiom store (database, and this is actually mostly an Axiom feature, although it has been argued that the code should be moved into Mantissa), and providing a "sharing" system to control access to the data in the users' stores. For the most part, Mantissa's functionality is currently web-based; but this is simply a matter of where the effort has been focused so far, and not a reflection of the underlying design — Mantissa's design is multi-protocol, allowing for an HTTP server sitting alongside anything else you might want (SMTP, IMAP, XMPP, SIP, whatever). This brings me to another important property that is pervasive within this stack; while Mantissa (and dependencies) may not have the popularity and thus the manpower that other projects enjoy, the components are typically designed in a fashion amenable to extension. If I need some functionality that nobody has had time to work on yet, I'll obviously have to write a fair amount of code to get the job done; but I typically won't have to overpower Mantissa in a brutal battle of wills just to get it to let me start working on what I want done. I simply provide the code for the parts that haven't been implemented yet, plugging them into the parts that *have* been implemented. This way I reap the benefits of the work that has been done by others without running into any nasty surprises later on once the need to go beyond that work arises; something that is essentially guaranteed to happen when working on any significantly large project.

In summary, I use Mantissa in a lot of my projects because it provides a lot of useful functionality that aids me in my quest to write good software, while at the same time getting out of my way when it comes time to go beyond the bounds of what it provides me with. Invariably, this choice means that certain architectural decisions have been made for me, decisions that can't easily be changed or reversed by building on top of the stack; for some projects, those decisions are inappropriate, and thus I'll c
hoose differently; but those decisions have usually been made by some pretty smart people, and are generally sound ones in the contexts in which they are appropriate.

Hopefully I've managed to convey the high-level attraction I have to Mantissa and related software; please note that I'm not trying to justify the low-level decisions involved here. Want to argue about how PostgreSQL is a superior database solution than SQLite? Why per-user database are a bad idea? How "COMET" and web applications that require JS are ruining the web? Sure, I'd be glad to have those discussions, but the point of this post was not to defend the specifics; think of it more as a window onto the development world I immerse myself in most of the time; and hopefully a little of my enthusiasm has been passed on to you!

1 Comment :, , , , , , more...

Mantissa rocks

by on Jun.27, 2008

Divmod Mantissa is an "application server" which I've been basing almost all of my applications on for a while now; this IRC conversation snippet is pretty illustrative of why I think Mantissa (and the rest of the stack, Twisted etc.) rocks.

< glyph> oubiwann: I think I've put my finger on why I don't like the serious use of acronyms like AJAX and LAMP
< glyph> oubiwann: they're just shibboleths for much simpler concepts
< oubiwann> glyph: why's that?
< glyph> oubiwann: AJAX is the magic word that means "Hey guys! Check this out! Javascript works now!"
< glyph> LAMP is the magic words that means "Hey uh, did you guys notice that PHP is almost completely broken without being attached to a database on the back end and a web server on the front end?"
< glyph> "AJAX" apps are Javascript apps, "LAMP" apps are PHP apps
< glyph> there's no additional information conveyed by the acronym, because all the peripheral crap that is pulled in doesn't make a difference
< glyph> lots of "LAMP" applications actually use postgres, or run on freebsd
< glyph> oubiwann: the reason we had so much trouble with *MAP, META, etc, is that there's actually no peripheral crap in our universe
< glyph> it's all part of the stack ;)

Leave a Comment :, , , more...

Search

Loading