(Fair) queuing, buffering, and "bufferbloat"

This is basically just a teaser, but I’ve been playing around with things on my home internet connection (PPPoE over ADSL) and have achieved some fairly good results (within the limitations of my internet connection). I’ve learned quite a lot about various bits and pieces along the way, so I decided I should blog it all for my own future reference as well as others. I’ll probably be duplicating some existing material along the way, but half of the challenge was just finding all the bits and pieces scattered around and putting them all together, so hopefully I’ll save somebody else some time in the future. I’ll also be uploading my actual test scripts, as well as the test results, for reference.

Debian Developer

I’m now a Debian Developer. Woot! And it only took me just 6 years and 17 days… although the overwhelming majority of the delay was due to my own lack of time / energy, rather than anyone else’s fault. On a related matter, my newish PGP key is sadly lacking in signatures. If you’re in or travelling to Johannesburg / Gauteng area in South Africa and want to exchange signatures, please get in touch (especially if you’re a Debian Developer or Debian Maintainer, but even if you’re not).

An introduction to Mantissa (part 3): Navigation powerups and other friends

This is the fourth post in a series of articles about Mantissa.

In the previous article I described how an offering can provide powerups to be included in a product, which will then be installed on a user store; in this installment, I will discuss what form these powerups can actually take, and how they allow you to expose your application's functionality to the user.

One of the most commonly-implemented powerup interfaces in Mantissa is INavigableElement. Mantissa has a somewhat generalized idea of "navigation", whereby a nested menu structure can be defined through INavigableElement powerups, and then displayed by different implementations for different protocols; for example, the web view has a visual dropdown menu system, whereas the SSH server presents a textual menu system. A typical INavigableElement powerup implementation will look something like this:

pony_navigable.py (Source)

from zope.interface import implements
from axiom.item import Item
from axiom.attributes import integer
from xmantissa.ixmantissa import INavigableElement
from xmantissa.webnav import Tab

class PonyCreator(Item):
    Powerup for creating and managing ponies.
    powerupInterfaces = [INavigableElement]

    ponyQuota = integer(allowNone=False, default=10)

    def getTabs(self):
        return [Tab('ZOMG PONIES!', self.storeID, 1.0)]

INavigableElement only has one method, getTabs, which returns a list of "tabs" or menu items to be presented in the nav. The primary components of a tab are a title (which is how the item is displayed in the UI), the storeID of an item in the same store which the tab points to, and a float between 0.0 and 1.0 indicating the sort priority of the tab (higher values sort sooner). In this case, we have the tab pointing directly at the PonyCreator item itself; in order for this to work, we'll need some extra code to allow PonyCreator to be exposed via the web.

In order for an item in a user's store to be privately accessible via the web by that user, it needs to be adaptable to the (somewhat poorly-named) INavigableFragment interface. This is almost always done by defining an adapter from the item type to INavigableFragment:

pony_view.py (Source)

from twisted.python.components import registerAdapter
from nevow.athena import LiveElement
from xmantissa.webtheme import ThemedDocumentFactory
from xmantissa.ixmantissa import INavigableFragment, ITemplateNameResolver

class PonyCreatorView(LiveElement):
    Web view for Pony management.

    title = u'Pony management'
    docFactory = ThemedDocumentFactory('pony-creator', 'resolver')

    def __init__(self, ponyCreator):
        super(PonyCreatorView, self).__init__()
        self.ponyCreator = ponyCreator
        self.resolver = ITemplateNameResolver(self.ponyCreator.store.parent)

registerAdapter(PonyCreatorView, PonyCreator, INavigableFragment)

Our element will be wrapped in the Mantissa shell when it is rendered, so we cannot control the page title directly from the template, but the title attribute provides a way for our element to specify the page title. ThemedDocumentFactory is used to retrieve the template through the theme system; the arguments are the name of the template ('pony-creator') and the name of the attribute holding the ITemplateNameResolver implementation used to retrieve the template. This attribute is set in __init__ using a slightly awkward method; the template resolver should really be passed in by Mantissa somehow, but currently there is no mechanism for doing this, so instead we retrieve the default resolver ourselves from the site store.

This is all that is needed for hooking some code up to the web view; any further UI behaviour would be implemented in HTML / JavaScript in PonyCreatorView, usually by invoking additional methods defined on PonyCreator.

Next up: Sharing, or "How do I publish public / shared content?"

NAT connection pinning with iproute2 / iptables

My home network has a somewhat complicated setup where I have multiple PPPoE sessions across my ADSL connection, with various different ISPs. This allows me to take advantage of varying ISP properties such as cost and latency, by routing different traffic over different connections. Naturally, each of these connections only affords me a single IPv4 address, so I make use of NAT to allow the rest of my network access to the Internet. A potential problem arises, however, when connections go down and come back up. In the simple case, with only one connection, MASQUERADE takes care of all the details; when the interface goes down, all of the NAT entries associated with the connection are removed, so when it comes back up, it’s not a problem that your IP address has changed, because all of the NAT entries associated with the old address are gone. This works just as well in the multiple connections scenario; if an interface goes down resulting in traffic being routed over another interface, all of the old NAT entries have been dropped, so new ones will be established associated with the interface they are now travelling over. The problem arises when the interface that went down comes back up; traffic will now be routed over the first interface again, while still being rewritten to the second interface’s address, and this traffic is almost guaranteed to be dropped by either your ISP, or their upstream provider.

What’s the solution? Well, if you absolutely definitely want to start routing traffic over the first interface as soon as it comes back up, you’re going to need to flush the associated conntrack NAT entries as soon as it comes up, and let all your users reconnect (since their connections will be interrupted); I’m not entirely sure how to do this. In my case, however, I’m more concerned with maintaining existing connections without interruption, even if that means continuing to route them over the “wrong” interface. This also applies to incoming connections; ordinarily if somebody tries to establish a connection to the public IP address of one of your connections, they will need to connect to the same interface that outbound traffic to them would be routed over, which can be somewhat inconvenient.

My solution is something I’m going to call “connection pinning”. The idea is that once an outbound interface has been selected for a particular connection (by the Linux routing table), we “pin” the connection to that interface, so that traffic associated with that connection always travels over that interface even if the routing table changes. In order to achieve this, we can use a combination of Linux policy routing (ip rule), as well as firewall / conntrack packet marking. When a connection is first established, we set a connmark, which is a value stored in the conntrack table entry for that connection. In the case of an incoming connection, we set the mark based on the interface the packet arrived on; in the case of an outgoing connection, we set the mark in POSTROUTING based on the outbound interface already selected by the routing table. Then, for future outgoing traffic associated with that connection (as determined by conntrack), we set an fwmark based on the connmark, and bypass the normal routing table using policy rules for traffic marked thusly.

This is implemented in three parts. Firewall rules added using iptables, for the netfilter/conntrack bits; an ip-up script for establishing policy rules and routes when a PPP connection is established; and an ip-down script for flushing them again when the PPP connection is terminated.

First, the firewall rules (using the excellent ferm tool):

 1 @def $DEV_PRIVATE = eth0;
 2 @def $NET_PRIVATE_V4 =;
 4 domain ip table mangle {
 5     # Only match new connections; established connections should
 6     # already have a connmark, which should not be overwritten.
 7     chain (INPUT FORWARD) {
 8         # Unfortunately the set-mark rules need to be duplicated for
 9         # each ppp interface we have.
10         mod conntrack ctstate NEW {
11             interface ppp0 CONNMARK set-mark 1;
12             interface ppp1 CONNMARK set-mark 2;
13             interface ppp2 CONNMARK set-mark 3;
14             interface ppp3 CONNMARK set-mark 4;
15             interface ppp4 CONNMARK set-mark 5;
16         }
17     }
18     chain POSTROUTING {
19         mod conntrack ctstate NEW {
20             outerface ppp0 CONNMARK set-mark 1;
21             outerface ppp1 CONNMARK set-mark 2;
22             outerface ppp2 CONNMARK set-mark 3;
23             outerface ppp3 CONNMARK set-mark 4;
24             outerface ppp4 CONNMARK set-mark 5;
25         }
26     }
27     chain PREROUTING {
28         # Copy the connmark to the fwmark in order to activate the
29         # policy rules for connection pinning. Only do this for
30         # traffic originating from the local network; other traffic
31         # (such as traffic going *to* the local network) should be
32         # left unmodified, to allow return traffic to be routed over
33         # the correct interface.
35         interface $DEV_PRIVATE daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
36     }
37     chain OUTPUT {
38         # Same as above, but for locally originating traffic.
40         daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
41     }
42 }
44 # I am assuming you already have something like this:
45 domain ip table nat {
46     chain POSTROUTING outerface (ppp0 ppp1 ppp2 ppp3 ppp4) MASQUERADE;
47 }

If you’re not using ferm, here’s what the raw iptables commands would be (these are exactly what ferm will install given the above, so this is just more verbose):

 1 iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
 2 iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
 3 iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
 4 iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
 5 iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
 6 iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
 7 iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
 8 iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
 9 iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
10 iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
11 iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp0 --jump CONNMARK --set-mark 1
12 iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp1 --jump CONNMARK --set-mark 2
13 iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp2 --jump CONNMARK --set-mark 3
14 iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp3 --jump CONNMARK --set-mark 4
15 iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp4 --jump CONNMARK --set-mark 5
16 iptables -t mangle -A PREROUTING --in-interface eth0 ! --destination --jump CONNMARK --restore-mark
17 iptables -t mangle -A OUTPUT ! --destination --jump CONNMARK --restore-mark
19 iptables -t nat -A POSTROUTING --out-interface ppp0 --jump MASQUERADE
20 iptables -t nat -A POSTROUTING --out-interface ppp1 --jump MASQUERADE
21 iptables -t nat -A POSTROUTING --out-interface ppp2 --jump MASQUERADE
22 iptables -t nat -A POSTROUTING --out-interface ppp3 --jump MASQUERADE
23 iptables -t nat -A POSTROUTING --out-interface ppp4 --jump MASQUERADE

Next, the ip-up script (to be placed in /etc/ppp/ip-up.d/ and made executable):

1 #!/bin/sh
3 MARK=$((${PPP_IFACE##ppp} + 1))
4 ip rule del lookup "$TABLE"
5 ip route flush table "$TABLE"
6 ip route add default dev "$PPP_IFACE" table "$TABLE"
7 ip rule add fwmark "$MARK" table "$TABLE"

Finally, the ip-down script (to be placed in /etc/ppp/ip-down.d/ and made executable):

1 #!/bin/sh
3 ip rule del lookup "$TABLE"
4 ip route flush table "$TABLE"

There are a couple of changes you will need to make to adapt these for your own network. In particular, you’ll need to duplicate the pppN iptables rules for each of the PPP interfaces you want to apply this to. Also, if you are already doing packet marking for some other reason, you’ll need to change the fwmark values I’ve used to ones that don’t interfere with your existing marks. I suspect there’s a better way to only mark outbound traffic than what I do above, but I wasn’t able to figure it out. If you have any improvements to suggest, feel free to mention them in the comments; I will try to keep this post updated with any improvements I make (either on my own, or based on other people’s suggestions).

Browser support for RFC 3749

RFC 3749 defines a mechanism for compressing a TLS connection using the DEFLATE compression mechanism. When used in conjunction with https, this fills a similar role to that of Content-Encoding: gzip, except that headers benefit from compression too (as the whole connection is compressed), and I suspect there is less chance of weird proxy / caching bugs. I decided to do some quick tests to see which browsers actually support this, as I found approximately zero information on the subject on the internet; the results, unfortunately, are rather dismal and depressing:

  • Chrome: supported (apparently since Chrome 9, but I only tested Chrome 15 on the dev channel)
  • Firefox: not supported (tested 8.0 Aurora)
  • Safari: not supported
  • Internet Explorer 9 on Windows 7: not supported
  • Android 2.3 default browser: not supported

Google Maps Navigation, South Africa: a review


In case you missed the news, Google Maps Navigation (Beta) for Mobile is now available in South Africa. I tried it out briefly, and thought I’d offer some thoughts on how it compares to the primary navigation software I use, Waze. Note that as an Android user (HTC Desire HD), this review will be fairly Android-specific. If you’re using an iPhone, please write your own review and let me know; if you’re using something else, please join the rest of us in the 21st century.


Navigation is part of the Google Maps application; as one of the core Google apps that virtually every Android phone ships with, it’s well integrated with the rest of the system, and in particular, with Maps / Places / People. Waze isn’t quite as well integrated; for example, there’s no easy way to grab a location out of  Places and have Waze navigate to it.

Voice prompts

These are generated entirely via TTS, using the system-configured speech engine. This is great for road names, not so great for actually being able to comprehend the prompts; I wish they had taken the Garmin approach, and only used TTS for the road names and such, not everything. By comparison, Waze only uses pre-recorded voice prompts, although they are currently testing out TTS functionality as well (I don’t have access to that, so I don’t yet know how it compares). As far as the actual prompting goes, they seem to be about equal in terms of usefulness.

Map data

This is a bit harder to quantify, as the quality and coverage of map data for both Waze and Google Maps varies drastically depending on where exactly you are in the country. In general, Google Maps has much more complete data; on the other hand, it tends to be several years or more out of date. In areas with active Area Managers on Waze, coverage is likely to be far more accurate, even going so far as to include temporary road detours during construction and so on. As such, your mileage may (and very likely will) vary.


Waze definitely wins this one, assuming you’re navigating in an area where the map is actually sufficiently complete to allow for sensible routing. Waze tracks the average travel time along each road segment and uses this as part of its routing calculations. In addition, if there is sufficient data, it seems that this will even be broken down by day of week / time of day, so Waze knows that what might be a crawling disaster at 4pm is actually smooth sailing at lunchtime. In addition, speed data is also handled in real-time; so if there’s a traffic jam right now, and some Waze users are stuck in it, it’ll detect that the average speed *right now* is much lower than it usually is, and route you around the problematic road segments if appropriate. Google Maps, by comparison, has a real-time traffic layer which can be used for routing decisions, but there currently seems to be no traffic data for South Africa, and I’m not sure if this information is used at all for long-term routing decisions. Even if it is, it’ll take a while for them to catch up with the existing data that has already been built up by Waze, so I guess we’ll have to see how that works out.

Display / UI

Google Maps wins this one. While driving, you can have the satellite layer displayed for the map, not just the road layer, which makes it a lot easier to match the map to what you’re seeing out of your windscreen, assuming the map isn’t horribly out of date. In addition, once you’re making your final approach to the destination, it will show you a Street View image of the destination, making it much easier to find the exact place you’re looking for, instead of trying to estimate distances on a map. By comparison, Waze offers only the usual abstract road map; this works, of course, but could be better.


In addition to using average speed data collected from users for routing decisions, Waze also does things like adjusting road segment positions, and toggling road directions (1-way to 2-way and vice-versa) automatically, based on collected driving data. Occasionally this results in errors, but it mostly saves a lot of work on the part of map editors who would otherwise have to manually fix the map up. There’s also the ability to report accidents, traffic cameras, potholes, speed traps, roadblocks, and so on, which other users will be able to see and avoid. Google Maps doesn’t really have anything similar, other than the real-time traffic info.


I’ll be sticking with Waze, for now; the real-time and routing functionality, as well as the ability to fix up the map myself, easily makes up for any of the other disadvantages. If Google Maps grows to encompass the functionality and userbase of Waze, this would definitely tip the scales in its favour, but that doesn’t look like it’s going to happen any time soon.

Mysterious Dialer battery usage

If you’re using an HTC Android (with HTC Sense, although I’m pretty sure they all have Sense) phone running Gingerbread (2.3.x), you may have noticed that the “Dialer” process seems to be inexplicably showing up at or near the top of your battery usage list. After seeing various people allude to the fact that the battery usage seems to be caused by sensors usage, I finally figured out what was causing it: In Settings -> Sound there is an option for “Pocket mode: Increase ring volume while in pocket or bag”. Having this active seems to keep the sensors in use all the time, presumably trying to determine whether the phone has been placed in your pocket / bag or not. After disabling this and rebooting (not sure why the reboot was required, but it seemed to make no difference before I rebooted), Dialer is now mostly gone from the battery usage list. Overall battery consumption doesn’t seem to be affected too much, but since I didn’t really find that feature to be very useful in the first place, I’ll probably just leave it off. In case it matters, my phone is the HTC Desire HD, but I suspect this will affect other Sense / Gingerbread phones in exactly the same way.

EDIT: Okay, this may be a red herring, and it’s just that the reboot fixes the problem; after receiving a phone call, the “problem” seems to be back. Apparently the latest HTC ROM base may fix the problem, but I haven’t had a chance to try it out yet.

Flow control with Deferreds, by example

[NOTE: If you are reading this post in an RSS reader, the formatting on the code examples may not be optimal; please read this in a web browser with JavaScript enabled for the optimal viewing experience]


I often find myself helping newer Twisted users come to grips with arranging their flow control when they first start writing code that uses Deferreds. While the Deferred Reference does a reasonable job of covering all of the details, it is often difficult to make the intuitive leap from the synchronous patterns one is used to, to their asynchronous equivalents. To that end, I often find that comparing sync and async versions is illustrative; there are some examples of this nature in the Deferred Reference, but some patterns are missing, and I've never actually put all of the examples down in one place, so I thought I'd do that in my blog post. Without any further ado, here they are:

EDIT: Added composition example

Call a function, and use the result

# Synchronous version
result = getSomething()

# Asynchronous version
d = getSomething()

Call a function and use the result, catching a particular exception

# Synchronous version
    result = getSomething()
except SomeException as e:

# Asynchronous version
def _eb(f):

d = getSomething()

Call a function and use the result, catching any exception

# Synchronous version
    result = getSomething()

# Asynchronous version
d = getSomething()

Call a function and use the result, catching exceptions raised by that function

# Synchronous version
    result = getSomething()

# Asynchronous version
d = getSomething()
d.addCallbacks(doSomething, log.err)

Call a function and use the result, recovering from a particular exception raised by the function

# Synchronous version
    result = getSomething()
except SomeException:
    result = 42

# Asynchronous version
def _eb(f):
    return 42

d = getSomething()

Call a function and use the result, performing cleanup if an exception occurs

# Synchronous version
    result = getSomething()

# Asynchronous version
d = getSomething()
d.addBoth(lambda ignored: cleanStuffUp())

Compose several functions

# Synchronous version
result = getSomething()
result2 = doStuff(result)
result3 = doMoreStuff(result2)

# Asynchronous version
d = getSomething()

If anyone has any suggestions for other examples I should add to this list, feel free to leave a comment or drop me a note, and I'll consider updating the post.

Quick SVN Tip

Q: How do I restore an SVN working copy to pristine condition?

A: Use bzr:

[plain]svn revert -R . && bzr clean-tree –ignored –unknown[/plain]