Shattered Crystalline Matrix

My home network has a somewhat complicated setup where I have multiple PPPoE sessions across my ADSL connection, with various different ISPs. This allows me to take advantage of varying ISP properties such as cost and latency, by routing different traffic over different connections. Naturally, each of these connections only affords me a single IPv4 address, so I make use of NAT to allow the rest of my network access to the Internet. A potential problem arises, however, when connections go down and come back up. In the simple case, with only one connection, MASQUERADE takes care of all the details; when the interface goes down, all of the NAT entries associated with the connection are removed, so when it comes back up, it’s not a problem that your IP address has changed, because all of the NAT entries associated with the old address are gone. This works just as well in the multiple connections scenario; if an interface goes down resulting in traffic being routed over another interface, all of the old NAT entries have been dropped, so new ones will be established associated with the interface they are now travelling over. The problem arises when the interface that went down comes back up; traffic will now be routed over the first interface again, while still being rewritten to the second interface’s address, and this traffic is almost guaranteed to be dropped by either your ISP, or their upstream provider.

What’s the solution? Well, if you absolutely definitely want to start routing traffic over the first interface as soon as it comes back up, you’re going to need to flush the associated conntrack NAT entries as soon as it comes up, and let all your users reconnect (since their connections will be interrupted); I’m not entirely sure how to do this. In my case, however, I’m more concerned with maintaining existing connections without interruption, even if that means continuing to route them over the “wrong” interface. This also applies to incoming connections; ordinarily if somebody tries to establish a connection to the public IP address of one of your connections, they will need to connect to the same interface that outbound traffic to them would be routed over, which can be somewhat inconvenient.

My solution is something I’m going to call “connection pinning”. The idea is that once an outbound interface has been selected for a particular connection (by the Linux routing table), we “pin” the connection to that interface, so that traffic associated with that connection always travels over that interface even if the routing table changes. In order to achieve this, we can use a combination of Linux policy routing (ip rule), as well as firewall / conntrack packet marking. When a connection is first established, we set a connmark, which is a value stored in the conntrack table entry for that connection. In the case of an incoming connection, we set the mark based on the interface the packet arrived on; in the case of an outgoing connection, we set the mark in POSTROUTING based on the outbound interface already selected by the routing table. Then, for future outgoing traffic associated with that connection (as determined by conntrack), we set an fwmark based on the connmark, and bypass the normal routing table using policy rules for traffic marked thusly.

This is implemented in three parts. Firewall rules added using iptables, for the netfilter/conntrack bits; an ip-up script for establishing policy rules and routes when a PPP connection is established; and an ip-down script for flushing them again when the PPP connection is terminated.

First, the firewall rules (using the excellent ferm tool):

@def $DEV_PRIVATE = eth0;
@def $NET_PRIVATE_V4 = 10.0.0.0/24;

domain ip table mangle {
    # Only match new connections; established connections should
    # already have a connmark, which should not be overwritten.
    chain (INPUT FORWARD) {
        # Unfortunately the set-mark rules need to be duplicated for
        # each ppp interface we have.
        mod conntrack ctstate NEW {
            interface ppp0 CONNMARK set-mark 1;
            interface ppp1 CONNMARK set-mark 2;
            interface ppp2 CONNMARK set-mark 3;
            interface ppp3 CONNMARK set-mark 4;
            interface ppp4 CONNMARK set-mark 5;
        }
    }
    chain POSTROUTING {
        mod conntrack ctstate NEW {
            outerface ppp0 CONNMARK set-mark 1;
            outerface ppp1 CONNMARK set-mark 2;
            outerface ppp2 CONNMARK set-mark 3;
            outerface ppp3 CONNMARK set-mark 4;
            outerface ppp4 CONNMARK set-mark 5;
        }
    }
    chain PREROUTING {
        # Copy the connmark to the fwmark in order to activate the
        # policy rules for connection pinning. Only do this for
        # traffic originating from the local network; other traffic
        # (such as traffic going *to* the local network) should be
        # left unmodified, to allow return traffic to be routed over
        # the correct interface.

        interface $DEV_PRIVATE daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
    }
    chain OUTPUT {
        # Same as above, but for locally originating traffic.

        daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
    }
}

# I am assuming you already have something like this:
domain ip table nat {
    chain POSTROUTING outerface (ppp0 ppp1 ppp2 ppp3 ppp4) MASQUERADE;
}

If you’re not using ferm, here’s what the raw iptables commands would be (these are exactly what ferm will install given the above, so this is just more verbose):

iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A PREROUTING --in-interface eth0 ! --destination 10.0.0.0/24 --jump CONNMARK --restore-mark
iptables -t mangle -A OUTPUT ! --destination 10.0.0.0/24 --jump CONNMARK --restore-mark

iptables -t nat -A POSTROUTING --out-interface ppp0 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp1 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp2 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp3 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp4 --jump MASQUERADE

Next, the ip-up script (to be placed in /etc/ppp/ip-up.d/ and made executable):

#!/bin/sh
TABLE="$PPP_IFACE"
MARK=$((${PPP_IFACE##ppp} + 1))
ip rule del lookup "$TABLE"
ip route flush table "$TABLE"
ip route add default dev "$PPP_IFACE" table "$TABLE"
ip rule add fwmark "$MARK" table "$TABLE"

Finally, the ip-down script (to be placed in /etc/ppp/ip-down.d/ and made executable):

#!/bin/sh
TABLE="$PPP_IFACE"
ip rule del lookup "$TABLE"
ip route flush table "$TABLE"

There are a couple of changes you will need to make to adapt these for your own network. In particular, you’ll need to duplicate the pppN iptables rules for each of the PPP interfaces you want to apply this to. Also, if you are already doing packet marking for some other reason, you’ll need to change the fwmark values I’ve used to ones that don’t interfere with your existing marks. I suspect there’s a better way to only mark outbound traffic than what I do above, but I wasn’t able to figure it out. If you have any improvements to suggest, feel free to mention them in the comments; I will try to keep this post updated with any improvements I make (either on my own, or based on other people’s suggestions).

My home network has a somewhat complicated setup where I have multiple PPPoE sessions across my ADSL connection, with various different ISPs. This allows me to take advantage of varying ISP properties such as cost and latency, by routing different traffic over different connections. Naturally, each of these connections only affords me a single ...

RFC 3749 defines a mechanism for compressing a TLS connection using the DEFLATE compression mechanism. When used in conjunction with https, this fills a similar role to that of Content-Encoding: gzip, except that headers benefit from compression too (as the whole connection is compressed), and I suspect there is less chance of weird proxy / caching bugs. I decided to do some quick tests to see which browsers actually support this, as I found approximately zero information on the subject on the internet; the results, unfortunately, are rather dismal and depressing:

• Chrome: supported (apparently since Chrome 9, but I only tested Chrome 15 on the dev channel)
• Firefox: not supported (tested 8.0 Aurora)
• Safari: not supported
• Internet Explorer 9 on Windows 7: not supported
• Android 2.3 default browser: not supported

RFC 3749 defines a mechanism for compressing a TLS connection using the DEFLATE compression mechanism. When used in conjunction with https, this fills a similar role to that of Content-Encoding: gzip, except that headers benefit from compression too (as the whole connection is compressed), and I suspect there is less chance of weird proxy ...

Introduction

In case you missed the news, Google Maps Navigation (Beta) for Mobile is now available in South Africa. I tried it out briefly, and thought I’d offer some thoughts on how it compares to the primary navigation software I use, Waze. Note that as an Android user (HTC Desire HD), this review will be fairly Android-specific. If you’re using an iPhone, please write your own review and let me know; if you’re using something else, please join the rest of us in the 21st century.

Integration

Navigation is part of the Google Maps application; as one of the core Google apps that virtually every Android phone ships with, it’s well integrated with the rest of the system, and in particular, with Maps / Places / People. Waze isn’t quite as well integrated; for example, there’s no easy way to grab a location out of  Places and have Waze navigate to it.

Voice prompts

These are generated entirely via TTS, using the system-configured speech engine. This is great for road names, not so great for actually being able to comprehend the prompts; I wish they had taken the Garmin approach, and only used TTS for the road names and such, not everything. By comparison, Waze only uses pre-recorded voice prompts, although they are currently testing out TTS functionality as well (I don’t have access to that, so I don’t yet know how it compares). As far as the actual prompting goes, they seem to be about equal in terms of usefulness.

Map data

This is a bit harder to quantify, as the quality and coverage of map data for both Waze and Google Maps varies drastically depending on where exactly you are in the country. In general, Google Maps has much more complete data; on the other hand, it tends to be several years or more out of date. In areas with active Area Managers on Waze, coverage is likely to be far more accurate, even going so far as to include temporary road detours during construction and so on. As such, your mileage may (and very likely will) vary.

Routing

Waze definitely wins this one, assuming you’re navigating in an area where the map is actually sufficiently complete to allow for sensible routing. Waze tracks the average travel time along each road segment and uses this as part of its routing calculations. In addition, if there is sufficient data, it seems that this will even be broken down by day of week / time of day, so Waze knows that what might be a crawling disaster at 4pm is actually smooth sailing at lunchtime. In addition, speed data is also handled in real-time; so if there’s a traffic jam right now, and some Waze users are stuck in it, it’ll detect that the average speed *right now* is much lower than it usually is, and route you around the problematic road segments if appropriate. Google Maps, by comparison, has a real-time traffic layer which can be used for routing decisions, but there currently seems to be no traffic data for South Africa, and I’m not sure if this information is used at all for long-term routing decisions. Even if it is, it’ll take a while for them to catch up with the existing data that has already been built up by Waze, so I guess we’ll have to see how that works out.

Display / UI

Google Maps wins this one. While driving, you can have the satellite layer displayed for the map, not just the road layer, which makes it a lot easier to match the map to what you’re seeing out of your windscreen, assuming the map isn’t horribly out of date. In addition, once you’re making your final approach to the destination, it will show you a Street View image of the destination, making it much easier to find the exact place you’re looking for, instead of trying to estimate distances on a map. By comparison, Waze offers only the usual abstract road map; this works, of course, but could be better.

Crowdsourcing

In addition to using average speed data collected from users for routing decisions, Waze also does things like adjusting road segment positions, and toggling road directions (1-way to 2-way and vice-versa) automatically, based on collected driving data. Occasionally this results in errors, but it mostly saves a lot of work on the part of map editors who would otherwise have to manually fix the map up. There’s also the ability to report accidents, traffic cameras, potholes, speed traps, roadblocks, and so on, which other users will be able to see and avoid. Google Maps doesn’t really have anything similar, other than the real-time traffic info.

Conclusion

I’ll be sticking with Waze, for now; the real-time and routing functionality, as well as the ability to fix up the map myself, easily makes up for any of the other disadvantages. If Google Maps grows to encompass the functionality and userbase of Waze, this would definitely tip the scales in its favour, but that doesn’t look like it’s going to happen any time soon.

Introduction In case you missed the news, Google Maps Navigation (Beta) for Mobile is now available in South Africa. I tried it out briefly, and thought I'd offer some thoughts on how it compares to the primary navigation software I use, Waze. Note that as an Android user (HTC Desire HD), this review will be ...

If you’re using an HTC Android (with HTC Sense, although I’m pretty sure they all have Sense) phone running Gingerbread (2.3.x), you may have noticed that the “Dialer” process seems to be inexplicably showing up at or near the top of your battery usage list. After seeing various people allude to the fact that the battery usage seems to be caused by sensors usage, I finally figured out what was causing it: In Settings -> Sound there is an option for “Pocket mode: Increase ring volume while in pocket or bag”. Having this active seems to keep the sensors in use all the time, presumably trying to determine whether the phone has been placed in your pocket / bag or not. After disabling this and rebooting (not sure why the reboot was required, but it seemed to make no difference before I rebooted), Dialer is now mostly gone from the battery usage list. Overall battery consumption doesn’t seem to be affected too much, but since I didn’t really find that feature to be very useful in the first place, I’ll probably just leave it off. In case it matters, my phone is the HTC Desire HD, but I suspect this will affect other Sense / Gingerbread phones in exactly the same way.

EDIT: Okay, this may be a red herring, and it’s just that the reboot fixes the problem; after receiving a phone call, the “problem” seems to be back. Apparently the latest HTC ROM base may fix the problem, but I haven’t had a chance to try it out yet.

If you're using an HTC Android (with HTC Sense, although I'm pretty sure they all have Sense) phone running Gingerbread (2.3.x), you may have noticed that the "Dialer" process seems to be inexplicably showing up at or near the top of your battery usage list. After seeing various people allude to the fact that ...

[NOTE: If you are reading this post in an RSS reader, the formatting on the code examples may not be optimal; please read this in a web browser with JavaScript enabled for the optimal viewing experience]

I often find myself helping newer Twisted users come to grips with arranging their flow control when they first start writing code that uses Deferreds. While the Deferred Reference does a reasonable job of covering all of the details, it is often difficult to make the intuitive leap from the synchronous patterns one is used to, to their asynchronous equivalents. To that end, I often find that comparing sync and async versions is illustrative; there are some examples of this nature in the Deferred Reference, but some patterns are missing, and I’ve never actually put all of the examples down in one place, so I thought I’d do that in my blog post. Without any further ado, here they are:

EDIT: Added composition example

Call a function, and use the result

# Synchronous version
result = getSomething()
doSomething(result)

# Asynchronous version
d = getSomething()
d.addCallback(doSomething)

Call a function and use the result, catching a particular exception

# Synchronous version
try:
    result = getSomething()
    doSomething(result)
except SomeException as e:
    handleError(e)

# Asynchronous version
def _eb(f):
    f.trap(SomeException)
    handleError(f)

d = getSomething()
d.addCallback(doSomething)
d.addErrback(_eb)

Call a function and use the result, catching any exception

# Synchronous version
try:
    result = getSomething()
    doSomething(result)
except:
    log.err()

# Asynchronous version
d = getSomething()
d.addCallback(doSomething)
d.addErrback(log.err)

Call a function and use the result, catching exceptions raised by that function

# Synchronous version
try:
    result = getSomething()
except:
    log.err()
else:
    doSomething(result)

# Asynchronous version
d = getSomething()
d.addCallbacks(doSomething, log.err)

Call a function and use the result, recovering from a particular exception raised by the function

# Synchronous version
try:
    result = getSomething()
except SomeException:
    result = 42
doSomething(result)

# Asynchronous version
def _eb(f):
    f.trap(SomeException)
    return 42

d = getSomething()
d.addErrback(_eb)
d.addCallback(doSomething)

Call a function and use the result, performing cleanup if an exception occurs

# Synchronous version
try:
    result = getSomething()
    doSomething(result)
finally:
    cleanStuffUp()

# Asynchronous version
d = getSomething()
d.addCallback(doSomething)
d.addBoth(lambda ignored: cleanStuffUp())

Compose several functions

# Synchronous version
result = getSomething()
result2 = doStuff(result)
result3 = doMoreStuff(result2)

# Asynchronous version
d = getSomething()
d.addCallback(doStuff)
d.addCallback(doMoreStuff)

If anyone has any suggestions for other examples I should add to this list, feel free to leave a comment or drop me a note, and I’ll consider updating the post.

I often find myself helping newer Twisted users come to grips with arranging their flow control when they first ...

Q: How do I restore an SVN working copy to pristine condition?

A: Use bzr:

svn revert -R . && bzr clean-tree --ignored --unknown

Q: How do I restore an SVN working copy to pristine condition? A: Use bzr:

I have finally gotten around to generating a new GPG key. See this signed message for details. My old key had exactly one signature from somebody that wasn’t me, and I’d like to do better with my new key; if you have the ability to verify my identity, please get in touch with me in order to arrange keysigning!

Is Mandela dead yet? Find out now!

Is Mandela dead yet? Find out now!

Jonathan just ran into this on his new HTC Desire (updated to 2.2 “Froyo”); I haven’t confirmed it on any other handset, but I imagine the problem is common to many HTC Sense + Android handsets. Anyhow, I figured I’d put this information out there to help anyone that runs into this issue. If anyone has a contact at HTC, please pass this along, so that they can look into a fix/workaround.

When “Facebook for HTC Sense” tries to connect to Facebook to retrieve contacts or profiles, the client sends Expect: 100-Continue in the HTTP header. Facebook’s HTTP servers do support 100 Continue, but some proxy servers (such as squid) do not. If you have configured your phone to use a proxy server, or are behind a transparent proxy (which many ISPs do inflict on their users), and the proxy server does not support this, it will (correctly) respond with 417 Expectation failed response. At this point, HTC’s client does not attempt to retry the request without requesting 100-Continue, but instead fails with an error message. This results in an empty Facebook directory / contacts list, and an error when you attempt to refresh it.

If the proxy server is on your own network, simply bypassing it should solve the issue for you. If this is your ISP’s proxy server, then you’ll either need to work around it somehow, or contact HTC and/or your ISP for assistance.

Jonathan just ran into this on his new HTC Desire (updated to 2.2 "Froyo"); I haven't confirmed it on any other handset, but I imagine the problem is common to many HTC Sense + Android handsets. Anyhow, I figured I'd put this information out there to help anyone that runs into this issue. If ...

I’m a little bored, and the whole internet (well, at least the portion of it composed of Desire owners) seems to be going crazy waiting around for the recently-announced rolling out of the Android 2.2 update for the HTC Desire, so I thought I’d poke around a little in the OTA update process.

UPDATE (2010/08/03): The update seems to be rolling out to most people, but some people still aren’t getting it, even though they have unlocked/unbranded phones. From what I’ve been able to discover, HTC appears to be employing an IMEI “blacklist” of some kind; checkins with specific IMEI values fail to return the 2.2 update, but anything else (including completely invalid IMEI values) returns the update. I’m still not sure whether this is just their way of staggering the update, or if there is some other reason for withholding the update from the affected devices.

UPDATE (2010/08/05): HTC now seems to either have a bad content server, or be in the process of pulling the update from their servers, including the update file itself on liveimage.htc.com. If it is deliberate, I have no idea why they’re doing it, and I can’t currently find any information that sheds any light.

The process seems to be relatively straightforward: when you hit “Check now”, the phone sends an HTTP POST to http://andchin.htc.com/android/checkin (not https, thus all in the clear, making it easy for me to sniff :P), with a Content-Type of “org/x-json” (wat), and a JSON-encoded payload. The payload looks as follows (I’ve pretty-printed it and stripped personal identifiers, but it’s still good enough to get a response):

{
  "locale": "en_GB",
  "checkin": {
    "build": {
      "product": "bravo",
      "serialno": "HTXXXXXXXXXX",
      "changelist": "174215",
      "build_type": "user",
      "carrier": "htc_wwe",
      "radio": "32.36.00.28U_4.06.00.02_2",
      "firmware_version": "1.21.405.2 CL174215 release-keys",
      "bootloader": "0.80.0000",
      "id": "htc_wwe/htc_bravo/bravo/bravo:2.1-update1/ERE27/174215:user/release-keys",
      "revision": "129"
    }
  },
  "model_number": "HTC Desire",
  "logging_id": 0,
  "imei": "",
  "id": 0,
  "digest": "8fccfd93dcbe837d072e5f1494da2"
}

Note the “carrier” value there; this is “htc_wwe” for phones that are not carrier-locked / otherwise carrier-customized. If you had a Sprint phone, for example, this would be “sprint”. I believe this value is ultimately determined by the the “androidboot.carrier” kernel parameter passed to the kernel by the boot loader.

I’m not sure what “digest” is of; it is 29 hex digits long, which doesn’t make any sense for a cryptographic digest. It doesn’t seem to be a personal identifier of any kind, though; if you modify it, you get some “upload crash” response back, which contains (among other things) the correct digest, so I can only guess that it corresponds to the product information in some way. I’ll be interested to see if it changes once I have the new update installed.

The response that comes back looks like this:

{"time_msec":"1280538364433","stats_ok":true,"intent":[{"action":"android.server.checkin.FOTA_CANCEL"}]}

The “intent” part will probably be obvious to Android developers (it’s specifying what the UI should do next), and time_msec is a UNIX timestamp in milliseconds, corresponding to the time the response was sent.

If an update *is* available, the response looks like this instead:

{
  "stats_ok": true,
  "intent": [
    {
      "action": "android.server.checkin.FOTA_UPDATE",
      "data_uri": "http://liveimage.htc.com/OTA_Bravo_HTC_WWE_1.21.405.2-1.15.405.4_release_P4ohim1mtp6zggcyon.zip",
      "extra": [
        {
          "name": "promptMessage",
          "value": "System Upgrade"
        },
        {
          "name": "promptFeature",
          "value": "Update:HTC application improvement"
        },
        {
          "name": "promptVersion",
          "value": "System upgrade1.21.405.2 (27.07 MB)We would recommend using a free Wi-Fi hotspot or an unlimited data plan to apply this update. If not, standard data connection charges may apply. Any questions? Contact us via http://www.htc.com/www/CA_Hotline.aspx"
        },
        {
          "name": "promptSize",
          "value": "27.07 MB"
        },
        {
          "name": "promptMinutes",
          "value": "30,..."
        },
        {
          "name": "timeoutSeconds",
          "value": "120"
        },
        {
          "name": "force_update",
          "value": "false"
        }
      ]
    }
  ],
  "time_msec": "1280513282008"
}

Now, it seems as if they are planning to do a rolling release of the 2.2 update, somehow. I assume this means that they will only be responding with the updated version to some requests, but I don’t know exactly what mechanism they’ll be using to do that. Oh well…

I'm a little bored, and the whole internet (well, at least the portion of it composed of Desire owners) seems to be going crazy waiting around for the recently-announced rolling out of the Android 2.2 update for the HTC Desire, so I thought I'd poke around a little in the OTA update process. UPDATE (2010/08/03): ...