Shattered Crystalline Matrix

I’m a little bored, and the whole internet (well, at least the portion of it composed of Desire owners) seems to be going crazy waiting around for the recently-announced rolling out of the Android 2.2 update for the HTC Desire, so I thought I’d poke around a little in the OTA update process.

The process seems to be relatively straightforward: when you hit “Check now”, the phone sends an HTTP POST to http://andchin.htc.com/android/checkin (not https, thus all in the clear, making it easy for me to sniff :P), with a Content-Type of “org/x-json” (wat), and a JSON-encoded payload. The payload looks as follows (I’ve pretty-printed it and stripped personal identifiers, but it’s still good enough to get a response):

{
  "locale": "en_GB",
  "checkin": {
    "build": {
      "product": "bravo",
      "serialno": "HTXXXXXXXXXX",
      "changelist": "174215",
      "build_type": "user",
      "carrier": "htc_wwe",
      "radio": "32.36.00.28U_4.06.00.02_2",
      "firmware_version": "1.21.405.2 CL174215 release-keys",
      "bootloader": "0.80.0000",
      "id": "htc_wwe/htc_bravo/bravo/bravo:2.1-update1/ERE27/174215:user/release-keys",
      "revision": "129"
    }
  },
  "model_number": "HTC Desire",
  "logging_id": 0,
  "imei": "",
  "id": 0,
  "digest": "8fccfd93dcbe837d072e5f1494da2"
}

Note the “carrier” value there; this is “htc_wwe” for phones that are not carrier-locked / otherwise carrier-customized. If you had a Sprint phone, for example, this would be “sprint”. I believe this value is ultimately determined by the the “androidboot.carrier” kernel parameter passed to the kernel by the boot loader.

I’m not sure what “digest” is of; it is 29 hex digits long, which doesn’t make any sense for a cryptographic digest. It doesn’t seem to be a personal identifier of any kind, though; if you modify it, you get some “upload crash” response back, which contains (among other things) the correct digest, so I can only guess that it corresponds to the product information in some way. I’ll be interested to see if it changes once I have the new update installed.

The response that comes back looks like this:

{"time_msec":"1280538364433","stats_ok":true,"intent":[{"action":"android.server.checkin.FOTA_CANCEL"}]}

The “intent” part will probably be obvious to Android developers (it’s specifying what the UI should do next), and time_msec is a UNIX timestamp in milliseconds, corresponding to the time the response was sent.

If an update *is* available, the response looks like this instead:

{
  "stats_ok": true,
  "intent": [
    {
      "action": "android.server.checkin.FOTA_UPDATE",
      "data_uri": "http://liveimage.htc.com/OTA_Bravo_HTC_WWE_1.21.405.2-1.15.405.4_release_P4ohim1mtp6zggcyon.zip",
      "extra": [
        {
          "name": "promptMessage",
          "value": "System Upgrade"
        },
        {
          "name": "promptFeature",
          "value": "Update:HTC application improvement"
        },
        {
          "name": "promptVersion",
          "value": "System upgrade1.21.405.2 (27.07 MB)We would recommend using a free Wi-Fi hotspot or an unlimited data plan to apply this update. If not, standard data connection charges may apply. Any questions? Contact us via http://www.htc.com/www/CA_Hotline.aspx"
        },
        {
          "name": "promptSize",
          "value": "27.07 MB"
        },
        {
          "name": "promptMinutes",
          "value": "30,..."
        },
        {
          "name": "timeoutSeconds",
          "value": "120"
        },
        {
          "name": "force_update",
          "value": "false"
        }
      ]
    }
  ],
  "time_msec": "1280513282008"
}

Now, it seems as if they are planning to do a rolling release of the 2.2 update, somehow. I assume this means that they will only be responding with the updated version to some requests, but I don’t know exactly what mechanism they’ll be using to do that. Oh well…

Encumbered forever by desire and ambition,
there’s a hunger still unsatisfied;
our weary eyes still stray to the horizon,
though down this road we’ve been so many times.

— Pink Floyd, High Hopes

Introduction

As some of you already know, I recently became the proud new owner of an HTC Desire. You can find the detailed specs elsewhere, but the phone is pretty much a Nexus One with an optical trackpad, real buttons, a little extra flash memory, and the HTC Sense software. This post isn’t exactly going to be an in-depth review, but I thought I’d give a brief rundown of my impressions, and also the apps I’m using on the phone at the moment.

Overall, I’m pretty happy with the phone. I can get 1-2 days of use between charges (depending on usage), which is good enough for my purposes, and that does include push email (quite a lot of email, in the case of my personal account) and syncing all sorts of stuff on a pretty regular schedule. The phone UI is generally way more responsive and snappy than my previous phones (Nokia E71, E65), and the touch screen works reasonably well. Also, I can actually get apps for a lot of the services I use now, whereas I couldn’t before (not many people are making Series 60 apps).

Input

The lack of a real keyboard turned out to be not as bad as I expected. Tap-typing with a single finger is not as fast as using both thumbs was on the E71, and holding the phone so that I can use both thumbs is somehow awkward enough that I never did it. Selecting predicted words is a lot easier with the touch screen, though (no need to scroll through the list). However, I am now using the Swype Beta, an alternative input method where you “swype” your finger through the letters of the word you want, without lifting it, along with a lot of other nice tweaks to the on-screen keyboard. It works pretty nicely; it doesn’t always select the right word, but you can finish your sentence (or paragraph) and then go back and select a different word where it made mistakes. If a word isn’t in the dictionary, or isn’t really a word at all, you can just tap it out; regular words will be added to the dictionary after you press space, but you can also add “non-words” like “http://” or “.co.za” for convenience. I’m fast enough on the Swype keyboard now that I can use it for taking notes during a conversation in real-time, which is great. Unfortunately, the beta is closed at the moment; if you didn’t have a chance to grab the app while it was open, you’ll have to wait around until they decide to let people have a crack at it again, or they open it up to everyone.

The multi-touch screen is great. A lot of apps (like Maps, or the browser) support the “pinch” and “spread” (what’s the real name for the opposite of pinch?) to zoom in and out, which easily lends itself to an “arbitrary precision” navigation style: zoom out to cover a lot of ground, then zoom back in to precisely select a button or other page element. The “momentum-based” scrolling mechanism (again, what the heck is this actually called?) is also used by most apps, and is similarly awesome. It sort of behaves like a globe: flick the screen to start scrolling; over time, it slowly slows down, but you can touch the screen to “catch” it once you get to where you want to be, or flick it again to keep going.

Multi-tasking

The Android approach to multi-tasking is awesome, and I kinda wish my desktop operating systems worked the same way. Under the hood, it’s basically just standard Linux processes being scheduled by the kernel, but with a bit of a twist. In terms of the UI, there’s no explicit process management. Some apps have an “exit” option, and you can burrow into the Applications list to “Force Stop” any app, but there isn’t even any built-in way to get a list of currently-running apps (although third-party apps for this purpose are available), simply because you don’t really need it. When the OS runs out of memory (think “OOM killer” if you’re familiar with Linux), it kills off the oldest apps to free up memory. In addition, provision is made for apps to save and restore their state. As a result, if you switch back and forth between apps, they’ll probably still be running, so switching will be immediate. If you open up an app that you haven’t used for a while, it may not be running anymore, so it’ll take a little longer (but usually not that much longer) for the app to start up again and restore the state. Most apps don’t do anything if they’re not in the foreground, so there’s no need to worry about wasting your battery running apps you’re not using. There are also “services”, which are for long-running background tasks that shouldn’t be killed off to reclaim memory; synchronization, for example.

Unfortunately, some people seem to be confused by all of this; as a result, there are a plethora of “advanced task killer” apps and suchlike on the Android market. These are all just about pointless; all you are likely to accomplish by running around killing random processes is breaking certain functionality like background synchronization that you turned on. If you want to stop your contacts syncing, for example, just go turn that off in the settings, instead of using an app that kills the process every time it starts up. Also, if an app really is draining your battery in the background, it’ll show up in the battery info screen, and you can then uninstall it or otherwise take steps to solve the problem. The only exception to the latter is built-in functionality that gets aggregated into the “Android Platform” entry; it may not be obvious what you need to disable or reconsider if “Android Platform” shows up as using too much juice.

Intents

Think of intents like file type associations on steroids. Instead of just “open this file which is of some type”, you also have “open this URL” and “dial this phone number” and “view these map coordinates” and “search for this text” and “display the home screen” and “the battery is running low” and just about anything else you can think of, including brand new intents created by third-party developers.

Since any application can register itself to handle an intent, this means that third-party applications can do anything from hooking in as a phone dialler (eg. a SIP client), to replacing the home screen; just about any aspect of the “default” OS software can be replaced. This can also be done selectively; for example, if you follow a link to a Google Maps URL (even from another web page in the browser), you’ll be given the option to open it in the “Maps” app as well as opening it in the browser.

The net result is that third-party software can replace and enhance the OS in a practically unlimited fashion; and different third-party apps can interoperate and integrate without a great deal of special effort being exerted on the part of the application developers.

GPS / Location

The Desire has A-GPS functionality. My E71 also supposedly had this functionality, but it never seemed to work; getting a GPS lock could take up to 5 minutes if I started the Ovi Maps app up in a different location to where I last closed it. On my Desire, under optimal conditions (outdoors, clear view), I can generally get a GPS lock in seconds, no matter how far I’ve moved since the last time I got a lock. I can even get a GPS lock in under 30 seconds standing indoors in the middle of my house! I guess this is probably a combination of a better A-GPS implementation and a better GPS receiver, but I’m definitely much happier with the results.

Battery lifetime

A few quick notes I wanted to make on battery lifetime. Firstly, the battery lifetime seems to go up after the first several charge-discharge cycles; I guess this probably applies to all batteries in this class, but I thought it was worth mentioning. Secondly, active wifi use generally uses less battery than an active cell data (GPRS/EDGE/3G/HSDPA) connection; but an idle wifi connection will use more battery than an idle cell data connection. You can adjust the wifi “sleep” policy to take advantage of this by disabling the wifi connection after a certain timeout, although you may wish to avoid the extra cell data charges. Also, if you are in an area with poor cell reception, the cell radio will use more battery as the transmit strength is increased in order to successfully transmit; this applies even if you are not making use of cell data.

My applications

You can see a list of all of the applications I have installed (thanks to AppBrain), but not all of them are worth mentioning, so I’ll run through the interesting ones here, with some commentary. Unfortunately paid apps are not currently available from the Android Market in South Africa, which some apps are either not available at all, while others are only available in free+ads form.

• Aldiko Book Reader
  A pretty decent eBook reader for ePub format books. It includes support for purchasing books from some sites (they have a mechanism whereby any online store can integrate their catalogue); otherwise you can download and import them from anywhere you can find them, or copy them onto the phone from your PC.
• Android Agenda Widget
  This is a ridiculously customizable home screen widget for viewing your calendar and tasks. It has support for “styles”, but you can also individually customize just about every aspect of the widget. It also provides widgets in almost every every possible size from 1×1 up to 4×4. It supports the various calendars (Google, HTC, etc.) as well as Astrid and some other tasks app that I can’t remember offhand.
• AppBrain App Market
  The name is a little misleading: this is not a separate market, but just a way to sync and share your installed apps list, as well as scheduling installs / uninstalls via the web interface (which will be performed the next time you sync using the phone app).
• Astrid Task/Todo List
  A nifty tasks list manager. You can schedule multiple reminders for tasks, tag them, track expected time / time spent, and sync with Remember The Milk.
• ASTRO File Manager
  Android does not come with any kind of file manager, although you won’t need one very often since apps tend to avoid explicit file management. Still, there is the occasional need to rename or move a file around; it can also be very useful to transfer files over the network, especially with the SMB module.
• Barcode Scanner
  Scan a variety of barcodes, including QR codes and UPC / EAN barcodes. Many sites provide download / market links for apps in the form of QR codes; scanning the code from another display is obviously easier than retyping the URL.
• Dropbox
  Nothing much to this app; it just lets you view / download files you have stored on Dropbox.
• Evernote
  Create and access your Evernote notes, including geotagging support.
• Foursquare
  Check in on Foursquare, get alerts when your friends check in nearby, view nearby venues, and search for venues.
• FoxyRing: Smart Ringtone
  I’m not actually using this app currently, but you may find it useful regardless. The app allows you to adjust your ringtone settings based on ambient noise levels, time of day, and location, as well as overriding them for selected contacts.
• Glympse
  Unlike Google Latitude, which shares your location to all of your contacts the whole time you have it active, Glympse provides more limited and granular sharing. When you create a “Glympse”, you specify a lifetime (eg. 30 minutes) after which the Glympse expires, and optionally start and destination locations; a unique code is then generated. You can then share the URL by text message or email, or you can post it on Facebook. The recipients can then view your location in realtime, until the Glympse expires, either when the specified time has elapsed, or by you manually expiring it at any time of your choosing.
• Google Goggles
  This app is described as a “visual search” application. You take a picture of anything, and the application attempts to recognise it. This might mean scanning a barcode / QR code, scanning and OCRing text (which you can then translate or search for), detecting a well-known landmark, a work of art, and more.
• GPS Status
  A simple app that allows you to visualize the GPS satellites in range, how many satellites you are locked onto, and see other information from the GPS receiver.
• Kindle for Android
  Amazon’s Kindle application. Great for reading Kindle books, but if you can get them in ePub, rather use Aldiko (see above), as the Kindle app is not nearly as featureful.
• Nimbuzz
  This is an IM / voice chat app. It has support for Skype, which is useful, as Fring no longer has Skype support, and the official Skype Android app is currently only available to Verizon subscribers through a special deal. It also supports a variety of IM networks, including Facebook, but unfortunately generic Jabber support is no longer available.
• Real Signal
  A simple app that gives you a configurable cell signal strength meter, as well as allowing you to see the actual numerical signal strength.
• Remember The Milk
  Only usable by RTM Pro subscribers (although you get a 14-day trial of Pro if you aren’t a subscriber). This lets you sync your RTM tasks to your phone, create tasks on your phone, and includes a widget for displaying tasks. Not strictly necessariy in conjuction with Astrid (see above), but I find it useful for managing additional task views (eg. my shopping list).
• ScrobbleDroid
  Just what it sounds like: scrobble your music activity to last.fm. Compatible with various music players.
• Shazam
  Want to know what song is currently playing? Capture an audio sample via the phone’s microphone with this app, and then search their online database to identify the song.
• Shopper
  This app overlaps with Google Goggles somewhat (see above). You can scan media (DVDs and CDs) or books by barcode or cover image, and then search for them online with Google Shopper; useful for doing some comparative shopping when you’re in a physical store.
• SpeedView
  A simple speedometer app that uses the GPS to track your speed.
• TripIt
  Sync your TripIt travel plans to your phone; allows for offline access, so you can get your important travel details even if you can’t get an internet connection from where you are.
• Wifi Analyzer
  Visualize the wifi networks in range, including the channel and signal strength. Useful for selecting an unused channel for your wifi network, or picking an internet access point.

People new to working with event-driven programming in general, and event loops in particular, often seem to have a hard time figuring out how to run their code when it needs to be run. For example, the event loop in Twisted is called the reactor (or perhaps more accurately, the event loop is part of the reactor). The reactor can be started and stopped, but the catch is that the current implementation only allows you to start one reactor once per process. Once you’ve stopped it, it can’t be started again. Now, this is really just an unfortunate consequence of two historical design artifacts; the reactor is a global singleton, so you can only have one, and it doesn’t manage the process-global state well enough to be able to start again once it has stopped. However, while it would be nice for those two issues to be fixed, the reality is that the usual answer to “how do I stop and start the reactor more than once?” is “you don’t need to, and you don’t want to.”

The idea of an event loop is, roughly, “don’t call us, we’ll call you”. When an event occurs, the event loop takes care of dispatching that event to the relevant code in order to handle the event. While there may be some layers of library / framework between your application code and the low-level event handler, ultimately it’ll usually be some part of your application code that gets invoked in response to the event. This means that instead of writing your program in the style of “do A, then do B, then do C”, you write it in a way that looks more like “when X happens, do A; when Y happens, do B; when Z happens, do C”.

So, how do you kick things off? If you fire up an event loop without doing anything else, generally no events will ever occur / be received, and so your program will just run forever doing absolutely nothing. Thus, in the startup phase of your program before you start the event loop, you will need to perform some operations that will cause events to be received later on. For example, an HTTP server would probably want to open a socket and start listening on port 80; alternatively, you might have some time-based event, along the lines of “run this code every 30 seconds”. The startup phase is not the only time that you can set your event sources up; you can also do this in any of your application code that handles other events that occur. For example, your web server might open a connection to a database server when it receives a request for a certain URL.

That’s really all there is to it; figure out which event sources you need to set up intially before you run the event loop, and figure out what other events will occur during the lifetime of your application that would require setting up additional event sources, no matter what those event sources might be (making connections, starting timers, completion of a prior operation, etc). While this post may be slightly Twisted-specific, the concepts are pretty general: any decent event loop will necessarily allow both cases (initial setup, and additional operations while the event loop is running). The key is to think of your application as code run in response to events, not just a linear series of operations.

This isn’t meant to last,
this is for right now.
— Nine Inch Nails, Last

Spent several hours this morning importing my old old blog posts into WordPress; the import is now complete. Early on in the life of the blog, I began generating an Atom feed directly from my hand-written XHTML; later on, I changed my markup to use the hAtom microformat, and ditched my custom transform in favour of hAtom2Atom. I used this again to run a conversion over each month’s posts, although I had to edit the oldest into hAtom form since they were using a custom class schema; just a trivial matter of assigning the correct classes. I then had to remove the doctype declaration from each page, because Saxon was failing to fetch the XHTMTL 1.1 DTD for some reason; since the DTD was completely unnecessary for this transform, I decided it would be easier to just ditch it.

Now that I had the ability to generate an Atom feed for each page, the next trick was importing this into WordPress somehow. WordPress supports importing an RSS 2.0 feed, so I tracked down another transform to convert the Atom to RSS 2.0: atom2rss-exslt.xsl. After hacking it slightly to run on Saxon 6 (the decode-uri function detection doesn’t work, since it checks for a later version of the Saxon processor), I had what looked like good RSS 2.0 output, which I imported. Unfortunately, this didn’t work out so well; the tags weren’t imported, and WordPress inserts a <br> tag for each newline; since the output of the transform had a bunch of extraneous newlines, this meant that my posts were now littered with extraneous line breaks. I edited the transform to get rid of the newlines, but still wasn’t very happy with this.

So… plan B! WordPress has an import/export format called WordPress eXtended RSS; basically, RSS 2.0 plus a whole whack of custom WordPress extension elements. I spent around an hour hacking the transform to generate WXR output. This was even more painful than it sounds; for example, encoded content is “supported” by stripping any CDATA section start/end markers, and then importing the content as-is. Even if there wasn’t a CDATA section. I guess they only care about reading their own output.

As a final touch, I hacked the transform a little more to insert an invisible anchor tag at the beginning of each post so that even my old permalinks will work. This was fortunately quite easy to do, since my old URL scheme was year/month, which happens to be supported by WordPress too; I just needed the anchors so that you would get taken to the correct post.