Shattered Crystalline Matrix

My home network has a somewhat complicated setup where I have multiple PPPoE sessions across my ADSL connection, with various different ISPs. This allows me to take advantage of varying ISP properties such as cost and latency, by routing different traffic over different connections. Naturally, each of these connections only affords me a single IPv4 address, so I make use of NAT to allow the rest of my network access to the Internet. A potential problem arises, however, when connections go down and come back up. In the simple case, with only one connection, MASQUERADE takes care of all the details; when the interface goes down, all of the NAT entries associated with the connection are removed, so when it comes back up, it’s not a problem that your IP address has changed, because all of the NAT entries associated with the old address are gone. This works just as well in the multiple connections scenario; if an interface goes down resulting in traffic being routed over another interface, all of the old NAT entries have been dropped, so new ones will be established associated with the interface they are now travelling over. The problem arises when the interface that went down comes back up; traffic will now be routed over the first interface again, while still being rewritten to the second interface’s address, and this traffic is almost guaranteed to be dropped by either your ISP, or their upstream provider.

What’s the solution? Well, if you absolutely definitely want to start routing traffic over the first interface as soon as it comes back up, you’re going to need to flush the associated conntrack NAT entries as soon as it comes up, and let all your users reconnect (since their connections will be interrupted); I’m not entirely sure how to do this. In my case, however, I’m more concerned with maintaining existing connections without interruption, even if that means continuing to route them over the “wrong” interface. This also applies to incoming connections; ordinarily if somebody tries to establish a connection to the public IP address of one of your connections, they will need to connect to the same interface that outbound traffic to them would be routed over, which can be somewhat inconvenient.

My solution is something I’m going to call “connection pinning”. The idea is that once an outbound interface has been selected for a particular connection (by the Linux routing table), we “pin” the connection to that interface, so that traffic associated with that connection always travels over that interface even if the routing table changes. In order to achieve this, we can use a combination of Linux policy routing (ip rule), as well as firewall / conntrack packet marking. When a connection is first established, we set a connmark, which is a value stored in the conntrack table entry for that connection. In the case of an incoming connection, we set the mark based on the interface the packet arrived on; in the case of an outgoing connection, we set the mark in POSTROUTING based on the outbound interface already selected by the routing table. Then, for future outgoing traffic associated with that connection (as determined by conntrack), we set an fwmark based on the connmark, and bypass the normal routing table using policy rules for traffic marked thusly.

This is implemented in three parts. Firewall rules added using iptables, for the netfilter/conntrack bits; an ip-up script for establishing policy rules and routes when a PPP connection is established; and an ip-down script for flushing them again when the PPP connection is terminated.

First, the firewall rules (using the excellent ferm tool):

@def $DEV_PRIVATE = eth0;
@def $NET_PRIVATE_V4 = 10.0.0.0/24;

domain ip table mangle {
    # Only match new connections; established connections should
    # already have a connmark, which should not be overwritten.
    chain (INPUT FORWARD) {
        # Unfortunately the set-mark rules need to be duplicated for
        # each ppp interface we have.
        mod conntrack ctstate NEW {
            interface ppp0 CONNMARK set-mark 1;
            interface ppp1 CONNMARK set-mark 2;
            interface ppp2 CONNMARK set-mark 3;
            interface ppp3 CONNMARK set-mark 4;
            interface ppp4 CONNMARK set-mark 5;
        }
    }
    chain POSTROUTING {
        mod conntrack ctstate NEW {
            outerface ppp0 CONNMARK set-mark 1;
            outerface ppp1 CONNMARK set-mark 2;
            outerface ppp2 CONNMARK set-mark 3;
            outerface ppp3 CONNMARK set-mark 4;
            outerface ppp4 CONNMARK set-mark 5;
        }
    }
    chain PREROUTING {
        # Copy the connmark to the fwmark in order to activate the
        # policy rules for connection pinning. Only do this for
        # traffic originating from the local network; other traffic
        # (such as traffic going *to* the local network) should be
        # left unmodified, to allow return traffic to be routed over
        # the correct interface.

        interface $DEV_PRIVATE daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
    }
    chain OUTPUT {
        # Same as above, but for locally originating traffic.

        daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
    }
}

# I am assuming you already have something like this:
domain ip table nat {
    chain POSTROUTING outerface (ppp0 ppp1 ppp2 ppp3 ppp4) MASQUERADE;
}

If you’re not using ferm, here’s what the raw iptables commands would be (these are exactly what ferm will install given the above, so this is just more verbose):

iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A PREROUTING --in-interface eth0 ! --destination 10.0.0.0/24 --jump CONNMARK --restore-mark
iptables -t mangle -A OUTPUT ! --destination 10.0.0.0/24 --jump CONNMARK --restore-mark

iptables -t nat -A POSTROUTING --out-interface ppp0 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp1 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp2 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp3 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp4 --jump MASQUERADE

Next, the ip-up script (to be placed in /etc/ppp/ip-up.d/ and made executable):

#!/bin/sh
TABLE="$PPP_IFACE"
MARK=$((${PPP_IFACE##ppp} + 1))
ip rule del lookup "$TABLE"
ip route flush table "$TABLE"
ip route add default dev "$PPP_IFACE" table "$TABLE"
ip rule add fwmark "$MARK" table "$TABLE"

Finally, the ip-down script (to be placed in /etc/ppp/ip-down.d/ and made executable):

#!/bin/sh
TABLE="$PPP_IFACE"
ip rule del lookup "$TABLE"
ip route flush table "$TABLE"

There are a couple of changes you will need to make to adapt these for your own network. In particular, you’ll need to duplicate the pppN iptables rules for each of the PPP interfaces you want to apply this to. Also, if you are already doing packet marking for some other reason, you’ll need to change the fwmark values I’ve used to ones that don’t interfere with your existing marks. I suspect there’s a better way to only mark outbound traffic than what I do above, but I wasn’t able to figure it out. If you have any improvements to suggest, feel free to mention them in the comments; I will try to keep this post updated with any improvements I make (either on my own, or based on other people’s suggestions).

Thanks to some trailblazing by Jonathan, I was able to set this up quickly the other day, and I thought I’d lay out the process to make it easier for anyone else trying to do the same thing.

First up, we need a netflow emitter and collector:
# aptitude install fprobe-ulog nfdump

Note that the default collector address/port in the fprobe-ulog configuration does not match the default port for nfcapd; you’ll probably want to enter “localhost:9995″ when prompted by debconf. Next, we’ll set nfcapd up by editing /etc/default/nfdump to look like this:

DAEMON_ARGS="-D -b 127.0.0.1 -l $DATA_BASE_DIR -S 1 -P $PIDFILE"
nfcapd_start=yes

We override the default DAEMON_ARGS to add some important options. -b 127.0.0.1 ensures that we only bind to localhost; -S 1 selects the year/month/day subdirectory hierarchy for the data files that get written out, which makes it easier to select a particular date range when using nfdump later.

Next, we need to add iptables rules to send traffic to ULOG so that fprobe-ulog can process it. Exactly how you accomplish this depends on what you’re using to set up your firewall rules, and what traffic you want to monitor; assuming everything, you would want rules like these:

# iptables -A INPUT -j ULOG --ulog-qthreshold 50 --ulog-cprange 48
# iptables -A FORWARD -j ULOG --ulog-qthreshold 50 --ulog-cprange 48
# iptables -A OUTPUT -j ULOG --ulog-qthreshold 50 --ulog-cprange 48

You should probably put them right at the end of the INPUT/FORWARD/OUTPUT chains, or wherever you ACCEPT traffic.

Make sure both daemons are started:
# invoke-rc.d fprobe-ulog start
# invoke-rc.d nfdump start

Now your flows are hopefully being logged. The data will only actually be written out to disk every 5 minutes, or so; grab a cup of coffee, and then try this command when you get back:

# nfdump -R /var/cache/nfdump -s port/bytes

You should get a list of the top 10 flows grouped by port, and sorted by bytes transferred. If there aren’t any results at all, make sure that your ULOG rules are actually running, by inspecting the counters in the output of iptables -nvL.

Hooray, you’re done! Read nfdump(1) for more information, including how to specify filters; you now have the ability to drill down into your traffic data in a very versatile way.

EDIT: Remove reference to lenny; I actually did this on sid.