Stefano Rivera (Posts about split-routing)

Split-Routing on Debian/Ubuntu

Stefano Rivera — Fri, 19 Sep 2008 15:13:58 GMT

My post on split-routing on OpenWRT has been incredibly popular, and led to many people implementing split-routing, whether or not they had OpenWRT. While it's fun to have an exercise as a reader, it led to me having to help lots of newbies through porting that setup to a Debian / Ubuntu environment. To save myself some time, here's how I do it on Debian:

Background, especially for non-South Africa readers: Bandwidth in South Africa is ridiculously expensive, especially International bandwidth. The point of this exercise is that we can buy "local-only" DSL accounts which only connect to South African networks. E.g. I have an account that gives me 30GB of local traffic / month, for the same cost as 2.5GB of International traffic account. Normally you'd change your username and password on your router to switch account when you wanted to do something like an Debian apt-upgrade, but that's irritating. There's no reason why you can't have a Linux-based router concurrently connected to both accounts via the same ADSL line.

Firstly, we have a DSL modem. Doesn't matter what it is, it just has to support bridged mode. If it won't work without a DSL account, you can use the Telkom guest account. My recommendation for a modem is to buy a Telkom-branded Billion modem (because Telkom sells everything with really big chunky, well-surge-protected power supplies).

For the sake of this example, we have the modem (IP 10.0.0.2/24) plugged into eth0 on our server, which is running Debian or Ubuntu, doesn't really matter much - personal preference. The modem has DHCP turned off, and we have our PCs on the same ethernet segment as the modem. Obviously this is all trivial to change.

You need these packages installed:

# aptitude install iproute pppoe wget awk findutils

You need ppp interfaces for your providers. I created /etc/ppp/peers/intl-dsl:

user intl-account@uber-isp.net
unit 1
pty "/usr/sbin/pppoe -I eth0 -T 80 -m 1452"
noipdefault
defaultroute
hide-password
lcp-echo-interval 20
lcp-echo-failure 3
noauth
persist
maxfail 0
mtu 1492
noaccomp
default-asyncmap

/etc/ppp/peer/local-dsl:

user local-account@uber-isp.net
unit 2
pty "/usr/sbin/pppoe -I eth0 -T 80 -m 1452"
noipdefault
hide-password
lcp-echo-interval 20
lcp-echo-failure 3
connect /bin/true
noauth
persist
maxfail 0
mtu 1492
noaccomp
default-asyncmap

unit 1 makes a connection always bind to "ppp1". Everything else is pretty standard. Note that only the international connection forces a default route.

To /etc/ppp/pap-secrets I added my username and password combinations:

# User                     Host Password
intl-account@uber-isp.net  *    s3cr3t
local-account@uber-isp.net *    passw0rd

You need custom iproute2 routing tables for each interface, for the source routing. This will ensure that incoming connections get responded to out of the correct interface. As your provider only lets you send packets from your assigned IP address, you can't send packets with the international address out of the local interface. We get around that with multiple routing tables. Add these lines to /etc/iproute2/rt_tables:

1       local-dsl
2       intl-dsl

Now for some magic. I create /etc/ppp/ip-up.d/20routing to set up routes when a connection comes up:

	`#!/bin/sh -e`

	`case "$PPP_IFACE" in`
	`"ppp1")`
	`IFACE="intl-dsl"`
	`;;`
	`"ppp2")`
	`IFACE="local-dsl"`
	`;;`
	`*)`
	`exit 0`
	`esac`

	`# Custom routes`
	`if [ -f "/etc/network/routes-$IFACE" ]; then`
	`cat "/etc/network/routes-$IFACE" \| while read route; do`
	`ip route add "$route" dev "$PPP_IFACE"`
	`done`
	`fi`

	`# Clean out old rules`
	`ip rule list \| grep "lookup $IFACE" \| cut -d: -f2 \| xargs -L 1 -I xx sh -c "ip rule del xx"`

	`# Source Routing`
	`ip route add "$PPP_REMOTE" dev "$PPP_IFACE" src "$address" table "$IFACE"`
	`ip route add default via "$PPP_REMOTE" table "$IFACE"`
	`ip rule add from "$PPP_LOCAL" table "$IFACE"`

	`# Make sure this interface is present in all the custom routing tables:`
	route=`ip route show dev "$PPP_IFACE" \| awk '/scope link src/ {print $1}'`
	`awk '/^[0-9]/ {if ($1 > 0 && $1 < 250) print $2}' /etc/iproute2/rt_tables \| while read table; do`
	`ip route add "$route" dev "$PPP_IFACE" table "$table"`
	`done`

That script loads routes from /etc/network/routes-intl-dsl and /etc/network/routes-local-dsl. It also sets up source routing so that incoming connections work as expected.

Now, we need those route files to exist and contain something useful. Create the script /etc/cron.daily/za-routes (and make it executable):

	`#!/bin/sh -e`
	`ROUTEFILE=/etc/network/routes-local-dsl`

	`wget -q http://mene.za.net/za-routes/latest.txt -O /tmp/zaroutes`
	size=`stat -c '%s' /tmp/zaroutes`

	`if [ $size -gt 0 ]; then`
	`mv /tmp/zaroutes "$ROUTEFILE"`
	`fi`

It downloads the routes file from cocooncrash's site (he gets them from local-route-server.is.co.za, aggregates them, and publishes every 6 hours). Run it now to seed that file.

Now some International-only routes. I use IS local DSL, so SAIX DNS queries should go through the SAIX connection even though the servers are local to ZA.

My /etc/network/routes-intl-dsl contains SAIX DNS servers and proxies:

196.25.255.3
196.25.1.9
196.25.1.11
196.43.1.14
196.43.1.11
196.43.34.190
196.43.38.190
196.43.42.190
196.43.45.190
196.43.46.190
196.43.50.190
196.43.53.190
196.43.9.21

Now we can tell /etc/network/interfaces about our connections so that they can get brought up automatically on bootup:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
        address 10.0.0.1
        netmask 255.255.255.0

auto local-dsl
iface local-dsl inet ppp
        provider local-dsl

auto intl-dsl
iface intl-dsl inet ppp
        provider intl-dsl

For DNS, I use dnsmasq, hardcoded to point to IS & SAIX upstreams. My machine's /etc/resolv.conf just points to this dnsmasq.

So something like /etc/resolv.conf:

nameserver 127.0.0.1

/etc/dnsmasq.conf:

no-resolv
# IS:
server=168.210.2.2
server=196.14.239.2
# SAIX:
server=196.43.34.190
server=196.43.46.190
server=196.25.1.11
domain=foobar.lan
dhcp-range=10.0.0.128,10.0.0.254,12h
dhcp-authoritative
no-negcache

If you haven't already, you'll need to turn on ip_forward. Add the following to /etc/sysctl.conf and then run sudo sysctl -p:

net.ipv4.ip_forward=1

Finally, you'll need masquerading set up in your firewall. Here is a trivial example firewall, put it in /etc/network/if-up.d/firewall and make it executable. You should probably change it to suit your needs or use something else, but this should work:

	`#!/bin/sh`
	`if [ $IFACE != "eth0" ]; then`
	`exit;`
	`fi`

	`iptables -F INPUT`
	`iptables -F FORWARD`
	`iptables -t nat -F POSTROUTING`
	`iptables -A INPUT -i lo -j ACCEPT`
	`iptables -A INPUT -i eth0 -s 10.0.0.0/24 -j ACCEPT`
	`iptables -A INPUT -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT`
	`iptables -A INPUT -j DROP`
	`iptables -A FORWARD -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT`
	`iptables -A FORWARD -i eth0 -o ppp+ -j ACCEPT`
	`iptables -A FORWARD -j DROP`
	`iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o ppp+ -j MASQUERADE`

Local only DSL

Stefano Rivera — Sat, 01 Sep 2007 21:06:15 GMT

Update: Debian/Ubuntu version

I've finally jumped onto the local only DSL bandwagon. If you haven't done it yet, it's a great way to save some bucks. The idea is that you get a local only account like this, which costs a fraction per GiB compared to normal account. Then you get your router to connect to both simultaneously, and route intelligently between them.

Most ADSL routers won't let you connect 2 concurrent ADSL connections on the same ATM circuit. The solution is to use a separate modem and router. I'm using a basic Billion modem, in bridged mode, and a WRT54GL, running OpenWRT/kamikaze, as the router.

OpenWRT doesn't support 2 PPPoE connections out of the box, but I've found the problems, and got a few changes committed upstream, that solve them:

Configurable default route: changeset 8569
Allow specifying ppp unit numbers: changeset 8563+8564
Support for multiple interfaces in UpdateDD (a DynDNS client): bug 2294

The firewall (/etc/init.d/firewall) needs to be modified with "WAN=ppp+" somewhere near the top, so that it masquerades all the ppp connections. This was a hack, apparently the firewall is being re-written soon.

There is also a bug that resets existing PPPoE connections on a ethernet interface when you fire up a new connection. This will apparently be fixed by the future interface aliasing support. For now, I just hacked around it in /lib/network/config.sh:

prepare_interface() {
    local iface="$1"
    local config="$2"

    #SR: We don't want to reset any pppoe connections
    config_get proto "$config" proto
    [ "$proto" = "pppoe" ] && return 0

and /sbin/ifdown:

config_get ifname "$cfg" ifname
config_get device "$cfg" device

[ ."${proto%oe}" == ."ppp" ] && device=
[ ."$device" != ."$ifname" ] || device=
for dev in $ifname $device; do
    ifconfig "$dev" 0.0.0.0 down >/dev/null 2>/dev/null
done

I got my local routes list from cocooncrash's site (he gets them from local-route-server.is.co.za, aggregates them, and publishes every 6 hours). OpenWRT already has a static routing configuration system, but it's very verbose. So I wrote my own, adding the new configuration option routefile. I used these hotplug scripts to set up routing and source routing, with the help of iproute2:

$ ipkg install ip

$ mkdir /etc/routes
$ wget http://mene.za.net/za-routes/latest.txt -O /etc/routes/zaroutes

You'll probably want to update that route file regularly. I don't run cron on my WRT54GL, so I do it manually. Up to you.

/etc/config/network:

# ...local interfaces...

#### WAN configuration
config interface    wan
    option ifname   "eth0.1"
    option proto    pppoe
    option username "xxxxx@international.co.za"
    option password "xxxxxx"
    option routefile "/etc/routes/exceptions"
    option defaultroute 1

config interface    localdsl
    option ifname   "eth0.1"
    option proto    pppoe
    option username "xxxxx@local.co.za"
    option password "xxxxxx"
    option routefile "/etc/routes/zaroutes"
    option defaultroute 0

/etc/iproute2/rt_tables:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
1   wan
2   localdsl

/etc/hotplug.d/iface/20-split-routes:

case "$ACTION" in
  ifup)
    . /etc/functions.sh
    include /lib/network
    scan_interfaces
    config_get routefile "$INTERFACE" routefile

    # Does this interface have custom routes?
    if [ -e "$routefile" ]; then

      # Add routes for this interface
      cat "$routefile" | while read route; do
        ip route add "$route" dev "$DEVICE"
      done

      # Set up source routing
      peer=`ip addr show dev $DEVICE | sed -n '/inet/ s#.* peer \([0-9.]*\)/.*#\1# p'`
      address=`ip addr show dev $DEVICE | sed -n '/inet/ s/.* inet \([0-9.]*\) .*/\1/ p'`

      ip route add "$peer" dev "$DEVICE" src "$address" table "$INTERFACE"
      ip route add default via "$peer" table "$INTERFACE"
      ip rule add from "$address" table "$INTERFACE"
    fi

    # Make sure this interface is present in all the custom routing tables:
    route=`ip route show dev "$DEVICE" | awk '/scope link  src/ {print $1}'`
    awk '/^[0-9]/ {if ($1 > 0 && $1 < 250) print $2}' /etc/iproute2/rt_tables | while read table; do
      ip route add "$route" dev "$DEVICE" table "$table"
    done
    ;;
esac

/etc/hotplug.d/net/20-split-routes:

case "$ACTION" in
  register)
    . /etc/functions.sh
    include /lib/network
    scan_interfaces

    # If this interface doesn't have a link local route, we don't need to bother
    route=`ip route show dev "$INTERFACE" | awk '/scope link  src/ {print $1}'`
    [ ."$route" = ."" ] && return 0

    # Add this interface's route to all custom routing tables
    awk '/^[0-9]/ {if ($1 > 0 && $1 < 250) print $2}' /etc/iproute2/rt_tables | while read table; do
      ip route add "$route" dev "$INTERFACE" table "$table"
    done
    ;;
esac

Now, lastly, it won't bring up both interfaces by default. That will be fixed by aliasing in the future, but for now:

/etc/init.d/network-multiwan:

	`#!/bin/sh`
	`ifup wan`
	`ifup localdsl`

	`$ chmod 755 /etc/init.d/network-multiwan`
	`$ ln -s ../init.d/network-multiwan /etc/rc.d/S49network-multiwan`

That's it, and it's working beautifully :-)

What is source routing, people ask? The problem is that your router now has 2 WAN IP addresses. IP1 is used for local traffic, and IP2 for international. So if somebody in ZA tries to connect to IP2, the reply (local destination) will go out of Interface 1. The ISP on the other end of Interface 1 will drop this reply, because it's not coming from IP1.

Source routing tells the router that replies must go out of the same interface that the request came in on. I'm doing it by creating separate routing tables for traffic origionating from each WAN interface.

Gotchas

If you are using 2 different ISPs (say SAIX international and IS local), you must make sure that DNS queries get routed out the right interface. SAIX won't accept queries on their servers from IS, and vice versa.
SAIX Web proxies, Mail servers, and News servers don't accept traffic from local accounts. (especially from another ISP)