Stefano Rivera (Posts about linux)

Fun with Squid and CDNs

Stefano Rivera — Wed, 18 Feb 2009 10:29:02 GMT

One neat upgrade in Debian's recent 5.0.0 release¹ was Squid 2.7. In this bandwidth-starved corner of the world, a caching proxy is a nice addition to a network, as it should shave at least 10% off your monthly bandwidth usage. However, the recent rise of CDNs has made many objects that should be highly cacheable, un-cacheable.

For example, a YouTube video has a static ID. The same piece of video will always have the same ID, it'll never be replaced by anything else (except a "sorry this is no longer available" notice). But it's served from one of many delivery servers. If I watch it once, it may come from

http://v3.cache.googlevideo.com/videoplayback?id=0123456789abcdef&itag=34&ip=1.2.3.4&region=0&signature=5B1BA40D8464F2303DDDD59B2586C10A0AEFAD19.169DA15A09AB88E824DE63DF138F0D835295463B&sver=2&expire=1234714137&key=yt1&ipbits=0

But the next time it may come from v15.cache.googlevideo.com. And that's not all, the signature parameter is unique (to protect against hot-linking) as well as other not-static parameters. Basically, any proxy will probably refuse to cache it (because of all the parameters) and if it did, it'd be a waste of space because the signature would ensure that no one would ever access that cached item again.

I came across a page on the squid wiki that addresses a solution to this. Squid 2.7 introduces the concept of a storeurl_rewrite_program which gets a chance to rewrite any URL before storing / accessing an item in the cache. Thus we could rewrite our example file to

http://cdn.googlevideo.com.SQUIDINTERNAL/videoplayback?id=0123456789abcdef&itag=34

We've normalised the URL and kept the only two parameters that matter, the video id and the itag which specifies the video quality level.

The squid wiki page I mentioned includes a sample perl script to perform this rewrite. They don't include the itag, and my perl isn't good enough to fix that without making a dog's breakfast of it, so I re-wrote it in Python. You can find it at the end of this post. Each line the rewrite program reads contains a concurrency ID, the URL to be rewritten, and some parameters. We output the concurrency ID and the URL to rewrite to.

The concurrency ID is a way to use a single script to process rewrites from different squid threads in parallel. The documentation is this is almost non-existant, but if you specify a non-zero storeurl_rewrite_concurrency each request and response will be prepended with a numeric ID. The perl script concatenated this directly before the re-written URL, but I separate them with a space. Both seem to work. (Bad documentation sucks)

All that's left is to tell Squid to use this, and to override the caching rules on these URLs.

storeurl_rewrite_program /usr/local/bin/storeurl-youtube.py
storeurl_rewrite_children 1
storeurl_rewrite_concurrency 10

#  The keyword for all youtube video files are "get_video?", "videodownload?" and "videoplaybeck?id"
#  The "\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)\?" is only for pictures and other videos
acl store_rewrite_list urlpath_regex \/(get_video\?|videodownload\?|videoplayback\?id) \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)\? \/ads\?
acl store_rewrite_list_web url_regex ^http:\/\/([A-Za-z-]+[0-9]+)*\.[A-Za-z]*\.[A-Za-z]*
acl store_rewrite_list_path urlpath_regex \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)$
acl store_rewrite_list_web_CDN url_regex ^http:\/\/[a-z]+[0-9]\.google\.com doubleclick\.net

# Rewrite youtube URLs
storeurl_access allow store_rewrite_list
# this is not related to youtube video its only for CDN pictures
storeurl_access allow store_rewrite_list_web_CDN
storeurl_access allow store_rewrite_list_web store_rewrite_list_path
storeurl_access deny all

# Default refresh_patterns
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0

# Updates (unrelated to this post, but useful settings to have):
refresh_pattern windowsupdate.com/.*\.(cab|exe)(\?|$) 518400 100% 518400 reload-into-ims
refresh_pattern update.microsoft.com/.*\.(cab|exe)(\?|$) 518400 100% 518400 reload-into-ims
refresh_pattern download.microsoft.com/.*\.(cab|exe)(\?|$) 518400 100% 518400 reload-into-ims
refresh_pattern (Release|Package(.gz)*)$        0       20%     2880
refresh_pattern \.deb$         518400   100%    518400 override-expire

# Youtube:
refresh_pattern -i (get_video\?|videodownload\?|videoplayback\?) 161280 50000% 525948 override-expire ignore-reload
# Other long-lived items
refresh_pattern -i \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)(\?|$) 161280 3000% 525948 override-expire reload-into-ims

refresh_pattern .               0       20%     4320

# All of the above can cause a redirect loop when the server
# doesn't send a "Cache-control: no-cache" header with a 302 redirect.
# This is a work-around.
minimum_object_size 512 bytes

Done. And it seems to be working relatively well. If only I'd set this up last year when I had pesky house-mates watching youtube all day ;-)

It should of course be noted that doing this instructs your Squid Proxy to break rules. Both override-expire and ignore-reload violate guarantees that the HTTP standards provide the browser and web-server about their communication with each other. They are relatively benign changes, but illegal nonetheless.

And it goes without saying that rewriting the URLs of stored objects could cause some major breakage by assuming that different objects (with different URLs) are the same. The provided regexes seem sane enough to not assume that this won't happen, but YMMV.

	`#!/usr/bin/env python`
	`# vim:et:ts=4:sw=4:`

	`import re`
	`import sys`
	`import urlparse`

	`youtube_getvid_res = [`
	`re.compile(r"^http:\/\/([A-Za-z]?)-(.?)\.(.)\.youtube\.com\/get_video\?video_id=(.?)&(.*?)$"),`
	`re.compile(r"^http:\/\/(.?)\/get_video\?video_id=(.?)&(.*?)$"),`
	`re.compile(r"^http:\/\/(.?)video_id=(.?)&(.*?)$"),`
	`]`

	`youtube_playback_re = re.compile(r"^http:\/\/(.?)\/videoplayback\?id=(.?)&(.*?)$")`

	`others = [`
	`(re.compile(r"^http:\/\/(.?)\/(ads)\?(?:.?)$"), "http://%s/%s"),`
	`(re.compile(r"^http:\/\/(?:.?)\.yimg\.com\/(?:.?)\.yimg\.com\/(.?)\?(?:.?)$"), "http://cdn.yimg.com/%s"),`
	`(re.compile(r"^http:\/\/(?:(?:[A-Za-z]+[0-9-.]+)?)\.(.?)\.(.?)\/(.?)\.(.?)\?(?:.?)$"), "http://cdn.%s.%s.SQUIDINTERNAL/%s.%s"),`
	`(re.compile(r"^http:\/\/(?:(?:[A-Za-z]+[0-9-.]+)?)\.(.?)\.(.?)\/(.?)\.(.{3,5})$"), "http://cdn.%s.%s.SQUIDINTERNAL/%s.%s"),`
	`(re.compile(r"^http:\/\/(?:(?:[A-Za-z]+[0-9-.]+)?)\.(.?)\.(.?)\/(.?)$"), "http://cdn.%s.%s.SQUIDINTERNAL/%s"),`
	`(re.compile(r"^http:\/\/(.?)\/(.?)\.(jp(?:e?g\|e\|2)\|gif\|png\|tiff?\|bmp\|ico\|flv)\?(?:.*?)$"), "http://%s/%s.%s"),`
	`(re.compile(r"^http:\/\/(.?)\/(.?)\;(?:.*?)$"), "http://%s/%s"),`
	`]`

	`def parse_params(url):`
	`"Convert a URL's set of GET parameters into a dictionary"`
	`params = {}`
	`for param in urlparse.urlsplit(url)[3].split("&"):`
	`if "=" in param:`
	`n, p = param.split("=", 1)`
	`params[n] = p`
	`return params`

	`while True:`
	`line = sys.stdin.readline()`
	`if line == "":`
	`break`
	`try:`
	`channel, url, other = line.split(" ", 2)`
	`matched = False`

	`for re in youtube_getvid_res:`
	`if re.match(url):`
	`params = parse_params(url)`
	`if "fmt" in params:`
	`print channel, "http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=%s&fmt=%s" % (params["video_id"], params["fmt"])`
	`else:`
	`print channel, "http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=%s" % params["video_id"]`
	`matched = True`
	`break`

	`if not matched and youtube_playback_re.match(url):`
	`params = parse_params(url)`
	`if "itag" in params:`
	`print channel, "http://video-srv.youtube.com.SQUIDINTERNAL/videoplayback?id=%s&itag=%s" % (params["id"], params["itag"])`
	`else:`
	`print channel, "http://video-srv.youtube.com.SQUIDINTERNAL/videoplayback?id=%s" % params["id"]`
	`matched = True`

	`if not matched:`
	`for re, pattern in others:`
	`m = re.match(url)`
	`if m:`
	`print channel, pattern % m.groups()`
	`matched = True`
	`break`

	`if not matched:`
	`print channel, url`

	`except Exception:`
	`# For Debugging only. In production we want this to never die.`
	`#raise`
	`print line`

	`sys.stdout.flush()`

Yes, Vhata, Debian released in 2009, I won the bet, you owe me a dinner now. ↩

The joy that is SysRq

Stefano Rivera — Mon, 06 Oct 2008 11:31:20 GMT

I'm constantly surprised when I come across long-time Linux users who don't know about SysRq. The Linux Magic System Request Key Hacks are a magic set of commands that you can get the Linux kernel to follow no matter what's going on (unless it has panicked or totally deadlocked).

Why is this useful? Well, there are many situations where you can't shut a system down properly, but you need to reboot. Examples:

You've had a kernel OOPS, which is not quite a panic but there could be memory corruption in the kernel, things are getting pretty weird, and quite honestly you don't want to be running in that condition for any longer than necessary.
You have reason to believe it won't be able to shut down properly.
Your system is almost-locked-up (i.e. the above point)
Your UPS has about 10 seconds worth of power left
Something is on fire (lp0 possibly?)
...Insert other esoteric failure modes here...

In any of those situations, grab a console keyboard, and type Alt+SysRq+s (sync), Alt+SysRq+u (unmount), wait for it to have synced, and finally Alt+SysRq+b (reboot NOW!). If you don't have a handy keyboard attached to said machine, or are on another continent, you can

# echo u > /proc/sysrq-trigger

In my books, the useful SysRq commands are:

b: Reboot
f: Call the oom_killer
h: Display SysRq help
l: Print a kernel stacktrace
o: Power Off
r: Set your keyboard to RAW mode (required after some X breakages)
s: Sync all filesystems
u: Remount all filesystems read-only
0-9: Change console logging level

In fact, read the rest of the SysRq documentation, print it out, and tape it above your bed. Next time you reach for the reset switch on a Linux box, stop your self, type the S,U,B sequence, and watch your system come up as if nothing untoward has happened.

Update: I previously recommended U,S,B but after a bit of digging, I think S,U,B may be correct.

Ctrl-Alt-shortcuts considered harmful

Stefano Rivera — Fri, 26 Sep 2008 18:32:12 GMT

I've written about this before, but Ctrl-Alt-workspace switching key-presses nail me routinely.

Let's go through some history:

We have Ctrl-Alt-Delete, the "three-fingered-salute", meaning reboot, right? That combination was designed to NEVER be pressed by accident. And it never used to be.

The X guys needed a kill-X key-press, as things can sometimes get broken in X. So they chose Ctrl-Alt-Backspace, which is also a pretty sensible combination. It's very similar to Ctrl-Alt-Delete, so we remember it, and backspace has milder connotations than delete, so we understand it to mean that it'll only kill a part of the system.

X also has some other Ctl-Alt- shortcuts. Some of these are also suitably obscure, i.e. NumPad+ and NumPad-. Others like Ctrl-Alt-F1 mean change to virtual console 1. That one might do by accident, if you are an old WordPerfect user, but should be safe enough otherwise. They were designed to look like big brothers of (and even work as) standard VT-changing behaviour.

For changing workspaces, Alt-F1 style key-presses were used, mimicing VT-changing key-presses. This is great for *nix users, but people coming from Windows expect Alt-F4 to close a program, not take them to workspace 4.

So GNOME came along, and decided that instead, they'd use Ctrl-Alt-Arrow key-presses to change workspace. That's fine, but it's a pretty common action, so I'm often holding down Ctrl-Alt without even thinking about it. If I start editing something and press delete/backspace, before I've released Ctrl-Alt, boom! And I run screaming and write a blog post.

Now, I know that Ctl-Alt-{Delete,Backspace} can be disabled (even if the latter is a little tricky to do), but I'd really like to change them. I like to be able to kill X without using another machine and ssh, I just don't like this to happen by accident. And no, the solution isn't for me to change my workspace-changing keys, because this problem must affect every GNOME user, not just me.

Dangerous key-presses should be really unlikely key-presses. Alt-SysRq- key-presses are good in this regard, they'll always be unlikely. (Oh, and they are insanely useful.)

Split-Routing on Debian/Ubuntu

Stefano Rivera — Fri, 19 Sep 2008 15:13:58 GMT

My post on split-routing on OpenWRT has been incredibly popular, and led to many people implementing split-routing, whether or not they had OpenWRT. While it's fun to have an exercise as a reader, it led to me having to help lots of newbies through porting that setup to a Debian / Ubuntu environment. To save myself some time, here's how I do it on Debian:

Background, especially for non-South Africa readers: Bandwidth in South Africa is ridiculously expensive, especially International bandwidth. The point of this exercise is that we can buy "local-only" DSL accounts which only connect to South African networks. E.g. I have an account that gives me 30GB of local traffic / month, for the same cost as 2.5GB of International traffic account. Normally you'd change your username and password on your router to switch account when you wanted to do something like an Debian apt-upgrade, but that's irritating. There's no reason why you can't have a Linux-based router concurrently connected to both accounts via the same ADSL line.

Firstly, we have a DSL modem. Doesn't matter what it is, it just has to support bridged mode. If it won't work without a DSL account, you can use the Telkom guest account. My recommendation for a modem is to buy a Telkom-branded Billion modem (because Telkom sells everything with really big chunky, well-surge-protected power supplies).

For the sake of this example, we have the modem (IP 10.0.0.2/24) plugged into eth0 on our server, which is running Debian or Ubuntu, doesn't really matter much - personal preference. The modem has DHCP turned off, and we have our PCs on the same ethernet segment as the modem. Obviously this is all trivial to change.

You need these packages installed:

# aptitude install iproute pppoe wget awk findutils

You need ppp interfaces for your providers. I created /etc/ppp/peers/intl-dsl:

user intl-account@uber-isp.net
unit 1
pty "/usr/sbin/pppoe -I eth0 -T 80 -m 1452"
noipdefault
defaultroute
hide-password
lcp-echo-interval 20
lcp-echo-failure 3
noauth
persist
maxfail 0
mtu 1492
noaccomp
default-asyncmap

/etc/ppp/peer/local-dsl:

user local-account@uber-isp.net
unit 2
pty "/usr/sbin/pppoe -I eth0 -T 80 -m 1452"
noipdefault
hide-password
lcp-echo-interval 20
lcp-echo-failure 3
connect /bin/true
noauth
persist
maxfail 0
mtu 1492
noaccomp
default-asyncmap

unit 1 makes a connection always bind to "ppp1". Everything else is pretty standard. Note that only the international connection forces a default route.

To /etc/ppp/pap-secrets I added my username and password combinations:

# User                     Host Password
intl-account@uber-isp.net  *    s3cr3t
local-account@uber-isp.net *    passw0rd

You need custom iproute2 routing tables for each interface, for the source routing. This will ensure that incoming connections get responded to out of the correct interface. As your provider only lets you send packets from your assigned IP address, you can't send packets with the international address out of the local interface. We get around that with multiple routing tables. Add these lines to /etc/iproute2/rt_tables:

1       local-dsl
2       intl-dsl

Now for some magic. I create /etc/ppp/ip-up.d/20routing to set up routes when a connection comes up:

	`#!/bin/sh -e`

	`case "$PPP_IFACE" in`
	`"ppp1")`
	`IFACE="intl-dsl"`
	`;;`
	`"ppp2")`
	`IFACE="local-dsl"`
	`;;`
	`*)`
	`exit 0`
	`esac`

	`# Custom routes`
	`if [ -f "/etc/network/routes-$IFACE" ]; then`
	`cat "/etc/network/routes-$IFACE" \| while read route; do`
	`ip route add "$route" dev "$PPP_IFACE"`
	`done`
	`fi`

	`# Clean out old rules`
	`ip rule list \| grep "lookup $IFACE" \| cut -d: -f2 \| xargs -L 1 -I xx sh -c "ip rule del xx"`

	`# Source Routing`
	`ip route add "$PPP_REMOTE" dev "$PPP_IFACE" src "$address" table "$IFACE"`
	`ip route add default via "$PPP_REMOTE" table "$IFACE"`
	`ip rule add from "$PPP_LOCAL" table "$IFACE"`

	`# Make sure this interface is present in all the custom routing tables:`
	route=`ip route show dev "$PPP_IFACE" \| awk '/scope link src/ {print $1}'`
	`awk '/^[0-9]/ {if ($1 > 0 && $1 < 250) print $2}' /etc/iproute2/rt_tables \| while read table; do`
	`ip route add "$route" dev "$PPP_IFACE" table "$table"`
	`done`

That script loads routes from /etc/network/routes-intl-dsl and /etc/network/routes-local-dsl. It also sets up source routing so that incoming connections work as expected.

Now, we need those route files to exist and contain something useful. Create the script /etc/cron.daily/za-routes (and make it executable):

	`#!/bin/sh -e`
	`ROUTEFILE=/etc/network/routes-local-dsl`

	`wget -q http://mene.za.net/za-routes/latest.txt -O /tmp/zaroutes`
	size=`stat -c '%s' /tmp/zaroutes`

	`if [ $size -gt 0 ]; then`
	`mv /tmp/zaroutes "$ROUTEFILE"`
	`fi`

It downloads the routes file from cocooncrash's site (he gets them from local-route-server.is.co.za, aggregates them, and publishes every 6 hours). Run it now to seed that file.

Now some International-only routes. I use IS local DSL, so SAIX DNS queries should go through the SAIX connection even though the servers are local to ZA.

My /etc/network/routes-intl-dsl contains SAIX DNS servers and proxies:

196.25.255.3
196.25.1.9
196.25.1.11
196.43.1.14
196.43.1.11
196.43.34.190
196.43.38.190
196.43.42.190
196.43.45.190
196.43.46.190
196.43.50.190
196.43.53.190
196.43.9.21

Now we can tell /etc/network/interfaces about our connections so that they can get brought up automatically on bootup:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
        address 10.0.0.1
        netmask 255.255.255.0

auto local-dsl
iface local-dsl inet ppp
        provider local-dsl

auto intl-dsl
iface intl-dsl inet ppp
        provider intl-dsl

For DNS, I use dnsmasq, hardcoded to point to IS & SAIX upstreams. My machine's /etc/resolv.conf just points to this dnsmasq.

So something like /etc/resolv.conf:

nameserver 127.0.0.1

/etc/dnsmasq.conf:

no-resolv
# IS:
server=168.210.2.2
server=196.14.239.2
# SAIX:
server=196.43.34.190
server=196.43.46.190
server=196.25.1.11
domain=foobar.lan
dhcp-range=10.0.0.128,10.0.0.254,12h
dhcp-authoritative
no-negcache

If you haven't already, you'll need to turn on ip_forward. Add the following to /etc/sysctl.conf and then run sudo sysctl -p:

net.ipv4.ip_forward=1

Finally, you'll need masquerading set up in your firewall. Here is a trivial example firewall, put it in /etc/network/if-up.d/firewall and make it executable. You should probably change it to suit your needs or use something else, but this should work:

	`#!/bin/sh`
	`if [ $IFACE != "eth0" ]; then`
	`exit;`
	`fi`

	`iptables -F INPUT`
	`iptables -F FORWARD`
	`iptables -t nat -F POSTROUTING`
	`iptables -A INPUT -i lo -j ACCEPT`
	`iptables -A INPUT -i eth0 -s 10.0.0.0/24 -j ACCEPT`
	`iptables -A INPUT -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT`
	`iptables -A INPUT -j DROP`
	`iptables -A FORWARD -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT`
	`iptables -A FORWARD -i eth0 -o ppp+ -j ACCEPT`
	`iptables -A FORWARD -j DROP`
	`iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o ppp+ -j MASQUERADE`

Just what is Universally Unique

Stefano Rivera — Sat, 16 Feb 2008 20:31:25 GMT

I had an interesting discussion with "bonnyrsa" in #ubuntu-za today. He'd re-arranged his partitions with gparted, and copied and pasted his / partition, so that he could move it to the end of the disk.

However this meant that he now had two partitions with the same UUID. While you can imagine that this is the correct result of a copy & paste operation, it now means that your universally unique ID is totally non-unique. Not in your PC, and no even on it's home drive.

Ubuntu mounts by UUID, so now how do we know which partition is being mounted?

"mount" said /dev/sda2
/proc/mounts said /dev/disk/by-uuid/c087bad7-5021-4f65-bb97-e0d3ea9d01a6 which was a symlink to /dev/sda2.

However neither were correct.

Mounting /dev/sda4 (ro) produced "/dev/sda4 already mounted or /mnt busy".

Aha, so we must be running from /dev/sda4.

/dev/sda2 mounted fine, but then wouldn't unmount: "it seems /dev/sda2 is mounted multiple times".

Aaaargh!

I got him to reboot, change /dev/sda2s UUID, and reboot again (sucks). Then everything was better.

This shouldn't have happened. Non-unique UUIDs is a really crap situation to be in. It brings out bugs in all sorts of unexpected places. I think parted should (by default) change the UUID of a copied partition (although if you are copying an entire disk, it shouldn't).

I've filed a bug on Launchpad, let's see if anyone bites.

PS: All UUIDs in this post have been changed to protect the identity of innocent Ubuntu systems (who aren't expecting a sudden attack of non-uniqueness).

Madwifi regdomain issues

Stefano Rivera — Mon, 11 Feb 2008 22:39:49 GMT

The CS Department at UCT has some Wireless APs on Channel 13. This is quite cool (for geeky reasons), but my MacBook (purchased in the US) did not agree. As far as it is concerned, the only 802.11g channels in existence are 1-11.

The reason for this is that my Atheros (madwifi) network card is a software-defined radio. Atheros interprets the FCC regulations to mean that it cannot provide an Open Source driver for this card, allowing it broadcast on any random channel. Thus the madwifi driver contains a binary HAL, produced by Atheros, which is responsible for regulating frequencies and power levels. (This HAL has been reverse-engineered by the OpenBSD people, but not for my card, unfortunately).

The card has two values stored in it's EEPROM, a "countrycode", and a "regdomain". The countrycode is overrideable in software (you modprobe ath_pci countrycode=710), but only if the countrycode you specify is valid for the card's regdomain. Some cards have a 0x00 or 0xFF regdomain (wildcard values), but mine had 0x64. This meant that whenever I tried to specify a country code, I'd get an error, and the madwifi module would refuse to load:

Feb 11 11:34:11 beethoven kernel: [ 2047.669023] MadWifi: ath_getchannels: Unable to collect channel list from HAL; regdomain likely 100 country code 710

There has been some success with changing the regdomain in the EEPROM, using the hard-to-find ar5k utility (or possible the ath_info utility?). However, again this didn't work with my model. But I found an e-mail from somebody who'd been playing with similar stuff. I mailed Salvatore, and he replied almost instantly, pointing me to a public Windows utility for changing regdomains. It depends on a special driver, available in the demo of "CommView for Wireless".

I installed Windows in my swap partition (it's not an operating system I normally have around). (Naturally, I forgot to have an Ubuntu CD handy, to rebuild my grub, but that was easily remedied.). After a few blue screens of death (install all necessary drivers first), I got my regdomain changed to 0x37, which is the regdomain for South Africa & Europe.

Now, I'm writing this from a couch in the CS department, using a channel 13 AP. Success.

Multiple IP addresses on Debian

Stefano Rivera — Wed, 05 Dec 2007 20:29:02 GMT

Quick post. If you have multiple IP addresses (i.e. a range) assigned to you server, and you want to listen on all of them (i.e. multiple SSL sites), then rather than using the ancient eth0:1 syntax, you can hack /etc/network/interfaces to use iproute2 properly.

Assuming the IP 10.2.3.4, with the extra range of 10.5.4.110-10.5.4.118 (yes these extra ranges often ignore class-boundries):

auto eth0
iface eth0 inet static
    address 10.2.3.4
    netmask 255.255.255.0
    network 10.2.3.0
    broadcast 10.2.3.255
    gateway 10.2.3.1
    # Extra IPs:
    post-up for last in `seq 110 118`; do ip addr add 10.5.4.$last/32 dev $IFACE; done || true
    pre-down for ip in `ip addr show dev $IFACE | sed -n 's@.* inet \([0-9.]*/32\) .*@\1@ p'`; do ip addr del $ip dev $IFACE; done || true

Yes, it's ugly as shit, but I can't think of a neater way to do it.

Update: Better solution

Gnome virtual desktop bindings

Stefano Rivera — Sat, 10 Nov 2007 20:34:10 GMT

This might sound like a silly topic, but it's infuriating.

It's only very recently (since July) that I've adopted the Ctrl-Alt-Arrow Virtual Desktop bindings. Call me an old todger, but up until now, I've always remapped Alt-F1 through Alt-F6 as my Virtual Desktop keys. That's how I always switched desktop, and I couldn't bring myself to change to the Ctrl-Alt style.

I try to slowly align myself with the new defaults, so that I have to do less customisation to feel comfortable when I sit down at a box. With the MacBook which infuriatingly requires Fn+ for the F-keys, I thought I'd switch. It took a bit of re-training to switch, and now I'm comfortable. But, every now an then, I accidentally press Ctrl-Alt-Backspace and kill X. This combination is supposed to be highly unlikely to be accidental, and used to be. But with the default gnome bindings, it's quite common. You just finish editing some text, and switch desktop, to find that your right hand hadn't fully released the backspace key before the Ctrl+Alt went down.

I think either the X kill key needs to be changed, or we have to get rid of this silly gnome desktop-switching binding.

OTOH, I'm almost entirely in line with the modern GNOME defaults. On a foreign machine, I need to set up Dvorak keyboard, change the terminal to grey-on-black, and I'm pretty much ready to go. (My .ssh/config is also nice to have, as are my firefox quick links)

An update on Laptop Hard Drives & Linux

Stefano Rivera — Sat, 27 Oct 2007 08:11:31 GMT

Launchpad bug 59695 has been gathering a huge amount of activity since I wrote about this issue. The issue seems to be that the hardware manufacturers (BIOS and HDD firmware) set very aggressive values for power management. And every other OS (Windows & Mac OSX) override these values to something more sane. The manufacturers only test their equipment in Windows, so they don't see any problems :-)

This kind of thing seems to happen to Linux quite regularly - we all remember the ACPI debacle caused by manufacturers using Microsoft's broken ASL compiler, which worked in Microsoft's broken ACPI environment (or was overridden with driver updates).

So in my opinion, Ubuntu (and every other distributor) has to step in and override these aggressive settings. And, by the look of the bug report, Gnome Power Manager should provide the user with a slider to set the balance between power savings and hardware lifetime.

There is already the obligatory ubuntu is killing your Hard Drive blog, and a closed for cooling off Ubuntu forums thread, if you want get involved and start swinging your battle-axe.

Oh, and the workaround I posted last week obviously doesn't cover the case of the machine resuming from suspend. You have to use an ACPI event script for that.

A serious warning to Linux Laptop users

Stefano Rivera — Mon, 22 Oct 2007 21:15:22 GMT

I've just come across a rather scary, and worryingly old launchpad bug, which talks about real hardware damage. There is more on the problem here. But basically, by default, Linux is far too optimistic with spinning laptop hard drives down, and you can reach number of spin-up/downs that your drive is rated for over it's entire life-time, in a few months.

My laptop (3 months old), is already at 160000 Load/Unloads: Around half it's rated life.

The easy solution is

# hdparm -B 180 /dev/sda

Or, the following in /etc/hdparm.conf

/dev/sda {
    apm = 180
}

Lets hope that this gets resolved soon, or the problem isn't as bad as it appears.

I see that Matthew Garrett (the Ubuntu Laptop Tzar) is subscribed to this bug, but doesn't seem to have commented on it. I find that a little odd, considering its seriousness.