I first read about on-line Software RAID reshaping a year ago on LWN. Today I tried it on a live system (that's too big to be backed-up first :-))

I added 2 250GB drives to my existing RAID5 array of 4, making for a 1.2TiB array. The reshape took a while...:

md0 : active raid5 sdb1[4] sda1[5] sdf1[2] sde1[3] sdd1[1] sdc1[0]
      732587712 blocks super 0.91 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
      [>....................]  reshape =  1.9% (4800784/244195904) finish=397.4min speed=10034K/sec

But when it was done:

$ df -h
/dev/md0              1.2T  657G  509G  57% /mnt/storage

This is one of the reasons why I love software RAID, while you have a kak load more I/O through the PCI(E) bus than you would with hardware RAID, you get the flexibility of the highest-end hardware controllers on a normal PC motherboard.

And of course, should things go pear-shaped, I don't need to find an identical controller, I just have to find a box with 6 SATA sockets.

HOWTO reshape

Lets say you have 4 SATA drives, /dev/sda to /dev/sdd, and you are adding a new one /dev/sde.

Check that everything is happy:

$ cat /proc/mdstat
md0 : active raid5 sdd1[3] sdc1[1] sdb1[2] sda1[0]
      937705728 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU]

Partition the new drive (clone sdas partition table onto sde):

# sfdisk -d /dev/sda | sfdisk /dev/sde

Add the new drive:

# mdadm -a /dev/md0 /dev/sde1

Grow the RAID:

# mdadm --grow -n5 /dev/md0

Watch the progress:

$ watch cat /proc/mdstat

Grow the filesystem:

# resize_reiserfs /dev/md0

See the extra space:

$ df -h

Planet GeekDinner has been filling up with lots of non-geek-dinner related cruft. So I've taken advantage of Planet Venus's cool filtering system.

Any feed that doesn't have a "GeekDinner" category feed, is being filtered, and only posts containing the regex [gG]eek[ -]?[dD]inner are being displayed. I.e. If you mention GeekDinner in your post, it will appear on the planet.

I'm also filtering Rafiq's geekdinner category, as he seems to post everything under "geekdinner" :-)

Oh, BTW, Nice job with the skin, Joe.

UPDATE: Rafiq is unfiltered again - I didn't read thoroughly enough to see the geekdinner reference as a footnote in your Dell IdeaStorm Post. You have quite a busy site, I thought I'd reached the end of the article when it said "REad more... | Digg Story").

BTW, Rafiq, seeing as you will probably see this, can you sort out your avatar on the CLUG Park planet? I had to resize it to be the standard size, and it now looks very pixellated.

People complain that I'm too fixated on being right. Sure I am, but I am right, dammit! :-)

In my CSC3002F lecture yesterday, our Networks lecturer asked the class to name an application protocol that uses UDP. Silence. Eventually, I piped up "DNS", as I get very bored in slow lectures, and just want them to get a move on...

No, he doesn't like that.

OK - maybe DNS isn't an application protocol, I mean, it's a function of the IP network... So I suggest VoIP.

"Am I sure?"

"Pretty damn sure!"

"Well, I think you're wrong, it would be TCP, because you don't want voice packets arriving out of order. The answer is SNMP, as I showed you in my foil on tuesday."

GRrrr! Some lecturers need to get out into the real world, and see what people are doing. VoIP is considered the textbook example for UDP, packets can be lost without too much trouble (humans have built-in error detection and correction), out of order packets can be dropped (for the same reason), and any attempt at flow control would be a problem (you'd need to change codec).

OK, so joe got me into sorting out a planet venus for geekdinner. And I've also tweaked their mediawiki a little.

Planet Venus is the first time I've used bzr, it's really quite a cool RCS, I think I'll use it more often... While I'm quite a subversion user, working away from home is a pain. SVK helps but it doesn't go as far as a real distributed RCS like bzr. Nice job ubunteros :-)

I've spent some time beefing up CLUG Park.

All the people who seem clueless about making avatars of the correct dimensions (ahem Rafiq), or cropping all the uncessary transparency around their avatar, I've sorted them out.

I've also switched us from Planet 2.0, to Planet Venus. This allows us to do funky things like filters (for you geeks with nasty RSS feeds). It also partitions out the theme far better, and above all does multi-threaded RSS harvesting.

Is anyone is keen on a tech-only (or CLUG-only) CLUG park, where we limit the subscriptions down to posts about CLUG or technology (using category RSS feeds), let me know, we can easily do this. While some of us like to read all about what CLUG Park members are up to, others probably only want to read tech-related articles?

I spent a lovely weekend morning setting up monitoring on servers - yes, what fun :-)

I like all my servers to run logcheck, smartmontools, sysstat, and lm-sensors.

logcheck means watching your email every hour, and adding in yet more ignore rules for things your server thinks it's perfectly OK to spit out.

smartmontools means waiting to see which attributes it's going to complain about, making sure it's set up to mail you about bad sectors, and getting this all in inside the 128-character line-length limit.

And lm-sensors, well that takes a lot of tweaking, to get all the alarms to stop ringing, labelling the right temperatures, and ignoring the disconnected pins.

Ugh, it's painful work, but it helps in the long-run...

I finally found a good blog post on the subject of getting Postfix to do SMTP-AUTH via SASL.

I went one step further, and instead of moving /var/run/saslauthd/ to the Postfix chroot, I did a bind mount:

/etc/fstab:

/var/run/saslauthd /var/spool/postfix/var/run/saslauthd none bind 0     0

Postfix was announcing methods like CRAM-MD5 which can't be supported by the PAM backend, so I restricted them down to PLAIN and LOGIN (over TLS only, obviously):

/etc/postfix/sasl/smtpd.conf:

pwcheck_method: saslauthd
mech_list: plain login

Now, it's working nicely, and I can IMAPS and SMTP-AUTH-TLS to my mail server from anywhere.

tea and tumbleweed were chatting one evening about having a shared dedicated server in london for #clug (or clug in general) members to use as MX, shell box, etc.

Basically Xen would be out of the question, because cheap boxes don't have enough RAM, it would have to be shared. This would mean agreeing on a common MTA (HORRORS!), and sharing root between a group of people... :-)

Some options:

• VPS: Bytemark
• Cheap Dedicated: UK2
• Good Dedicated: Black Cat

tumbleweed has a "Value" server with uk2.net, who's lease expires on 22 Feb. If 11 other people were interested, we could renew it, for the cost of about R600 pa, each. This would save on setup costs.

Anyone keen? We have a wiki page on the subject, add yourself.

I've been enjoying our server at UK2.net. It's a pretty speedy machine (although a little light on RAM - I suspect that they don't want people running Xen), and it's connected to a fat pipe. But I've been experiencing a lot of bad lockups.

I traced the problem to postmaping the uceprotect.net RBL file. They recommend that you rsync this file from them, and then postmap it into a fast lookup database for postfix, rather than using their DNSRBL service. But running the postmap was taking my box 40 mins. The same operation, on a loaded, lower-spec, 2 year old server took 2 mins (yes this server also has RAID1 on the volume concerned). On my UK2 box, while the postmap was running, the machine became totally unresponsive, and it could take a minute or two to log in, serve a web page, or even execute a basic command like ps.

Clearly something wasn't right. And it was something in the IO system. The only answer is the 3ware RAID controller. (It's a 8006-2, doing RAID-1) I know these controllers have a big buffer, so I looked up the 3ware website, for tuning guidance. I followed it to the letter, and things didn't really improve. I tried the deadline scheduler, and tweaking the buffers, but it only got marginally better.

Personally, I've always used software RAID, even for RAID-5, and I've never had bad performance like that. And having the RAID in a portable format has really helped with recovery in the past. I understand that Windows monkeys have to use hardware RAID (because their software RAID sucks so much), but is this kind of performance normal?

I've asked UK2 to chuck my controller and give me software RAID :-)

Update

I've now got software RAID 1, and postmap runs in 25 seconds. That's what I call a 60x speed improvement :-)

Oh, and the system is totally responsive while the postmap runs.

My cheapo desktop Planet Gigabit switch died of heatstroke today. (or at least the port my desktop PC plugs into did)

The sooner this heatwave ends, the better.