Method to "unstick" buggy raid1 devices

Method to "unstick" buggy raid1 devices

[William M. Shubert]

Once I found out why my "dd" program was stuck, and what the workaround 
was, it was easy to see that no reboot was necessary to fix the problem. 
If you can easily reboot with a new kernel that doesn't have this bug, 
then of course that is the best solution, but for irritating reasons 
rebooting would have been difficult for me. If anybody else ever has the 
same problem, here is a reboot-less fix. It wasn't hard to write - I 
just tested it on a crashable system, it worked, so I installed it on my 
production system that was suffering from a frozen "dd" process.

When you need this: If you have a system running a kernel with a buggy 
raid1 driver and you can't easily switch to a non-buggy kernel. Vanilla 
2.4.9 is buggy. Red hat 2.4.9 is not. I do not know which other kernels 
are/are not buggy. If you get hit with the bug, the symptom will be that 
one or more programs is "frozen", can't be killed, and never does 
anything. If you check the "wchan" parameter of "top" or "ps", it is 
stuck in "raid1_alloc_r1bh". This will happen very rarely - only if a 
process is doing a disk write at the exact moment that the system is 
temporarily out of physical memory. If your system is under heavy load 
it will happen much more often.

After you apply the bug, once per minute this module will wake up, look 
for stuck raid devices, and "unstick" them if necessary. So at most your 
processes will freeze for 1 minute. You can also make it check more 
often if you want.

Not sure if anybody but me will need it, but just in case - here it is!

#define __KERNEL__ 1
#define MODULE 1

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/init.h>
#include <linux/kdev_t.h>
#include <linux/raid/md.h>
#include <linux/raid/md_k.h>
#include <linux/raid/raid1.h>

static struct timer_list timer;

#define CHECK_INTERVAL (60 * HZ) /* Once per minute. */
#define NUM_RAIDS 3 /* # of raid1 partitions. Make sure this is correct! */

static void doFix(unsigned long x) {
  int i;
  raid1_conf_t *conf;
  for (i = 0; i < NUM_RAIDS; ++i) {
    conf = mddev_to_conf(kdev_to_mddev(MKDEV(9, i)));
    if (conf->freer1_blocked == 1) {
      printk(KERN_WARNING "fixRaid: MD system %d is stuck. Fixing.\n", i);
      conf->freer1_blocked = 0;
      wake_up(&conf->wait_buffer);
    }
  }
  timer.expires = jiffies + CHECK_INTERVAL;
  add_timer(&timer);
}

int fixRaid_init(void) {
  init_timer(&timer);
  timer.expires = jiffies + CHECK_INTERVAL;
  timer.data = 0;
  timer.function = &doFix;
  add_timer(&timer);
  return(0);
}

void fixRaid_cleanup(void) {
  del_timer(&timer);
}

module_init(fixRaid_init);
module_exit(fixRaid_cleanup);
-- 

Bill Shubert (wms@igoweb.org) <mailto:wms@igoweb.org>
http://www.igoweb.org/~wms/ <http://igoweb.org/%7Ewms/>