[linux-lvm] Troubleshooting LVM IO errors on debian 3.0

[Kirk Ismay <captain@netidea.com>]

Hi everyone,

I have a Dell Poweredge 2550 running debian 3.0 (woody), kernel 2.4.25.
I've recently installed a second set of 36GB SCSI disks, configured as a
RAID-1 mirror using the PERC controller and mounted as /home. Its
formatted as ext2 (originally ext3, but I converted back thinking that
was the issue). I've had this happen about 4 times now, twice before the
conversion back to ext2.  The server is a web, mail, and nfs server for
an ISP, and needs to as reliable as possible.

I wanted to be able to use LVM to combine my new /home partition with
the old one, giving me more space.

I was first alerted to the problem when a customer reported that they
were unable to view their website.

ls -l gave me errors like this:
/bin/ls: .addressbook: Input/output error
/bin/ls: .lynx_cookies: Input/output error
/bin/ls: .procmailrc: Input/output error
/bin/ls: .kermrc: Input/output error

The console was spewing LVM IO errors (sorry, I didn't write them down).
    One disk had an amber blinking light, but that dissappeared after
the reboot. When I halted the system, I got the 'VFS: Busy inodes after
unmount. Self-destruct in 5 seconds' message, which was quite alarming.

My questions:
1. Where should I be looking for the problem? Is it LVM?
2. What are some solutions? Should I upgrade my kernel?
3. How stable is LVM on kernel 2.4?
4. What's up with the "Self destruct" message?  Is it really that severe?
5. How can I capture the kernel error messages to a remote log server?
(I'm using syslog-ng)

Additional Information, including dmesg vgdisplay and kernel config:
http://community.netidea.com/admin/jan2006/

Thanks.

-- 
Sincerely,
Kirk Ismay
System Administrator

--
Net Idea
201-625 Front Street Nelson, BC V1L 4B6
P:250-352-3512 | F:250-352-9780 | TF:888-246-4222

10 Years of Service Excellence!

Visit us online at:
www.netidea.com | www.netidea.biz