XFS repair on / in a hosted environment

XFS repair on / in a hosted environment

[Rupa Schomaker]

I have a virtual server hosted in a XEN environment.  Kernel is
2.6.16.13-XenU and I can't change it. :( Host provider is not responsive
to getting us a newer kernel.

Anyway, during my nightly rsync backup, I started seeing the following:

Jul 18 12:51:01 hosted kernel: 0x0: 00 00 00 00 01 00 00 00 23 00 00 00
00 00 00 00
Jul 18 12:51:01 hosted kernel: Filesystem "sda1": XFS internal error
xfs_da_do_buf(2) at line 2221 of file fs/xfs/xfs_da_btree.c.  Caller
0xc02288fd
Jul 18 12:51:01 hosted kernel:  [<c0237b71>] xfs_corruption_error+0xbb/0xc6
Jul 18 12:51:01 hosted kernel:  [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel:  [<c025e4e5>] xfs_buf_read_flags+0x1f/0x82
Jul 18 12:51:01 hosted kernel:  [<c025b1aa>] kmem_zone_alloc+0x43/0x88
Jul 18 12:51:01 hosted kernel:  [<c02287bf>] xfs_da_do_buf+0x5df/0x6a9
Jul 18 12:51:01 hosted kernel:  [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel:  [<c0169d88>] dput+0x1b/0x11c
Jul 18 12:51:01 hosted kernel:  [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel:  [<c022cc12>]
xfs_dir2_block_getdents+0xe0/0x2e4
Jul 18 12:51:01 hosted kernel:  [<c022cc12>]
xfs_dir2_block_getdents+0xe0/0x2e4
Jul 18 12:51:01 hosted kernel:  [<c013e7ee>]
get_page_from_freelist+0xa8/0x34b
Jul 18 12:51:01 hosted kernel:  [<c0219919>] xfs_bmap_last_offset+0xd5/0xe8
Jul 18 12:51:01 hosted kernel:  [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel:  [<c022c17e>] xfs_dir2_isblock+0x1c/0x65
Jul 18 12:51:01 hosted kernel:  [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel:  [<c022c452>] xfs_dir2_getdents+0xb0/0xe4
Jul 18 12:51:01 hosted kernel:  [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel:  [<c02559d4>] xfs_readdir+0x3f/0x51
Jul 18 12:51:01 hosted kernel:  [<c025ea5e>] linvfs_readdir+0xfd/0x1d2
Jul 18 12:51:01 hosted kernel:  [<c0165d46>] vfs_readdir+0x42/0x6f
Jul 18 12:51:01 hosted kernel:  [<c0165e59>] filldir64+0x0/0xc6
Jul 18 12:51:01 hosted kernel:  [<c0165f82>] sys_getdents64+0x63/0xa9
Jul 18 12:51:01 hosted kernel:  [<c0165e59>] filldir64+0x0/0xc6
Jul 18 12:51:01 hosted kernel:  [<c0104869>] syscall_call+0x7/0xb


I've narrowed this down to a single directory which cannot be removed:

/usr/src/linux-2.6.16-xen/sound/pci/au88x0

any attempt to get the contents of the directory results in an error and
the trace:

hosted:/usr/src/linux-2.6.16-xen/sound/pci# ls -l au88x0/
ls: reading directory au88x0/: Unknown error 990


Everything is on / so I can't easily get it unmounted to run xfs_repair.

I try booting into single user mode, mount -o remount ,ro /.  xfs_repair
-d /dev/sda1 still complains about a mounted writable partition.

===

On boot, mount shows:

/dev/sda1 on / type xfs (rw)

# mount -o remount,ro /

after the remount, mount still shows / as rw

# mount
/dev/sda1 on / type xfs (rw)

but attempts to modify it result in:

hosted:~# touch foo
touch: cannot touch `foo': Read-only file system


Ok, so try xfs_repair:

hosted:~# xfs_repair -d /dev/sda1
xfs_repair: /dev/sda1 contains a mounted and writable filesystem

fatal error -- couldn't initialize XFS library

xfs_repair thinks we are writable.

I tried doing this as early as I could in the boot process by putting a
script in

/etc/rcS.d/S00fixroot

but that is not early enough.

Any thoughts?

-Rupa

Re: XFS repair on / in a hosted environment

[David Chinner]

On Wed, Jul 18, 2007 at 12:24:59PM -0700, Rupa Schomaker wrote:
> I have a virtual server hosted in a XEN environment.  Kernel is
> 2.6.16.13-XenU and I can't change it. :( Host provider is not responsive
> to getting us a newer kernel.
> 
> Anyway, during my nightly rsync backup, I started seeing the following:
.....
> 
> On boot, mount shows:
> 
> /dev/sda1 on / type xfs (rw)
> 
> # mount -o remount,ro /
> 
> after the remount, mount still shows / as rw
> 
> # mount
> /dev/sda1 on / type xfs (rw)
> 

what does /proc/mounts tell you (rather than /etc/mtab)?

If /proc/mounts says ro, then try upgrading your xfsprogs package.
and then retrying the repair....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS repair on / in a hosted environment

[Rupa Schomaker]

On 7/18/2007 7:34 PM, David Chinner wrote:
> On Wed, Jul 18, 2007 at 12:24:59PM -0700, Rupa Schomaker wrote:
>> I have a virtual server hosted in a XEN environment.  Kernel is
>> 2.6.16.13-XenU and I can't change it. :( Host provider is not responsive
>> to getting us a newer kernel.
>>
>> Anyway, during my nightly rsync backup, I started seeing the following:
> .....
> 
> what does /proc/mounts tell you (rather than /etc/mtab)?
> 
> If /proc/mounts says ro, then try upgrading your xfsprogs package.
> and then retrying the repair....

/proc/mounts shows ro.

What version is new enough?  I'm running debian sarge, so have:

# xfs_repair -V
xfs_repair version 2.9.0

ftp://oss.sgi.com/projects/xfs/cmd_tars/

only shows a xfs_progs up to 2.8.21-1.

> Cheers,
> 
> Dave.

Thanks,

-Rupa

Re: XFS repair on / in a hosted environment

[Rupa Schomaker]

On 7/19/2007 6:09 AM, Rupa Schomaker wrote:
> 
> On 7/18/2007 7:34 PM, David Chinner wrote:
>> On Wed, Jul 18, 2007 at 12:24:59PM -0700, Rupa Schomaker wrote:
>>> I have a virtual server hosted in a XEN environment.  Kernel is
>>> 2.6.16.13-XenU and I can't change it. :( Host provider is not responsive
>>> to getting us a newer kernel.
>>>
>>> Anyway, during my nightly rsync backup, I started seeing the following:
>> .....
>>
>> what does /proc/mounts tell you (rather than /etc/mtab)?
>>
>> If /proc/mounts says ro, then try upgrading your xfsprogs package.
>> and then retrying the repair....
> 
> /proc/mounts shows ro.

I built the latest xfsprogs from cvs.  Still no go.  This is perhaps
something strange with the Xen environment?

hosted:/# mount
/dev/sda1 on / type xfs (rw)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)

hosted:/# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / xfs ro 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid 0 0
proc /proc proc rw,nosuid,nodev,noexec 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec 0 0
none /dev/shm tmpfs rw 0 0


mount and /proc/mounts don't agree.  Thre is no /dev/root.  I tried the
mount -o remount,ro on both / and /dev/sda1.  No good, only /dev/root is
being changed to ro.

<boggle>

Ok, created a /dev/root.  xfs_repair works as expected.  Now to figure
out what got dumped into lost+found.

-Rupa

Re: XFS repair on / in a hosted environment

[David Chinner]

On Thu, Jul 19, 2007 at 09:25:14AM -0700, Rupa Schomaker wrote:
> On 7/19/2007 6:09 AM, Rupa Schomaker wrote:
> > 
> > On 7/18/2007 7:34 PM, David Chinner wrote:
> >> On Wed, Jul 18, 2007 at 12:24:59PM -0700, Rupa Schomaker wrote:
> >>> I have a virtual server hosted in a XEN environment.  Kernel is
> >>> 2.6.16.13-XenU and I can't change it. :( Host provider is not responsive
> >>> to getting us a newer kernel.
> >>>
> >>> Anyway, during my nightly rsync backup, I started seeing the following:
> >> .....
> >>
> >> what does /proc/mounts tell you (rather than /etc/mtab)?
> >>
> >> If /proc/mounts says ro, then try upgrading your xfsprogs package.
> >> and then retrying the repair....
> > 
> > /proc/mounts shows ro.
> 
> I built the latest xfsprogs from cvs.  Still no go.  This is perhaps
> something strange with the Xen environment?

The above /etc/mtab vs /proc/mounts problem was the reason I mentioned
you should get a recent xfsprogs - we changed it to look at /proc/mounts
rather than /etc/mtab for this reason....

> mount and /proc/mounts don't agree.  Thre is no /dev/root.  I tried the
> mount -o remount,ro on both / and /dev/sda1.  No good, only /dev/root is
> being changed to ro.
> 
> <boggle>
> 
> Ok, created a /dev/root.

That sounds like an init script or initrd setup problem....

> xfs_repair works as expected.  Now to figure
> out what got dumped into lost+found.

Oh, good to know you made it past the first (high) hurdle.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group