Is it possible the check an frozen XFS filesytem to avoid downtime

Is it possible the check an frozen XFS filesytem to avoid downtime

[Martin Steigerwald]

Hi!

We seen in-memory corruption on two XFS filesystem on a server heartbeat 
cluster of one of our customers:


XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file 
fs/xfs/xfs_alloc.c.  Caller 0xffffffff8824eb5d

Call Trace:
 [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
 [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
 [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
 [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
 [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
 [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
 [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
 [<ffffffff8022a06d>] notify_change+0x156/0x2ef
 [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
 [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
 [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
 [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
 [<ffffffff8025b20b>] __down_read+0x12/0x9a
 [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
 [<ffffffff80255f38>] child_rip+0xa/0x12
 [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
 [<ffffffff80255f2e>] child_rip+0x0/0x12

xfs_force_shutdown(dm-1,0x8) called from line 4261 of file fs/xfs/xfs_bmap.c.  
Return address = 0xffffffff88258673
Filesystem "dm-1": Corruption of in-memory data detected.  Shutting down 
filesystem: dm-1
Please umount the filesystem, and rectify the problem(s)

on

Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1) (nobse@backports.org) 
(gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Tue Jun 5 
07:43:32 UTC 2007


We plan to do a takeover so that the server which appears to have memory 
errors can be memtested. 

After the takeover we would like to make sure that the XFS filesystems are 
intact. Is it possible to do so without taking the filesystem completely 
offline?

I thought about mounting read only and it might be the best choice available, 
but then it will *fail* write accesses. I would prefer if these are just 
stalled.

I tried xfs_freeze -f on my laptop home directory, but then did not machine to 
get it check via xfs_check or xfs_repair -nd... is it possible at all?

Ciao,
-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Timothy Shimmin]

Hi Martin,

Martin Steigerwald wrote:
> Hi!
> 
> We seen in-memory corruption on two XFS filesystem on a server heartbeat 
> cluster of one of our customers:
> 
> 
> XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file 
> fs/xfs/xfs_alloc.c.  Caller 0xffffffff8824eb5d
> 
> Call Trace:
>  [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
>  [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
>  [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
>  [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
>  [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
>  [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
>  [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
>  [<ffffffff8022a06d>] notify_change+0x156/0x2ef
>  [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
>  [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
>  [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
>  [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
>  [<ffffffff8025b20b>] __down_read+0x12/0x9a
>  [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
>  [<ffffffff80255f38>] child_rip+0xa/0x12
>  [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
>  [<ffffffff80255f2e>] child_rip+0x0/0x12
> 
> xfs_force_shutdown(dm-1,0x8) called from line 4261 of file fs/xfs/xfs_bmap.c.  
> Return address = 0xffffffff88258673
> Filesystem "dm-1": Corruption of in-memory data detected.  Shutting down 
> filesystem: dm-1
> Please umount the filesystem, and rectify the problem(s)
> 
> on
> 
> Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1) (nobse@backports.org) 
> (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Tue Jun 5 
> 07:43:32 UTC 2007
> 
> 
> We plan to do a takeover so that the server which appears to have memory 
> errors can be memtested. 
> 
> After the takeover we would like to make sure that the XFS filesystems are 
> intact. Is it possible to do so without taking the filesystem completely 
> offline?
> 
> I thought about mounting read only and it might be the best choice available, 
> but then it will *fail* write accesses. I would prefer if these are just 
> stalled.
> 
> I tried xfs_freeze -f on my laptop home directory, but then did not machine to 
> get it check via xfs_check or xfs_repair -nd... is it possible at all?
> 
> Ciao,


When I last tried (and I don't think Barry has done anything to it to change
things) it wouldn't work.
However, I think it could/should be changed to make it work.

My notes from the SGI bug:

958642: running xfs_check and "xfs_repair -n" on a frozen xfs filesystem
> We've been asked a few times about the possibility of running xfs_check
> or xfs_repair -n on a frozen filesystem.
> And a while back I looked into what some of the hinderances were.
> And now I've forgotten ;-))
> 
> I think there are hinderances for libxfs_init (check_open()) and
> for having a dirty log.
> 
> For libxfs_init, I found that I couldn't run the tools without error'ing out.
> I think I found out that I needed the INACTIVE flag,
> without READONLY/DANGEROUSLY, like xfs_logprint does.
> 
> ----------------------------------------
> Date: Thu, 19 Oct 2006 11:24:06 +1000
> From: Timothy Shimmin <tes@sgi.com>
> To: lachlan@sgi.com
> cc: xfs-dev@sgi.com
> Subject: Re: init.c patch
> ------------------------------------------------------
>   Ok, my understanding of the READONLY/DANGEROUSLY flags were wrong.
>   I thought they were just overriding flags when you were guaranteeing you were only reading
>   and it would be more permissive,
>   but they are for doing stuff on readonly (ro) mounts.
>   They are rather confusing to me. When you go with defaults for repair and db then
>   it doesn't set the INACTIVE flag.
>   It means if I do _not_ want to be fatal then I need to set INACTIVE but not set READONLY or
>   DANGEROUSLY - which is what logprint does.
> 
>   I would have thought they'd be an option which for commands which don't modify anything,
>   that they can read from a non-ro mounted filesystem (at the users risk) -
>   which is what logprint does. i.e an option which just sets INACTIVE and only
>   produces a warning.
> 
>   The other alternative is to be able to test for a frozen fs as you suggested.
> ----------------------------------------------------------
> 
> Lachlan suggested using a check_isfrozen() routine instead of overriding
> check_isactive().
> 
> 
> And as far as the dirty log is concerned...
> It will be dirty when it is frozen, but in a special way.
> It will have an unmount record followed by a dummy record -
> solely used so that when mounted again it can do
> the unlinked list processing.
> So we could add code to test if the log just had an unmount record
> followed by a dummy record and continue anyway knowing that
> the metadata was consistent.
> e.g. in xfs_repair/phase2.c:zero_log() it calls xlog_find_tail()
> and tests if (head_blk != tail_blk) to know if the log is dirty.
> I think libxfs should provide a routine: libxfs_dirty_log
> or in the libxlog code with a suitable name,
> which could say how dirty the log is ;-)
> Is it dirty such that we have real transactions to replay or
> does it just have to do the unlinked processing as in the case of
> a frozen filesystem.
> It would be nice anyway to have an abstraction here because
> it is finding out the head and tail blocks solely for this purpose
> and doesn't really care what they are.
> 
> --Tim


--Tim

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Martin Steigerwald]

Am Dienstag, 15. Juli 2008 05:38:23 schrieb Timothy Shimmin:
> Hi Martin,

Hi Tim,

> Martin Steigerwald wrote:
> > Hi!
> >
> > We seen in-memory corruption on two XFS filesystem on a server heartbeat
> > cluster of one of our customers:
> >
> >
> > XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file
> > fs/xfs/xfs_alloc.c.  Caller 0xffffffff8824eb5d
> >
> > Call Trace:
> >  [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
> >  [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
> >  [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
> >  [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
> >  [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
> >  [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
> >  [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
> >  [<ffffffff8022a06d>] notify_change+0x156/0x2ef
> >  [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
> >  [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
> >  [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
> >  [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
> >  [<ffffffff8025b20b>] __down_read+0x12/0x9a
> >  [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
> >  [<ffffffff80255f38>] child_rip+0xa/0x12
> >  [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
> >  [<ffffffff80255f2e>] child_rip+0x0/0x12
> >
> > xfs_force_shutdown(dm-1,0x8) called from line 4261 of file
> > fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673
> > Filesystem "dm-1": Corruption of in-memory data detected.  Shutting down
> > filesystem: dm-1
> > Please umount the filesystem, and rectify the problem(s)
> >
> > on
> >
> > Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1)
> > (nobse@backports.org) (gcc version 4.1.2 20061115 (prerelease) (Debian
> > 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007
> >
> >
> > We plan to do a takeover so that the server which appears to have memory
> > errors can be memtested.
> >
> > After the takeover we would like to make sure that the XFS filesystems
> > are intact. Is it possible to do so without taking the filesystem
> > completely offline?
> >
> > I thought about mounting read only and it might be the best choice
> > available, but then it will *fail* write accesses. I would prefer if
> > these are just stalled.
> >
> > I tried xfs_freeze -f on my laptop home directory, but then did not
> > machine to get it check via xfs_check or xfs_repair -nd... is it possible
> > at all?
> >
> > Ciao,
>
> When I last tried (and I don't think Barry has done anything to it to
> change things) it wouldn't work.
> However, I think it could/should be changed to make it work.

Okay... we recommended the customer to do it the safe way unmounting the 
filesystem completely. He did and the filesystem appear to be intact *phew*. 
XFS appeared to detect the in memory corruption early enough.

Its a bit strange however, cause we now know that the server sports ECC RAM. 
Well we will see what memtest86+ has to say about it.

> My notes from the SGI bug:
>
> 958642: running xfs_check and "xfs_repair -n" on a frozen xfs filesystem
>
> > We've been asked a few times about the possibility of running xfs_check
> > or xfs_repair -n on a frozen filesystem.
> > And a while back I looked into what some of the hinderances were.
> > And now I've forgotten ;-))
> >
> > I think there are hinderances for libxfs_init (check_open()) and
> > for having a dirty log.
> >
> > For libxfs_init, I found that I couldn't run the tools without error'ing
> > out. I think I found out that I needed the INACTIVE flag,
> > without READONLY/DANGEROUSLY, like xfs_logprint does.
> >
> > ----------------------------------------
> > Date: Thu, 19 Oct 2006 11:24:06 +1000
> > From: Timothy Shimmin <tes@sgi.com>
> > To: lachlan@sgi.com
> > cc: xfs-dev@sgi.com
> > Subject: Re: init.c patch
> > ------------------------------------------------------
> >   Ok, my understanding of the READONLY/DANGEROUSLY flags were wrong.
> >   I thought they were just overriding flags when you were guaranteeing
> > you were only reading and it would be more permissive,
> >   but they are for doing stuff on readonly (ro) mounts.
> >
> >   They are rather confusing to me. When you go with defaults for repair
> > and db then it doesn't set the INACTIVE flag.
> >   It means if I do _not_ want to be fatal then I need to set INACTIVE but
> > not set READONLY or DANGEROUSLY - which is what logprint does.
> >

I think that there should be different options for readonly / frozen fs 
checking and dangerous repair... since I think readonly checks are a 
different thing than repairing a mounted filesystem and hoping that the 
running XFS will not choke upon the filesystem that xfs_repair changes under 
its hood.

I expected a "-r" for read only in xfs_check and xfs_repair, well but for 
xfs_repair this option is already taken for specifying the realtime volume.

Ciao,
-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Eric Sandeen]

Martin Steigerwald wrote:

> Okay... we recommended the customer to do it the safe way unmounting the 
> filesystem completely. He did and the filesystem appear to be intact *phew*. 
> XFS appeared to detect the in memory corruption early enough.
> 
> Its a bit strange however, cause we now know that the server sports ECC RAM. 
> Well we will see what memtest86+ has to say about it.

in-memory corruption could mean, but certainly does not absolutely mean,
problematic memory.  It could be, and usually is, a plain ol' bug (in
xfs or elsewhere).

-Eric

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Martin Steigerwald]

Am Dienstag, 15. Juli 2008 17:27:39 schrieb Eric Sandeen:
> Martin Steigerwald wrote:
> > Okay... we recommended the customer to do it the safe way unmounting the
> > filesystem completely. He did and the filesystem appear to be intact
> > *phew*. XFS appeared to detect the in memory corruption early enough.
> >
> > Its a bit strange however, cause we now know that the server sports ECC
> > RAM. Well we will see what memtest86+ has to say about it.
>
> in-memory corruption could mean, but certainly does not absolutely mean,
> problematic memory.  It could be, and usually is, a plain ol' bug (in
> xfs or elsewhere).

Thanks. 

Yes, I thought about this, too. But then the machine ran over one year without 
any visible issues. And it happened only on one server, not on the other. It 
happened on the server that does NFS tough... could be an NFS related issue 
then. The other one does MySQL with the database stored on a XFS volume, too. 
But there haven't been any visible issues.

Well we will see whether it happens again on the server that has taken over 
and is now doing both MySQL and NFS. If it does, I think we will update to 
one of the lastest Debian Etch backport kernels (2.6.24 or even 2.6.25) on 
one of the servers and see whether that helps.

-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Martin Steigerwald]

Am Dienstag, 15. Juli 2008 schrieb Eric Sandeen:
> Martin Steigerwald wrote:
> > Okay... we recommended the customer to do it the safe way unmounting the
> > filesystem completely. He did and the filesystem appear to be intact
> > *phew*. XFS appeared to detect the in memory corruption early enough.
> >
> > Its a bit strange however, cause we now know that the server sports ECC
> > RAM. Well we will see what memtest86+ has to say about it.
>
> in-memory corruption could mean, but certainly does not absolutely mean,
> problematic memory.  It could be, and usually is, a plain ol' bug (in
> xfs or elsewhere).

Ok, just as a follow up:

Now we got similar XFS errors on the second backend server, this time on a 
local hardware RAID1 while on the first backend server it was on logical 
volumes on a soft RAID spread over two dislocated external hardware RAID 
boxes.

So this appears to be an XFS bug to me. Maybe when running for long time it 
corrupts its in-memory structures. Fortunately we did not see errors in 
on-disk structures.

A colleague did a kernel update on the inactive backend 1 server from 2.6.21 
to 2.6.26 kernel from backports.org, tommorow backend 2 will follow. Let's 
see whether that solves the issue.

Anyway it seems to be a hard to trigger bug and before bugging you with 
something in kernel 2.6.21, we at least update to the latest backports.org 
kernel.

-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Eric Sandeen]

Martin Steigerwald wrote:

> A colleague did a kernel update on the inactive backend 1 server from 2.6.21 
> to 2.6.26 kernel from backports.org, tommorow backend 2 will follow. Let's 
> see whether that solves the issue.
> 
> Anyway it seems to be a hard to trigger bug and before bugging you with 
> something in kernel 2.6.21, we at least update to the latest backports.org 
> kernel.

Honestly, I'd try a 2.6.27 kernel if you can, a few more problems were
fixed there.

-Eric

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Martin Steigerwald]

Am Monday 27 October 2008 18:15:19 schrieb Eric Sandeen:
> Martin Steigerwald wrote:
> > A colleague did a kernel update on the inactive backend 1 server from
> > 2.6.21 to 2.6.26 kernel from backports.org, tommorow backend 2 will
> > follow. Let's see whether that solves the issue.
> >
> > Anyway it seems to be a hard to trigger bug and before bugging you with
> > something in kernel 2.6.21, we at least update to the latest
> > backports.org kernel.
>
> Honestly, I'd try a 2.6.27 kernel if you can, a few more problems were
> fixed there.

Thanks for the hint. backports.org doesn't contain a 2.6.26 and AFAIK lenny 
will ship with 2.6.26 as well, thus I'd like to try that one before doing a 
backport of 2.6.27 from sid myself. Thus I hope the issue that is triggered 
on those servers is fixed with 2.6.26 or some prior kernel since 2.6.21 
already. If not, we will consider backporting 2.6.27.

At least on my notebook I didn't have any issues so far with any kernel since 
2.6.17.7. But even with TuxOnIce it doesn't have uptimes of 100 days or more, 
since I compile a new kernel for it once in a while.

-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Timothy Shimmin]

Martin Steigerwald wrote:
> Am Dienstag, 15. Juli 2008 05:38:23 schrieb Timothy Shimmin:
>> Hi Martin,
> 
> Hi Tim,
> 
>> Martin Steigerwald wrote:
>>> Hi!
>>>
>>> We seen in-memory corruption on two XFS filesystem on a server heartbeat
>>> cluster of one of our customers:
>>>
>>>
>>> XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file
>>> fs/xfs/xfs_alloc.c.  Caller 0xffffffff8824eb5d
>>>
>>> Call Trace:
>>>  [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
>>>  [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
>>>  [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
>>>  [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
>>>  [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
>>>  [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
>>>  [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
>>>  [<ffffffff8022a06d>] notify_change+0x156/0x2ef
>>>  [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
>>>  [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
>>>  [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
>>>  [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
>>>  [<ffffffff8025b20b>] __down_read+0x12/0x9a
>>>  [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
>>>  [<ffffffff80255f38>] child_rip+0xa/0x12
>>>  [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
>>>  [<ffffffff80255f2e>] child_rip+0x0/0x12
>>>
>>> xfs_force_shutdown(dm-1,0x8) called from line 4261 of file
>>> fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673
>>> Filesystem "dm-1": Corruption of in-memory data detected.  Shutting down
>>> filesystem: dm-1
>>> Please umount the filesystem, and rectify the problem(s)
>>>
>>> on
>>>
>>> Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1)
>>> (nobse@backports.org) (gcc version 4.1.2 20061115 (prerelease) (Debian
>>> 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007
>>>
>>>
>>> We plan to do a takeover so that the server which appears to have memory
>>> errors can be memtested.
>>>
>>> After the takeover we would like to make sure that the XFS filesystems
>>> are intact. Is it possible to do so without taking the filesystem
>>> completely offline?
>>>
>>> I thought about mounting read only and it might be the best choice
>>> available, but then it will *fail* write accesses. I would prefer if
>>> these are just stalled.
>>>
>>> I tried xfs_freeze -f on my laptop home directory, but then did not
>>> machine to get it check via xfs_check or xfs_repair -nd... is it possible
>>> at all?
>>>
>>> Ciao,
>> When I last tried (and I don't think Barry has done anything to it to
>> change things) it wouldn't work.
>> However, I think it could/should be changed to make it work.
> 
> Okay... we recommended the customer to do it the safe way unmounting the 
> filesystem completely. He did and the filesystem appear to be intact *phew*. 
> XFS appeared to detect the in memory corruption early enough.
> 
> Its a bit strange however, cause we now know that the server sports ECC RAM. 
> Well we will see what memtest86+ has to say about it.
> 
>> My notes from the SGI bug:
>>
>> 958642: running xfs_check and "xfs_repair -n" on a frozen xfs filesystem
>>
>>> We've been asked a few times about the possibility of running xfs_check
>>> or xfs_repair -n on a frozen filesystem.
>>> And a while back I looked into what some of the hinderances were.
>>> And now I've forgotten ;-))
>>>
>>> I think there are hinderances for libxfs_init (check_open()) and
>>> for having a dirty log.
>>>
>>> For libxfs_init, I found that I couldn't run the tools without error'ing
>>> out. I think I found out that I needed the INACTIVE flag,
>>> without READONLY/DANGEROUSLY, like xfs_logprint does.
>>>
>>> ----------------------------------------
>>> Date: Thu, 19 Oct 2006 11:24:06 +1000
>>> From: Timothy Shimmin <tes@sgi.com>
>>> To: lachlan@sgi.com
>>> cc: xfs-dev@sgi.com
>>>   It means if I do _not_ want to be fatal then I need to set INACTIVE but
>>> not set READONLY or DANGEROUSLY - which is what logprint does.
>>>
> 
> I think that there should be different options for readonly / frozen fs 
> checking and dangerous repair... since I think readonly checks are a 
> different thing than repairing a mounted filesystem and hoping that the 
> running XFS will not choke upon the filesystem that xfs_repair changes under 
> its hood.
> 
> I expected a "-r" for read only in xfs_check and xfs_repair, well but for 
> xfs_repair this option is already taken for specifying the realtime volume.
> 
> Ciao,


My comments mentioned above were for the c flags used in
libxfs and the commands.
I was curious to know why logprint was able to continue printing and doing
its stuff but repair and check couldn't and would exit.
What actual _command_ options/flags we use is a different matter and I probably
confused the issue by using the term flags.

Looking at the code:

#define LIBXFS_ISREADONLY       0x0002  /* disallow all mounted filesystems */
#define LIBXFS_ISINACTIVE       0x0004  /* allow mounted only if mounted ro */
#define LIBXFS_DANGEROUSLY      0x0008  /* repairing a device mounted ro    */


       -n     No modify mode.  Specifies that xfs_repair should not modify the
              filesystem but should only scan the filesystem and indicate what
              repairs would have been made.

       -d     Repair dangerously. Allow xfs_repair to repair an XFS filesystem
              mounted read only. This is typically done on  a  root  fileystem
              from single user mode, immediately followed by a reboot.

        if (no_modify) /* -n */
                args->isreadonly = (LIBXFS_ISREADONLY  | LIBXFS_ISINACTIVE);
        else if (dangerously) /* -d */
                args->isreadonly = (LIBXFS_DANGEROUSLY | LIBXFS_ISINACTIVE);

The comment for LIBXFS_ISINACTIVE seems a bit misleading as I believe
it will allow things to continue on a filesystem which is not marked
as readonly as xfs_logprint does.

Looking at:

init.c:	
        if (!readonly && !inactive && platform_check_ismounted(path, *blockfile, NULL, 1))
                return 0;
        
        if (inactive && check_isactive(path, *blockfile, ((readonly|dangerously)?1:0)))
                return 0; 

static int
check_isactive(char *name, char *block, int fatal)
{      
....
        return platform_check_iswritable(name, block, &st, fatal);
}

Okay, so this is why it worked fine for just ISINACTIVE but not when
either ISREADONLY or DANGEROUSLY are set.

Yes, it would be nice for the commands to detect that the filesystem is
frozen and then allow things to proceed in a non modifying mode.

--Tim

Re: Is it possible the check an frozen XFS filesytem to avoid downtime

[Martin Steigerwald]

Am Dienstag, 15. Juli 2008 05:38:23 schrieb Timothy Shimmin:
> Hi Martin,
>
> Martin Steigerwald wrote:
> > Hi!
> >
> > We seen in-memory corruption on two XFS filesystem on a server heartbeat
> > cluster of one of our customers:
> >
> >
> > XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file
> > fs/xfs/xfs_alloc.c.  Caller 0xffffffff8824eb5d
> >
> > Call Trace:
> >  [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
> >  [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
> >  [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
> >  [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
> >  [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
> >  [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
> >  [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
> >  [<ffffffff8022a06d>] notify_change+0x156/0x2ef
> >  [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
> >  [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
> >  [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
> >  [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
> >  [<ffffffff8025b20b>] __down_read+0x12/0x9a
> >  [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
> >  [<ffffffff80255f38>] child_rip+0xa/0x12
> >  [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
> >  [<ffffffff80255f2e>] child_rip+0x0/0x12
> >
> > xfs_force_shutdown(dm-1,0x8) called from line 4261 of file
> > fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673
> > Filesystem "dm-1": Corruption of in-memory data detected.  Shutting down
> > filesystem: dm-1
> > Please umount the filesystem, and rectify the problem(s)
> >
> > on
> >
> > Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1)
> > (nobse@backports.org) (gcc version 4.1.2 20061115 (prerelease) (Debian
> > 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007
> >
> >
> > We plan to do a takeover so that the server which appears to have memory
> > errors can be memtested.
> >
> > After the takeover we would like to make sure that the XFS filesystems
> > are intact. Is it possible to do so without taking the filesystem
> > completely offline?
> >
> > I thought about mounting read only and it might be the best choice
> > available, but then it will *fail* write accesses. I would prefer if
> > these are just stalled.
> >
> > I tried xfs_freeze -f on my laptop home directory, but then did not
> > machine to get it check via xfs_check or xfs_repair -nd... is it possible
> > at all?
> >
> > Ciao,
>
> When I last tried (and I don't think Barry has done anything to it to
> change things) it wouldn't work.
> However, I think it could/should be changed to make it work.

I am wondering whether it would need to set an option at all.

Shouldn't checking a filesystem that is not being written too be safe? So 
xfs_check and xfs_repair -n could just check whether fs is readonly or frozen 
and if so continue without requiring a special option? They can print the 
fact aka

Checking a frozen filesystem

or 

Checking a read only filesystem

in the beginning tough.

Only thing is the log itself... when that is not cleared upon a freeze or 
readonly mount, it might be a problem for xfs_check and xfs_repair -n.

Ciao,
-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90