From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: with ECARTIS (v1.0.0; list xfs); Tue, 15 Jul 2008 00:43:13 -0700 (PDT) Received: from cuda.sgi.com ([192.48.176.15]) by oss.sgi.com (8.12.11.20060308/8.12.11/SuSE Linux 0.7) with ESMTP id m6F7hB4W009682 for ; Tue, 15 Jul 2008 00:43:11 -0700 Received: from rproxy.teamix.net (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 0ACB11BB86BD for ; Tue, 15 Jul 2008 00:44:15 -0700 (PDT) Received: from rproxy.teamix.net (postman.teamix.net [194.150.191.120]) by cuda.sgi.com with ESMTP id JcVCWNswDKaIefHX for ; Tue, 15 Jul 2008 00:44:15 -0700 (PDT) From: Martin Steigerwald Subject: Re: Is it possible the check an frozen XFS filesytem to avoid downtime Date: Tue, 15 Jul 2008 09:44:12 +0200 References: <200807141542.51613.ms@teamix.de> <487C1BAF.2030404@sgi.com> In-Reply-To: <487C1BAF.2030404@sgi.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200807150944.13277.ms@teamix.de> Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com List-Id: xfs To: Timothy Shimmin Cc: xfs@oss.sgi.com Am Dienstag, 15. Juli 2008 05:38:23 schrieb Timothy Shimmin: > Hi Martin, Hi Tim, > Martin Steigerwald wrote: > > Hi! > > > > We seen in-memory corruption on two XFS filesystem on a server heartbeat > > cluster of one of our customers: > > > > > > XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file > > fs/xfs/xfs_alloc.c. Caller 0xffffffff8824eb5d > > > > Call Trace: > > [] :xfs:xfs_free_ag_extent+0x1a6/0x6b5 > > [] :xfs:xfs_free_extent+0xa9/0xc9 > > [] :xfs:xfs_bmap_finish+0xf0/0x169 > > [] :xfs:xfs_itruncate_finish+0x180/0x2c1 > > [] :xfs:xfs_setattr+0x841/0xe59 > > [] sock_common_recvmsg+0x30/0x45 > > [] :xfs:xfs_vn_setattr+0x121/0x144 > > [] notify_change+0x156/0x2ef > > [] :nfsd:nfsd_setattr+0x334/0x4b1 > > [] :nfsd:nfsd3_proc_setattr+0xa2/0xae > > [] :nfsd:nfsd_dispatch+0xdd/0x19e > > [] :sunrpc:svc_process+0x3cb/0x6d9 > > [] __down_read+0x12/0x9a > > [] :nfsd:nfsd+0x192/0x2b0 > > [] child_rip+0xa/0x12 > > [] :nfsd:nfsd+0x0/0x2b0 > > [] child_rip+0x0/0x12 > > > > xfs_force_shutdown(dm-1,0x8) called from line 4261 of file > > fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673 > > Filesystem "dm-1": Corruption of in-memory data detected. Shutting down > > filesystem: dm-1 > > Please umount the filesystem, and rectify the problem(s) > > > > on > > > > Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1) > > (nobse@backports.org) (gcc version 4.1.2 20061115 (prerelease) (Debian > > 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007 > > > > > > We plan to do a takeover so that the server which appears to have memory > > errors can be memtested. > > > > After the takeover we would like to make sure that the XFS filesystems > > are intact. Is it possible to do so without taking the filesystem > > completely offline? > > > > I thought about mounting read only and it might be the best choice > > available, but then it will *fail* write accesses. I would prefer if > > these are just stalled. > > > > I tried xfs_freeze -f on my laptop home directory, but then did not > > machine to get it check via xfs_check or xfs_repair -nd... is it possible > > at all? > > > > Ciao, > > When I last tried (and I don't think Barry has done anything to it to > change things) it wouldn't work. > However, I think it could/should be changed to make it work. Okay... we recommended the customer to do it the safe way unmounting the filesystem completely. He did and the filesystem appear to be intact *phew*. XFS appeared to detect the in memory corruption early enough. Its a bit strange however, cause we now know that the server sports ECC RAM. Well we will see what memtest86+ has to say about it. > My notes from the SGI bug: > > 958642: running xfs_check and "xfs_repair -n" on a frozen xfs filesystem > > > We've been asked a few times about the possibility of running xfs_check > > or xfs_repair -n on a frozen filesystem. > > And a while back I looked into what some of the hinderances were. > > And now I've forgotten ;-)) > > > > I think there are hinderances for libxfs_init (check_open()) and > > for having a dirty log. > > > > For libxfs_init, I found that I couldn't run the tools without error'ing > > out. I think I found out that I needed the INACTIVE flag, > > without READONLY/DANGEROUSLY, like xfs_logprint does. > > > > ---------------------------------------- > > Date: Thu, 19 Oct 2006 11:24:06 +1000 > > From: Timothy Shimmin > > To: lachlan@sgi.com > > cc: xfs-dev@sgi.com > > Subject: Re: init.c patch > > ------------------------------------------------------ > > Ok, my understanding of the READONLY/DANGEROUSLY flags were wrong. > > I thought they were just overriding flags when you were guaranteeing > > you were only reading and it would be more permissive, > > but they are for doing stuff on readonly (ro) mounts. > > > > They are rather confusing to me. When you go with defaults for repair > > and db then it doesn't set the INACTIVE flag. > > It means if I do _not_ want to be fatal then I need to set INACTIVE but > > not set READONLY or DANGEROUSLY - which is what logprint does. > > I think that there should be different options for readonly / frozen fs checking and dangerous repair... since I think readonly checks are a different thing than repairing a mounted filesystem and hoping that the running XFS will not choke upon the filesystem that xfs_repair changes under its hood. I expected a "-r" for read only in xfs_check and xfs_repair, well but for xfs_repair this option is already taken for specifying the realtime volume. Ciao, -- Martin Steigerwald - team(ix) GmbH - http://www.teamix.de gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90