* Re: xfs_growfs doesn't resize @ 2011-07-06 22:51 kkeller 2011-07-07 18:25 ` Keith Keller 0 siblings, 1 reply; 8+ messages in thread From: kkeller @ 2011-07-06 22:51 UTC (permalink / raw) To: Eric Sandeen; +Cc: xfs Hello again XFS folks, I have finally made the time to revisit this, after copying most of my data elsewhere. On Sun 03/07/11 9:41 PM , Eric Sandeen <sandeen@sandeen.net> wrote: > On 7/3/11 11:34 PM, kkeller@sonic.net wrote: > > How safe is running xfs_db with -r on my mounted filesystem? I > > it's safe. At worst it might read inconsistent data, but it's > perfectly safe. So, here is my xfs_db output. This is still on a mounted filesystem. # xfs_db -r -c 'sb 0' -c 'print' /dev/mapper/saharaVG-saharaLV magicnum = 0x58465342 blocksize = 4096 dblocks = 5371061248 rblocks = 0 rextents = 0 uuid = 1bffcb88-0d9d-4228-93af-83ec9e208e88 logstart = 2147483652 rootino = 128 rbmino = 129 rsumino = 130 rextsize = 1 agblocks = 91552192 agcount = 59 rbmblocks = 0 logblocks = 32768 versionnum = 0x30e4 sectsize = 512 inodesize = 256 inopblock = 16 fname = "\000\000\000\000\000\000\000\000\000\000\000\000" blocklog = 12 sectlog = 9 inodelog = 8 inopblog = 4 agblklog = 27 rextslog = 0 inprogress = 0 imax_pct = 25 icount = 19556544 ifree = 1036 fdblocks = 2634477046 frextents = 0 uquotino = 131 gquotino = 132 qflags = 0x7 flags = 0 shared_vn = 0 inoalignmt = 2 unit = 0 width = 0 dirblklog = 0 logsectlog = 0 logsectsize = 0 logsunit = 0 features2 = 0 # xfs_db -r -c 'sb 1' -c 'print' /dev/mapper/saharaVG-saharaLV magicnum = 0x58465342 blocksize = 4096 dblocks = 2929670144 rblocks = 0 rextents = 0 uuid = 1bffcb88-0d9d-4228-93af-83ec9e208e88 logstart = 2147483652 rootino = 128 rbmino = 129 rsumino = 130 rextsize = 1 agblocks = 91552192 agcount = 32 rbmblocks = 0 logblocks = 32768 versionnum = 0x30e4 sectsize = 512 inodesize = 256 inopblock = 16 fname = "\000\000\000\000\000\000\000\000\000\000\000\000" blocklog = 12 sectlog = 9 inodelog = 8 inopblog = 4 agblklog = 27 rextslog = 0 inprogress = 0 imax_pct = 25 icount = 19528640 ifree = 15932 fdblocks = 170285408 frextents = 0 uquotino = 131 gquotino = 132 qflags = 0x7 flags = 0 shared_vn = 0 inoalignmt = 2 unit = 0 width = 0 dirblklog = 0 logsectlog = 0 logsectsize = 0 logsunit = 0 features2 = 0 I can immediately see with a diff that dblocks and agcount are different. Some other variables are also different, namely icount, ifree, and fdblocks, which I am unclear how to interpret. But judging from the other threads I quoted, it seems that dblocks and agcount are using values for a 20TB filesystem, and that therefore on a umount the filesystem will become (at least temporarily) unmountable. I've seen two different routes for trying to correct this issue--either use xfs_db to manipulate the values directly, or using xfs_repair on a frozen ro-mounted filesystem with a dump from xfs_metadata. My worry about the latter is twofold--will I even be able to do a remount? And will I have space for a dump from xfs_metadata of an 11TB filesystem? I have also seen advice in some of the other threads that xfs_repair can actually make the damage worse (though presumably xfs_repair -n should be safe). If xfs_db is a better way to go, and if the values xfs_db returns on a umount don't change, would I simply do this? # xfs_db -x /dev/mapper/saharaVG-saharaLV sb 0 w dblocks = 2929670144 w agcount = 32 and then do an xfs_repair -n? A route I have used many ages ago, on ext2 filesystems, was to specify an alternate superblock when running e2fsck. Can xfs_repair do this? > Get a recent xfsprogs too, if you haven't already, it scales better > than the really old versions. I think I may have asked this in another post, but would you suggest compiling 3.0 from source? The version that CentOS distributes is marked as 2.9.4, but I don't know what patches they've applied (if any). Would 3.0 be more likely to help recover the fs? Thanks all for your patience! --keith -- kkeller@sonic.net _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: xfs_growfs doesn't resize 2011-07-06 22:51 xfs_growfs doesn't resize kkeller @ 2011-07-07 18:25 ` Keith Keller 2011-07-07 19:34 ` Eric Sandeen 0 siblings, 1 reply; 8+ messages in thread From: Keith Keller @ 2011-07-07 18:25 UTC (permalink / raw) To: xfs Hi all, First, I hope that this message fixes my mail client breaking threading. I am sorry for following up my own post (again), but I realized this morning that there may be another possible risk I had not considered: On Wed, Jul 06, 2011 at 03:51:32PM -0700, kkeller@sonic.net wrote: > > So, here is my xfs_db output. This is still on a mounted filesystem. How safe/risky is it to leave this filesystem mounted and in use? I'm not too concerned about new data, since it won't be a huge amount, but I am wondering if data that's already been written may be at risk. Or, it it a reasonable guess that the kernel is still working completely with the old filesystem geometry, and so won't write anything beyond the old limits while it's still mounted? df certainly seems to use the old fs size, not the new one. Thanks again, --keith -- kkeller@sonic.net _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: xfs_growfs doesn't resize 2011-07-07 18:25 ` Keith Keller @ 2011-07-07 19:34 ` Eric Sandeen 2011-07-07 22:23 ` Keith Keller 0 siblings, 1 reply; 8+ messages in thread From: Eric Sandeen @ 2011-07-07 19:34 UTC (permalink / raw) To: Keith Keller; +Cc: xfs On 7/7/11 1:25 PM, Keith Keller wrote: > Hi all, > > First, I hope that this message fixes my mail client breaking threading. > > I am sorry for following up my own post (again), but I realized this > morning that there may be another possible risk I had not considered: > > On Wed, Jul 06, 2011 at 03:51:32PM -0700, kkeller@sonic.net wrote: >> >> So, here is my xfs_db output. This is still on a mounted filesystem. > > How safe/risky is it to leave this filesystem mounted and in use? > I'm not too concerned about new data, since it won't be a huge amount, > but I am wondering if data that's already been written may be at risk. > Or, it it a reasonable guess that the kernel is still working completely > with the old filesystem geometry, and so won't write anything beyond the > old limits while it's still mounted? df certainly seems to use the old > fs size, not the new one. I don't remember all the implications of this very old bug... It seems like you need an "exit strategy" - you probably cannot leave your fs mounted -forever- ... If it were me, if possible, I'd make backups of the fs as it's mounted now, then umount it and make an xfs_metadump of it, restore that metadump to a new sparse file, and point xfs_repair at that metadata image file, to see what repair might do with it. If repair eats it alive, then we can look into more xfs_db type surgery to fix things up more nicely... -Eric > Thanks again, > > > --keith > > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: xfs_growfs doesn't resize 2011-07-07 19:34 ` Eric Sandeen @ 2011-07-07 22:23 ` Keith Keller 2011-07-07 22:30 ` Eric Sandeen 0 siblings, 1 reply; 8+ messages in thread From: Keith Keller @ 2011-07-07 22:23 UTC (permalink / raw) To: Eric Sandeen; +Cc: xfs On Thu, Jul 07, 2011 at 02:34:12PM -0500, Eric Sandeen wrote: > > It seems like you need an "exit strategy" - you probably cannot leave > your fs mounted -forever- ... Yes, of course. I didn't mean to imply that I'd leave it like this indefinitely. :) I will be on vacation next week, and it's not really reschedulable. So my plan was to ride the filesystem through next week, hope for the best, then fix it when I return, rather than attempt to start a fix now and risk ending up with a broken filesystem. Ideally I would preemptively switch to a warm backup before I leave, but that won't be ready in time. (I currently do have all the important data backed up, but it is to various different spaces where I had free disk space. The warm backup should be ready early next week if the filesystem does go belly-up before I return.) > If it were me, if possible, I'd make backups of the fs as it's mounted > now, then umount it and make an xfs_metadump of it, restore that metadump > to a new sparse file, and point xfs_repair at that metadata image file, > to see what repair might do with it. > > If repair eats it alive, then we can look into more xfs_db type surgery > to fix things up more nicely... This sounds like a reasonable plan. It looks like xfs_metadump needs a umounted or readonly filesystem in order to work properly; is there any way to estimate how long such a dump would take, and how large it would be from an almost-full 11TB fs with however many inodes it has (~19 million IIRC)? I want to plan downtime and usable disk space accordingly. Would xfs_metadump create the same dump from a filesystem remounted ro as from a filesystem not mounted? I think you suggested this idea in an earlier post. In a very optimistic scenario, I could imagine remounting the original filesystem ro, taking the metadump, then being able to remount rw so that I could put it back into service while I work with the metadump. Then, once I knew more about the metadump, I could do an actual umount and fix the filesystem using the information gathered from the metadump testing. If they will produce the same metadump, then it could be a win-win if it's able to successfully remount rw afterwards; and if it can't, it wasn't any additional effort or risk to try. Will xfsprogs 3.1.5 work with the older kernel, and will it make a better dump than 2.9.4? I have built xfsprogs from source, but if it might have problems working with the kmod-xfs kernel module I can use the 2.9.4 tools instead. (Again, in keeping with the hope-for-the-best scenario above, if avoiding a reboot won't be too harmful it'd be convenient.) I think you also mentioned that xfs_metadump can not dump frozen filesystems, but the man page for 3.1.5 says it can. FWIW xfs_metadump refused to work on a frozen filesystem on my test machine, which is version 2.9.4 (though from an older CentOS base). (xfs_freeze does look like a nice tool though!) --keith -- kkeller@sonic.net _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: xfs_growfs doesn't resize 2011-07-07 22:23 ` Keith Keller @ 2011-07-07 22:30 ` Eric Sandeen 2011-07-20 19:08 ` xfs_growfs doesn't resize (update) Keith Keller 0 siblings, 1 reply; 8+ messages in thread From: Eric Sandeen @ 2011-07-07 22:30 UTC (permalink / raw) To: Keith Keller; +Cc: xfs On 7/7/11 5:23 PM, Keith Keller wrote: > On Thu, Jul 07, 2011 at 02:34:12PM -0500, Eric Sandeen wrote: ... >> If it were me, if possible, I'd make backups of the fs as it's mounted >> now, then umount it and make an xfs_metadump of it, restore that metadump >> to a new sparse file, and point xfs_repair at that metadata image file, >> to see what repair might do with it. >> >> If repair eats it alive, then we can look into more xfs_db type surgery >> to fix things up more nicely... > > This sounds like a reasonable plan. It looks like xfs_metadump needs a > umounted or readonly filesystem in order to work properly; is there any > way to estimate how long such a dump would take, and how large it would > be from an almost-full 11TB fs with however many inodes it has (~19 million > IIRC)? I want to plan downtime and usable disk space accordingly. well, I'm looking at an image of a 4T fs right now, with 208k inodes, and the image itself took up 800M (a 4T sparse file when restored, of course, but only using 800M) > Would xfs_metadump create the same dump from a filesystem remounted ro > as from a filesystem not mounted? I think you suggested this idea in yes, looks like it works, with recent tools anyway. > an earlier post. In a very optimistic scenario, I could imagine > remounting the original filesystem ro, taking the metadump, then being > able to remount rw so that I could put it back into service while I > work with the metadump. Then, once I knew more about the metadump, I I think that should work. > could do an actual umount and fix the filesystem using the information > gathered from the metadump testing. If they will produce the same > metadump, then it could be a win-win if it's able to successfully > remount rw afterwards; and if it can't, it wasn't any additional effort > or risk to try. agreed. > Will xfsprogs 3.1.5 work with the older kernel, and will it make a > better dump than 2.9.4? I have built xfsprogs from source, but if it 2.9.4 won't have xfs_metadump ... and no problems with newer tools on older kernels. It's just reading the block device, in any case. No unique kernel interaction. > might have problems working with the kmod-xfs kernel module I can use > the 2.9.4 tools instead. (Again, in keeping with the hope-for-the-best > scenario above, if avoiding a reboot won't be too harmful it'd be > convenient.) I think you can. > I think you also mentioned that xfs_metadump can not dump frozen > filesystems, but the man page for 3.1.5 says it can. FWIW xfs_metadump > refused to work on a frozen filesystem on my test machine, which is > version 2.9.4 (though from an older CentOS base). (xfs_freeze does look > like a nice tool though!) it should(?) but: # xfs_metadump /dev/loop0 blah xfs_metadump: /dev/loop0 contains a mounted and writable filesystem fatal error -- couldn't initialize XFS library # xfs_freeze -f mnt/ # xfs_metadump /dev/loop0 blah xfs_metadump: /dev/loop0 contains a mounted and writable filesystem fatal error -- couldn't initialize XFS library # xfs_freeze -u mnt # mount -o remount,ro mnt # xfs_metadump /dev/loop0 blah <proceeds w/o problems> I think we should make the tools work with freeze, but remount,ro works fine too. -Eric > --keith > > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: xfs_growfs doesn't resize (update) 2011-07-07 22:30 ` Eric Sandeen @ 2011-07-20 19:08 ` Keith Keller 2011-07-25 18:28 ` Keith Keller 0 siblings, 1 reply; 8+ messages in thread From: Keith Keller @ 2011-07-20 19:08 UTC (permalink / raw) To: xfs Hello again list! Back from my vacation, with no problems on the filesystem I potentially messed up by running xfs_growfs with an old kernel a few weeks ago. I was able to take a metadump of the filesystem by remounting ro, running xfs_metadump, and remounting rw, which did not seem to cause any obvious problems. Since then I have been throwing the various xfs tools at it to see what I could find out. The last thing I tried gave an interesting result: I simply did an mdrestore and attempted a mount over loopback, on a different machine that is running the latest kernel available from the CentOS repository (2.6.18-238.9.1.el5). To my great surprise, it mounted cleanly, and reported the correct filesystem size! A very cursory glance at the filesystem didn't uncover any glaring errors (though I forgot to use -o when I ran xfs_metadump, so filenames are mangled). The reason I tried this at all was that I got some unexpected results from xfs_repair in various permutations. First, just running it against the restored fs image, I got this error (using a locally-compiled xfsprogs-3.1.5, not the xfsprogs CentOS package--all references to xfsprogs programs will be this version unless stated otherwise): Phase 1 - find and verify superblock... would write modified primary superblock Primary superblock would have been modified. Cannot proceed further in no_modify mode. Exiting now. That wasn't a huge surprise, so I reran xfs_repair without -n, and got many errors like so: bad directory block magic # 0 in block 0 for directory inode 16359537 corrupt block 0 in directory inode 16359537 will junk block When I examined the files added to lost+found, there were a large number of them, but they all were owned by the same user and generated within a few days of each other. (Are owner and group UIDs preserved by a metadump? It would seem so.) I thought this was curious, and wasn't sure exactly what was going on, so I ran the xfs_db alterations that I asked about in a previous post (basically resetting dblocks and agcount to the values from sb 1) and ran xfs_repair on the modified image. This time, running with -n succeeded fine, but reported that it would fix almost the same set of errors that the first xfs_repair did report that it tried to fix. So, I am wondering if those files are truly damaged, and something else caused the filesystem inconsistency, not xfs_growfs. (The dates on the files in lost+found are about a week after we had a power outage, but months before my failed xfs_growfs attempt.) At this point, I really don't know what to try next. I still have the metadump, so I can do pretty much anything I want to it to try to narrow down the best course of action on the actual filesystem. What information would be most helpful for me to report? If it would help to see the actual output from xfs_repair, it's almost certainly too big to post to the list, but I can make them available over http. Just for posterity's sake, xfs_metadump on my almost-full 11TB filesystem took about 15 minutes, and generated a ~6.7GB dump file. Restoring this dump to a sparse file takes about 4 minutes. xfs_repair takes about 30-45 minutes on these images (I haven't been tracking the times that carefully). I'm sure running these tools on an actual 11TB filesystem would take a whole lot longer, so having the metadump is really a great tool for this kind of work. Thanks for reading! --keith -- kkeller@sonic.net _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: xfs_growfs doesn't resize (update) 2011-07-20 19:08 ` xfs_growfs doesn't resize (update) Keith Keller @ 2011-07-25 18:28 ` Keith Keller 2011-08-01 4:46 ` xfs_growfs doesn't resize (partial resolution) Keith Keller 0 siblings, 1 reply; 8+ messages in thread From: Keith Keller @ 2011-07-25 18:28 UTC (permalink / raw) To: xfs Hi again all, I thought about this a bit more over the past few days, and did some more testing this morning. I am currently thinking that I really don't have as many paths to follow as I originally thought. It seems like, whether I modify sb 0 with xfs_db or not, xfs_repair still wants to see an 11TB filesystem--I did an mdrestore and mount on the metadump image, which saw a 21TB filesystem, then did a umount and xfs_repair, which modified the superblock. On mounting again, the filesystem was back to 11TB. So I think there must be a definite risk of data loss if I try to mount what the latest kernel thinks is a 21TB filesystem, then need to run a repair at a later date, and therefore I have to run an xfs_repair before trying to use the new free space. So, here is what I think is my plan for the actual filesystem: --take another backup --umount all XFS filesystems (the OS filesystems are ext3) --remove the kmod-xfs CentOS package --update to the latest CentOS kernel and reboot, making sure the target XFS fs does not have a mount attempted --run xfs_repair from xfsprogs-3.1.5 --cross fingers :) --mount and check what's in lost+found --if all seems well, attempt another xfs_growfs using xfsprogs-3.1.5 Does this seem like a reasonable plan of attack? If so, is there a way to estimate how long the actual xfs_repair will take from my xfs_repair sessions on the metadump image? Obviously the hardware isn't the same, but I'd just hope for a back of the envelope estimate, not necessarily something terribly accurate. Finally, are there other things I can try on the metadump image first to give me more information on what'll happen on the live filesystem? Thanks again! --keith -- kkeller@wombat.san-francisco.ca.us _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: xfs_growfs doesn't resize (partial resolution) 2011-07-25 18:28 ` Keith Keller @ 2011-08-01 4:46 ` Keith Keller 0 siblings, 0 replies; 8+ messages in thread From: Keith Keller @ 2011-08-01 4:46 UTC (permalink / raw) To: xfs Hello again everyone, I was able to perform the steps I proposed last week: On Mon, Jul 25, 2011 at 11:28:51AM -0700, Keith Keller wrote: > --take another backup > --umount all XFS filesystems (the OS filesystems are ext3) > --remove the kmod-xfs CentOS package > --update to the latest CentOS kernel and reboot, making sure > the target XFS fs does not have a mount attempted > --run xfs_repair from xfsprogs-3.1.5 > --cross fingers :) > --mount and check what's in lost+found For posterity sake, I thought I should post my results. Everything went very smoothly, actually--even though the repair of the metadump I took reported errors, when I worked with the actual filesystem, it fixed superblock 0 but otherwise found no errors. The xfs_repair took about 42 minutes, compared to about 35-40 minutes to operate (on similar but different hardware) on the metadump. That's really nice performance--I had to e2fsck a ~5TB filesystem the other day, and it took about 5 hours total. > --if all seems well, attempt another xfs_growfs using xfsprogs-3.1.5 I haven't yet attempted this final step. I am going to wait till I return from another vacation to do so. :) But I am hopeful that with the latest kernel available from the CentOS repos that I should be fine. Is there a way to directly query the running module for version information, so that I can try to verify that I have a version that will work? Here's my uname, if it's helpful: Linux xxxxxxxxxx 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux --keith -- kkeller@wombat.san-francisco.ca.us _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2011-08-01 4:46 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-07-06 22:51 xfs_growfs doesn't resize kkeller 2011-07-07 18:25 ` Keith Keller 2011-07-07 19:34 ` Eric Sandeen 2011-07-07 22:23 ` Keith Keller 2011-07-07 22:30 ` Eric Sandeen 2011-07-20 19:08 ` xfs_growfs doesn't resize (update) Keith Keller 2011-07-25 18:28 ` Keith Keller 2011-08-01 4:46 ` xfs_growfs doesn't resize (partial resolution) Keith Keller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox