Re: xfs_growfs doesn't resize

Re: xfs_growfs doesn't resize

[kkeller]

Hello again XFS folks,

I have finally made the time to revisit this, after copying most of my data
elsewhere.

On Sun 03/07/11  9:41 PM , Eric Sandeen <sandeen@sandeen.net> wrote:
> On 7/3/11 11:34 PM, kkeller@sonic.net wrote:

> > How safe is running xfs_db with -r on my mounted filesystem? I
> 
> it's safe. At worst it might read inconsistent data, but it's
> perfectly safe.

So, here is my xfs_db output.  This is still on a mounted filesystem.

# xfs_db -r -c 'sb 0' -c 'print' /dev/mapper/saharaVG-saharaLV
magicnum = 0x58465342
blocksize = 4096
dblocks = 5371061248
rblocks = 0
rextents = 0
uuid = 1bffcb88-0d9d-4228-93af-83ec9e208e88
logstart = 2147483652
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 91552192
agcount = 59
rbmblocks = 0
logblocks = 32768
versionnum = 0x30e4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 27
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 19556544
ifree = 1036
fdblocks = 2634477046
frextents = 0
uquotino = 131
gquotino = 132
qflags = 0x7
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0


#  xfs_db -r -c 'sb 1' -c 'print' /dev/mapper/saharaVG-saharaLV
magicnum = 0x58465342
blocksize = 4096
dblocks = 2929670144
rblocks = 0
rextents = 0
uuid = 1bffcb88-0d9d-4228-93af-83ec9e208e88
logstart = 2147483652
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 91552192
agcount = 32
rbmblocks = 0
logblocks = 32768
versionnum = 0x30e4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 27
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 19528640
ifree = 15932
fdblocks = 170285408
frextents = 0
uquotino = 131
gquotino = 132
qflags = 0x7
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0


I can immediately see with a diff that dblocks and agcount are
different.  Some other variables are also different, namely icount,
ifree, and fdblocks, which I am unclear how to interpret.  But judging
from the other threads I quoted, it seems that dblocks and agcount
are using values for a 20TB filesystem, and that therefore on a umount
the filesystem will become (at least temporarily) unmountable.

I've seen two different routes for trying to correct this issue--either use
xfs_db to manipulate the values directly, or using xfs_repair on a frozen
ro-mounted filesystem with a dump from xfs_metadata.  My worry about
the latter is twofold--will I even be able to do a remount?  And will I
have space for a dump from xfs_metadata of an 11TB filesystem?  I have
also seen advice in some of the other threads that xfs_repair can actually
make the damage worse (though presumably xfs_repair -n should be safe).

If xfs_db is a better way to go, and if the values xfs_db returns on a
umount don't change, would I simply do this?

# xfs_db -x /dev/mapper/saharaVG-saharaLV
sb 0 w dblocks = 2929670144 w agcount = 32

and then do an xfs_repair -n?

A route I have used many ages ago, on ext2 filesystems, was to specify
an alternate superblock when running e2fsck.  Can xfs_repair do this?

> Get a recent xfsprogs too, if you haven't already, it scales better
> than the really old versions.

I think I may have asked this in another post, but would you suggest
compiling 3.0 from source?  The version that CentOS distributes is marked
as 2.9.4, but I don't know what patches they've applied (if any).  Would 3.0
be more likely to help recover the fs?

Thanks all for your patience!

--keith

-- 
kkeller@sonic.net


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't resize

[Keith Keller]

Hi all,

First, I hope that this message fixes my mail client breaking threading.

I am sorry for following up my own post (again), but I realized this
morning that there may be another possible risk I had not considered:

On Wed, Jul 06, 2011 at 03:51:32PM -0700, kkeller@sonic.net wrote:
> 
> So, here is my xfs_db output.  This is still on a mounted filesystem.

How safe/risky is it to leave this filesystem mounted and in use?
I'm not too concerned about new data, since it won't be a huge amount,
but I am wondering if data that's already been written may be at risk.
Or, it it a reasonable guess that the kernel is still working completely
with the old filesystem geometry, and so won't write anything beyond the
old limits while it's still mounted?  df certainly seems to use the old
fs size, not the new one.

Thanks again,


--keith


-- 
kkeller@sonic.net

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't resize

[Eric Sandeen]

On 7/7/11 1:25 PM, Keith Keller wrote:
> Hi all,
> 
> First, I hope that this message fixes my mail client breaking threading.
> 
> I am sorry for following up my own post (again), but I realized this
> morning that there may be another possible risk I had not considered:
> 
> On Wed, Jul 06, 2011 at 03:51:32PM -0700, kkeller@sonic.net wrote:
>>
>> So, here is my xfs_db output.  This is still on a mounted filesystem.
> 
> How safe/risky is it to leave this filesystem mounted and in use?
> I'm not too concerned about new data, since it won't be a huge amount,
> but I am wondering if data that's already been written may be at risk.
> Or, it it a reasonable guess that the kernel is still working completely
> with the old filesystem geometry, and so won't write anything beyond the
> old limits while it's still mounted?  df certainly seems to use the old
> fs size, not the new one.

I don't remember all the implications of this very old bug...

It seems like you need an "exit strategy" - you probably cannot leave
your fs mounted -forever- ...

If it were me, if possible, I'd make backups of the fs as it's mounted
now, then umount it and make an xfs_metadump of it, restore that metadump
to a new sparse file, and point xfs_repair at that metadata image file,
to see what repair might do with it.

If repair eats it alive, then we can look into more xfs_db type surgery
to fix things up more nicely...

-Eric

> Thanks again,
> 
> 
> --keith
> 
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't resize

[Keith Keller]

On Thu, Jul 07, 2011 at 02:34:12PM -0500, Eric Sandeen wrote:
> 
> It seems like you need an "exit strategy" - you probably cannot leave
> your fs mounted -forever- ...

Yes, of course.  I didn't mean to imply that I'd leave it like this
indefinitely.  :)

I will be on vacation next week, and it's not really reschedulable.  So
my plan was to ride the filesystem through next week, hope for the best,
then fix it when I return, rather than attempt to start a fix now and
risk ending up with a broken filesystem.  Ideally I would preemptively
switch to a warm backup before I leave, but that won't be ready in time.
(I currently do have all the important data backed up, but it is to
various different spaces where I had free disk space.  The warm backup
should be ready early next week if the filesystem does go belly-up
before I return.)

> If it were me, if possible, I'd make backups of the fs as it's mounted
> now, then umount it and make an xfs_metadump of it, restore that metadump
> to a new sparse file, and point xfs_repair at that metadata image file,
> to see what repair might do with it.
>
> If repair eats it alive, then we can look into more xfs_db type surgery
> to fix things up more nicely...

This sounds like a reasonable plan.  It looks like xfs_metadump needs a
umounted or readonly filesystem in order to work properly; is there any
way to estimate how long such a dump would take, and how large it would
be from an almost-full 11TB fs with however many inodes it has (~19 million
IIRC)?  I want to plan downtime and usable disk space accordingly.

Would xfs_metadump create the same dump from a filesystem remounted ro
as from a filesystem not mounted?  I think you suggested this idea in
an earlier post.  In a very optimistic scenario, I could imagine
remounting the original filesystem ro, taking the metadump, then being
able to remount rw so that I could put it back into service while I
work with the metadump.  Then, once I knew more about the metadump, I
could do an actual umount and fix the filesystem using the information
gathered from the metadump testing.  If they will produce the same
metadump, then it could be a win-win if it's able to successfully
remount rw afterwards; and if it can't, it wasn't any additional effort
or risk to try.

Will xfsprogs 3.1.5 work with the older kernel, and will it make a
better dump than 2.9.4?  I have built xfsprogs from source, but if it
might have problems working with the kmod-xfs kernel module I can use
the 2.9.4 tools instead.  (Again, in keeping with the hope-for-the-best
scenario above, if avoiding a reboot won't be too harmful it'd be
convenient.)

I think you also mentioned that xfs_metadump can not dump frozen
filesystems, but the man page for 3.1.5 says it can.  FWIW xfs_metadump
refused to work on a frozen filesystem on my test machine, which is
version 2.9.4 (though from an older CentOS base).  (xfs_freeze does look
like a nice tool though!)

--keith


-- 
kkeller@sonic.net

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't resize

[Eric Sandeen]

On 7/7/11 5:23 PM, Keith Keller wrote:
> On Thu, Jul 07, 2011 at 02:34:12PM -0500, Eric Sandeen wrote:

...

>> If it were me, if possible, I'd make backups of the fs as it's mounted
>> now, then umount it and make an xfs_metadump of it, restore that metadump
>> to a new sparse file, and point xfs_repair at that metadata image file,
>> to see what repair might do with it.
>>
>> If repair eats it alive, then we can look into more xfs_db type surgery
>> to fix things up more nicely...
> 
> This sounds like a reasonable plan.  It looks like xfs_metadump needs a
> umounted or readonly filesystem in order to work properly; is there any
> way to estimate how long such a dump would take, and how large it would
> be from an almost-full 11TB fs with however many inodes it has (~19 million
> IIRC)?  I want to plan downtime and usable disk space accordingly.

well, I'm looking at an image of a 4T fs right now, with 208k inodes,
and the image itself took up 800M (a 4T sparse file when restored,
of course, but only using 800M)

> Would xfs_metadump create the same dump from a filesystem remounted ro
> as from a filesystem not mounted?  I think you suggested this idea in

yes, looks like it works, with recent tools anyway.

> an earlier post.  In a very optimistic scenario, I could imagine
> remounting the original filesystem ro, taking the metadump, then being
> able to remount rw so that I could put it back into service while I
> work with the metadump.  Then, once I knew more about the metadump, I

I think that should work.

> could do an actual umount and fix the filesystem using the information
> gathered from the metadump testing.  If they will produce the same
> metadump, then it could be a win-win if it's able to successfully
> remount rw afterwards; and if it can't, it wasn't any additional effort
> or risk to try.

agreed.

> Will xfsprogs 3.1.5 work with the older kernel, and will it make a
> better dump than 2.9.4?  I have built xfsprogs from source, but if it

2.9.4 won't have xfs_metadump ... and no problems with newer tools on
older kernels.  It's just reading the block device, in any case.
No unique kernel interaction.

> might have problems working with the kmod-xfs kernel module I can use
> the 2.9.4 tools instead.  (Again, in keeping with the hope-for-the-best
> scenario above, if avoiding a reboot won't be too harmful it'd be
> convenient.)

I think you can.

> I think you also mentioned that xfs_metadump can not dump frozen
> filesystems, but the man page for 3.1.5 says it can.  FWIW xfs_metadump
> refused to work on a frozen filesystem on my test machine, which is
> version 2.9.4 (though from an older CentOS base).  (xfs_freeze does look
> like a nice tool though!)

it should(?) but:

# xfs_metadump /dev/loop0 blah
xfs_metadump: /dev/loop0 contains a mounted and writable filesystem

fatal error -- couldn't initialize XFS library

# xfs_freeze -f mnt/
# xfs_metadump /dev/loop0 blah
xfs_metadump: /dev/loop0 contains a mounted and writable filesystem

fatal error -- couldn't initialize XFS library

# xfs_freeze -u mnt
# mount -o remount,ro mnt
# xfs_metadump /dev/loop0 blah

<proceeds w/o problems>

I think we should make the tools work with freeze, but remount,ro works fine too.

-Eric

> --keith
> 
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't resize (update)

[Keith Keller]

Hello again list!

Back from my vacation, with no problems on the filesystem I
potentially messed up by running xfs_growfs with an old kernel
a few weeks ago.  I was able to take a metadump of the filesystem
by remounting ro, running xfs_metadump, and remounting rw, which did
not seem to cause any obvious problems.  Since then I have been 
throwing the various xfs tools at it to see what I could find out.

The last thing I tried gave an interesting result: I simply
did an mdrestore and attempted a mount over loopback, on a
different machine that is running the latest kernel available
from the CentOS repository (2.6.18-238.9.1.el5).  To my great 
surprise, it mounted cleanly, and reported the correct filesystem
size!  A very cursory glance at the filesystem didn't uncover
any glaring errors (though I forgot to use -o when I ran
xfs_metadump, so filenames are mangled).

The reason I tried this at all was that I got some unexpected
results from xfs_repair in various permutations.  First, just
running it against the restored fs image, I got this error (using
a locally-compiled xfsprogs-3.1.5, not the xfsprogs CentOS
package--all references to xfsprogs programs will be this version
unless stated otherwise):

Phase 1 - find and verify superblock...
would write modified primary superblock
Primary superblock would have been modified.
Cannot proceed further in no_modify mode.
Exiting now.

That wasn't a huge surprise, so I reran xfs_repair without -n, and
got many errors like so:

bad directory block magic # 0 in block 0 for directory inode 16359537
corrupt block 0 in directory inode 16359537
        will junk block

When I examined the files added to lost+found, there were a large number
of them, but they all were owned by the same user and generated within
a few days of each other.  (Are owner and group UIDs preserved by a
metadump?  It would seem so.)  I thought this was curious, and wasn't
sure exactly what was going on, so I ran the xfs_db alterations that
I asked about in a previous post (basically resetting dblocks and agcount
to the values from sb 1) and ran xfs_repair on the modified image.  This
time, running with -n succeeded fine, but reported that it would fix
almost the same set of errors that the first xfs_repair did report that
it tried to fix.  So, I am wondering if those files are truly damaged,
and something else caused the filesystem inconsistency, not xfs_growfs.
(The dates on the files in lost+found are about a week after we had a
power outage, but months before my failed xfs_growfs attempt.)

At this point, I really don't know what to try next.  I still have the
metadump, so I can do pretty much anything I want to it to try to narrow
down the best course of action on the actual filesystem.  What information
would be most helpful for me to report?  If it would help to see the
actual output from xfs_repair, it's almost certainly too big to post to
the list, but I can make them available over http.

Just for posterity's sake, xfs_metadump on my almost-full 11TB filesystem
took about 15 minutes, and generated a ~6.7GB dump file.  Restoring this
dump to a sparse file takes about 4 minutes.  xfs_repair takes about 30-45
minutes on these images (I haven't been tracking the times that
carefully).  I'm sure running these tools on an actual 11TB filesystem
would take a whole lot longer, so having the metadump is really a great
tool for this kind of work.

Thanks for reading!

--keith



-- 
kkeller@sonic.net

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't resize (update)

[Keith Keller]

Hi again all,

I thought about this a bit more over the past few days, and did 
some more testing this morning.  I am currently thinking that I
really don't have as many paths to follow as I originally thought.

It seems like, whether I modify sb 0 with xfs_db or not, xfs_repair
still wants to see an 11TB filesystem--I did an mdrestore and mount
on the metadump image, which saw a 21TB filesystem, then did a umount
and xfs_repair, which modified the superblock.  On mounting again, the
filesystem was back to 11TB.  So I think there must be a definite
risk of data loss if I try to mount what the latest kernel thinks is
a 21TB filesystem, then need to run a repair at a later date, and
therefore I have to run an xfs_repair before trying to use the new
free space.

So, here is what I think is my plan for the actual filesystem:

--take another backup
--umount all XFS filesystems (the OS filesystems are ext3)
--remove the kmod-xfs CentOS package
--update to the latest CentOS kernel and reboot, making sure
  the target XFS fs does not have a mount attempted
--run xfs_repair from xfsprogs-3.1.5
--cross fingers :)
--mount and check what's in lost+found
--if all seems well, attempt another xfs_growfs using xfsprogs-3.1.5

Does this seem like a reasonable plan of attack?  If so, is there
a way to estimate how long the actual xfs_repair will take from my
xfs_repair sessions on the metadump image?  Obviously the hardware
isn't the same, but I'd just hope for a back of the envelope estimate,
not necessarily something terribly accurate.

Finally, are there other things I can try on the metadump image first
to give me more information on what'll happen on the live filesystem?

Thanks again!

--keith

-- 
kkeller@wombat.san-francisco.ca.us

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't resize (partial resolution)

[Keith Keller]

Hello again everyone,

I was able to perform the steps I proposed last week:

On Mon, Jul 25, 2011 at 11:28:51AM -0700, Keith Keller wrote:
> --take another backup
> --umount all XFS filesystems (the OS filesystems are ext3)
> --remove the kmod-xfs CentOS package
> --update to the latest CentOS kernel and reboot, making sure
>   the target XFS fs does not have a mount attempted
> --run xfs_repair from xfsprogs-3.1.5
> --cross fingers :)
> --mount and check what's in lost+found

For posterity sake, I thought I should post my results.  Everything went
very smoothly, actually--even though the repair of the metadump I took
reported errors, when I worked with the actual filesystem, it fixed
superblock 0 but otherwise found no errors.  The xfs_repair took about
42 minutes, compared to about 35-40 minutes to operate (on similar but
different hardware) on the metadump.  That's really nice performance--I
had to e2fsck a ~5TB filesystem the other day, and it took about 5 hours
total.

> --if all seems well, attempt another xfs_growfs using xfsprogs-3.1.5

I haven't yet attempted this final step.  I am going to wait till I
return from another vacation to do so.  :)  But I am hopeful that with
the latest kernel available from the CentOS repos that I should be fine.
Is there a way to directly query the running module for version
information, so that I can try to verify that I have a version that will
work?  Here's my uname, if it's helpful:

Linux xxxxxxxxxx 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

--keith

-- 
kkeller@wombat.san-francisco.ca.us

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs