problems with xfs_growfs after lvextend

problems with xfs_growfs after lvextend

[Jimmy Dorff]

Hello,

I'm having a problem with a corrupt xfs filesystem after attempting to 
grow the filesystem.

The problem is very similar to this post:
http://www.redhat.com/archives/linux-lvm/2005-November/msg00026.html
Except that xfs_repair never finds a secondary superblock.

Details:
CentOS Linux,
kernel 2.6.18-128.1.16.el5.centos.plus x86_64
lvm2-2.02.40-6.el5
originally: xfsprogs-2.9.4-1.el5.centos.x86_64

This sever has been up and running for a few months w/o problems. Today 
we added some disks to a 3ware controller. The disks were all tested 
individually before installation.

Used "tw_cli" to configure a new RAID volume, which appeared in Linux as 
normal (/dev/sdd).

# pvcreate /dev/sdd
# vgextend array_vg /dev/sdd
# lvextend /dev/array_vg/data --size +12T

This all worked w/o any errors. vgdisplay and lvdisplay all report the 
correct info and sizes.

The 6TB xfs filesystem on /dev/array_vg/data was mounted as "/srv/data"
# xfs_growfs /srv/data

The size of the filesystem didn't change. I unmounted it and tried 
again, but no change. However, now I can't mount the filesystem at all.

xfs_check causes xfs_db to use so much memory as to hang up the system.
xfs_repair reports:
Phase 1 - find and verify superblock...
superblock read failed, offset 19791209299968, size 2048, ag 96, rval 0

fatal error -- Invalid argument

Also, I've noticed this in syslog:
kernel: attempt to access beyond end of device
kernel: dm-1: rw=0, want=64424509440, limit=38654705664
kernel: I/O error in filesystem ("dm-1") meta-data dev dm-1 block 
0xeffffffff       ("xfs_read_buf") error 5 buf count 512
kernel: XFS: size check 2 failed


I've trying using xfs_repair from xfsprogs 3.0.3, but it made no 
difference.

Any suggestions ? Any help understanding why this didn't work ?


Thanks,
Jimmy Dorff

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: problems with xfs_growfs after lvextend

[Eric Sandeen]

Jimmy Dorff wrote:
> Hello,
> 
> I'm having a problem with a corrupt xfs filesystem after attempting to 
> grow the filesystem.
> 
> The problem is very similar to this post:
> http://www.redhat.com/archives/linux-lvm/2005-November/msg00026.html
> Except that xfs_repair never finds a secondary superblock.
> 
> Details:
> CentOS Linux,
> kernel 2.6.18-128.1.16.el5.centos.plus x86_64
> lvm2-2.02.40-6.el5
> originally: xfsprogs-2.9.4-1.el5.centos.x86_64

> This sever has been up and running for a few months w/o problems. Today 
> we added some disks to a 3ware controller. The disks were all tested 
> individually before installation.
> 
> Used "tw_cli" to configure a new RAID volume, which appeared in Linux as 
> normal (/dev/sdd).
> 
> # pvcreate /dev/sdd
> # vgextend array_vg /dev/sdd
> # lvextend /dev/array_vg/data --size +12T

that centos kernel had a bug where growing more than 2T (I Think ...) at
a time overflowed.  So for +12T you probably hit it.

That's probably your bug, and cattelan on this list has helped people
recover from it before, I'd suggest not trying to "fix" anything just
yet (don't force repair, etc).

It's fixed in the RHEL5.4 xfs code from RH, FWIW.

-Eric


> 
> This all worked w/o any errors. vgdisplay and lvdisplay all report the 
> correct info and sizes.
> 
> The 6TB xfs filesystem on /dev/array_vg/data was mounted as "/srv/data"
> # xfs_growfs /srv/data
> 
> The size of the filesystem didn't change. I unmounted it and tried 
> again, but no change. However, now I can't mount the filesystem at all.
> 
> xfs_check causes xfs_db to use so much memory as to hang up the system.
> xfs_repair reports:
> Phase 1 - find and verify superblock...
> superblock read failed, offset 19791209299968, size 2048, ag 96, rval 0
> 
> fatal error -- Invalid argument
> 
> Also, I've noticed this in syslog:
> kernel: attempt to access beyond end of device
> kernel: dm-1: rw=0, want=64424509440, limit=38654705664
> kernel: I/O error in filesystem ("dm-1") meta-data dev dm-1 block 
> 0xeffffffff       ("xfs_read_buf") error 5 buf count 512
> kernel: XFS: size check 2 failed
> 
> 
> I've trying using xfs_repair from xfsprogs 3.0.3, but it made no 
> difference.
> 
> Any suggestions ? Any help understanding why this didn't work ?
> 
> 
> Thanks,
> Jimmy Dorff
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: problems with xfs_growfs after lvextend

[Russell Cattelan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jimmy Dorff wrote:
> Hello,
>
> I'm having a problem with a corrupt xfs filesystem after attempting
>  to grow the filesystem.
>
> The problem is very similar to this post:
> http://www.redhat.com/archives/linux-lvm/2005-November/msg00026.html
>  Except that xfs_repair never finds a secondary superblock.
>
> Details: CentOS Linux, kernel 2.6.18-128.1.16.el5.centos.plus
> x86_64 lvm2-2.02.40-6.el5 originally:
> xfsprogs-2.9.4-1.el5.centos.x86_64
>
> This sever has been up and running for a few months w/o problems.
> Today we added some disks to a 3ware controller. The disks were all
>  tested individually before installation.
>
> Used "tw_cli" to configure a new RAID volume, which appeared in
> Linux as normal (/dev/sdd).
>
> # pvcreate /dev/sdd # vgextend array_vg /dev/sdd # lvextend
> /dev/array_vg/data --size +12T
>
> This all worked w/o any errors. vgdisplay and lvdisplay all report
> the correct info and sizes.
>
> The 6TB xfs filesystem on /dev/array_vg/data was mounted as
> "/srv/data" # xfs_growfs /srv/data
>
> The size of the filesystem didn't change. I unmounted it and tried
> again, but no change. However, now I can't mount the filesystem at
> all.
>
> xfs_check causes xfs_db to use so much memory as to hang up the
> system. xfs_repair reports: Phase 1 - find and verify superblock...
>  superblock read failed, offset 19791209299968, size 2048, ag 96,
> rval 0
>
> fatal error -- Invalid argument
>
> Also, I've noticed this in syslog: kernel: attempt to access beyond
> end of device kernel: dm-1: rw=0, want=64424509440,
> limit=38654705664 kernel: I/O error in filesystem ("dm-1")
> meta-data dev dm-1 block 0xeffffffff       ("xfs_read_buf") error 5
> buf count 512 kernel: XFS: size check 2 failed
>
>
> I've trying using xfs_repair from xfsprogs 3.0.3, but it made no
> difference.
>
> Any suggestions ? Any help understanding why this didn't work ?
Sounds like you hit a known bug with xfs_growfs, actually it is a bug
with the kernel interface to growfs, the
size used to be a 32 bit number and thus would overflow if trying to
grow the filesystem by more than 2TB.

The good new is you can easily restore the the values of the primary
super block from one of the other superblocks
using the xfs_db cmd.

The bad news is that if you have run repair you may have messed things up.

basically pull up xfs_db
> sb 0 p
... output ..
> sb 1 p
... output ...

Send us the result of that and we can figure out which values need to
be restored.

- -Russell
>
>
> Thanks, Jimmy Dorff
>
> _______________________________________________ xfs mailing list
> xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKupUjNRmM+OaGhBgRAhH0AJ9Z98xpNsATGPeJAezQNwSB/sxGZACfRlwy
FFgr2iTYOYCHd0BkCsAOW6k=
=SAIq
-----END PGP SIGNATURE-----

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: problems with xfs_growfs after lvextend

[Jimmy Dorff]

On Wed, 23 Sep 2009, Russell Cattelan wrote:
> The good new is you can easily restore the the values of the primary
> super block from one of the other superblocks
> using the xfs_db cmd.
>
> The bad news is that if you have run repair you may have messed things up.
>
> basically pull up xfs_db
>> sb 0 p
> ... output ..
>> sb 1 p
> ... output ...
>
> Send us the result of that and we can figure out which values need to
> be restored.

Here you go. Thanks for your help!

xfs_db> sb 0
xfs_db> p
magicnum = 0x58465342
blocksize = 4096
dblocks = 8053063680
rblocks = 0
rextents = 0
uuid = bada0c85-0739-4fdb-99fb-6bb7777eb4d0
logstart = 1073741828
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 50331648
agcount = 160
rbmblocks = 0
logblocks = 32768
versionnum = 0x3084
sectsize = 512
inodesize = 256
inopblock = 16
fname = "DATA\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 26
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 103936
ifree = 269
fdblocks = 6499724269
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0
bad_features2 = 0
xfs_db> sb 1
xfs_db> p
magicnum = 0x58465342
blocksize = 4096
dblocks = 1610612736
rblocks = 0
rextents = 0
uuid = bada0c85-0739-4fdb-99fb-6bb7777eb4d0
logstart = 1073741828
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 50331648
agcount = 32
rbmblocks = 0
logblocks = 32768
versionnum = 0x3084
sectsize = 512
inodesize = 256
inopblock = 16
fname = "DATA\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 26
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 103936
ifree = 269
fdblocks = 57273829
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0
bad_features2 = 0

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: problems with xfs_growfs after lvextend

[Russell Cattelan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jimmy Dorff wrote:
> On Wed, 23 Sep 2009, Russell Cattelan wrote:
>> The good new is you can easily restore the the values of the
>> primary super block from one of the other superblocks using the
>> xfs_db cmd.
>>
>> The bad news is that if you have run repair you may have messed
>> things up.
>>
>> basically pull up xfs_db
>>> sb 0 p
>> ... output ..
>>> sb 1 p
>> ... output ...
>>
>> Send us the result of that and we can figure out which values
>> need to be restored.
>
> Here you go. Thanks for your help!
Ok good things look ok.
run xfs_db -x to put it into expert mode
use the write cmd to modify your sb 0 values back to what they were.

> sb 0 w dblocks = 1610612736 w agcount = 32

That should get the file system back into a state xfs_repair can
understand.
I would run xfs_repair -n and see what is says about things.
The old last AG might have been resized out to a full AG if it was a
partial AG previously
xfs_epair should be able to fix that up, but again run it with -n
first to see what is will do.

Once things are back to normal update your xfs kernel module and
re-run the grow cmd
and you should be in good shape.

- -Russell

>
> xfs_db> sb 0 xfs_db> p magicnum = 0x58465342 blocksize = 4096
> dblocks = 8053063680 rblocks = 0 rextents = 0 uuid =
> bada0c85-0739-4fdb-99fb-6bb7777eb4d0 logstart = 1073741828 rootino
> = 128 rbmino = 129 rsumino = 130 rextsize = 1 agblocks = 50331648
> agcount = 160 rbmblocks = 0 logblocks = 32768 versionnum = 0x3084
> sectsize = 512 inodesize = 256 inopblock = 16 fname =
> "DATA\000\000\000\000\000\000\000\000" blocklog = 12 sectlog = 9
> inodelog = 8 inopblog = 4 agblklog = 26 rextslog = 0 inprogress = 0
>  imax_pct = 25 icount = 103936 ifree = 269 fdblocks = 6499724269
> frextents = 0 uquotino = 0 gquotino = 0 qflags = 0 flags = 0
> shared_vn = 0 inoalignmt = 2 unit = 0 width = 0 dirblklog = 0
> logsectlog = 0 logsectsize = 0 logsunit = 0 features2 = 0
> bad_features2 = 0 xfs_db> sb 1 xfs_db> p magicnum = 0x58465342
> blocksize = 4096 dblocks = 1610612736 rblocks = 0 rextents = 0 uuid
> = bada0c85-0739-4fdb-99fb-6bb7777eb4d0 logstart = 1073741828
> rootino = 128 rbmino = 129 rsumino = 130 rextsize = 1 agblocks =
> 50331648 agcount = 32 rbmblocks = 0 logblocks = 32768 versionnum =
> 0x3084 sectsize = 512 inodesize = 256 inopblock = 16 fname =
> "DATA\000\000\000\000\000\000\000\000" blocklog = 12 sectlog = 9
> inodelog = 8 inopblog = 4 agblklog = 26 rextslog = 0 inprogress = 0
>  imax_pct = 25 icount = 103936 ifree = 269 fdblocks = 57273829
> frextents = 0 uquotino = 0 gquotino = 0 qflags = 0 flags = 0
> shared_vn = 0 inoalignmt = 2 unit = 0 width = 0 dirblklog = 0
> logsectlog = 0 logsectsize = 0 logsunit = 0 features2 = 0
> bad_features2 = 0

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKurasNRmM+OaGhBgRAt4YAJ9aHdZF3iX5hjcrcm6FqS/avk+bBQCfft0U
HTAySeS7kYxNvmnwsmalgKw=
=f80e
-----END PGP SIGNATURE-----

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: problems with xfs_growfs after lvextend

[Jimmy Dorff]

On Wed, 23 Sep 2009, Russell Cattelan wrote:
> Ok good things look ok.
> run xfs_db -x to put it into expert mode
> use the write cmd to modify your sb 0 values back to what they were.
>
>> sb 0 w dblocks = 1610612736 w agcount = 32
>

That worked perfectly! Much thanks!!

-Jimmy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: problems with xfs_growfs after lvextend

[Michael Monnerie]

On Mittwoch 23 September 2009 Jimmy Dorff wrote:
> Any suggestions ? Any help understanding why this didn't work ?

Hm, hasn't there been a problem with growing more than 2TB at once?

I search in my local box and found this, might help:
May 09: [PATCH] fix overflow in xfs_growfs_data_privat

Maybe kernel update?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs