Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Lance Reed]

I recently experienced a problem trying to expand an existing LVM + XFS installation.
The core problem was that xfs_growfs did not correctly resize the XFS filesystem while trying to expand from 11 TB to about 21 TB.

The previous setup had 5 x 2.18 TB LUNs using LVM2 for a total of just under 11 TB.
This is a 64bit Linux system.
Linux nfs3 2.6.18-8.1.15.el5 #1 SMP Mon Oct 22 08:32:28 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
CentOS release 5 (Final)

XFS versions:
xfsprogs-2.9.4-1.el5.centos
xfsdump-2.2.46-1.el5.centos
kmod-xfs-0.4-1.2.6.18_8.1.15.el5

LVM:
lvm2-2.02.16-3.el5

The plan was to expand to 5 more 2.18 TB LUNs for a total just under 21 TB.  This should be allowed since this is a 64 bit install.

Five new Physical LVM volumes were created and added into the LVM VG.
I extended the LVM Logical Volume by 1 TB to test.

# lvextend -L+1T  /dev/VolGroupNAS200/LogVolNAS200
  Extending logical volume LogVolNAS200 to 11.39 TB
  Logical volume LogVolNAS200 successfully resized

Then used xfs_growfs to extend the XFS filesystem.

# xfs_growfs /nas2
meta-data=/dev/VolGroupNAS200/LogVolNAS200 isize=256    agcount=34, agsize=83886080 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=2789212160, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2789212160 to 3057647616

This seemed fine and checked the XFS filesystem:

# df -Ph /nas2
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupNAS200-LogVolNAS200   12T  9.0T  2.5T  79% /nas2

Then decided to add in the rest of the remaining space to the LVM Logical Volume (saving about 512 GB)

# lvextend -L+10179G  /dev/VolGroupNAS200/LogVolNAS200
  Extending logical volume LogVolNAS200 to 21.33 TB
  Logical volume LogVolNAS200 successfully resized

[root@nfs4.arc ~]# vgdisplay VolGroupNAS200
  --- Volume group ---
  VG Name               VolGroupNAS200
  System ID
  Format                lvm2
  Metadata Areas        10
  Metadata Sequence No  10
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                10
  Act PV                10
  VG Size               21.83 TB
  PE Size               4.00 MB
  Total PE              5722830
  Alloc PE / Size       5591808 / 21.33 TB
  Free  PE / Size       131022 / 511.80 GB
  VG UUID               iNQ6VK-tdaO-fDrk-WwXk-uL3s-2zmw-j5zAcm

Then attempted to grow XFS:

~]# xfs_growfs /nas2
meta-data=/dev/VolGroupNAS200/LogVolNAS200 isize=256    agcount=37, agsize=83886080 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=3057647616, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

PROBLEM:
The xfs_growfs  returned but did not give the "data blocks changed from X to Y".

A quick google search confirmed that this would be a problem.
Attempting to umount and remount the file system proved to be a problem.

When the mount was attempted:

Apr 25 00:13:19 nfs3 kernel: SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no
debug enabled
Apr 25 00:13:19 nfs3 kernel: SGI XFS Quota Management subsystem
Apr 25 00:13:19 nfs3 kernel: attempt to access beyond end of device
Apr 25 00:13:19 nfs3 kernel: dm-5: rw=0, want=67155001344, limit=45808091136
Apr 25 00:13:19 nfs3 kernel: I/O error in filesystem ("dm-5") meta-data dev dm-5 block 0xfa2bfffff       ("xf
s_read_buf") error 5 buf count 512
Apr 25 00:13:19 nfs3 kernel: XFS: size check 2 failed
Apr 25 00:13:19 nfs3 Filesystem[4236]: [4293]: ERROR: Couldn't mount filesystem /dev/VolGroupNAS200/LogVolNAS
200 on /nas2
Apr 25 00:13:19 nfs3 Filesystem[4225]: [4294]: ERROR:  Generic error

The first attempt to run xfs_repair  did not succeed, as the terminal lost connection (network).
The second attempt to run  xfs_repair did succeed .  In this case xfs_repaired the sizes and the volume was able to be mounted.
However the filesystem is at the size of the last successful xfs_grow of 11664G (Just under 12 TB).

XFS mounting filesystem dm-5
Ending clean XFS mount for filesystem: dm-5

So, XFS did not expand correctly.  I do not know why.
I don't think this is a problem with LVM, as I have used large volumes before on LVM, and this was an expansion from an existing LVM setup that basically went from 5 x 2.18 TB LUNs to 10 x 2.18 TB LUNs.

If anyone has any ideas it would be great.
I do not know if I should attempt to expand the filesystem in 1 TB amounts using " -D size or -R size ?" option?
Should I shrink the LVM Logical volume down to something more reasonable?


Thanks in advance for any help....








[[HTML alternate version deleted]]

Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Eric Sandeen]

Lance Reed wrote:
> I recently experienced a problem trying to expand an existing LVM + XFS installation.
> The core problem was that xfs_growfs did not correctly resize the XFS filesystem while trying to expand from 11 TB to about 21 TB.
> 
> The previous setup had 5 x 2.18 TB LUNs using LVM2 for a total of just under 11 TB.
> This is a 64bit Linux system.
> Linux nfs3 2.6.18-8.1.15.el5 #1 SMP Mon Oct 22 08:32:28 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5 (Final)
> 
> XFS versions:
> xfsprogs-2.9.4-1.el5.centos
> xfsdump-2.2.46-1.el5.centos
> kmod-xfs-0.4-1.2.6.18_8.1.15.el5
> 
...

> Thanks in advance for any help....

You probably hit:

TAKE 959978 - growing an XFS filesystem by more than 2TB is broken
http://oss.sgi.com/archives/xfs/2007-01/msg00053.html

which unfortunately probably was never backported to the centos5 xfs
module.  Hmm maybe I should do that.  :(

Dave, do you remember what the fs looked like after hitting that bug?

You may want to read through the "Repairing a possibly incomplete
xfs_growfs command?" thread from earlier, on this list...

I assume you need to try to recover this fs...?  If so I'd not try
shrinking or doing anything else to the fs until there's a chance to see
what shape it's in at this point.

-Eric

RE: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Lance Reed]

Thanks for the quick response.

Actually, I was able to run an xfs_repair and all was well.(took 45 minutes...)

But I would love to be able to expand the XFS file system out to the max.

I guess I could expand it in < 2TB increments then maybe?

Thanks for the update and I will look around.

I did find this which I think is related.

http://oss.sgi.com/archives/xfs/2008-01/msg00097.html


Thanks again.

Lance


-----Original Message-----

Subject: Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

Lance Reed wrote:
> I recently experienced a problem trying to expand an existing LVM + XFS installation.
> The core problem was that xfs_growfs did not correctly resize the XFS filesystem while trying to expand from 11 TB to about 21 TB.
>
> The previous setup had 5 x 2.18 TB LUNs using LVM2 for a total of just under 11 TB.
> This is a 64bit Linux system.
> Linux nfs3 2.6.18-8.1.15.el5 #1 SMP Mon Oct 22 08:32:28 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5 (Final)
>
> XFS versions:
> xfsprogs-2.9.4-1.el5.centos
> xfsdump-2.2.46-1.el5.centos
> kmod-xfs-0.4-1.2.6.18_8.1.15.el5
>
...

> Thanks in advance for any help....

You probably hit:

TAKE 959978 - growing an XFS filesystem by more than 2TB is broken
http://oss.sgi.com/archives/xfs/2007-01/msg00053.html

which unfortunately probably was never backported to the centos5 xfs
module.  Hmm maybe I should do that.  :(

Dave, do you remember what the fs looked like after hitting that bug?

You may want to read through the "Repairing a possibly incomplete
xfs_growfs command?" thread from earlier, on this list...

I assume you need to try to recover this fs...?  If so I'd not try
shrinking or doing anything else to the fs until there's a chance to see
what shape it's in at this point.

-Eric

Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Eric Sandeen]

Lance Reed wrote:
> Thanks for the quick response.
> 
> Actually, I was able to run an xfs_repair and all was well.(took 45 minutes...)
> 
> But I would love to be able to expand the XFS file system out to the max.
> 
> I guess I could expand it in < 2TB increments then maybe?
> 
> Thanks for the update and I will look around.
> 
> I did find this which I think is related.
> 
> http://oss.sgi.com/archives/xfs/2008-01/msg00097.html

Yep, I think expanding it in, say, 1T increments should be fine, and it
 should all (I think...) end up the same as if you'd done it all at once
(modulo the bug, of course...)

-Eric

Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Mark Goodwin]

Eric Sandeen wrote:
> Lance Reed wrote:
>> Thanks for the quick response.
>>
>> Actually, I was able to run an xfs_repair and all was well.(took 45 minutes...)
>>
>> But I would love to be able to expand the XFS file system out to the max.
>>
>> I guess I could expand it in < 2TB increments then maybe?
>>
>> Thanks for the update and I will look around.
>>
>> I did find this which I think is related.
>>
>> http://oss.sgi.com/archives/xfs/2008-01/msg00097.html
>
> Yep, I think expanding it in, say, 1T increments should be fine, and it
>  should all (I think...) end up the same as if you'd done it all at once
> (modulo the bug, of course...)

I think we verified a while back that growing in 2T increments is an
effective workaround - the bug is due to a signed 32 bit overflow.
The fix has been available for some time now, but apparently hasn't
made it's way into Centos yet.

Cheers

-- 

  Mark Goodwin                                  markgw@sgi.com
  Engineering Manager for XFS and PCP    Phone: +61-3-99631937
  SGI Australian Software Group           Cell: +61-4-18969583
-------------------------------------------------------------

Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Eric Sandeen]

Mark Goodwin wrote:
> 
> Eric Sandeen wrote:
>> Lance Reed wrote:
>>> Thanks for the quick response.
>>>
>>> Actually, I was able to run an xfs_repair and all was well.(took 45 minutes...)
>>>
>>> But I would love to be able to expand the XFS file system out to the max.
>>>
>>> I guess I could expand it in < 2TB increments then maybe?
>>>
>>> Thanks for the update and I will look around.
>>>
>>> I did find this which I think is related.
>>>
>>> http://oss.sgi.com/archives/xfs/2008-01/msg00097.html
>> Yep, I think expanding it in, say, 1T increments should be fine, and it
>>  should all (I think...) end up the same as if you'd done it all at once
>> (modulo the bug, of course...)
> 
> I think we verified a while back that growing in 2T increments is an
> effective workaround - the bug is due to a signed 32 bit overflow.
> The fix has been available for some time now, but apparently hasn't
> made it's way into Centos yet.

Honestly, I think nothing makes its way back to Centos... I really don't
have the time to maintain it.  If anyone on the list uses Centos + xfs
and wants to backport patches that appear to be bugfixes, I'm sure it'd
be welcomed.  I'd be happy to facilitate w/ review or whatnot, but
probably won't have time to actually take on this task myself (despite
being the instigator of the centos module originally, which was, in
retrospect, perhaps a tad irresponsible...)  :)

-Eric

RE: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Lance Reed]

Thanks so much for all the info!

I am currently looking over the
http://oss.sgi.com/archives/xfs/2008-01/msg00097.html
post to figure  out if I need to do antyhing manual to finish expanding the filesystem.

If it is possible to expand in increments less than 2 TB, is there a way to do this via xfs_growfs?  I am a bit confused by the -D and -R options and which one would be relevant here.  Or do I have to shrink the LVM Logical Volume first and then just run "xfs_grow /mntpoint".
Also, can I run with the "-n" option to confirm if the xfs_growfs command "should" complete correctly each time? (looking for docs on this...)

Is there any reason I couldn't grab a newerer release of the XFS tools, etc, and compile for CentOs 5.0 and use them in this case?


Thanks in advance for the help.


Lance

-----Original Message-----
From: Eric Sandeen [mailto:sandeen@sandeen.net]
Sent: Monday, April 28, 2008 8:44 PM
To: markgw@sgi.com
Cc: Lance Reed; xfs@oss.sgi.com
Subject: Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

Mark Goodwin wrote:
>
> Eric Sandeen wrote:
>> Lance Reed wrote:
>>> Thanks for the quick response.
>>>
>>> Actually, I was able to run an xfs_repair and all was well.(took 45 minutes...)
>>>
>>> But I would love to be able to expand the XFS file system out to the max.
>>>
>>> I guess I could expand it in < 2TB increments then maybe?
>>>
>>> Thanks for the update and I will look around.
>>>
>>> I did find this which I think is related.
>>>
>>> http://oss.sgi.com/archives/xfs/2008-01/msg00097.html
>> Yep, I think expanding it in, say, 1T increments should be fine, and it
>>  should all (I think...) end up the same as if you'd done it all at once
>> (modulo the bug, of course...)
>
> I think we verified a while back that growing in 2T increments is an
> effective workaround - the bug is due to a signed 32 bit overflow.
> The fix has been available for some time now, but apparently hasn't
> made it's way into Centos yet.

Honestly, I think nothing makes its way back to Centos... I really don't
have the time to maintain it.  If anyone on the list uses Centos + xfs
and wants to backport patches that appear to be bugfixes, I'm sure it'd
be welcomed.  I'd be happy to facilitate w/ review or whatnot, but
probably won't have time to actually take on this task myself (despite
being the instigator of the centos module originally, which was, in
retrospect, perhaps a tad irresponsible...)  :)

-Eric

Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Eric Sandeen]

Lance Reed wrote:
> Thanks so much for all the info!
> 
> I am currently looking over the
> http://oss.sgi.com/archives/xfs/2008-01/msg00097.html
> post to figure  out if I need to do antyhing manual to finish expanding the filesystem.
> 
> If it is possible to expand in increments less than 2 TB, is there a way to do this via xfs_growfs?  I am a bit confused by the -D and -R options and which one would be relevant here.  Or do I have to shrink the LVM Logical Volume first and then just run "xfs_grow /mntpoint".
> Also, can I run with the "-n" option to confirm if the xfs_growfs command "should" complete correctly each time? (looking for docs on this...)
> 
> Is there any reason I couldn't grab a newerer release of the XFS tools, etc, and compile for CentOs 5.0 and use them in this case?

Well, it's there's a kernel bug when growing by >2T so it's not just a
tools issue...

>From what Mark said it sounds like growing 2T at a time is fine.

you can specify sizes on the xfs_growfs commandline, as specified in the
man page.

-Eric

RE: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Lance Reed]

Thanks,

Sorry, I am a bit confused on the "data section" vs. the "real-time section"?

Is it enough to just run "xfs_growfs -D XXX  /mntpoint" and the rest should fall into place.

Again, sorry for being dense.
I really appraciate the rapid feedback.

Thanks,

Lance

-----Original Message-----
From: Eric Sandeen [mailto:sandeen@sandeen.net]
Sent: Tuesday, April 29, 2008 12:44 PM
To: Lance Reed
Cc: markgw@sgi.com; xfs@oss.sgi.com
Subject: Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

Lance Reed wrote:
> Thanks so much for all the info!
>
> I am currently looking over the
> http://oss.sgi.com/archives/xfs/2008-01/msg00097.html
> post to figure  out if I need to do antyhing manual to finish expanding the filesystem.
>
> If it is possible to expand in increments less than 2 TB, is there a way to do this via xfs_growfs?  I am a bit confused by the -D and -R options and which one would be relevant here.  Or do I have to shrink the LVM Logical Volume first and then just run "xfs_grow /mntpoint".
> Also, can I run with the "-n" option to confirm if the xfs_growfs command "should" complete correctly each time? (looking for docs on this...)
>
> Is there any reason I couldn't grab a newerer release of the XFS tools, etc, and compile for CentOs 5.0 and use them in this case?

Well, it's there's a kernel bug when growing by >2T so it's not just a
tools issue...

>From what Mark said it sounds like growing 2T at a time is fine.

you can specify sizes on the xfs_growfs commandline, as specified in the
man page.

-Eric

Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Eric Sandeen]

Lance Reed wrote:
> Thanks,
> 
> Sorry, I am a bit confused on the "data section" vs. the "real-time section"?
> 
> Is it enough to just run "xfs_growfs -D XXX  /mntpoint" and the rest should fall into place.
> 
> Again, sorry for being dense.
> I really appraciate the rapid feedback.
> 

Unless you specifically made a filesystem with a realtime subvol, just
ignore it, it's not created by default.

So yes, -D <size> (in blocks) is what you want.  A little cumbersome but
not too bad :)

-Eric

RE: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Lance Reed]

Great!
That is exactly what I needed to know.

One follow up question:

Can I assume that the bug:
TAKE 959978 - growing an XFS filesystem by more than 2TB is broken
is a problem only with the the xfs_growfs code?  The reason I asked is that when I first made the original filesystem, I created it using mkfs.xfs and it succeeded fine for 10.5 TB.

# mkfs.xfs /dev/VolGroupNAS200/LogVolNAS200
meta-data=/dev/VolGroupNAS200/LogVolNAS200 isize=256    agcount=32, agsize=83886080 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=2684354560, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=

so I am assuming the rest of the XFS setup can handle large Filesystems fine.  I am just trying to confirm that the probem is TAKE 959978 and that doing it in less than 2 TB increments should be fine.

Thanks again to the List for all the assitance.

Lance

-----Original Message-----
From: Eric Sandeen [mailto:sandeen@sandeen.net]
Sent: Tuesday, April 29, 2008 12:55 PM
To: Lance Reed
Cc: markgw@sgi.com; xfs@oss.sgi.com
Subject: Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

Lance Reed wrote:
> Thanks,
>
> Sorry, I am a bit confused on the "data section" vs. the "real-time section"?
>
> Is it enough to just run "xfs_growfs -D XXX  /mntpoint" and the rest should fall into place.
>
> Again, sorry for being dense.
> I really appraciate the rapid feedback.
>

Unless you specifically made a filesystem with a realtime subvol, just
ignore it, it's not created by default.

So yes, -D <size> (in blocks) is what you want.  A little cumbersome but
not too bad :)

-Eric

Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[Eric Sandeen]

Lance Reed wrote:
> Great!
> That is exactly what I needed to know.
> 
> One follow up question:
> 
> Can I assume that the bug:
> TAKE 959978 - growing an XFS filesystem by more than 2TB is broken
> is a problem only with the the xfs_growfs code?  The reason I asked is that when I first made the original filesystem, I created it using mkfs.xfs and it succeeded fine for 10.5 TB.
> 
> # mkfs.xfs /dev/VolGroupNAS200/LogVolNAS200
> meta-data=/dev/VolGroupNAS200/LogVolNAS200 isize=256    agcount=32, agsize=83886080 blks
>          =                       sectsz=512   attr=0
> data     =                       bsize=4096   blocks=2684354560, imaxpct=25
>          =                       sunit=0      swidth=0 blks, unwritten=1
> naming   =version 2              bsize=4096
> log      =internal log           bsize=4096   blocks=32768, version=1
>          =                       sectsz=512   sunit=0 blks, lazy-count=0
> realtime =none                   extsz=4096   blocks=0, rtextents=
> 
> so I am assuming the rest of the XFS setup can handle large Filesystems fine.  I am just trying to confirm that the probem is TAKE 959978 and that doing it in less than 2 TB increments should be fine.

yes, it was just in the growth path.

-Eric

CentOS project? <- (Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed)

[Tru Huynh]

On Mon, Apr 28, 2008 at 07:43:46PM -0500, Eric Sandeen wrote:
...
> 
> Honestly, I think nothing makes its way back to Centos... I really don't
> have the time to maintain it.  If anyone on the list uses Centos + xfs
> and wants to backport patches that appear to be bugfixes, I'm sure it'd
> be welcomed.  I'd be happy to facilitate w/ review or whatnot, but
> probably won't have time to actually take on this task myself (despite
> being the instigator of the centos module originally, which was, in
> retrospect, perhaps a tad irresponsible...)  :)

Let's start a new CentOS project http://wiki.centos.org/Projects
for kmod-xfs CentOS-4 and 5.

Cheers,

Tru
-- 
Dr Tru Huynh          | http://www.pasteur.fr/recherche/unites/Binfs/
mailto:tru@pasteur.fr | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France  

Re: CentOS project? <- (Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed)

[Eric Sandeen]

Tru Huynh wrote:
> On Mon, Apr 28, 2008 at 07:43:46PM -0500, Eric Sandeen wrote:
> ...
>> Honestly, I think nothing makes its way back to Centos... I really don't
>> have the time to maintain it.  If anyone on the list uses Centos + xfs
>> and wants to backport patches that appear to be bugfixes, I'm sure it'd
>> be welcomed.  I'd be happy to facilitate w/ review or whatnot, but
>> probably won't have time to actually take on this task myself (despite
>> being the instigator of the centos module originally, which was, in
>> retrospect, perhaps a tad irresponsible...)  :)
> 
> Let's start a new CentOS project http://wiki.centos.org/Projects
> for kmod-xfs CentOS-4 and 5.

Sounds good to me.  I think Karanbir is going to set this up.

-Eric

> Cheers,
> 
> Tru

Re: Problems with xfs_grow on large LVM + XFS filesystem 20TB size check 2 failed

[David Chinner]

On Mon, Apr 28, 2008 at 05:34:29PM -0500, Eric Sandeen wrote:
> Lance Reed wrote:
> > I recently experienced a problem trying to expand an existing LVM + XFS installation.
> > The core problem was that xfs_growfs did not correctly resize the XFS filesystem while trying to expand from 11 TB to about 21 TB.
> > 
> > The previous setup had 5 x 2.18 TB LUNs using LVM2 for a total of just under 11 TB.
> > This is a 64bit Linux system.
> > Linux nfs3 2.6.18-8.1.15.el5 #1 SMP Mon Oct 22 08:32:28 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
> > CentOS release 5 (Final)
> > 
> > XFS versions:
> > xfsprogs-2.9.4-1.el5.centos
> > xfsdump-2.2.46-1.el5.centos
> > kmod-xfs-0.4-1.2.6.18_8.1.15.el5
> > 
> ...
> 
> > Thanks in advance for any help....
> 
> You probably hit:
> 
> TAKE 959978 - growing an XFS filesystem by more than 2TB is broken
> http://oss.sgi.com/archives/xfs/2007-01/msg00053.html
> 
> which unfortunately probably was never backported to the centos5 xfs
> module.  Hmm maybe I should do that.  :(
> 
> Dave, do you remember what the fs looked like after hitting that bug?

Not exactly. I'd have to read this thread:
> 
> You may want to read through the "Repairing a possibly incomplete
> xfs_growfs command?" thread from earlier, on this list...

To remind myself. IIRC it shows all the information you need to gather
and all the commands needed to put the pieces back together again....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group