Probems with xfs_repair on large filesystem and 32bit OS.

Probems with xfs_repair on large filesystem and 32bit OS.

[Lance Reed]

So I am having trouble fixing a corrupted <16 TB file system on a 32bit Linux system:


The problem is that xfs_repair dies due to xfs_repair: libxfs_initbuf can't memalign 4096 bytes: Cannot allocate memory:

I found the below post, but the links are dead.

I tried upping the system RAM to 12 GB, but the problem still exists, which I assume is the per process limit of 1-4 GB.


So I am looking for advice on how to fix a large 32bit filesystem.  I see options about running on a 64bit host.
Can I do this with a newer version of xfs and kernel?
We have many hosts that run 64bit Linux (2.6.18-8.1.15.el5  x86_64) and updated xfs progs.
Is it possible to mount the existing LVM volumes on a new 64bit host with a newer OS and xfs progs and not risk losing data.
Os details for old and possible new host below:

Thanks in advance for any possible ideas.
Really, THANKS!
Lance

Basic Os details: (Problem host used to mount XFS filesystem).
CentOS release 4.4 (Final)
2.6.9-42.ELsmp  i686 i686 i386 GNU/Linux

XFS info:

xfsprogs-2.8.11-1.c4
xfsdump-2.2.42-1.c4
kernel-module-xfs-2.6.9-42.ELsmp-0.1-3

Possible new host.:

CentOS release 5 (Final)
2.6.18-8.1.15.el5  x86_64
xfsprogs-2.9.4-1.el5.centos
xfsdump-2.2.46-1.el5.centos
kmod-xfs-0.4-1.2.6.18_8.1.15.el5


Previous posts on list:




#> I had 2GB RAM and a 300GB swap partition and was monitoring

#> memory consumption with "top": it never went over 3BG before

#> failing.

#

#The default 3GiB address space limit per process on a 32 bit

#system is rather well documented, for a summary:

#

#  http://WWW.sabi.co.UK/Notes/#060821c

#

#Let's mention this classic entry again:

#

#  http://OSS.SGI.com/archives/linux-xfs/2005-08/msg00045.html

#

#   <> > Your filesystem (8TiB) may simply bee too large for your

#    > > system to be able to repair. Try mounting it on a 64bit

#    > > system with more RAM in it and repairing it from there.

#

#    Now that linux supports larger than 2TiB filesystems on 32

#    bit systems, this is true for Linux as well.>

#

#A previous entry in the same thread might help too:

#

#  http://OSS.SGI.com/archives/linux-xfs/2005-08/msg00037.html

#

#   <> I try xfs_check and xfs_ncheck (and more progs) with

#    > +200GB swap, but no different! less than 1 second and get

#    > out of memory.

#

#    Swap won't help if you're running an ia32 (32bit) kernel -

#    you have a per-process memory limit of 1-4GiB (depending on

#    kernel and config). The amount of physical memory and swap

#    does not change this limitation.>

#

#Trying hard to avoid looking at what <has been discussed many

#times previously> is a great time saving strategy! :-)
#












[[HTML alternate version deleted]]

RE: Probems with xfs_repair on large filesystem and 32bit OS.

[Lance Reed]

SO, I take it from what you say that that I can running any newer version of xfs progs and a 64bit host (fiber attached, easy to do this), and run xfs_repair on the 32bit XFS volumes without fear of data corruption?
Is this because XFS is not version specific, and xfs_repair will honor the 32bit file data structures?

Again thanks so much for your response!


Lance
-----Original Message-----
From: Tru Huynh [mailto:tru@pasteur.fr]
Sent: Tuesday, June 10, 2008 7:51 AM
To: Lance Reed
Subject: Re: Probems with xfs_repair on large filesystem and 32bit OS.

On Tue, Jun 10, 2008 at 07:28:21AM -0400, Lance Reed wrote:
> So I am having trouble fixing a corrupted <16 TB file system on a 32bit Linux
> system:
>
>
> The problem is that xfs_repair dies due to xfs_repair: libxfs_initbuf can't
> memalign 4096 bytes: Cannot allocate memory:
>
> I found the below post, but the links are dead.
>
> I tried upping the system RAM to 12 GB, but the problem still exists, which I
> assume is the per process limit of 1-4 GB.
you are on a 32 bits OS, so I would say are hitting a hard limit per process.
>
> So I am looking for advice on how to fix a large 32bit filesystem.  I see
> options about running on a 64bit host.  Can I do this with a newer version of
> xfs and kernel?  We have many hosts that run 64bit Linux (2.6.18-8.1.15.el5
> x86_64) and updated xfs progs.  Is it possible to mount the existing LVM
> volumes on a new 64bit host with a newer OS and xfs progs and not risk losing
> data.  Os details for old and possible new host below:
>
My 0.2 cents.
- you can install a x86_64 CentOS-4 machine and fix it from there
or
- you can install a x86_64 CentOS-5 machine and fix it from there
or
- upgrade to the **development** versions for CentOS-4 i386 at
https://projects.centos.org/trac/xfs/

http://people.centos.org/tru/XFS/centos-4/RPMS/i386/xfsprogs-2.9.8-2.c4.i686.rpm
http://people.centos.org/tru/XFS/centos-4/RPMS/i386/xfsdump-2.2.48-1.c4.i686.rpm
and the right kmod-xfs http://people.centos.org/tru/XFS/centos-4/RPMS/i386/

> CentOS release 4.4 (Final)
you should be running 4.6
> 2.6.9-42.ELsmp  i686 i686 i386 GNU/Linux
and 2.6.9-67.0.7.ELsmp...
>
> Possible new host.:
>
> CentOS release 5 (Final)
> 2.6.18-8.1.15.el5  x86_64
2.6.18-53.1.21.el5

> xfsprogs-2.9.4-1.el5.centos
> xfsdump-2.2.46-1.el5.centos
or
http://people.centos.org/tru/XFS/centos-5/RPMS/x86_64/xfsprogs-2.9.8-1.c5.x86_64.rpm
http://people.centos.org/tru/XFS/centos-5/RPMS/x86_64/xfsdump-2.2.48-1.c5.x86_64.rpm

> kmod-xfs-0.4-1.2.6.18_8.1.15.el5
or
http://people.centos.org/tru/XFS/centos-5/RPMS/x86_64/kmod-xfs-0.4-2.2.6.18_53.1.21.el5.x86_64.rpm

Cheers,

Tru
--
Dr Tru Huynh          | http://www.pasteur.fr/recherche/unites/Binfs/
mailto:tru@pasteur.fr | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France

Re: Probems with xfs_repair on large filesystem and 32bit OS.

[Christoph Hellwig]

On Tue, Jun 10, 2008 at 08:07:18AM -0400, Lance Reed wrote:
> SO, I take it from what you say that that I can running any newer version of xfs progs and a 64bit host (fiber attached, easy to do this), and run xfs_repair on the 32bit XFS volumes without fear of data corruption?
> Is this because XFS is not version specific, and xfs_repair will honor the 32bit file data structures?

Yes, this is fine.  All the actual filesystem structures are endian and
32/64bit clean.  The log needs to be in the same endianess and had 32bit
vs 64bit problems on x86 for a while, but it needs to be recovered
before you run xfs_repair anyway.

Re: Probems with xfs_repair on large filesystem and 32bit OS.

[Lance Reed]

Thanks!

Yes the log died and was zeroed already with xfs_repair already.

So I plan to put a 64bit head on the volumes and leave it that way.

I asuume I need to still keep it under 16 tb since the fs datastuctures are 32bit?

Thanks again for the help!!!

----- Original Message -----
From: Christoph Hellwig <hch@infradead.org>
To: Lance Reed
Cc: Tru Huynh <tru@pasteur.fr>; 'xfs@oss.sgi.com' <xfs@oss.sgi.com>
Sent: Tue Jun 10 08:28:11 2008
Subject: Re: Probems with xfs_repair on large filesystem and 32bit OS.

On Tue, Jun 10, 2008 at 08:07:18AM -0400, Lance Reed wrote:
> SO, I take it from what you say that that I can running any newer version of xfs progs and a 64bit host (fiber attached, easy to do this), and run xfs_repair on the 32bit XFS volumes without fear of data corruption?
> Is this because XFS is not version specific, and xfs_repair will honor the 32bit file data structures?

Yes, this is fine.  All the actual filesystem structures are endian and
32/64bit clean.  The log needs to be in the same endianess and had 32bit
vs 64bit problems on x86 for a while, but it needs to be recovered
before you run xfs_repair anyway.

Re: Probems with xfs_repair on large filesystem and 32bit OS.

['hch@infradead.org']

On Tue, Jun 10, 2008 at 08:50:30AM -0400, Lance Reed wrote:
> Thanks!
> 
> Yes the log died and was zeroed already with xfs_repair already.
> 
> So I plan to put a 64bit head on the volumes and leave it that way.
> 
> I asuume I need to still keep it under 16 tb since the fs datastuctures are 32bit?
> 

All XFS data structures (which matter for this) are 64bit.  But the
pagecache size in 32bit x86 systems is indeeed limited, so a larger
filesystem won't work there.

Re: Probems with xfs_repair on large filesystem and 32bit OS.

[Andi Kleen]

Christoph Hellwig <hch@infradead.org> writes:

> On Tue, Jun 10, 2008 at 08:07:18AM -0400, Lance Reed wrote:
>> SO, I take it from what you say that that I can running any newer version of xfs progs and a 64bit host (fiber attached, easy to do this), and run xfs_repair on the 32bit XFS volumes without fear of data corruption?
>> Is this because XFS is not version specific, and xfs_repair will honor the 32bit file data structures?
>
> Yes, this is fine.  All the actual filesystem structures are endian and
> 32/64bit clean.  The log needs to be in the same endianess and had 32bit
> vs 64bit problems on x86 for a while, but it needs to be recovered
> before you run xfs_repair anyway.

Actually there used to be some bugs in old versions where 32bit couldn't 
replay 64bit logs or vice versa.  Probably all fixed in uptodate kernels.

-Andi

Re: Probems with xfs_repair on large filesystem and 32bit OS.

[Barry Naujok]

On Tue, 10 Jun 2008 22:07:18 +1000, Lance Reed <lreed@brightcove.com>  
wrote:

> SO, I take it from what you say that that I can running any newer  
> version of xfs progs and a 64bit host (fiber attached, easy to do this),  
> and run xfs_repair on the 32bit XFS volumes without fear of data  
> corruption?
> Is this because XFS is not version specific, and xfs_repair will honor  
> the 32bit file data structures?
>
> Again thanks so much for your response!

As Christoph said, you can run 64-bit kernel/xfsprogs on a filesystem
used in a 32-bit kernel without problems.

You'll also find that xfsprogs 2.9.4 and later is more memory optimised
than the 2.8.x series. It may even work on your filesystem in 32-bit
mode. I can't guarantee that though as you never actually specified
the size of your filesystem and how many inodes are on it. I can repair a
9TB filesystem on a 2GB machine without using swap.

> Lance
> -----Original Message-----
> From: Tru Huynh [mailto:tru@pasteur.fr]
> Sent: Tuesday, June 10, 2008 7:51 AM
> To: Lance Reed
> Subject: Re: Probems with xfs_repair on large filesystem and 32bit OS.
>
> On Tue, Jun 10, 2008 at 07:28:21AM -0400, Lance Reed wrote:
>> So I am having trouble fixing a corrupted <16 TB file system on a 32bit  
>> Linux
>> system:
>>
>>
>> The problem is that xfs_repair dies due to xfs_repair: libxfs_initbuf  
>> can't
>> memalign 4096 bytes: Cannot allocate memory:
>>
>> I found the below post, but the links are dead.
>>
>> I tried upping the system RAM to 12 GB, but the problem still exists,  
>> which I
>> assume is the per process limit of 1-4 GB.
> you are on a 32 bits OS, so I would say are hitting a hard limit per  
> process.
>>
>> So I am looking for advice on how to fix a large 32bit filesystem.  I  
>> see
>> options about running on a 64bit host.  Can I do this with a newer  
>> version of
>> xfs and kernel?  We have many hosts that run 64bit Linux  
>> (2.6.18-8.1.15.el5
>> x86_64) and updated xfs progs.  Is it possible to mount the existing LVM
>> volumes on a new 64bit host with a newer OS and xfs progs and not risk  
>> losing
>> data.  Os details for old and possible new host below:
>>
> My 0.2 cents.
> - you can install a x86_64 CentOS-4 machine and fix it from there
> or
> - you can install a x86_64 CentOS-5 machine and fix it from there
> or
> - upgrade to the **development** versions for CentOS-4 i386 at
> https://projects.centos.org/trac/xfs/
>
> http://people.centos.org/tru/XFS/centos-4/RPMS/i386/xfsprogs-2.9.8-2.c4.i686.rpm
> http://people.centos.org/tru/XFS/centos-4/RPMS/i386/xfsdump-2.2.48-1.c4.i686.rpm
> and the right kmod-xfs  
> http://people.centos.org/tru/XFS/centos-4/RPMS/i386/
>
>> CentOS release 4.4 (Final)
> you should be running 4.6
>> 2.6.9-42.ELsmp  i686 i686 i386 GNU/Linux
> and 2.6.9-67.0.7.ELsmp...
>>
>> Possible new host.:
>>
>> CentOS release 5 (Final)
>> 2.6.18-8.1.15.el5  x86_64
> 2.6.18-53.1.21.el5
>
>> xfsprogs-2.9.4-1.el5.centos
>> xfsdump-2.2.46-1.el5.centos
> or
> http://people.centos.org/tru/XFS/centos-5/RPMS/x86_64/xfsprogs-2.9.8-1.c5.x86_64.rpm
> http://people.centos.org/tru/XFS/centos-5/RPMS/x86_64/xfsdump-2.2.48-1.c5.x86_64.rpm
>
>> kmod-xfs-0.4-1.2.6.18_8.1.15.el5
> or
> http://people.centos.org/tru/XFS/centos-5/RPMS/x86_64/kmod-xfs-0.4-2.2.6.18_53.1.21.el5.x86_64.rpm
>
> Cheers,
>
> Tru
> --
> Dr Tru Huynh          | http://www.pasteur.fr/recherche/unites/Binfs/
> mailto:tru@pasteur.fr | tel/fax +33 1 45 68 87 37/19
> Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France
>
>