xfsrepair memory consumption

xfsrepair memory consumption

[Daniele P.]

Hi all,
I'm just asking if any work has been done/is in progress to improve
xfs_repair memory consumption.
I know that xfs tools use a lot of memory but IIRC someone wrote on this
mailing list that s/he'll work on this issue.
But now I see that the memory requirements are increasing instead of
lowering.

I discovered this because running xfs_repair 2.6.20-1 on a 300GB file
system fills up the entire memory on a debian sarge (256MB Mem/256MB
swap) and eventually the process is killed

....
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- ensuring existence of lost+found directory
- traversing filesystem starting at / ...
Killed

Next I tried a self compiled xfsprogs 2.8.18-1 from cvs but things get
worse.
So I increased the memory to 512MB, and made sure not to use the multi
thread but still no luck:

....
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
Killed

Finally using the *old* version 2.6.20-1 with 512MB of memory xfs_repair
finished with success, but using near all the available memory.

More info:

enceladus:~# xfs_info /dev/sdb1
meta-data=/media/iomega300       isize=256    agcount=16, agsize=4578901 blks
         =                       sectsz=512
data     =                       bsize=4096   blocks=73262416, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0
enceladus:~# df /dev/sdb1
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdb1            292918592 164698268 128220324  57% /media/300
enceladus:~# df -i /dev/sdb1
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1            293049664 6511481 286538183    3% /media/300

Thanks in Advance,
Daniele P.

Re: xfsrepair memory consumption

[David Chinner]

On Tue, Mar 20, 2007 at 03:32:05PM +0100, Daniele P. wrote:
> Hi all,
> I'm just asking if any work has been done/is in progress to improve
> xfs_repair memory consumption.

Work is in progress, but it won't really solve your problem.
See, you've got:

> enceladus:~# df -i /dev/sdb1
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/sdb1            293049664 6511481 286538183    3% /media/300

6 million inodes in your filesystem, and a certain points in
repair we have to hold indexes of them all (plus some state) in
memory. Phase 6 is one of these points.

In terms of inode count, I generally use the rule that for every
10million inodes you need a gigabyte of RAM for repair - you needed
about 500MB for 6million inodes.

We have been trimming bits and pieces off this per-inode usage
but there comes a point where you just need more memory. That
is, as filesystem size grows, so does the amount of memory
needed to repair it in a finite time....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

RE: xfsrepair memory consumption

[Barry Naujok]

Hi Daniele,

The nlink/phase 7 patch I recently sent out does reduce this 
inode memory requirements which should address the issue you 
see.

I'm going to make one further optimisation in this patch 
which I will repost most likely next week.

Regards,
Barry.

> -----Original Message-----
> From: xfs-bounce@oss.sgi.com [mailto:xfs-bounce@oss.sgi.com] 
> On Behalf Of David Chinner
> Sent: Wednesday, 21 March 2007 9:13 AM
> To: Daniele P.
> Cc: xfs@oss.sgi.com
> Subject: Re: xfsrepair memory consumption
> 
> On Tue, Mar 20, 2007 at 03:32:05PM +0100, Daniele P. wrote:
> > Hi all,
> > I'm just asking if any work has been done/is in progress to improve
> > xfs_repair memory consumption.
> 
> Work is in progress, but it won't really solve your problem.
> See, you've got:
> 
> > enceladus:~# df -i /dev/sdb1
> > Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> > /dev/sdb1            293049664 6511481 286538183    3% /media/300
> 
> 6 million inodes in your filesystem, and a certain points in
> repair we have to hold indexes of them all (plus some state) in
> memory. Phase 6 is one of these points.
> 
> In terms of inode count, I generally use the rule that for every
> 10million inodes you need a gigabyte of RAM for repair - you needed
> about 500MB for 6million inodes.
> 
> We have been trimming bits and pieces off this per-inode usage
> but there comes a point where you just need more memory. That
> is, as filesystem size grows, so does the amount of memory
> needed to repair it in a finite time....
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> Principal Engineer
> SGI Australian Software Group
> 
> 

Re: xfsrepair memory consumption

[Daniele P.]

On Wednesday 21 March 2007 00:48, you wrote:
> Hi Daniele,
>
> The nlink/phase 7 patch I recently sent out does reduce this
> inode memory requirements which should address the issue you
> see.

Hi Barry, 
maybe this could help. A simple ps -u during xfs_repair show
an increasing memory usage in phase 6 and 7 for 2.8.18-1.
Unfortunately with 2.6.20-1 the memory usage is exploding in
phase 2.

Regards,
Daniele P.

Re: xfsrepair memory consumption

[Daniele P.]

On Tuesday 20 March 2007 23:13, you wrote:
> On Tue, Mar 20, 2007 at 03:32:05PM +0100, Daniele P. wrote:
> 6 million inodes in your filesystem, and a certain points in
> repair we have to hold indexes of them all (plus some state) in
> memory. Phase 6 is one of these points.

Hi David,
thanks for the explanation. 

> In terms of inode count, I generally use the rule that for every
> 10million inodes you need a gigabyte of RAM for repair - you needed
> about 500MB for 6million inodes.

This is true for 2.6.20-1, but 2.8.18-1 use a lot of memory in
phase 2.
I just want to point out the *increasing* memory requirements rather
than the total amount of memory needed to repair an xfs file system.

Regards,
Daniele P.

Re: xfsrepair memory consumption

[Daniele P.]

On Wednesday 21 March 2007 09:48, Barry Naujok wrote:
> This could be libxfs caching in action.
>
> Another option you can try is:
>
> -o bhash=256
>
> Also, -M and -P options may reduce memory consumption.
> I recommend -M (I'm in the middle of making that default).

Hi Barry,
In the previous test I already used the -M option (no thread).
Using also -P and -o bhash=256 make the memory usage of the new version
of xfs_repair close to the old version with no option for phases
1-5.
This is really useful.
Phase 6 still uses little more memory (killed!).
I will add more memory and will test later.

Thanks,
Daniele P.

Re: xfsrepair memory consumption

[Chris Wedgwood]

On Wed, Mar 21, 2007 at 12:08:27PM +0100, Daniele P. wrote:

> Phase 6 still uses little more memory (killed!).
> I will add more memory and will test later.

Stupid question (Sorry, I didn't read the thread that carefully), is
there any reason you can't just add swap-space and let it thrash a
little?

Re: xfsrepair memory consumption

[Daniele P.]

On Wednesday 21 March 2007 22:36, Chris Wedgwood wrote:
> On Wed, Mar 21, 2007 at 12:08:27PM +0100, Daniele P. wrote:
> > Phase 6 still uses little more memory (killed!).
> > I will add more memory and will test later.
>
> Stupid question (Sorry, I didn't read the thread that carefully), is
> there any reason you can't just add swap-space and let it thrash a
> little?

Oh, it's less expensive (less typing) to add ram because it's a
virtual machine.
Finally using the 2.8.20-1 version requires 1 GB of memory vs 750 MB for
2.6.20-1.
Next time I will add more swap by default.

Regards,
Daniele P.