Re: Memory allocation failed, e2fsck: aborted

[Andre Noll <maan@systemlinux.org>]

[-- Attachment #1: Type: text/plain, Size: 4064 bytes --]

On Wed, Aug 18, 14:20, Andreas Dilger wrote:
> > This is an old 32 bit system with only 1G of ram and a 2.6.24 distro
> > kernel. I added _lots_ of swap but this did not help.
> 
> Yeah, it is possible to have filesystems that are too large for the
> node they are running on.  There are low-priority discussions for how
> to reduce memory usage of e2fsck, but they have never been a priority
> to implement.

But e2fsck runs entirely in user space, so all memory should be
swappable, no? I think the system did not swap at all.

> > Since the file system is corrupt anyway, it is maybe easiest
> > to delete inode 245859898 with debugfs, but maybe there is
> > a better option. Moreover, since this might be some kind of
> > e2fsck-trusts-corrupt-data issue, you might be interested in looking
> > at this.
> 
> No, I don't think this will help.  The problem is not with that inode,
> just that it is needing to allocate a structure because of nlinks=2
> (this is normal).
> 
> In theory it might be possible to avoid allocating icount structures
> for every directory inode (which have icount == 2 normally), if we
> used the "fs->inode_dir_map" bit as "+1" for the inode link count.
> 
> In any case, this is a non-trivial fix.

I'm not sure I can follow. Are you saying we currently allocate two
struct ext2_icount for a directory inode even if .  and .. are the
only two references? So we could just omit this allocation in the
common icount == 2 case because we know it is a directory inode
(so we have one additional reference) if fs->inode_dir_map is not NULL.

> > Further info: The ext3 file system lives on a lv within a vg whose
> > single pv is the 12 disk raid6 array. The file system stores hard
> > link based backups, so it contains _lots_ of hard links.
> 
> Ah, that is also a major user of memory, and not necessarily one that
> optimizing the internal bitmap will help significantly.  It may well
> be that your swap cannot be used if a single allocation is in the same
> neighbourhood as the total RAM size.

Is playing with the memory overcommit knobs likely going to help?

> Every file with nlink > 1 will need an additional 8 bytes of data, and
> the insert_icount_el() function reallocates this structure every 100
> elements, so it can use AT MOST 1/2 of all memory before the new copy
> and the old one fill all available memory.
> 
> It would probably make sense to modify the internal icount structure
> to hold a 2-level tree of arrays of e.g. 8kB chunks, or other advanced
> data structure so that it doesn't force reallocation and average .51
> memory copies of the WHOLE LIST on every insert.  This is probably
> doable with some light understanding of e2fsprogs, since the icount
> interface is well encapsulated, but it isn't something I have time for
> now.

I'm interested in having a look at the icount structure and see what
can be done to reduce memory usage. Here's a first question: There is
ext2fs_create_icount() and ext2fs_create_icount_tdb(). Is is correct
that they do the same thing, the only difference being that the tdb
variant uses an on-disk database while ext2fs_create_icount stores
everything in memory?

If so we might want to discuss first whether it is more important
to improve the performance of the on-disk database or to decrease
the memory requirements of the in-memory variant. The answer likely
depends on the amounts of disk space and RAM a typical system will
have in 5 or 10 years from now.

> If you are interested to hack/improve e2fsprogs I'd be willing to
> guide you, but if not I'd just suggest connecting this array to
> another node to run e2fsck, and consider spending the $200 needed to
> get a 64-bit system with a few GB of RAM.

Yeah right. There is already a 64bit system waiting to replace the
old one. Moving the old disks would be a PITA though because of the
lack of hot swap bays...

Thanks for your help
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]