resize2fs: Memory allocation failed while trying to resize

resize2fs: Memory allocation failed while trying to resize

[Tim]

Hi,

I used resize2fs to do an online grow of from 8TB to 12TB and that worked fine.
now I'm trying to offline shrink back down to 8TB and resize2fs is
unable to do so.

Output:
resize2fs -M /dev/mapper/
crypt_left
resize2fs 1.42.5 (29-Jul-2012)
Resizing the filesystem on /dev/mapper/crypt_left to 1885633106 (4k) blocks.
resize2fs: Memory allocation failed while trying to resize
/dev/mapper/crypt_left
Please run 'e2fsck -fy /dev/mapper/crypt_left' to fix the filesystem
after the aborted resize operation.

I am using a raspberrypi so it's 32-bit and limited to 256MB and I
have 2x that in swap.

uname -a
Linux test 3.6.11+ #474 PREEMPT Thu Jun 13 17:14:42 BST 2013 armv6l GNU/Linux

I was watching top and never saw it use more that 53% of the RAM but I
may have missed the crucial moment.
(taken during a "e2fsck -fy ")
free
             total used free shared buffers cached
Mem: 237648 225440 12208 0 63240 11796
-/+ buffers/cache: 150404 87244
Swap: 475132 13444 461688

Please let me know if there is anything that would help solve this issue.

Re: resize2fs: Memory allocation failed while trying to resize

[Eric Sandeen]

On 8/28/13 10:29 AM, Tim wrote:
> Hi,
> 
> I used resize2fs to do an online grow of from 8TB to 12TB and that worked fine.
> now I'm trying to offline shrink back down to 8TB and resize2fs is
> unable to do so.
> 
> Output:
> resize2fs -M /dev/mapper/
> crypt_left
> resize2fs 1.42.5 (29-Jul-2012)
> Resizing the filesystem on /dev/mapper/crypt_left to 1885633106 (4k) blocks.
> resize2fs: Memory allocation failed while trying to resize
> /dev/mapper/crypt_left
> Please run 'e2fsck -fy /dev/mapper/crypt_left' to fix the filesystem
> after the aborted resize operation.
> 
> I am using a raspberrypi so it's 32-bit and limited to 256MB and I
> have 2x that in swap.

Ok, now, hold on right there ;)

On the one hand it's worth understanding which memory allocation failed
and whether there's anything to be done about it, but:

Hoping to administer a 12T filesystem on a box with 256M of memory is
almost certainly wishful thinking.  (ok, 768M with swap)


A crucial part of deploying systems with very large filesystems is ensuring
that there is enough system memory to go along with it; in this case,
there almost certainly is not.

> uname -a
> Linux test 3.6.11+ #474 PREEMPT Thu Jun 13 17:14:42 BST 2013 armv6l GNU/Linux
> 
> I was watching top and never saw it use more that 53% of the RAM but I
> may have missed the crucial moment.
> (taken during a "e2fsck -fy ")
> free
>              total used free shared buffers cached
> Mem: 237648 225440 12208 0 63240 11796
> -/+ buffers/cache: 150404 87244
> Swap: 475132 13444 461688
> 
> Please let me know if there is anything that would help solve this issue.

You might use gdb, or an instrumented resize2fs, to find out which memory
allocation is failing and/or where the memory allocations are going.
The sad truth may just be that 256M + 512M swap isn't going to do the
trick.

-Eric

Re: resize2fs: Memory allocation failed while trying to resize

[Theodore Ts'o]

On Thu, Aug 29, 2013 at 11:04:30AM -0500, Eric Sandeen wrote:
> You might use gdb, or an instrumented resize2fs, to find out which memory
> allocation is failing and/or where the memory allocations are going.
> The sad truth may just be that 256M + 512M swap isn't going to do the
> trick.

You might try applying the following patch to the e2fsprogs sources,
compiling it with "make CFLAGS=-g" so debuging information is enabled,
and optimizations are disabled, and then running resize2fs under gdb,
after setting a breakpoint at the function, ext2fs_breakpoint().  That
will tell us where it's crashing, which might be of academic interest,
but as Eric has said (and as I have commented on the Ubuntu launchpad
website), it may be extremely unrealistic to expect this to work.

Oh, also try running resize2fs with the options "-p -d 31" and
redirect the output to a file.  This will give us more information
about how far it got before it ran out of memory.

Cheers,

					- Ted

diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index ff088bb..58074ea 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -1462,6 +1462,8 @@ extern int ext2fs_open_file(const char *pathname, int flags, mode_t mode);
 extern int ext2fs_stat(const char *path, ext2fs_struct_stat *buf);
 extern int ext2fs_fstat(int fd, ext2fs_struct_stat *buf);
 
+extern void ext2fs_breakpoint(void);
+
 /*
  * The actual inlined functions definitions themselves...
  *
@@ -1489,8 +1491,10 @@ _INLINE_ errcode_t ext2fs_get_mem(unsigned long size, void *ptr)
 	void *pp;
 
 	pp = malloc(size);
-	if (!pp)
+	if (!pp) {
+		ext2fs_breakpoint();
 		return EXT2_ET_NO_MEMORY;
+	}
 	memcpy(ptr, &pp, sizeof (pp));
 	return 0;
 }
@@ -1500,8 +1504,10 @@ _INLINE_ errcode_t ext2fs_get_memzero(unsigned long size, void *ptr)
 	void *pp;
 
 	pp = malloc(size);
-	if (!pp)
+	if (!pp) {
+		ext2fs_breakpoint();
 		return EXT2_ET_NO_MEMORY;
+	}
 	memset(pp, 0, size);
 	memcpy(ptr, &pp, sizeof(pp));
 	return 0;
@@ -1554,8 +1560,10 @@ _INLINE_ errcode_t ext2fs_resize_mem(unsigned long EXT2FS_ATTR((unused)) old_siz
 	 * with C99 strict type aliasing rules. */
 	memcpy(&p, ptr, sizeof(p));
 	p = realloc(p, size);
-	if (!p)
+	if (!p) {
+		ext2fs_breakpoint();
 		return EXT2_ET_NO_MEMORY;
+	}
 	memcpy(ptr, &p, sizeof(p));
 	return 0;
 }
diff --git a/lib/ext2fs/inline.c b/lib/ext2fs/inline.c
index eef3dda..9ab1c67 100644
--- a/lib/ext2fs/inline.c
+++ b/lib/ext2fs/inline.c
@@ -37,6 +37,10 @@
 #define INCLUDE_INLINE_FUNCS
 #include "ext2fs.h"
 
+void ext2fs_breakpoint(void)
+{
+}
+
 /*
  * We used to define this as an inline, but since we are now using
  * autoconf-defined #ifdef's, we need to export this as a