RFC - patch to reduce peak memory usage by disabling compression

RFC - patch to reduce peak memory usage by disabling compression

[Øyvind Harboe]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

Rationale:

Reduce RAM usage by removing compression/decompression altogether.

Even trivial compression schemes require a working buffer,
hence all compression schemes and not only the most memory
hungry are removed.

I've introduced a define JFFS2_COMPRESS which is 1 in os-linux.h,
and which eCos can make configurable via its CDL files.



Øyvind

[-- Attachment #2: lessmem.txt --]
[-- Type: text/plain, Size: 4146 bytes --]

? lessmem.txt
Index: compr.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/compr.c,v
retrieving revision 1.26
diff -a -w -u -r1.26 compr.c
--- compr.c	12 Jan 2003 13:21:28 -0000	1.26
+++ compr.c	21 Aug 2003 12:26:50 -0000
@@ -28,6 +28,8 @@
 
 #include <linux/jffs2.h>
 
+#if defined(JFFS2_COMPRESS)
+
 int jffs2_zlib_compress(unsigned char *data_in, unsigned char *cpage_out, uint32_t *sourcelen, uint32_t *dstlen);
 void jffs2_zlib_decompress(unsigned char *data_in, unsigned char *cpage_out, uint32_t srclen, uint32_t destlen);
 int jffs2_rtime_compress(unsigned char *data_in, unsigned char *cpage_out, uint32_t *sourcelen, uint32_t *dstlen);
@@ -136,3 +138,5 @@
 	}
 	return 0;
 }
+
+#endif
Index: gc.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/gc.c,v
retrieving revision 1.103
diff -a -w -u -r1.103 gc.c
--- gc.c	22 May 2003 18:01:02 -0000	1.103
+++ gc.c	21 Aug 2003 12:26:51 -0000
@@ -1085,9 +1085,11 @@
 
 		writebuf = pg_ptr + (offset & (PAGE_CACHE_SIZE -1));
 
+#if defined(JFFS2_COMPRESS)
 		if (comprbuf) {
 			comprtype = jffs2_compress(writebuf, comprbuf, &datalen, &cdatalen);
 		}
+#endif
 		if (comprtype) {
 			writebuf = comprbuf;
 		} else {
Index: os-linux.h
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/os-linux.h,v
retrieving revision 1.28
diff -a -w -u -r1.28 os-linux.h
--- os-linux.h	2 Jul 2003 10:46:00 -0000	1.28
+++ os-linux.h	21 Aug 2003 12:26:51 -0000
@@ -189,6 +189,12 @@
 /* super.c */
 
 
+// by disabling compression, some working memory can be saved.
+// for embedded systems(e.g. eCos) even 50k ram is substantial
+// note that all forms of compression is disabled, since even
+// simple run-length encoding allocates working space.
+#define JFFS2_COMPRESS 1
+
 #endif /* __JFFS2_OS_LINUX_H__ */
 
 
Index: read.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/read.c,v
retrieving revision 1.33
diff -a -w -u -r1.33 read.c
--- read.c	17 Jul 2003 08:44:12 -0000	1.33
+++ read.c	21 Aug 2003 12:26:51 -0000
@@ -93,6 +93,7 @@
 		}
 	}
 	if (ri->compr != JFFS2_COMPR_NONE) {
+#if defined(JFFS2_COMPRESS)
 		if (len < je32_to_cpu(ri->dsize)) {
 			decomprbuf = kmalloc(je32_to_cpu(ri->dsize), GFP_KERNEL);
 			if (!decomprbuf) {
@@ -102,6 +103,10 @@
 		} else {
 			decomprbuf = buf;
 		}
+#else
+		ret = -EINVAL;
+		goto out_readbuf;
+#endif
 	} else {
 		decomprbuf = readbuf;
 	}
@@ -124,6 +129,8 @@
 		goto out_decomprbuf;
 	}
 	D2(printk(KERN_DEBUG "Data CRC matches calculated CRC %08x\n", crc));
+
+#if defined(JFFS2_COMPRESS)
 	if (ri->compr != JFFS2_COMPR_NONE) {
 		D2(printk(KERN_DEBUG "Decompress %d bytes from %p to %d bytes at %p\n",
 			  je32_to_cpu(ri->csize), readbuf, je32_to_cpu(ri->dsize), decomprbuf)); 
@@ -133,7 +140,7 @@
 			goto out_decomprbuf;
 		}
 	}
-
+#endif
 	if (len < je32_to_cpu(ri->dsize)) {
 		memcpy(buf, decomprbuf+ofs, len);
 	}
Index: write.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/write.c,v
retrieving revision 1.65
diff -a -w -u -r1.65 write.c
--- write.c	21 Jan 2003 18:11:29 -0000	1.65
+++ write.c	21 Aug 2003 12:26:51 -0000
@@ -300,6 +300,7 @@
 		datalen = writelen;
 		cdatalen = min_t(uint32_t, alloclen - sizeof(*ri), writelen);
 
+#if defined(JFFS2_COMPRESS)
 		comprbuf = kmalloc(cdatalen, GFP_KERNEL);
 		if (comprbuf) {
 			comprtype = jffs2_compress(buf, comprbuf, &datalen, &cdatalen);
@@ -308,9 +309,12 @@
 			/* Either compression failed, or the allocation of comprbuf failed */
 			if (comprbuf)
 				kfree(comprbuf);
+#endif
 			comprbuf = buf;
 			datalen = cdatalen;
+#if defined(CYGPKG_FS_JFFS2_COMPRESS)
 		}
+#endif
 		/* Now comprbuf points to the data to be written, be it compressed or not.
 		   comprtype holds the compression type, and comprtype == JFFS2_COMPR_NONE means
 		   that the comprbuf doesn't need to be kfree()d. 

Re: RFC - patch to reduce peak memory usage by disabling compression

[David Woodhouse]

On Thu, 2003-08-21 at 14:31 +0200, Øyvind Harboe wrote:
> Reduce RAM usage by removing compression/decompression altogether.

You could make it use a static buffer and serialise compression, and it
really would be only 4KiB you're saving.

I sincerely hope this is done because you're using uncompressible data,
rather than trading about 4KiB of cheap RAM for a doubling in expensive
flash space :)

If you're doing this, I'd prefer to change jffs2_compress() to allocate
its _own_ buffer (or optionally use a static one), then have a simple
definition of jffs2_compress which does nothing in the
!CONFIG_JFFS2_FS_COMPRESSION case. 

You need to handle the case where compression is configured out and a
filesystem created with compressed nodes is encountered -- it looks like
it'll die horribly at the moment.

I'd also like to see individual compression routines selectable, and
also at runtime with something akin to chattr.

-- 
dwmw2

RE: RFC - patch to reduce peak memory usage by disablingcompression

[Øyvind Harboe]

>You could make it use a static buffer and serialise 
>compression, and it really would be only 4KiB you're saving.
>
>I sincerely hope this is done because you're using uncompressible 
>data, rather than trading about 4KiB of cheap RAM for a doubling 
>in expensive flash space :)
>
>If you're doing this, I'd prefer to change jffs2_compress() to 
>allocate its _own_ buffer (or optionally use a static one), 
>then have a simple definition of jffs2_compress which does 
>nothing in the !CONFIG_JFFS2_FS_COMPRESSION case. 

These are bigger changes than I want to attempt to JFFS2 at the
moment. 

>You need to handle the case where compression is configured out 
>and a filesystem created with compressed nodes is encountered -- 
>it looks like it'll die horribly at the moment.

I don't know my way around JFFS2 too well, but I thought I reported
an error in that case? 

What else can be done in this case, except to fail to read the file?

In my case, it is my own firmware that writes the JFFS2, so the
situation
will never arise. 

I can't say that I think it makes much sense to support the
configuration
of JFFS2 where it does not compress, but decompress. 

>I'd also like to see individual compression routines selectable, 
>and also at runtime with something akin to chattr.

What you describe is higher resolution of configurability, but for
my purposes the resolution is good enough.

Too many options isn't a good thing either. There will be a lot of
implied
orthogonality that isn't there(since all configurations will never be
tested).

Øyvind