[PATCH 0/4] mtd-utils: adding support for lzma in jffs2reader via liblzma

[PATCH 0/4] mtd-utils: adding support for lzma in jffs2reader via liblzma

[Frederic Germain]

Hi,

This is my first time sending patch, I probably do some things wrong somewhere.

A while ago I, unfortunately, removed a file from my gl-inet 4g openwrt router.
That file is on a jffs2 partition, with LZMA compression.

LZMA support on OpenWRT is activated via a patch which seems never made it to the mainline : jffs2_make_lzma_available.patch.  There might be a problem with the jffs2 integration of LZMA, if it was not integrated into the kernel? The filesystem support should probably be first integrated in the main kernel before these patch in the user tools...

I think openwrt/LEDE is a big use case for jffs2, with maybe a lot of them with LZMA supports (I'm cannot say how much though...)
So it would be cool to have this integrated so people can recover the files they lost.

There are two opensource LZMA libraries that I found on Ubuntu 20.04...
- The source SDK from https://www.7-zip.org/sdk.html . On ubuntu, it's in the p7zip source package, which provides p7zip & p7zip-full package. p7zip-full provides a library, /usr/lib/p7zip/7z.so, but it's not integrated the Linux/GNU way...
- liblzma: from https://tukaani.org/xz/, which integrates an older version of the LZMA SDK provided by www.7-zip.org, but on a standard Linux/GNU library.

It seems LEDE already has patches for mtd-utils, but it seems they integrate source files from www.7-zip.org in a static way.
https://patchwork.ozlabs.org/project/lede/patch/20180330145639.15824-1-hauke@hauke-m.de/

I have to fiddle around to integrate with liblzma, as the low-level decoder is not exposed, but I got help from the XZ project author  https://sourceforge.net/p/lzmautils/discussion/708858/thread/d02ebb9386/

I pushed the patches on https://github.com/fredericgermain/mtd-utils too, main branch. It might just help some people to recover their file anyway.

---
There is also a problem that jffs2reader selects the earliest version of an inode instead of the latest? I've made a patch to take the latest one with data. But I still needed to patch the mtd dump to remove the DIRENT entry that remove the file from the FS. 

Maybe it would be nice that jffs2reader takes inode/version of the file to dump instead of a path? It's easy to get these parameters with jffs2dump.

Best regards,

Frederic Germain (4):
  .gitignore: add new ubiscan utility
  Fix warning about unaligned pointer in jffs2reader
  jffsX-utils: add lzma support on jffs2reader
  find_raw_inode returns latest version with data of inode

 .gitignore                |  1 +
 Makefile.am               |  4 +++
 configure.ac              | 24 +++++++++++++++
 include/linux/jffs2.h     |  2 ++
 include/mtd/jffs2-user.h  |  1 +
 jffsX-utils/Makemodule.am |  2 +-
 jffsX-utils/jffs2reader.c | 63 ++++++++++++++++++++++++++++++++++-----
 7 files changed, 88 insertions(+), 9 deletions(-)

-- 
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

[PATCH 1/4] .gitignore: add new ubiscan utility

[Frederic Germain]

Signed-off-by: Frederic Germain <frederic.germain@gmail.com>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 1644ef0..8c51ee5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -100,6 +100,7 @@ ubinize
 ubirename
 ubirmvol
 ubirsvol
+ubiscan
 ubiupdatevol
 volrefcnt
 mtdlib_test
-- 
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

[PATCH 2/4] Fix warning about unaligned pointer in jffs2reader

[Frederic Germain]

gcc 9.3.0-17ubuntu1~20.04 warning on time_t ctime value is :

Fix taking address of packed member of ‘struct jffs2_raw_inode’ may
result in an unaligned pointer value [-Waddress-of-packed-member].

Signed-off-by: Frederic Germain <frederic.germain@gmail.com>
---
 jffsX-utils/jffs2reader.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/jffsX-utils/jffs2reader.c b/jffsX-utils/jffs2reader.c
index 083500e..33c5577 100644
--- a/jffsX-utils/jffs2reader.c
+++ b/jffsX-utils/jffs2reader.c
@@ -336,8 +336,9 @@ static void printdir(char *o, size_t size, struct dir *d, const char *path,
 			d = d->next;
 			continue;
 		}
-
-		filetime = ctime((const time_t *) &(ri->ctime));
+		time_t _ctime;
+		memcpy(&_ctime, &(ri->ctime), sizeof(time_t));
+		filetime = ctime(&_ctime);
 		age = time(NULL) - je32_to_cpu(ri->ctime);
 		mode.v32 = ri->mode.m;
 		printf("%s %-4d %-8d %-8d ", mode_string(je32_to_cpu(mode)),
-- 
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

[PATCH 3/4] jffsX-utils: add lzma support on jffs2reader

[Frederic Germain]

Signed-off-by: Frederic Germain <frederic.germain@gmail.com>
---
 Makefile.am               |  4 ++++
 configure.ac              | 24 +++++++++++++++++++++
 include/linux/jffs2.h     |  2 ++
 include/mtd/jffs2-user.h  |  1 +
 jffsX-utils/Makemodule.am |  2 +-
 jffsX-utils/jffs2reader.c | 45 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 77 insertions(+), 1 deletion(-)

diff --git a/Makefile.am b/Makefile.am
index 9da38af..02a3073 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -11,6 +11,10 @@ if WITHOUT_LZO
 AM_CPPFLAGS += -DWITHOUT_LZO
 endif
 
+if WITHOUT_LZMA
+AM_CPPFLAGS += -DWITHOUT_LZMA
+endif
+
 if WITHOUT_ZSTD
 AM_CPPFLAGS += -DWITHOUT_ZSTD
 endif
diff --git a/configure.ac b/configure.ac
index f47100f..e5035c0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -65,6 +65,7 @@ need_clock_gettime="no"
 need_pthread="no"
 need_uuid="no"
 need_zlib="no"
+need_lzma=""
 need_lzo="no"
 need_zstd="no"
 need_xattr="no"
@@ -158,6 +159,14 @@ AC_ARG_WITH([lzo],
 	*) AC_MSG_ERROR([bad value ${withval} for --without-lzo]) ;;
 	esac])
 
+AC_ARG_WITH([lzma],
+	[AS_HELP_STRING([--without-lzma], [Disable support for LZMA compression])],
+	[case "${withval}" in
+	yes) need_lzma="yes";;
+	no) need_lzma="no" ;;
+	*) AC_MSG_ERROR([bad value ${withval} for --without-lzma]) ;;
+	esac])
+
 AC_ARG_WITH([zstd],
 	[AS_HELP_STRING([--without-zstd], [Disable support for ZSTD compression])],
 	[case "${withval}" in
@@ -190,6 +199,7 @@ clock_gettime_missing="no"
 pthread_missing="no"
 uuid_missing="no"
 zlib_missing="no"
+lzma_missing="no"
 lzo_missing="no"
 zstd_missing="no"
 xattr_missing="no"
@@ -219,6 +229,14 @@ if test "x$need_pthread" = "xyes"; then
 	AX_PTHREAD([], [pthread_missing="yes"])
 fi
 
+if test "x$need_lzma" != "xno"; then
+	AC_ARG_VAR([LZMA_CFLAGS], [C compiler flags for lzma])
+	AC_ARG_VAR([LZMA_LIBS], [linker flags for lzma])
+	AC_CHECK_LIB([lzma], [lzma_alone_decoder], [LZMA_LIBS="-llzma"],
+		[lzma_missing="yes"]
+	)
+fi
+
 if test "x$need_lzo" = "xyes"; then
 	AC_ARG_VAR([LZO_CFLAGS], [C compiler flags for lzo])
 	AC_ARG_VAR([LZO_LIBS], [linker flags for lzo])
@@ -287,6 +305,11 @@ if test "x$zlib_missing" = "xyes"; then
 	dep_missing="yes"
 fi
 
+if test "x$need_lzma" = "xyes" -a "x$lzma_missing" = "xyes"; then
+	AC_MSG_WARN([cannot find LZMA library required by options])
+	dep_missing="yes"
+fi
+
 if test "x$lzo_missing" = "xyes"; then
 	AC_MSG_WARN([cannot find LZO library required for mkfs programs])
 	AC_MSG_NOTICE([mtd-utils can optionally be built without mkfs.ubifs])
@@ -339,6 +362,7 @@ fi
 ##### generate output #####
 
 AM_CONDITIONAL([WITHOUT_LZO], [test "x$need_lzo" != "xyes"])
+AM_CONDITIONAL([WITHOUT_LZMA], [test "x$need_lzma" == "xno" -o "x$missing_lzma" == "xyes" ])
 AM_CONDITIONAL([WITHOUT_ZSTD], [test "x$need_zstd" != "xyes"])
 AM_CONDITIONAL([WITHOUT_XATTR], [test "x$need_xattr" != "xyes"])
 AM_CONDITIONAL([WITH_SELINUX], [test "x$need_selinux" == "xyes"])
diff --git a/include/linux/jffs2.h b/include/linux/jffs2.h
index 7306f86..47abf62 100644
--- a/include/linux/jffs2.h
+++ b/include/linux/jffs2.h
@@ -47,6 +47,8 @@
 #define JFFS2_COMPR_DYNRUBIN	0x05
 #define JFFS2_COMPR_ZLIB	0x06
 #define JFFS2_COMPR_LZO		0x07
+// diff --git a/target/linux/generic/patches-4.1/530-jffs2_make_lzma_available.patch b/target/linux/generic/patches-4.1/530-jffs2_make_lzma_available.patch
+#define JFFS2_COMPR_LZMA	0x08 // https://git.openwrt.org/project/ubus.git;git://git.openwrt.org/project/ubox.git?p=openwrt/openwrt.git;a=commitdiff;h=a0c49ef46f7caf5eb02c635d446218201008ecff;hp=518ab154e059c9f39684f8c78159c9caf333716b
 /* Compatibility flags. */
 #define JFFS2_COMPAT_MASK 0xc000      /* What do to if an unknown nodetype is found */
 #define JFFS2_NODE_ACCURATE 0x2000
diff --git a/include/mtd/jffs2-user.h b/include/mtd/jffs2-user.h
index bc5d99a..7228b5f 100644
--- a/include/mtd/jffs2-user.h
+++ b/include/mtd/jffs2-user.h
@@ -36,6 +36,7 @@ extern int target_endian;
 #define le32_to_cpu(x)	(__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_32(x))
 #define cpu_to_le16(x)	(__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_16(x))
 #define cpu_to_le32(x)	(__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_32(x))
+#define cpu_to_le64(x)	(__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_64(x))
 
 /* XATTR/POSIX-ACL related definition */
 /* Namespaces copied from xattr.h and posix_acl_xattr.h */
diff --git a/jffsX-utils/Makemodule.am b/jffsX-utils/Makemodule.am
index 96389f6..0c93c18 100644
--- a/jffsX-utils/Makemodule.am
+++ b/jffsX-utils/Makemodule.am
@@ -16,7 +16,7 @@ mkfs_jffs2_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS)
 mkfs_jffs2_CPPFLAGS = $(AM_CPPFLAGS) $(ZLIB_CFLAGS) $(LZO_CFLAGS)
 
 jffs2reader_SOURCES = jffsX-utils/jffs2reader.c	include/mtd/jffs2-user.h
-jffs2reader_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS)
+jffs2reader_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS) $(LZMA_LIBS)
 jffs2reader_CPPFLAGS = $(AM_CPPFLAGS) $(ZLIB_CFLAGS) $(LZO_CFLAGS)
 
 jffs2dump_SOURCES = jffsX-utils/jffs2dump.c include/mtd/jffs2-user.h
diff --git a/jffsX-utils/jffs2reader.c b/jffsX-utils/jffs2reader.c
index 33c5577..2aa2137 100644
--- a/jffsX-utils/jffs2reader.c
+++ b/jffsX-utils/jffs2reader.c
@@ -76,6 +76,9 @@ BUGS:
 #include <sys/stat.h>
 #include <dirent.h>
 #include <zlib.h>
+#ifndef WITHOUT_LZMA
+#include <lzma.h>
+#endif
 
 #include "mtd/jffs2-user.h"
 #include "common.h"
@@ -147,6 +150,48 @@ static void putblock(char *b, size_t bsize, size_t * rsize,
 			bzero(b + je32_to_cpu(n->offset), dlen);
 			break;
 
+#ifndef WITHOUT_LZMA
+		case JFFS2_COMPR_LZMA: {
+			// https://sourceforge.net/p/lzmautils/discussion/708858/thread/d02ebb9386/
+			lzma_stream strm = LZMA_STREAM_INIT;
+			size_t csize = je32_to_cpu(n->csize);
+			Bytef *in = (Bytef *) ((char *) n) + sizeof(struct jffs2_raw_inode);
+			Bytef *out = (Bytef *) b + je32_to_cpu(n->offset);
+			Bytef *compressed_with_header = xmalloc(13 + csize);
+
+			// LZMA properties byte (lc/lp/pb). typical 3/0/2 is 0x5D.
+			// But here works with 0/0/0
+			compressed_with_header[0] = 0;
+
+			// dictionary size as bytes[4], using value from preset 1
+			lzma_options_lzma opt_lzma;
+			if (lzma_lzma_preset(&opt_lzma, 1)) {
+				fprintf(stderr, "preset failed\n");
+				exit(1);
+			}
+			uint32_t dict_size_le = cpu_to_le32(opt_lzma.dict_size);
+			memcpy(&compressed_with_header[1], &dict_size_le, sizeof(dict_size_le));
+
+			// uncompressed size as unsigned 60-bit little endian integer
+			uint64_t uncompressed_size_le = cpu_to_le64(dlen);
+			memcpy(&compressed_with_header[5], &uncompressed_size_le, sizeof(uncompressed_size_le));
+
+			memcpy(&compressed_with_header[13], in, csize);
+			lzma_ret ret = lzma_alone_decoder(&strm, UINT64_MAX);
+			// lzma_ret ret = lzma_alone_decoder(&strm, hardware_memlimit_get(MODE_DECOMPRESS));
+			strm.next_in = compressed_with_header;
+			strm.avail_in = 13 + csize;
+			strm.next_out = out;
+			strm.avail_out = dlen;
+			ret = lzma_code(&strm, LZMA_RUN);
+			if (ret == LZMA_OK) {
+				printf("lzma_code ret:%d in:%ld\n", ret, strm.total_in);
+			}
+			free(compressed_with_header);
+		}
+			break;
+#endif
+
 			/* [DYN]RUBIN support required! */
 
 		default:
-- 
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

[PATCH 4/4] find_raw_inode returns latest version with data of inode

[Frederic Germain]

Signed-off-by: Frederic Germain <frederic.germain@gmail.com>
---
 jffsX-utils/jffs2reader.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/jffsX-utils/jffs2reader.c b/jffsX-utils/jffs2reader.c
index 2aa2137..c0c61cf 100644
--- a/jffsX-utils/jffs2reader.c
+++ b/jffsX-utils/jffs2reader.c
@@ -484,18 +484,19 @@ static struct jffs2_raw_inode *find_raw_inode(char *o, size_t size,
 
 		if (n < e && je16_to_cpu(n->u.magic) == JFFS2_MAGIC_BITMASK) {
 			if (je16_to_cpu(n->u.nodetype) == JFFS2_NODETYPE_INODE &&
-				je32_to_cpu(n->i.ino) == ino && (v = je32_to_cpu(n->i.version)) > vcur) {
+				je32_to_cpu(n->i.ino) == ino && (v = je32_to_cpu(n->i.version)) > vcur && je32_to_cpu(n->i.csize)) {
 				/* XXX crc check */
 
-				if (vmaxt < v)
+				if (vmaxt < v) {
 					vmaxt = v;
-				if (vmint > v) {
-					vmint = v;
 					mp = n;
 				}
+					if (vmint > v) {
+					vmint = v;
+				}
 
-				if (v == (vcur + 1))
-					return (&(n->i));
+//				if (v == (vcur + 1))
+//					return (&(n->i));
 			}
 
 			ADD_BYTES(n, ((je32_to_cpu(n->u.totlen) + 3) & ~3));
-- 
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/