[PATCH 1/2] initramfs: don't include filenames from the initramfs for make goals (dist)clean

[PATCH 1/2] initramfs: don't include filenames from the initramfs for make goals (dist)clean

[Alexander Holler]

Various unusual filenames might break the make process when they are included
as dependencies (e.g. filenames with a colon).

This has the disadvantage that not even a make (dist)clean does work afterwards.
What makes the matter worse is that git clean -df doesn't remove such a possible
broken dependency list too.

Avoid that by not including this dependency list for make goals (dist)clean.

Signed-off-by: Alexander Holler <holler@ahsoftware.de>
Cc: <stable@vger.kernel.org>
---
 usr/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/usr/Makefile b/usr/Makefile
index e767f01..5d5b6aa 100644
--- a/usr/Makefile
+++ b/usr/Makefile
@@ -49,9 +49,11 @@ ramfs-args  := \
 # in initramfs and to detect if any files are added/removed.
 # Removed files are identified by directory timestamp being updated
 # The dependency list is generated by gen_initramfs.sh -l
+ifndef clean
 ifneq ($(wildcard $(obj)/.initramfs_data.cpio.d),)
 	include $(obj)/.initramfs_data.cpio.d
 endif
+endif
 
 quiet_cmd_initfs = GEN     $@
       cmd_initfs = $(initramfs) -o $@ $(ramfs-args) $(ramfs-input)
-- 
1.8.3.1

[PATCH 2/2 v3] initramfs: print error and shell out for unsupported content

[Alexander Holler]

The initramfs generation is broken for file and directory names which contain
colons, spaces and other unusual characters. Print an error and don't try to
continue.

Some tests:

cd linux
make defconfig
echo 'CONFIG_BLK_DEV_INITRD=y' >> .config
echo 'CONFIG_INITRAMFS_ROOT_UID=0' >>.config
echo 'CONFIG_INITRAMFS_ROOT_GID=0' >>.config
echo 'CONFIG_INITRAMFS_COMPRESSION_NONE=y' >>.config
echo 'CONFIG_INITRAMFS_SOURCE="/tmp/bugroot"' >>.config

Problem with colons:

mkdir -p /tmp/bugroot/a:b
make -j4 bzImage # no error
make bzImage # try again, oops

Problem with spaces:

mkdir -p /tmp/bugroot/a\ b
make -j4 bzImage # no error
zcat usr/initramfs_data.cpio.gz | cpio --extract --list # oops, no content

Signed-off-by: Alexander Holler <holler@ahsoftware.de>
Cc: <stable@vger.kernel.org>
---
 Changes in v2, v3: check for \n, \r and \t too, updated error msg.

 scripts/gen_initramfs_list.sh | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh
index 17fa901..2c613e7 100644
--- a/scripts/gen_initramfs_list.sh
+++ b/scripts/gen_initramfs_list.sh
@@ -171,6 +171,19 @@ dir_filelist() {
 	${dep_list}header "$1"
 
 	srcdir=$(echo "$1" | sed -e 's://*:/:g')
+
+	# Files and directories with spaces and colons are unsupported.
+	local unsupported=$(find "${srcdir}" -regex '.*\(:\| \|\n\|\r\|\t\).*')
+	if [ ! -z "${unsupported}" ]; then
+		printf "ERROR: Unable to handle files/directories with " >&2
+		printf "unsupported characters (spaces, :, \\\n, \\\r, " >&2
+		printf "\\\t).\n Please use other ways to generate a " >&2
+		printf "cpio-archive with such names.\n" >&2
+		printf "Unsupported files and directories are:\n" >&2
+		printf "$unsupported\n" >&2
+		exit 1
+	fi
+
 	dirlist=$(find "${srcdir}" -printf "%p %m %U %G\n")
 
 	# If $dirlist is only one line, then the directory is empty
-- 
1.8.3.1

Re: [PATCH 2/2 v3] initramfs: print error and shell out for unsupported content

[Alexander Holler]

Am 22.03.2014 13:29, schrieb Alexander Holler:
> The initramfs generation is broken for file and directory names which contain
> colons, spaces and other unusual characters. Print an error and don't try to
> continue.

(...)
> +	# Files and directories with spaces and colons are unsupported.
> +	local unsupported=$(find "${srcdir}" -regex '.*\(:\| \|\n\|\r\|\t\).*')

I've just noted that -regex isn't POSIX. I don't know the kernel rules 
regarding this, and I don't care. But it might be a blocker for this patch.

Regards,

Alexander Holler

Re: [PATCH 2/2 v3] initramfs: print error and shell out for unsupported content

[Michal Marek]

Dne 22.3.2014 19:22, Alexander Holler napsal(a):
> Am 22.03.2014 13:29, schrieb Alexander Holler:
>> The initramfs generation is broken for file and directory names which
>> contain
>> colons, spaces and other unusual characters. Print an error and don't
>> try to
>> continue.
> 
> (...)
>> +    # Files and directories with spaces and colons are unsupported.
>> +    local unsupported=$(find "${srcdir}" -regex '.*\(:\|
>> \|\n\|\r\|\t\).*')
> 
> I've just noted that -regex isn't POSIX. I don't know the kernel rules
> regarding this, and I don't care. But it might be a blocker for this patch.

The bigger problem is that there is no C-style quoting in regexps or
character classes matches any file with 'n', 'r' or 't' in its name. How
about

  -name '*[:[:space:]]*'

?

Michal

Re: [PATCH 2/2 v3] initramfs: print error and shell out for unsupported content

[Alexander Holler]

Am 31.03.2014 22:31, schrieb Michal Marek:
> Dne 22.3.2014 19:22, Alexander Holler napsal(a):
>> Am 22.03.2014 13:29, schrieb Alexander Holler:
>>> The initramfs generation is broken for file and directory names which
>>> contain
>>> colons, spaces and other unusual characters. Print an error and don't
>>> try to
>>> continue.
>>
>> (...)
>>> +    # Files and directories with spaces and colons are unsupported.
>>> +    local unsupported=$(find "${srcdir}" -regex '.*\(:\|
>>> \|\n\|\r\|\t\).*')
>>
>> I've just noted that -regex isn't POSIX. I don't know the kernel rules
>> regarding this, and I don't care. But it might be a blocker for this patch.
>
> The bigger problem is that there is no C-style quoting in regexps or
> character classes matches any file with 'n', 'r' or 't' in its name. How
> about
>
>    -name '*[:[:space:]]*'
>
> ?

Hmm, I wasn't aware that find supports such expressions, and I don't 
know how compatible that is. It would do the trick too.

But as I already said, trying to use arbitrary filenames in a Makefile 
doesn't really work. There are still other problems, e.g. filenames with 
a / (directory separator), *, or some of the special make variables like $<.

So the really working solution would be to get rid of that generated 
(and hidden) include. Modifying gen_init_cpio.c to browse and collect 
all filenames and types itself doesn't look like much work (an evening 
should be enough).

The drawback is that such a solution would build the initramfs every 
time make is called (while CONFIG_INITRAMFS_SOURCE points to a 
directory), but I think that time is negligible.
Another drawback is that it wouldn't be a simple patch anymore. Thus it 
wouldn't qualify for any stable tree (and nobody promises such a patch 
would be accepted at all, so it might be a waste of time at all).

Regards,

Alexander Holler

Re: [PATCH 2/2 v3] initramfs: print error and shell out for unsupported content

[Michal Marek]

On 2014-04-01 13:23, Alexander Holler wrote:
> But as I already said, trying to use arbitrary filenames in a Makefile 
> doesn't really work. There are still other problems, e.g. filenames with 
> a / (directory separator), *, or some of the special make variables like $<.
> 
> So the really working solution would be to get rid of that generated 
> (and hidden) include. Modifying gen_init_cpio.c to browse and collect 
> all filenames and types itself doesn't look like much work (an evening 
> should be enough).
> 
> The drawback is that such a solution would build the initramfs every 
> time make is called (while CONFIG_INITRAMFS_SOURCE points to a 
> directory), but I think that time is negligible.

Well, the cpio is embedded in the kernel image, so a rebuild of the cpio
means a relink of the kernel. One option would be to implement the
timestamp checking in gen_init_cpio.c, but I'm not sure if that's worth it.

Michal

Re: [PATCH 2/2 v3] initramfs: print error and shell out for unsupported content

[Alexander Holler]

Am 01.04.2014 14:23, schrieb Michal Marek:
> On 2014-04-01 13:23, Alexander Holler wrote:
>> But as I already said, trying to use arbitrary filenames in a Makefile
>> doesn't really work. There are still other problems, e.g. filenames with
>> a / (directory separator), *, or some of the special make variables like $<.
>>
>> So the really working solution would be to get rid of that generated
>> (and hidden) include. Modifying gen_init_cpio.c to browse and collect
>> all filenames and types itself doesn't look like much work (an evening
>> should be enough).
>>
>> The drawback is that such a solution would build the initramfs every
>> time make is called (while CONFIG_INITRAMFS_SOURCE points to a
>> directory), but I think that time is negligible.
>
> Well, the cpio is embedded in the kernel image, so a rebuild of the cpio
> means a relink of the kernel. One option would be to implement the
> timestamp checking in gen_init_cpio.c, but I'm not sure if that's worth it.

I can't answer that too, but if someone spends the time implement the 
directory traversing and collecting stats, it would just be one stat 
more (to get the timestamp of an maybe already built cpio archive).

But I'm fine with just avoiding the most common problems (in my case 
colons and spaces). And the little patch which makes sure that make 
clean, distclean or mrproper will work always, avoids the need to search 
how to fix the problem (deleting that hidden include) once a broken 
include was generated.

Regards,

Alexander Holler

[PATCH 1/2 v4] initramfs: don't include filenames from the initramfs for make goals (dist)clean

[Alexander Holler]

Various unusual filenames might break the make process when they are included
as dependencies (e.g. filenames with a colon).

This has the disadvantage that not even a make (dist)clean does work afterwards.
What makes the matter worse is that git clean -df doesn't remove such a possible
broken dependency list too.

Avoid that by not including this dependency list for make goals (dist)clean.

Signed-off-by: Alexander Holler <holler@ahsoftware.de>
Cc: <stable@vger.kernel.org>
---

 Changes to v3: none

 usr/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/usr/Makefile b/usr/Makefile
index e767f01..5d5b6aa 100644
--- a/usr/Makefile
+++ b/usr/Makefile
@@ -49,9 +49,11 @@ ramfs-args  := \
 # in initramfs and to detect if any files are added/removed.
 # Removed files are identified by directory timestamp being updated
 # The dependency list is generated by gen_initramfs.sh -l
+ifndef clean
 ifneq ($(wildcard $(obj)/.initramfs_data.cpio.d),)
 	include $(obj)/.initramfs_data.cpio.d
 endif
+endif
 
 quiet_cmd_initfs = GEN     $@
       cmd_initfs = $(initramfs) -o $@ $(ramfs-args) $(ramfs-input)
-- 
1.8.3.1

[PATCH 2/2 v4] initramfs: print error and shell out for unsupported content

[Alexander Holler]

The initramfs generation is broken for file and directory names which contain
colons, spaces and other unusual characters which can't be included in names
for dependencies in traditional Makefiles (which gen_initramfs_list.sh
generates). Print an error for the most common unsupported characters (colons
and whitespace) and don't try to continue if these are discovered.

Some tests:

cd linux
make defconfig
echo 'CONFIG_BLK_DEV_INITRD=y' >> .config
echo 'CONFIG_INITRAMFS_ROOT_UID=0' >>.config
echo 'CONFIG_INITRAMFS_ROOT_GID=0' >>.config
echo 'CONFIG_INITRAMFS_COMPRESSION_NONE=y' >>.config
echo 'CONFIG_INITRAMFS_SOURCE="/tmp/bugroot"' >>.config

Problem with colons:

mkdir -p /tmp/bugroot/a:b
make -j4 bzImage # no error
make bzImage # try again, oops

Problem with spaces:

mkdir -p /tmp/bugroot/a\ b
make -j4 bzImage # no error
zcat usr/initramfs_data.cpio.gz | cpio --extract --list # oops, no content

Cc: <stable@vger.kernel.org>
---

 Changes to v3:
   - Uses find with -name and an expression suggested by Michael Marek
     instead of find -regex. Therefor I've added his Signed-off-by.
   - Detail the unsupported characters which which will be discovered in
     the commit message.

 scripts/gen_initramfs_list.sh | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh
index 17fa901..53398d5 100755
--- a/scripts/gen_initramfs_list.sh
+++ b/scripts/gen_initramfs_list.sh
@@ -171,6 +171,19 @@ dir_filelist() {
 	${dep_list}header "$1"
 
 	srcdir=$(echo "$1" | sed -e 's://*:/:g')
+
+	# Files and directories with spaces and colons are unsupported.
+	local unsupported=$(find "${srcdir}" -name '*[:[:space:]]*')
+	if [ ! -z "${unsupported}" ]; then
+		printf "ERROR: Unable to handle files/directories with " >&2
+		printf "unsupported characters (spaces, :, \\\n, \\\r, " >&2
+		printf "\\\t).\n Please use other ways to generate a " >&2
+		printf "cpio-archive with such names.\n" >&2
+		printf "Unsupported files and directories are:\n" >&2
+		printf "$unsupported\n" >&2
+		exit 1
+	fi
+
 	dirlist=$(find "${srcdir}" -printf "%p %m %U %G\n")
 
 	# If $dirlist is only one line, then the directory is empty
-- 
1.8.3.1