bpfilter causes a leftover kernel process

bpfilter causes a leftover kernel process

[Olivier Brunel]

Hi,

(Please cc me as I'm not subscribed to the list, thanks.)

I'm running an Arch Linux x86_64 system, and recently updated to a 3.18
kernel, which led me to encounter what I believe to be a kernel bug
related to the bpfilter framework.

What happens is that upon boot, there's a "leftover kernel process"
running (shown as "[none]" in ps), which doesn't seem to do anything
(anymore) but does have references/fds open to the root fs, and so when
trying to shutdown the system umounting the root fs fails (EBUSY)
because of it, leading to expected issues.

Simply killing that process allows umounting the root fs fine and
"resolves" all issues. This process/behavior wasn't there with any
previous kernel, and is there with all tried kernels from 4.18.0 to
4.18.4, without any other change to the system -- although this is due
to CONFIG_BPFILTER=y in the kernel config.

Indeed I managed to compile a kernel 4.18.4 myself using the Arch Linux
config[1] with a single change of unsetting CONFIG_BPFILTER, and with
the resulting kernel I don't have this "leftover kernel process"
anymore, everything is back to normal.

Now, about this process, here's a few outputs to try and describe what
it is:

rafus# pgrep none
920

rafus# cd /proc/920

rafus# readlink exe
/ (deleted)

rafus# ls -l fd
total 0
lr-x------ 1 root root 64 Aug 26 17:17 0 -> 'pipe:[13366]'
l-wx------ 1 root root 64 Aug 26 17:17 1 -> 'pipe:[13367]'
lrwx------ 1 root root 64 Aug 26 17:17 2 -> /dev/console

rafus# cat status
Name:	none
Umask:	0022
State:	S (sleeping)
Tgid:	920
Ngid:	0
Pid:	920
PPid:	2
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	64
Groups:	 
NStgid:	920
NSpid:	920
NSpgid:	0
NSsid:	0
VmPeak:	    2296 kB
VmSize:	    2296 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	     748 kB
VmRSS:	     748 kB
RssAnon:	      60 kB
RssFile:	     684 kB
RssShmem:	       4 kB
VmData:	     176 kB
VmStk:	     132 kB
VmExe:	       8 kB
VmLib:	    1452 kB
VmPTE:	      44 kB
VmSwap:	       0 kB
HugetlbPages:	       0 kB
CoreDumping:	0
Threads:	1
SigQ:	0/7861
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	0000000000000000
CapInh:	0000000000000000
CapPrm:	0000003fffffffff
CapEff:	0000003fffffffff
CapBnd:	0000003fffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
Speculation_Store_Bypass:	vulnerable
Cpus_allowed:	1
Cpus_allowed_list:	0
Mems_allowed:	00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	65
nonvoluntary_ctxt_switches:	1

rafus# cat stack
[<0>] pipe_wait+0x6c/0xb0
[<0>] pipe_read+0x20a/0x2d0
[<0>] __vfs_read+0x13a/0x180
[<0>] vfs_read+0x8a/0x130
[<0>] ksys_read+0x4f/0xb0
[<0>] do_syscall_64+0x5b/0x170
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff

rafus# file -L exe
exe: ELF 64-bit LSB pie executable x86-64, version 1 (SYSV),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for
GNU/Linux 3.2.0,
BuildID[sha1]=b247cedd3f8daaea3eee38477aa641d84b77f0ba, not stripped

rafus# stat -L exe
  File: exe
  Size: 16832     	Blocks: 40         IO Block: 4096   regular
file Device: 1h/1d	Inode: 13361       Links: 0
Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-08-26 17:17:37.334261924 +0200
Modify: 2018-08-26 17:14:27.787595262 +0200
Change: 2018-08-26 17:14:27.787595262 +0200
 Birth: -

rafus# sha1sum exe
723d59584abe5e1e9917f0ec41d7e9120514afe7  exe

rafus# strings exe|grep bpf
Started bpfilter


I'm not actually sure what the process is, I'm guessing some kind of
helper is spawned at some point during boot, and for some reason it
never ends.

Although I can reproduce it (it happens on every boot with a kernel
4.18 and CONFIG_BPFILTER=y), I'm unfortunately not sure what is
actually needed to be done in order to trigger it.
I don't use bpfilter myself/directly, as said this happens with the
exact same system as with previous kernels, but I obviously have some
network configuration (done using iptables/iproute2) set up during boot.

Let me know if you need more information or need me to test things, and
I'll do my best.

Thank you.


[1]
https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/linux

Re: bpfilter causes a leftover kernel process

[Olivier Brunel]

Small addentum: Of course I failed to realize this bpfilter helper is
also mentioned in the kernel log:

kern.info: [    8.997711] bpfilter: Loaded bpfilter_umh pid 920


It also seems to be absolutely required when CONFIG_BPFILTER is
enabled, that is I tried blacklisting the module bpfilter, but then
other things (e.g. iptables-restore) just fail to work.

So the process is required, never ends and prevents umouting the
rootfs on shutdown. Unless I'm missing something, there's definitely a
bug there?

Thanks,

Re: bpfilter causes a leftover kernel process

[Alexei Starovoitov]

On Mon, Aug 27, 2018 at 06:31:22PM +0200, Olivier Brunel wrote:
> 
> So the process is required, never ends and prevents umouting the

it's not required. It's not doing anything useful at the moment
and defaults to 'n' in kconfig. Please disable it your kernel.

> rootfs on shutdown. Unless I'm missing something, there's definitely a
> bug there?

I'm also running Arch Linux in my VM, but I'm not able to reproduce umount issue.
I'm guessing it's somehow related to non-static build and libc.so being busy
with old systemd.
Typical shutdown should have done:
[   73.498022] shutdown[1]: Sending SIGTERM to remaining processes...
[   73.505501] shutdown[1]: Sending SIGKILL to remaining processes...
[   73.512783] shutdown[1]: Unmounting file systems.
And at the time of umount / no processes are alive other than systemd.

Re: bpfilter causes a leftover kernel process

[Olivier Brunel]

On Mon, 27 Aug 2018 20:35:02 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> I'm also running Arch Linux in my VM, but I'm not able to reproduce
> umount issue. I'm guessing it's somehow related to non-static build
> and libc.so being busy with old systemd.

Oh, I mentioned it in a previous draft of my original mail but it
seems it got lost in rewrites, I don't actually use systemd. Not that
it should matter here though.


> Typical shutdown should have done:
> [   73.498022] shutdown[1]: Sending SIGTERM to remaining processes...
> [   73.505501] shutdown[1]: Sending SIGKILL to remaining processes...
> [   73.512783] shutdown[1]: Unmounting file systems.
> And at the time of umount / no processes are alive other than systemd.

Yeah, I have a similar thing happening on shutdown, except that we're
talking about a kernel thread here, so that process is ignored by the
mentionned killing spree as a result, thus leaving that process running.

>From a shell opened after the umounting fails I can see that only my
pid1 & the opened shell are running, except of course that this
bpfilter process is there, as a kernel thread "[none]"

Manually killing it at that point works, i.e. allows the umounting to
succeed and a proper shutdown to complete.

Re: bpfilter causes a leftover kernel process

[Alexei Starovoitov]

On Tue, Aug 28, 2018 at 01:23:38PM +0200, Olivier Brunel wrote:
> On Mon, 27 Aug 2018 20:35:02 -0700
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> 
> > I'm also running Arch Linux in my VM, but I'm not able to reproduce
> > umount issue. I'm guessing it's somehow related to non-static build
> > and libc.so being busy with old systemd.
> 
> Oh, I mentioned it in a previous draft of my original mail but it
> seems it got lost in rewrites, I don't actually use systemd. Not that
> it should matter here though.
> 
> 
> > Typical shutdown should have done:
> > [   73.498022] shutdown[1]: Sending SIGTERM to remaining processes...
> > [   73.505501] shutdown[1]: Sending SIGKILL to remaining processes...
> > [   73.512783] shutdown[1]: Unmounting file systems.
> > And at the time of umount / no processes are alive other than systemd.
> 
> Yeah, I have a similar thing happening on shutdown, except that we're
> talking about a kernel thread here, so that process is ignored by the
> mentionned killing spree as a result, thus leaving that process running.

it's not a kernel thread and sounds like there is a bug in your pid 1
that is worth fixing.

Re: bpfilter causes a leftover kernel process

[Olivier Brunel]

On Tue, 28 Aug 2018 22:35:38 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> > Yeah, I have a similar thing happening on shutdown, except that
> > we're talking about a kernel thread here, so that process is
> > ignored by the mentionned killing spree as a result, thus leaving
> > that process running.  
> 
> it's not a kernel thread and sounds like there is a bug in your pid 1
> that is worth fixing.
 
Well, it sure does look like one. By which I mean that looking at
its /proc/$PID entry, one can see that it has an empty command line and
kthreadd as parent (ppid 2), which usually are only true for kernel
threads (unless I'm mistaken).

The tool I'm using to send the signals on shutdown does indeed not use
kill(-1,sig) but instead scans /proc and determines whether or not to
signal a process (thus allowing to ignore a few specific processes to
be killed later on).

Kernel threads are obviously to be skipped, and to identify them it
uses the classic method of checking whether or not it has an empty
command line. As I said, this bpfilter helper does and that's why it is
considered a kernel thread & thus not killed.

This is also what other tools do, like ps or top, which is indeed why
they also show it as a kernel thread ("[none]").

So is this way of doing this wrong? And if so, what would be a better
way to identify kernel threads?

Out of curiosity I also had a look at how systemd does things, and it
looks to me like it does the exact same thing[1], also identifying
kernel threads by their empty command line. So I would think that a
similar issue could be observed under systemd as well.

Thanks.


[1]
https://github.com/systemd/systemd/blob/master/src/core/killall.c#L44

Re: bpfilter causes a leftover kernel process

[Olivier Brunel]

Hi,

Quick follow-up on this:

- first off, Arch devs have updated their kernel config so the next
  kernel will not have bpfilter enabled anymore, thus avoiding any
  issue.

- having said that, I've found a neasy way to reproduce it in an Arch
  VM, in case you're interested :

Boot the latest Arch ISO[1] which now contains a kernel 4.18.5 and do a
very basic installation, pretty much just:

# pacstrap /mnt base
# genfstab -U /mnt >> /mnt/etc/fstab

And of course install your boot loader of choice.

Then boot the brand new system, log in and make sure the helper is
actually started, i.e. `modprobe bpfilter` -- Now halt.

You'll see in the end that systemd complains that it can't
unmount /oldroot (EBUSY), aka the root fs; and that's because of the
bpfilter helper, which wasn't killed because it's seen as a kernel
thread due to its empty command line and therefore not signaled.


Cheers,



[1] https://www.archlinux.org/download/

Re: bpfilter causes a leftover kernel process

[Alexei Starovoitov]

On Wed, Sep 5, 2018 at 5:05 PM Olivier Brunel <jjk@jjacky.com> wrote:
>
> You'll see in the end that systemd complains that it can't
> unmount /oldroot (EBUSY), aka the root fs; and that's because of the
> bpfilter helper, which wasn't killed because it's seen as a kernel
> thread due to its empty command line and therefore not signaled.

thanks for tracking it down.
can somebody send a patch to give bpfilter non-empty cmdline?
I think that would be a better fix than tweaking all pid1s.

[PATCH 0/2] Re: bpfilter causes a leftover kernel process

[Olivier Brunel]

On Tue, 16 Oct 2018 16:38:56 +0000
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> On Wed, Sep 5, 2018 at 5:05 PM Olivier Brunel <jjk@jjacky.com> wrote:
> >
> > You'll see in the end that systemd complains that it can't
> > unmount /oldroot (EBUSY), aka the root fs; and that's because of the
> > bpfilter helper, which wasn't killed because it's seen as a kernel
> > thread due to its empty command line and therefore not signaled.  
> 
> thanks for tracking it down.
> can somebody send a patch to give bpfilter non-empty cmdline?
> I think that would be a better fix than tweaking all pid1s.

So I'm not a kernel dev and this would be my first atttempt at a kernel patch,
but I did have a look and came up with the following patch(es) to fix this.
Hopefully I did things right.

It adds a default command line ("usermodehelper") to such processes, so any &
all such helpers will be seen as user process and not kernel threads, but
there's also the possibility to specify a command line to use, here
"bpfilter_umh"

Cheers,


Olivier Brunel (2):
  umh: Add command line to user mode helpers
  net: bpfilter: Set user mode helper's command line

 include/linux/umh.h          |  1 +
 kernel/umh.c                 | 16 ++++++++++++++--
 net/bpfilter/bpfilter_kern.c |  1 +
 3 files changed, 16 insertions(+), 2 deletions(-)

-- 
2.19.0

[PATCH 1/2] umh: Add command line to user mode helpers

[Olivier Brunel]

User mode helpers were spawned without a command line, and because
an empty command line is used by many tools to identify processes as
kernel threads, this could cause some issues.

Notably during killing spree on shutdown, since such helper would then
be skipped (i.e. not killed) which would result in the process remaining
alive, and thus preventing unmouting of the rootfs (as experienced with
the bpfilter umh).

Fixes: 449325b52b7a ("umh: introduce fork_usermode_blob() helper")
Signed-off-by: Olivier Brunel <jjk@jjacky.com>
---
 include/linux/umh.h |  1 +
 kernel/umh.c        | 16 ++++++++++++++--
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/include/linux/umh.h b/include/linux/umh.h
index 5c812acbb..235f51b62 100644
--- a/include/linux/umh.h
+++ b/include/linux/umh.h
@@ -44,6 +44,7 @@ struct subprocess_info *call_usermodehelper_setup_file(struct file *file,
 			  int (*init)(struct subprocess_info *info, struct cred *new),
 			  void (*cleanup)(struct subprocess_info *), void *data);
 struct umh_info {
+	const char *cmdline;
 	struct file *pipe_to_umh;
 	struct file *pipe_from_umh;
 	pid_t pid;
diff --git a/kernel/umh.c b/kernel/umh.c
index c44985894..0baa672e0 100644
--- a/kernel/umh.c
+++ b/kernel/umh.c
@@ -405,11 +405,19 @@ struct subprocess_info *call_usermodehelper_setup_file(struct file *file,
 		void (*cleanup)(struct subprocess_info *info), void *data)
 {
 	struct subprocess_info *sub_info;
+	struct umh_info *info = data;
+	const char *cmdline = (info->cmdline) ? info->cmdline : "usermodehelper";
 
 	sub_info = kzalloc(sizeof(struct subprocess_info), GFP_KERNEL);
 	if (!sub_info)
 		return NULL;
 
+	sub_info->argv = argv_split(GFP_KERNEL, cmdline, NULL);
+	if (!sub_info->argv) {
+		kfree(sub_info);
+		return NULL;
+	}
+
 	INIT_WORK(&sub_info->work, call_usermodehelper_exec_work);
 	sub_info->path = "none";
 	sub_info->file = file;
@@ -458,10 +466,11 @@ static int umh_pipe_setup(struct subprocess_info *info, struct cred *new)
 	return 0;
 }
 
-static void umh_save_pid(struct subprocess_info *info)
+static void umh_clean_and_save_pid(struct subprocess_info *info)
 {
 	struct umh_info *umh_info = info->data;
 
+	argv_free(info->argv);
 	umh_info->pid = info->pid;
 }
 
@@ -471,6 +480,9 @@ static void umh_save_pid(struct subprocess_info *info)
  * @len: length of the blob
  * @info: information about usermode process (shouldn't be NULL)
  *
+ * If info->cmdline is set it will be used as command line for the
+ * user process, else "usermodehelper" is used.
+ *
  * Returns either negative error or zero which indicates success
  * in executing a blob of bytes as a usermode process. In such
  * case 'struct umh_info *info' is populated with two pipes
@@ -500,7 +512,7 @@ int fork_usermode_blob(void *data, size_t len, struct umh_info *info)
 
 	err = -ENOMEM;
 	sub_info = call_usermodehelper_setup_file(file, umh_pipe_setup,
-						  umh_save_pid, info);
+						  umh_clean_and_save_pid, info);
 	if (!sub_info)
 		goto out;
 
-- 
2.19.0

Re: [PATCH 1/2] umh: Add command line to user mode helpers

[David Miller]

From: Olivier Brunel <jjk@jjacky.com>
Date: Sat, 20 Oct 2018 19:39:56 +0200

> User mode helpers were spawned without a command line, and because
> an empty command line is used by many tools to identify processes as
> kernel threads, this could cause some issues.
> 
> Notably during killing spree on shutdown, since such helper would then
> be skipped (i.e. not killed) which would result in the process remaining
> alive, and thus preventing unmouting of the rootfs (as experienced with
> the bpfilter umh).
> 
> Fixes: 449325b52b7a ("umh: introduce fork_usermode_blob() helper")
> Signed-off-by: Olivier Brunel <jjk@jjacky.com>

Applied.

[PATCH 2/2] net: bpfilter: Set user mode helper's command line

[Olivier Brunel]

Signed-off-by: Olivier Brunel <jjk@jjacky.com>
---
 net/bpfilter/bpfilter_kern.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/net/bpfilter/bpfilter_kern.c b/net/bpfilter/bpfilter_kern.c
index 94e88f510..7acfc8308 100644
--- a/net/bpfilter/bpfilter_kern.c
+++ b/net/bpfilter/bpfilter_kern.c
@@ -92,6 +92,7 @@ static int __init load_umh(void)
 	int err;
 
 	/* fork usermode process */
+	info.cmdline = "bpfilter_umh";
 	err = fork_usermode_blob(&bpfilter_umh_start,
 				 &bpfilter_umh_end - &bpfilter_umh_start,
 				 &info);
-- 
2.19.0

Re: [PATCH 2/2] net: bpfilter: Set user mode helper's command line

[David Miller]

aFrom: Olivier Brunel <jjk@jjacky.com>
Date: Sat, 20 Oct 2018 19:39:57 +0200

> Signed-off-by: Olivier Brunel <jjk@jjacky.com>

Applied.