[PATCH v2 bpf-next 0/4] cgroup bpf auto-detachment

[PATCH v2 bpf-next 0/4] cgroup bpf auto-detachment

[Roman Gushchin]

This patchset implements a cgroup bpf auto-detachment functionality:
bpf programs are attached as soon as possible after removal of the
cgroup, without waiting for the release of all associated resources.

Patches 2 and 3 are required to implement a corresponding kselftest
in patch 4.

v2:
  1) removed a bogus check in patch 4
  2) moved buf[len] = 0 in patch 2


Roman Gushchin (4):
  bpf: decouple the lifetime of cgroup_bpf from cgroup itself
  selftests/bpf: convert test_cgrp2_attach2 example into kselftest
  selftests/bpf: enable all available cgroup v2 controllers
  selftests/bpf: add auto-detach test

 include/linux/bpf-cgroup.h                    |   8 +-
 include/linux/cgroup.h                        |  18 +++
 kernel/bpf/cgroup.c                           |  25 ++-
 kernel/cgroup/cgroup.c                        |  11 +-
 samples/bpf/Makefile                          |   2 -
 tools/testing/selftests/bpf/Makefile          |   4 +-
 tools/testing/selftests/bpf/cgroup_helpers.c  |  57 +++++++
 .../selftests/bpf/test_cgroup_attach.c        | 145 ++++++++++++++++--
 8 files changed, 243 insertions(+), 27 deletions(-)
 rename samples/bpf/test_cgrp2_attach2.c => tools/testing/selftests/bpf/test_cgroup_attach.c (79%)

-- 
2.20.1

[PATCH v2 bpf-next 2/4] selftests/bpf: convert test_cgrp2_attach2 example into kselftest

[Roman Gushchin]

Convert test_cgrp2_attach2 example into a proper test_cgroup_attach
kselftest. It's better because we do run kselftest on a constant
basis, so there are better chances to spot a potential regression.

Also make it slightly less verbose to conform kselftests output style.

Output example:
  $ ./test_cgroup_attach
  #override:PASS
  #multi:PASS
  test_cgroup_attach:PASS

Signed-off-by: Roman Gushchin <guro@fb.com>
---
 samples/bpf/Makefile                          |  2 -
 tools/testing/selftests/bpf/Makefile          |  4 +-
 .../selftests/bpf/test_cgroup_attach.c        | 48 ++++++++++++-------
 3 files changed, 35 insertions(+), 19 deletions(-)
 rename samples/bpf/test_cgrp2_attach2.c => tools/testing/selftests/bpf/test_cgroup_attach.c (92%)

diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
index 4f0a1cdbfe7c..253e5a2856be 100644
--- a/samples/bpf/Makefile
+++ b/samples/bpf/Makefile
@@ -26,7 +26,6 @@ hostprogs-y += map_perf_test
 hostprogs-y += test_overhead
 hostprogs-y += test_cgrp2_array_pin
 hostprogs-y += test_cgrp2_attach
-hostprogs-y += test_cgrp2_attach2
 hostprogs-y += test_cgrp2_sock
 hostprogs-y += test_cgrp2_sock2
 hostprogs-y += xdp1
@@ -81,7 +80,6 @@ map_perf_test-objs := bpf_load.o map_perf_test_user.o
 test_overhead-objs := bpf_load.o test_overhead_user.o
 test_cgrp2_array_pin-objs := test_cgrp2_array_pin.o
 test_cgrp2_attach-objs := test_cgrp2_attach.o
-test_cgrp2_attach2-objs := test_cgrp2_attach2.o $(CGROUP_HELPERS)
 test_cgrp2_sock-objs := test_cgrp2_sock.o
 test_cgrp2_sock2-objs := bpf_load.o test_cgrp2_sock2.o
 xdp1-objs := xdp1_user.o
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 66f2dca1dee1..e09f419f4d7e 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -23,7 +23,8 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test
 	test_align test_verifier_log test_dev_cgroup test_tcpbpf_user \
 	test_sock test_btf test_sockmap test_lirc_mode2_user get_cgroup_id_user \
 	test_socket_cookie test_cgroup_storage test_select_reuseport test_section_names \
-	test_netcnt test_tcpnotify_user test_sock_fields test_sysctl
+	test_netcnt test_tcpnotify_user test_sock_fields test_sysctl \
+	test_cgroup_attach
 
 BPF_OBJ_FILES = $(patsubst %.c,%.o, $(notdir $(wildcard progs/*.c)))
 TEST_GEN_FILES = $(BPF_OBJ_FILES)
@@ -96,6 +97,7 @@ $(OUTPUT)/test_cgroup_storage: cgroup_helpers.c
 $(OUTPUT)/test_netcnt: cgroup_helpers.c
 $(OUTPUT)/test_sock_fields: cgroup_helpers.c
 $(OUTPUT)/test_sysctl: cgroup_helpers.c
+$(OUTPUT)/test_cgroup_attach: cgroup_helpers.c
 
 .PHONY: force
 
diff --git a/samples/bpf/test_cgrp2_attach2.c b/tools/testing/selftests/bpf/test_cgroup_attach.c
similarity index 92%
rename from samples/bpf/test_cgrp2_attach2.c
rename to tools/testing/selftests/bpf/test_cgroup_attach.c
index 0bb6507256b7..93d4fe295e7d 100644
--- a/samples/bpf/test_cgrp2_attach2.c
+++ b/tools/testing/selftests/bpf/test_cgroup_attach.c
@@ -1,3 +1,5 @@
+// SPDX-License-Identifier: GPL-2.0
+
 /* eBPF example program:
  *
  * - Creates arraymap in kernel with 4 bytes keys and 8 byte values
@@ -25,20 +27,27 @@
 #include <sys/resource.h>
 #include <sys/time.h>
 #include <unistd.h>
+#include <linux/filter.h>
 
 #include <linux/bpf.h>
 #include <bpf/bpf.h>
 
-#include "bpf_insn.h"
+#include "bpf_util.h"
 #include "bpf_rlimit.h"
 #include "cgroup_helpers.h"
 
 #define FOO		"/foo"
 #define BAR		"/foo/bar/"
-#define PING_CMD	"ping -c1 -w1 127.0.0.1 > /dev/null"
+#define PING_CMD	"ping -q -c1 -w1 127.0.0.1 > /dev/null"
 
 char bpf_log_buf[BPF_LOG_BUF_SIZE];
 
+#ifdef DEBUG
+#define debug(args...) printf(args)
+#else
+#define debug(args...)
+#endif
+
 static int prog_load(int verdict)
 {
 	int ret;
@@ -89,7 +98,7 @@ static int test_foo_bar(void)
 		goto err;
 	}
 
-	printf("Attached DROP prog. This ping in cgroup /foo should fail...\n");
+	debug("Attached DROP prog. This ping in cgroup /foo should fail...\n");
 	assert(system(PING_CMD) != 0);
 
 	/* Create cgroup /foo/bar, get fd, and join it */
@@ -100,7 +109,7 @@ static int test_foo_bar(void)
 	if (join_cgroup(BAR))
 		goto err;
 
-	printf("Attached DROP prog. This ping in cgroup /foo/bar should fail...\n");
+	debug("Attached DROP prog. This ping in cgroup /foo/bar should fail...\n");
 	assert(system(PING_CMD) != 0);
 
 	if (bpf_prog_attach(allow_prog, bar, BPF_CGROUP_INET_EGRESS,
@@ -109,7 +118,7 @@ static int test_foo_bar(void)
 		goto err;
 	}
 
-	printf("Attached PASS prog. This ping in cgroup /foo/bar should pass...\n");
+	debug("Attached PASS prog. This ping in cgroup /foo/bar should pass...\n");
 	assert(system(PING_CMD) == 0);
 
 	if (bpf_prog_detach(bar, BPF_CGROUP_INET_EGRESS)) {
@@ -117,7 +126,7 @@ static int test_foo_bar(void)
 		goto err;
 	}
 
-	printf("Detached PASS from /foo/bar while DROP is attached to /foo.\n"
+	debug("Detached PASS from /foo/bar while DROP is attached to /foo.\n"
 	       "This ping in cgroup /foo/bar should fail...\n");
 	assert(system(PING_CMD) != 0);
 
@@ -132,7 +141,7 @@ static int test_foo_bar(void)
 		goto err;
 	}
 
-	printf("Attached PASS from /foo/bar and detached DROP from /foo.\n"
+	debug("Attached PASS from /foo/bar and detached DROP from /foo.\n"
 	       "This ping in cgroup /foo/bar should pass...\n");
 	assert(system(PING_CMD) == 0);
 
@@ -199,9 +208,9 @@ static int test_foo_bar(void)
 	close(bar);
 	cleanup_cgroup_environment();
 	if (!rc)
-		printf("### override:PASS\n");
+		printf("#override:PASS\n");
 	else
-		printf("### override:FAIL\n");
+		printf("#override:FAIL\n");
 	return rc;
 }
 
@@ -441,19 +450,26 @@ static int test_multiprog(void)
 	close(cg5);
 	cleanup_cgroup_environment();
 	if (!rc)
-		printf("### multi:PASS\n");
+		printf("#multi:PASS\n");
 	else
-		printf("### multi:FAIL\n");
+		printf("#multi:FAIL\n");
 	return rc;
 }
 
 int main(int argc, char **argv)
 {
-	int rc = 0;
+	int (*tests[])(void) = {test_foo_bar, test_multiprog};
+	int errors = 0;
+	int i;
 
-	rc = test_foo_bar();
-	if (rc)
-		return rc;
+	for (i = 0; i < ARRAY_SIZE(tests); i++)
+		if (tests[i]())
+			errors++;
+
+	if (errors)
+		printf("test_cgroup_attach:FAIL\n");
+	else
+		printf("test_cgroup_attach:PASS\n");
 
-	return test_multiprog();
+	return errors ? EXIT_FAILURE : EXIT_SUCCESS;
 }
-- 
2.20.1

[PATCH v2 bpf-next 3/4] selftests/bpf: enable all available cgroup v2 controllers

[Roman Gushchin]

Enable all available cgroup v2 controllers when setting up
the environment for the bpf kselftests. It's required to properly test
the bpf prog auto-detach feature. Also it will generally increase
the code coverage.

Signed-off-by: Roman Gushchin <guro@fb.com>
---
 tools/testing/selftests/bpf/cgroup_helpers.c | 57 ++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/tools/testing/selftests/bpf/cgroup_helpers.c b/tools/testing/selftests/bpf/cgroup_helpers.c
index 6692a40a6979..4efe57c171cd 100644
--- a/tools/testing/selftests/bpf/cgroup_helpers.c
+++ b/tools/testing/selftests/bpf/cgroup_helpers.c
@@ -33,6 +33,60 @@
 	snprintf(buf, sizeof(buf), "%s%s%s", CGROUP_MOUNT_PATH, \
 		 CGROUP_WORK_DIR, path)
 
+/**
+ * enable_all_controllers() - Enable all available cgroup v2 controllers
+ *
+ * Enable all available cgroup v2 controllers in order to increase
+ * the code coverage.
+ *
+ * If successful, 0 is returned.
+ */
+int enable_all_controllers(char *cgroup_path)
+{
+	char path[PATH_MAX + 1];
+	char buf[PATH_MAX];
+	char *c, *c2;
+	int fd, cfd;
+	size_t len;
+
+	snprintf(path, sizeof(path), "%s/cgroup.controllers", cgroup_path);
+	fd = open(path, O_RDONLY);
+	if (fd < 0) {
+		log_err("Opening cgroup.controllers: %s", path);
+		return -1;
+	}
+
+	len = read(fd, buf, sizeof(buf) - 1);
+	if (len < 0) {
+		close(fd);
+		log_err("Reading cgroup.controllers: %s", path);
+		return -1;
+	}
+	buf[len] = 0;
+	close(fd);
+
+	/* No controllers available? We're probably on cgroup v1. */
+	if (len == 0)
+		return 0;
+
+	snprintf(path, sizeof(path), "%s/cgroup.subtree_control", cgroup_path);
+	cfd = open(path, O_RDWR);
+	if (cfd < 0) {
+		log_err("Opening cgroup.subtree_control: %s", path);
+		return -1;
+	}
+
+	for (c = strtok_r(buf, " ", &c2); c; c = strtok_r(NULL, " ", &c2)) {
+		if (dprintf(cfd, "+%s\n", c) <= 0) {
+			log_err("Enabling controller %s: %s", c, path);
+			close(cfd);
+			return -1;
+		}
+	}
+	close(cfd);
+	return 0;
+}
+
 /**
  * setup_cgroup_environment() - Setup the cgroup environment
  *
@@ -71,6 +125,9 @@ int setup_cgroup_environment(void)
 		return 1;
 	}
 
+	if (enable_all_controllers(cgroup_workdir))
+		return 1;
+
 	return 0;
 }
 
-- 
2.20.1

[PATCH v2 bpf-next 4/4] selftests/bpf: add auto-detach test

[Roman Gushchin]

Add a kselftest to cover bpf auto-detachment functionality.
The test creates a cgroup, associates some resources with it,
attaches a couple of bpf programs and deletes the cgroup.

Then it checks that bpf programs are going away in 5 seconds.

Expected output:
  $ ./test_cgroup_attach
  #override:PASS
  #multi:PASS
  #autodetach:PASS
  test_cgroup_attach:PASS

On a kernel without auto-detaching:
  $ ./test_cgroup_attach
  #override:PASS
  #multi:PASS
  #autodetach:FAIL
  test_cgroup_attach:FAIL

Signed-off-by: Roman Gushchin <guro@fb.com>
---
 .../selftests/bpf/test_cgroup_attach.c        | 99 ++++++++++++++++++-
 1 file changed, 98 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/bpf/test_cgroup_attach.c b/tools/testing/selftests/bpf/test_cgroup_attach.c
index 93d4fe295e7d..bc5bd0f1728e 100644
--- a/tools/testing/selftests/bpf/test_cgroup_attach.c
+++ b/tools/testing/selftests/bpf/test_cgroup_attach.c
@@ -456,9 +456,106 @@ static int test_multiprog(void)
 	return rc;
 }
 
+static int test_autodetach(void)
+{
+	__u32 prog_cnt = 4, attach_flags;
+	int allow_prog[2] = {0};
+	__u32 prog_ids[2] = {0};
+	int cg = 0, i, rc = -1;
+	void *ptr = NULL;
+	int attempts;
+
+
+	for (i = 0; i < ARRAY_SIZE(allow_prog); i++) {
+		allow_prog[i] = prog_load_cnt(1, 1 << i);
+		if (!allow_prog[i])
+			goto err;
+	}
+
+	if (setup_cgroup_environment())
+		goto err;
+
+	/* create a cgroup, attach two programs and remember their ids */
+	cg = create_and_get_cgroup("/cg_autodetach");
+	if (cg < 0)
+		goto err;
+
+	if (join_cgroup("/cg_autodetach"))
+		goto err;
+
+	for (i = 0; i < ARRAY_SIZE(allow_prog); i++) {
+		if (bpf_prog_attach(allow_prog[i], cg, BPF_CGROUP_INET_EGRESS,
+				    BPF_F_ALLOW_MULTI)) {
+			log_err("Attaching prog[%d] to cg:egress", i);
+			goto err;
+		}
+	}
+
+	/* make sure that programs are attached and run some traffic */
+	assert(bpf_prog_query(cg, BPF_CGROUP_INET_EGRESS, 0, &attach_flags,
+			      prog_ids, &prog_cnt) == 0);
+	assert(system(PING_CMD) == 0);
+
+	/* allocate some memory (4Mb) to pin the original cgroup */
+	ptr = malloc(4 * (1 << 20));
+	if (!ptr)
+		goto err;
+
+	/* close programs and cgroup fd */
+	for (i = 0; i < ARRAY_SIZE(allow_prog); i++) {
+		close(allow_prog[i]);
+		allow_prog[i] = 0;
+	}
+
+	close(cg);
+	cg = 0;
+
+	/* leave the cgroup and remove it. don't detach programs */
+	cleanup_cgroup_environment();
+
+	/* wait for the asynchronous auto-detachment.
+	 * wait for no more than 5 sec and give up.
+	 */
+	for (i = 0; i < ARRAY_SIZE(prog_ids); i++) {
+		for (attempts = 5; attempts >= 0; attempts--) {
+			int fd = bpf_prog_get_fd_by_id(prog_ids[i]);
+
+			if (fd < 0)
+				break;
+
+			/* don't leave the fd open */
+			close(fd);
+
+			if (!attempts)
+				goto err;
+
+			sleep(1);
+		}
+	}
+
+	rc = 0;
+err:
+	for (i = 0; i < ARRAY_SIZE(allow_prog); i++)
+		if (allow_prog[i] > 0)
+			close(allow_prog[i]);
+	if (cg)
+		close(cg);
+	free(ptr);
+	cleanup_cgroup_environment();
+	if (!rc)
+		printf("#autodetach:PASS\n");
+	else
+		printf("#autodetach:FAIL\n");
+	return rc;
+}
+
 int main(int argc, char **argv)
 {
-	int (*tests[])(void) = {test_foo_bar, test_multiprog};
+	int (*tests[])(void) = {
+		test_foo_bar,
+		test_multiprog,
+		test_autodetach,
+	};
 	int errors = 0;
 	int i;
 
-- 
2.20.1

Re: [PATCH v2 bpf-next 0/4] cgroup bpf auto-detachment

[Yonghong Song]

On 5/22/19 4:20 PM, Roman Gushchin wrote:
> This patchset implements a cgroup bpf auto-detachment functionality:
> bpf programs are attached as soon as possible after removal of the
typo here "attached" => "detached"?

> cgroup, without waiting for the release of all associated resources.
> 
> Patches 2 and 3 are required to implement a corresponding kselftest
> in patch 4.
> 
> v2:
>    1) removed a bogus check in patch 4
>    2) moved buf[len] = 0 in patch 2
> 
> 
> Roman Gushchin (4):
>    bpf: decouple the lifetime of cgroup_bpf from cgroup itself
>    selftests/bpf: convert test_cgrp2_attach2 example into kselftest
>    selftests/bpf: enable all available cgroup v2 controllers
>    selftests/bpf: add auto-detach test
> 
>   include/linux/bpf-cgroup.h                    |   8 +-
>   include/linux/cgroup.h                        |  18 +++
>   kernel/bpf/cgroup.c                           |  25 ++-
>   kernel/cgroup/cgroup.c                        |  11 +-
>   samples/bpf/Makefile                          |   2 -
>   tools/testing/selftests/bpf/Makefile          |   4 +-
>   tools/testing/selftests/bpf/cgroup_helpers.c  |  57 +++++++
>   .../selftests/bpf/test_cgroup_attach.c        | 145 ++++++++++++++++--
>   8 files changed, 243 insertions(+), 27 deletions(-)
>   rename samples/bpf/test_cgrp2_attach2.c => tools/testing/selftests/bpf/test_cgroup_attach.c (79%)
> 

Re: [PATCH v2 bpf-next 1/4] bpf: decouple the lifetime of cgroup_bpf from cgroup itself

[Yonghong Song]

On 5/22/19 4:20 PM, Roman Gushchin wrote:
> Currently the lifetime of bpf programs attached to a cgroup is bound
> to the lifetime of the cgroup itself. It means that if a user
> forgets (or intentionally avoids) to detach a bpf program before
> removing the cgroup, it will stay attached up to the release of the
> cgroup. Since the cgroup can stay in the dying state (the state
> between being rmdir()'ed and being released) for a very long time, it
> leads to a waste of memory. Also, it blocks a possibility to implement
> the memcg-based memory accounting for bpf objects, because a circular
> reference dependency will occur. Charged memory pages are pinning the
> corresponding memory cgroup, and if the memory cgroup is pinning
> the attached bpf program, nothing will be ever released.
> 
> A dying cgroup can not contain any processes, so the only chance for
> an attached bpf program to be executed is a live socket associated
> with the cgroup. So in order to release all bpf data early, let's
> count associated sockets using a new percpu refcounter. On cgroup
> removal the counter is transitioned to the atomic mode, and as soon
> as it reaches 0, all bpf programs are detached.
> 
> The reference counter is not socket specific, and can be used for any
> other types of programs, which can be executed from a cgroup-bpf hook
> outside of the process context, had such a need arise in the future.
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Cc: jolsa@redhat.com

The logic looks sound to me. With one nit below,
Acked-by: Yonghong Song <yhs@fb.com>

> ---
>   include/linux/bpf-cgroup.h |  8 ++++++--
>   include/linux/cgroup.h     | 18 ++++++++++++++++++
>   kernel/bpf/cgroup.c        | 25 ++++++++++++++++++++++---
>   kernel/cgroup/cgroup.c     | 11 ++++++++---
>   4 files changed, 54 insertions(+), 8 deletions(-)
> 
[...]
> @@ -167,7 +178,12 @@ int cgroup_bpf_inherit(struct cgroup *cgrp)
>    */
>   #define	NR ARRAY_SIZE(cgrp->bpf.effective)
>   	struct bpf_prog_array __rcu *arrays[NR] = {};
> -	int i;
> +	int ret, i;
> +
> +	ret = percpu_ref_init(&cgrp->bpf.refcnt, cgroup_bpf_release, 0,
> +			      GFP_KERNEL);
> +	if (ret)
> +		return -ENOMEM;
Maybe return "ret" here instead of -ENOMEM. Currently, percpu_ref_init
only return error code is -ENOMEM. But in the future, it could
change?

>   
>   	for (i = 0; i < NR; i++)
>   		INIT_LIST_HEAD(&cgrp->bpf.progs[i]);
> @@ -183,6 +199,9 @@ int cgroup_bpf_inherit(struct cgroup *cgrp)
>   cleanup:
>   	for (i = 0; i < NR; i++)
>   		bpf_prog_array_free(arrays[i]);
> +
> +	percpu_ref_exit(&cgrp->bpf.refcnt);
> +
>   	return -ENOMEM;
>   }
>   
[...]

Re: [PATCH v2 bpf-next 2/4] selftests/bpf: convert test_cgrp2_attach2 example into kselftest

[Yonghong Song]

On 5/22/19 4:20 PM, Roman Gushchin wrote:
> Convert test_cgrp2_attach2 example into a proper test_cgroup_attach
> kselftest. It's better because we do run kselftest on a constant
> basis, so there are better chances to spot a potential regression.
> 
> Also make it slightly less verbose to conform kselftests output style.
> 
> Output example:
>    $ ./test_cgroup_attach
>    #override:PASS
>    #multi:PASS
>    test_cgroup_attach:PASS
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>
Ack except the nit below.
Acked-by: Yonghong Song <yhs@fb.com>
> ---
>   samples/bpf/Makefile                          |  2 -
>   tools/testing/selftests/bpf/Makefile          |  4 +-
>   .../selftests/bpf/test_cgroup_attach.c        | 48 ++++++++++++-------
>   3 files changed, 35 insertions(+), 19 deletions(-)
>   rename samples/bpf/test_cgrp2_attach2.c => tools/testing/selftests/bpf/test_cgroup_attach.c (92%)
> 
> diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
> index 4f0a1cdbfe7c..253e5a2856be 100644
> --- a/samples/bpf/Makefile
> +++ b/samples/bpf/Makefile
> @@ -26,7 +26,6 @@ hostprogs-y += map_perf_test
>   hostprogs-y += test_overhead
>   hostprogs-y += test_cgrp2_array_pin
>   hostprogs-y += test_cgrp2_attach
> -hostprogs-y += test_cgrp2_attach2
>   hostprogs-y += test_cgrp2_sock
>   hostprogs-y += test_cgrp2_sock2
>   hostprogs-y += xdp1
> @@ -81,7 +80,6 @@ map_perf_test-objs := bpf_load.o map_perf_test_user.o
>   test_overhead-objs := bpf_load.o test_overhead_user.o
>   test_cgrp2_array_pin-objs := test_cgrp2_array_pin.o
>   test_cgrp2_attach-objs := test_cgrp2_attach.o
> -test_cgrp2_attach2-objs := test_cgrp2_attach2.o $(CGROUP_HELPERS)
>   test_cgrp2_sock-objs := test_cgrp2_sock.o
>   test_cgrp2_sock2-objs := bpf_load.o test_cgrp2_sock2.o
>   xdp1-objs := xdp1_user.o
> diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
> index 66f2dca1dee1..e09f419f4d7e 100644
> --- a/tools/testing/selftests/bpf/Makefile
> +++ b/tools/testing/selftests/bpf/Makefile
> @@ -23,7 +23,8 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test
>   	test_align test_verifier_log test_dev_cgroup test_tcpbpf_user \
>   	test_sock test_btf test_sockmap test_lirc_mode2_user get_cgroup_id_user \
>   	test_socket_cookie test_cgroup_storage test_select_reuseport test_section_names \
> -	test_netcnt test_tcpnotify_user test_sock_fields test_sysctl
> +	test_netcnt test_tcpnotify_user test_sock_fields test_sysctl \
> +	test_cgroup_attach
>   
>   BPF_OBJ_FILES = $(patsubst %.c,%.o, $(notdir $(wildcard progs/*.c)))
>   TEST_GEN_FILES = $(BPF_OBJ_FILES)
> @@ -96,6 +97,7 @@ $(OUTPUT)/test_cgroup_storage: cgroup_helpers.c
>   $(OUTPUT)/test_netcnt: cgroup_helpers.c
>   $(OUTPUT)/test_sock_fields: cgroup_helpers.c
>   $(OUTPUT)/test_sysctl: cgroup_helpers.c
> +$(OUTPUT)/test_cgroup_attach: cgroup_helpers.c
>   
>   .PHONY: force
>   
> diff --git a/samples/bpf/test_cgrp2_attach2.c b/tools/testing/selftests/bpf/test_cgroup_attach.c
> similarity index 92%
> rename from samples/bpf/test_cgrp2_attach2.c
> rename to tools/testing/selftests/bpf/test_cgroup_attach.c
> index 0bb6507256b7..93d4fe295e7d 100644
> --- a/samples/bpf/test_cgrp2_attach2.c
> +++ b/tools/testing/selftests/bpf/test_cgroup_attach.c
> @@ -1,3 +1,5 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
>   /* eBPF example program:
>    *
>    * - Creates arraymap in kernel with 4 bytes keys and 8 byte values
> @@ -25,20 +27,27 @@
>   #include <sys/resource.h>
>   #include <sys/time.h>
>   #include <unistd.h>
> +#include <linux/filter.h>
>   
>   #include <linux/bpf.h>
>   #include <bpf/bpf.h>
>   
> -#include "bpf_insn.h"
> +#include "bpf_util.h"
>   #include "bpf_rlimit.h"
>   #include "cgroup_helpers.h"
>   
>   #define FOO		"/foo"
>   #define BAR		"/foo/bar/"
> -#define PING_CMD	"ping -c1 -w1 127.0.0.1 > /dev/null"
> +#define PING_CMD	"ping -q -c1 -w1 127.0.0.1 > /dev/null"
>   
>   char bpf_log_buf[BPF_LOG_BUF_SIZE];
>   
> +#ifdef DEBUG
> +#define debug(args...) printf(args)
> +#else
> +#define debug(args...)
> +#endif
> +
>   static int prog_load(int verdict)
>   {
>   	int ret;
> @@ -89,7 +98,7 @@ static int test_foo_bar(void)
>   		goto err;
>   	}
>   
> -	printf("Attached DROP prog. This ping in cgroup /foo should fail...\n");
> +	debug("Attached DROP prog. This ping in cgroup /foo should fail...\n");
>   	assert(system(PING_CMD) != 0);
>   
>   	/* Create cgroup /foo/bar, get fd, and join it */
> @@ -100,7 +109,7 @@ static int test_foo_bar(void)
>   	if (join_cgroup(BAR))
>   		goto err;
>   
> -	printf("Attached DROP prog. This ping in cgroup /foo/bar should fail...\n");
> +	debug("Attached DROP prog. This ping in cgroup /foo/bar should fail...\n");
>   	assert(system(PING_CMD) != 0);
>   
>   	if (bpf_prog_attach(allow_prog, bar, BPF_CGROUP_INET_EGRESS,
> @@ -109,7 +118,7 @@ static int test_foo_bar(void)
>   		goto err;
>   	}
>   
> -	printf("Attached PASS prog. This ping in cgroup /foo/bar should pass...\n");
> +	debug("Attached PASS prog. This ping in cgroup /foo/bar should pass...\n");
>   	assert(system(PING_CMD) == 0);
>   
>   	if (bpf_prog_detach(bar, BPF_CGROUP_INET_EGRESS)) {
> @@ -117,7 +126,7 @@ static int test_foo_bar(void)
>   		goto err;
>   	}
>   
> -	printf("Detached PASS from /foo/bar while DROP is attached to /foo.\n"
> +	debug("Detached PASS from /foo/bar while DROP is attached to /foo.\n"
>   	       "This ping in cgroup /foo/bar should fail...\n");
>   	assert(system(PING_CMD) != 0);
>   
> @@ -132,7 +141,7 @@ static int test_foo_bar(void)
>   		goto err;
>   	}
>   
> -	printf("Attached PASS from /foo/bar and detached DROP from /foo.\n"
> +	debug("Attached PASS from /foo/bar and detached DROP from /foo.\n"
>   	       "This ping in cgroup /foo/bar should pass...\n");
>   	assert(system(PING_CMD) == 0);
>   
> @@ -199,9 +208,9 @@ static int test_foo_bar(void)
>   	close(bar);
>   	cleanup_cgroup_environment();
>   	if (!rc)
> -		printf("### override:PASS\n");
> +		printf("#override:PASS\n");
>   	else
> -		printf("### override:FAIL\n");
> +		printf("#override:FAIL\n");
>   	return rc;
>   }
>   
> @@ -441,19 +450,26 @@ static int test_multiprog(void)
>   	close(cg5);
>   	cleanup_cgroup_environment();
>   	if (!rc)
> -		printf("### multi:PASS\n");
> +		printf("#multi:PASS\n");
>   	else
> -		printf("### multi:FAIL\n");
> +		printf("#multi:FAIL\n");
>   	return rc;
>   }
>   
>   int main(int argc, char **argv)

argc and argv are not used.

currently, the debug output is controlled by macro DEBUG.
you could enable debug output with command line option.
But I am okay with the current mechanism as well.

>   {
> -	int rc = 0;
> +	int (*tests[])(void) = {test_foo_bar, test_multiprog};
> +	int errors = 0;
> +	int i;
>   
> -	rc = test_foo_bar();
> -	if (rc)
> -		return rc;
> +	for (i = 0; i < ARRAY_SIZE(tests); i++)
> +		if (tests[i]())
> +			errors++;
> +
> +	if (errors)
> +		printf("test_cgroup_attach:FAIL\n");
> +	else
> +		printf("test_cgroup_attach:PASS\n");
>   
> -	return test_multiprog();
> +	return errors ? EXIT_FAILURE : EXIT_SUCCESS;
>   }
> 

Re: [PATCH v2 bpf-next 3/4] selftests/bpf: enable all available cgroup v2 controllers

[Yonghong Song]

On 5/22/19 4:20 PM, Roman Gushchin wrote:
> Enable all available cgroup v2 controllers when setting up
> the environment for the bpf kselftests. It's required to properly test
> the bpf prog auto-detach feature. Also it will generally increase
> the code coverage.
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>
Looks good to me. Ack with one nit below.
Acked-by: Yonghong Song <yhs@fb.com>

> ---
>   tools/testing/selftests/bpf/cgroup_helpers.c | 57 ++++++++++++++++++++
>   1 file changed, 57 insertions(+)
> 
> diff --git a/tools/testing/selftests/bpf/cgroup_helpers.c b/tools/testing/selftests/bpf/cgroup_helpers.c
> index 6692a40a6979..4efe57c171cd 100644
> --- a/tools/testing/selftests/bpf/cgroup_helpers.c
> +++ b/tools/testing/selftests/bpf/cgroup_helpers.c
> @@ -33,6 +33,60 @@
>   	snprintf(buf, sizeof(buf), "%s%s%s", CGROUP_MOUNT_PATH, \
>   		 CGROUP_WORK_DIR, path)
>   
> +/**
> + * enable_all_controllers() - Enable all available cgroup v2 controllers
> + *
> + * Enable all available cgroup v2 controllers in order to increase
> + * the code coverage.
> + *
> + * If successful, 0 is returned.
> + */
> +int enable_all_controllers(char *cgroup_path)
> +{
> +	char path[PATH_MAX + 1];
> +	char buf[PATH_MAX];
> +	char *c, *c2;
> +	int fd, cfd;
> +	size_t len;
> +
> +	snprintf(path, sizeof(path), "%s/cgroup.controllers", cgroup_path);
> +	fd = open(path, O_RDONLY);
> +	if (fd < 0) {
> +		log_err("Opening cgroup.controllers: %s", path);
> +		return -1;
It looks like either -1 or 1 could be returned to indicate an error
in this file. Maybe, at least for the consistency of this file,
always returning 1 is preferred as setup_cgroup_environment()
has the following comments:
  * This function will print an error to stderr and return 1 if it is unable
  * to setup the cgroup environment. If setup is successful, 0 is returned.

> +	}
> +
> +	len = read(fd, buf, sizeof(buf) - 1);
> +	if (len < 0) {
> +		close(fd);
> +		log_err("Reading cgroup.controllers: %s", path);
> +		return -1;
> +	}
> +	buf[len] = 0;
> +	close(fd);
> +
> +	/* No controllers available? We're probably on cgroup v1. */
> +	if (len == 0)
> +		return 0;
> +
> +	snprintf(path, sizeof(path), "%s/cgroup.subtree_control", cgroup_path);
> +	cfd = open(path, O_RDWR);
> +	if (cfd < 0) {
> +		log_err("Opening cgroup.subtree_control: %s", path);
> +		return -1;
> +	}
> +
> +	for (c = strtok_r(buf, " ", &c2); c; c = strtok_r(NULL, " ", &c2)) {
> +		if (dprintf(cfd, "+%s\n", c) <= 0) {
> +			log_err("Enabling controller %s: %s", c, path);
> +			close(cfd);
> +			return -1;
> +		}
> +	}
> +	close(cfd);
> +	return 0;
> +}
> +
>   /**
>    * setup_cgroup_environment() - Setup the cgroup environment
>    *
> @@ -71,6 +125,9 @@ int setup_cgroup_environment(void)
>   		return 1;
>   	}
>   
> +	if (enable_all_controllers(cgroup_workdir))
> +		return 1;
> +
>   	return 0;
>   }
>   
> 

Re: [PATCH v2 bpf-next 4/4] selftests/bpf: add auto-detach test

[Yonghong Song]

On 5/22/19 4:20 PM, Roman Gushchin wrote:
> Add a kselftest to cover bpf auto-detachment functionality.
> The test creates a cgroup, associates some resources with it,
> attaches a couple of bpf programs and deletes the cgroup.
> 
> Then it checks that bpf programs are going away in 5 seconds.
> 
> Expected output:
>    $ ./test_cgroup_attach
>    #override:PASS
>    #multi:PASS
>    #autodetach:PASS
>    test_cgroup_attach:PASS
> 
> On a kernel without auto-detaching:
>    $ ./test_cgroup_attach
>    #override:PASS
>    #multi:PASS
>    #autodetach:FAIL
>    test_cgroup_attach:FAIL

I ran this problem without both old and new kernels and
both get all PASSes. My testing environment is a VM.
Could you specify how to trigger the above failure?

> 
> Signed-off-by: Roman Gushchin <guro@fb.com>
> ---
>   .../selftests/bpf/test_cgroup_attach.c        | 99 ++++++++++++++++++-
>   1 file changed, 98 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/bpf/test_cgroup_attach.c b/tools/testing/selftests/bpf/test_cgroup_attach.c
> index 93d4fe295e7d..bc5bd0f1728e 100644
> --- a/tools/testing/selftests/bpf/test_cgroup_attach.c
> +++ b/tools/testing/selftests/bpf/test_cgroup_attach.c
> @@ -456,9 +456,106 @@ static int test_multiprog(void)
>   	return rc;
>   }
>   
> +static int test_autodetach(void)
> +{
> +	__u32 prog_cnt = 4, attach_flags;
> +	int allow_prog[2] = {0};
> +	__u32 prog_ids[2] = {0};
> +	int cg = 0, i, rc = -1;
> +	void *ptr = NULL;
> +	int attempts;
> +
> +
> +	for (i = 0; i < ARRAY_SIZE(allow_prog); i++) {
> +		allow_prog[i] = prog_load_cnt(1, 1 << i);
> +		if (!allow_prog[i])
> +			goto err;
> +	}
> +
> +	if (setup_cgroup_environment())
> +		goto err;
> +
> +	/* create a cgroup, attach two programs and remember their ids */
> +	cg = create_and_get_cgroup("/cg_autodetach");
> +	if (cg < 0)
> +		goto err;
> +
> +	if (join_cgroup("/cg_autodetach"))
> +		goto err;
> +
> +	for (i = 0; i < ARRAY_SIZE(allow_prog); i++) {
> +		if (bpf_prog_attach(allow_prog[i], cg, BPF_CGROUP_INET_EGRESS,
> +				    BPF_F_ALLOW_MULTI)) {
> +			log_err("Attaching prog[%d] to cg:egress", i);
> +			goto err;
> +		}
> +	}
> +
> +	/* make sure that programs are attached and run some traffic */
> +	assert(bpf_prog_query(cg, BPF_CGROUP_INET_EGRESS, 0, &attach_flags,
> +			      prog_ids, &prog_cnt) == 0);
> +	assert(system(PING_CMD) == 0);
> +
> +	/* allocate some memory (4Mb) to pin the original cgroup */
> +	ptr = malloc(4 * (1 << 20));
> +	if (!ptr)
> +		goto err;
> +
> +	/* close programs and cgroup fd */
> +	for (i = 0; i < ARRAY_SIZE(allow_prog); i++) {
> +		close(allow_prog[i]);
> +		allow_prog[i] = 0;
> +	}
> +
> +	close(cg);
> +	cg = 0;
> +
> +	/* leave the cgroup and remove it. don't detach programs */
> +	cleanup_cgroup_environment();
> +
> +	/* wait for the asynchronous auto-detachment.
> +	 * wait for no more than 5 sec and give up.
> +	 */
> +	for (i = 0; i < ARRAY_SIZE(prog_ids); i++) {
> +		for (attempts = 5; attempts >= 0; attempts--) {
> +			int fd = bpf_prog_get_fd_by_id(prog_ids[i]);
> +
> +			if (fd < 0)
> +				break;
> +
> +			/* don't leave the fd open */
> +			close(fd);
> +
> +			if (!attempts)
> +				goto err;
> +
> +			sleep(1);
> +		}
> +	}
> +
> +	rc = 0;
> +err:
> +	for (i = 0; i < ARRAY_SIZE(allow_prog); i++)
> +		if (allow_prog[i] > 0)
> +			close(allow_prog[i]);
> +	if (cg)
> +		close(cg);
> +	free(ptr);
> +	cleanup_cgroup_environment();
> +	if (!rc)
> +		printf("#autodetach:PASS\n");
> +	else
> +		printf("#autodetach:FAIL\n");
> +	return rc;
> +}
> +
>   int main(int argc, char **argv)
>   {
> -	int (*tests[])(void) = {test_foo_bar, test_multiprog};
> +	int (*tests[])(void) = {
> +		test_foo_bar,
> +		test_multiprog,
> +		test_autodetach,
> +	};
>   	int errors = 0;
>   	int i;
>   
> 

Re: [PATCH v2 bpf-next 4/4] selftests/bpf: add auto-detach test

[Yonghong Song]

On 5/22/19 4:20 PM, Roman Gushchin wrote:
> Add a kselftest to cover bpf auto-detachment functionality.
> The test creates a cgroup, associates some resources with it,
> attaches a couple of bpf programs and deletes the cgroup.
> 
> Then it checks that bpf programs are going away in 5 seconds.
> 
> Expected output:
>    $ ./test_cgroup_attach
>    #override:PASS
>    #multi:PASS
>    #autodetach:PASS
>    test_cgroup_attach:PASS
> 
> On a kernel without auto-detaching:
>    $ ./test_cgroup_attach
>    #override:PASS
>    #multi:PASS
>    #autodetach:FAIL
>    test_cgroup_attach:FAIL
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>
> ---
>   .../selftests/bpf/test_cgroup_attach.c        | 99 ++++++++++++++++++-
>   1 file changed, 98 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/bpf/test_cgroup_attach.c b/tools/testing/selftests/bpf/test_cgroup_attach.c
> index 93d4fe295e7d..bc5bd0f1728e 100644
> --- a/tools/testing/selftests/bpf/test_cgroup_attach.c
> +++ b/tools/testing/selftests/bpf/test_cgroup_attach.c
> @@ -456,9 +456,106 @@ static int test_multiprog(void)
>   	return rc;
>   }
>   
> +static int test_autodetach(void)
> +{
> +	__u32 prog_cnt = 4, attach_flags;
> +	int allow_prog[2] = {0};
> +	__u32 prog_ids[2] = {0};
> +	int cg = 0, i, rc = -1;
> +	void *ptr = NULL;
> +	int attempts;
> +
> +
Also extra line here.

> +	for (i = 0; i < ARRAY_SIZE(allow_prog); i++) {
> +		allow_prog[i] = prog_load_cnt(1, 1 << i);
> +		if (!allow_prog[i])
> +			goto err;
> +	}
> +
> +	if (setup_cgroup_environment())
> +		goto err;
> +
> +	/* create a cgroup, attach two programs and remember their ids */
> +	cg = create_and_get_cgroup("/cg_autodetach");
[...]
> +
>   int main(int argc, char **argv)
>   {
> -	int (*tests[])(void) = {test_foo_bar, test_multiprog};
> +	int (*tests[])(void) = {
> +		test_foo_bar,
> +		test_multiprog,
> +		test_autodetach,
> +	};
>   	int errors = 0;
>   	int i;
>   
> 

Re: [PATCH v2 bpf-next 4/4] selftests/bpf: add auto-detach test

[Roman Gushchin]

On Wed, May 22, 2019 at 10:47:24PM -0700, Yonghong Song wrote:
> 
> 
> On 5/22/19 4:20 PM, Roman Gushchin wrote:
> > Add a kselftest to cover bpf auto-detachment functionality.
> > The test creates a cgroup, associates some resources with it,
> > attaches a couple of bpf programs and deletes the cgroup.
> > 
> > Then it checks that bpf programs are going away in 5 seconds.
> > 
> > Expected output:
> >    $ ./test_cgroup_attach
> >    #override:PASS
> >    #multi:PASS
> >    #autodetach:PASS
> >    test_cgroup_attach:PASS
> > 
> > On a kernel without auto-detaching:
> >    $ ./test_cgroup_attach
> >    #override:PASS
> >    #multi:PASS
> >    #autodetach:FAIL
> >    test_cgroup_attach:FAIL
> 
> I ran this problem without both old and new kernels and
> both get all PASSes. My testing environment is a VM.
> Could you specify how to trigger the above failure?

Most likely you're running cgroup v1, so the memory controller
is not enabled on unified hierarchy. You need to pass
"cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1"
as boot time options to run fully on cgroup v2.

But generally speaking, the lifecycle of a dying cgroup is
completely implementation-defined. No guarantees are provided.
So false positives are fine here, and shouldn't be considered as
something bad.

At the end all we want it to detach programs in a reasonable time
after rmdir.

Btw, thank you for the careful review of the patchset. I'll
address your comments, add acks and will send out v3.

Thanks!

Re: [PATCH v2 bpf-next 4/4] selftests/bpf: add auto-detach test

[Yonghong Song]

On 5/23/19 10:58 AM, Roman Gushchin wrote:
> On Wed, May 22, 2019 at 10:47:24PM -0700, Yonghong Song wrote:
>>
>>
>> On 5/22/19 4:20 PM, Roman Gushchin wrote:
>>> Add a kselftest to cover bpf auto-detachment functionality.
>>> The test creates a cgroup, associates some resources with it,
>>> attaches a couple of bpf programs and deletes the cgroup.
>>>
>>> Then it checks that bpf programs are going away in 5 seconds.
>>>
>>> Expected output:
>>>     $ ./test_cgroup_attach
>>>     #override:PASS
>>>     #multi:PASS
>>>     #autodetach:PASS
>>>     test_cgroup_attach:PASS
>>>
>>> On a kernel without auto-detaching:
>>>     $ ./test_cgroup_attach
>>>     #override:PASS
>>>     #multi:PASS
>>>     #autodetach:FAIL
>>>     test_cgroup_attach:FAIL
>>
>> I ran this problem without both old and new kernels and
>> both get all PASSes. My testing environment is a VM.
>> Could you specify how to trigger the above failure?
> 
> Most likely you're running cgroup v1, so the memory controller
> is not enabled on unified hierarchy. You need to pass
> "cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1"
> as boot time options to run fully on cgroup v2.

I tested on a cgroup v2 machine and it indeed failed without
the core patch. Thanks!

> 
> But generally speaking, the lifecycle of a dying cgroup is
> completely implementation-defined. No guarantees are provided.
> So false positives are fine here, and shouldn't be considered as
> something bad.
> 
> At the end all we want it to detach programs in a reasonable time
> after rmdir.
> 
> Btw, thank you for the careful review of the patchset. I'll
> address your comments, add acks and will send out v3.
> 
> Thanks!
>