* [PATCH bpf-next v2 1/8] bpf: btf: refactor btf_int_bits_seq_show()
2018-12-14 23:34 [PATCH bpf-next v2 0/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
@ 2018-12-14 23:34 ` Yonghong Song
2018-12-15 16:44 ` Martin Lau
2018-12-14 23:34 ` [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
` (6 subsequent siblings)
7 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-14 23:34 UTC (permalink / raw)
To: netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team,
Martin KaFai Lau
Refactor function btf_int_bits_seq_show() by creating
function btf_bitfield_seq_show() which has no dependence
on btf and btf_type. The function btf_bitfield_seq_show()
will be in later patch to directly dump bitfield member values.
Signed-off-by: Yonghong Song <yhs@fb.com>
---
kernel/bpf/btf.c | 35 +++++++++++++++++++++--------------
1 file changed, 21 insertions(+), 14 deletions(-)
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 8fa0bf1c33fd..72caa799e82f 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -1068,26 +1068,16 @@ static void btf_int_log(struct btf_verifier_env *env,
btf_int_encoding_str(BTF_INT_ENCODING(int_data)));
}
-static void btf_int_bits_seq_show(const struct btf *btf,
- const struct btf_type *t,
- void *data, u8 bits_offset,
- struct seq_file *m)
+static void btf_bitfield_seq_show(void *data, u8 bits_offset,
+ u8 nr_bits, struct seq_file *m)
{
u16 left_shift_bits, right_shift_bits;
- u32 int_data = btf_type_int(t);
- u8 nr_bits = BTF_INT_BITS(int_data);
- u8 total_bits_offset;
u8 nr_copy_bytes;
u8 nr_copy_bits;
u64 print_num;
- /*
- * bits_offset is at most 7.
- * BTF_INT_OFFSET() cannot exceed 64 bits.
- */
- total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
- data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
- bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
+ data += BITS_ROUNDDOWN_BYTES(bits_offset);
+ bits_offset = BITS_PER_BYTE_MASKED(bits_offset);
nr_copy_bits = nr_bits + bits_offset;
nr_copy_bytes = BITS_ROUNDUP_BYTES(nr_copy_bits);
@@ -1107,6 +1097,23 @@ static void btf_int_bits_seq_show(const struct btf *btf,
seq_printf(m, "0x%llx", print_num);
}
+static void btf_int_bits_seq_show(const struct btf *btf,
+ const struct btf_type *t,
+ void *data, u8 bits_offset,
+ struct seq_file *m)
+{
+ u32 int_data = btf_type_int(t);
+ u8 nr_bits = BTF_INT_BITS(int_data);
+ u8 total_bits_offset;
+
+ /*
+ * bits_offset is at most 7.
+ * BTF_INT_OFFSET() cannot exceed 64 bits.
+ */
+ total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
+ btf_bitfield_seq_show(data, total_bits_offset, nr_bits, m);
+}
+
static void btf_int_seq_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct seq_file *m)
--
2.17.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-14 23:34 [PATCH bpf-next v2 0/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
2018-12-14 23:34 ` [PATCH bpf-next v2 1/8] bpf: btf: refactor btf_int_bits_seq_show() Yonghong Song
@ 2018-12-14 23:34 ` Yonghong Song
2018-12-15 16:37 ` Martin Lau
2018-12-15 22:37 ` Daniel Borkmann
2018-12-14 23:34 ` [PATCH bpf-next v2 3/8] bpf: enable cgroup local storage map pretty print " Yonghong Song
` (5 subsequent siblings)
7 siblings, 2 replies; 30+ messages in thread
From: Yonghong Song @ 2018-12-14 23:34 UTC (permalink / raw)
To: netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team,
Martin KaFai Lau
This patch fixed two issues with BTF. One is related to
struct/union bitfield encoding and the other is related to
forward type.
Issue #1 and solution:
======================
Current btf encoding of bitfield follows what pahole generates.
For each bitfield, pahole will duplicate the type chain and
put the bitfield size at the final int or enum type.
Since the BTF enum type cannot encode bit size,
pahole workarounds the issue by generating
an int type whenever the enum bit size is not 32.
For example,
-bash-4.4$ cat t.c
typedef int ___int;
enum A { A1, A2, A3 };
struct t {
int a[5];
___int b:4;
volatile enum A c:4;
} g;
-bash-4.4$ gcc -c -O2 -g t.c
The current kernel supports the following BTF encoding:
$ pahole -JV t.o
[1] TYPEDEF ___int type_id=2
[2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
[3] ENUM A size=4 vlen=3
A1 val=0
A2 val=1
A3 val=2
[4] STRUCT t size=24 vlen=3
a type_id=5 bits_offset=0
b type_id=9 bits_offset=160
c type_id=11 bits_offset=164
[5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
[6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
[7] VOLATILE (anon) type_id=3
[8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
[9] TYPEDEF ___int type_id=8
[10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
[11] VOLATILE (anon) type_id=10
Two issues are in the above:
. by changing enum type to int, we lost the original
type information and this will not be ideal later
when we try to convert BTF to a header file.
. the type duplication for bitfields will cause
BTF bloat. Duplicated types cannot be deduplicated
later if the bitfield size is different.
To fix this issue, this patch implemented a compatible
change for BTF struct type encoding:
. the bit 31 of struct_type->info, previously reserved,
now is used to indicate whether bitfield_size is
encoded in btf_member or not.
. if bit 31 of struct_type->info is set,
btf_member->offset will encode like:
bit 0 - 23: bit offset
bit 24 - 31: bitfield size
if bit 31 is not set, the old behavior is preserved:
bit 0 - 31: bit offset
So if the struct contains a bit field, the maximum bit offset
will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
bitfield size will be 256 which is enough for today as maximum
bitfield in compiler can be 128 where int128 type is supported.
This kernel patch intends to support the new BTF encoding:
$ pahole -JV t.o
[1] TYPEDEF ___int type_id=2
[2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
[3] ENUM A size=4 vlen=3
A1 val=0
A2 val=1
A3 val=2
[4] STRUCT t kind_flag=1 size=24 vlen=3
a type_id=5 bitfield_size=0 bits_offset=0
b type_id=1 bitfield_size=4 bits_offset=160
c type_id=7 bitfield_size=4 bits_offset=164
[5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
[6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
[7] VOLATILE (anon) type_id=3
Issue #2 and solution:
======================
Current forward type in BTF does not specify whether the original
type is struct or union. This will not work for type pretty print
and BTF-to-header-file conversion as struct/union must be specified.
$ cat tt.c
struct t;
union u;
int foo(struct t *t, union u *u) { return 0; }
$ gcc -c -g -O2 tt.c
$ pahole -JV tt.o
[1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
[2] FWD t type_id=0
[3] PTR (anon) type_id=2
[4] FWD u type_id=0
[5] PTR (anon) type_id=4
To fix this issue, similar to issue #1, type->info bit 31
is used. If the bit is set, it is union type. Otherwise, it is
a struct type.
$ pahole -JV tt.o
[1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
[2] FWD t kind_flag=0 type_id=0
[3] PTR (anon) kind_flag=0 type_id=2
[4] FWD u kind_flag=1 type_id=0
[5] PTR (anon) kind_flag=0 type_id=4
Pahole/LLVM change:
===================
The new kind_flag functionality has been implemented in pahole
and llvm:
https://github.com/yonghong-song/pahole/tree/bitfield
https://github.com/yonghong-song/llvm/tree/bitfield
Note that pahole hasn't implemented func/func_proto kind
and .BTF.ext. So to print function signature with bpftool,
the llvm compiler should be used.
Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Yonghong Song <yhs@fb.com>
---
include/uapi/linux/btf.h | 15 ++-
kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
2 files changed, 267 insertions(+), 22 deletions(-)
diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
index 14f66948fc95..34aba40ed926 100644
--- a/include/uapi/linux/btf.h
+++ b/include/uapi/linux/btf.h
@@ -34,7 +34,9 @@ struct btf_type {
* bits 0-15: vlen (e.g. # of struct's members)
* bits 16-23: unused
* bits 24-27: kind (e.g. int, ptr, array...etc)
- * bits 28-31: unused
+ * bits 28-30: unused
+ * bit 31: kind_flag, currently used by
+ * struct, union and fwd
*/
__u32 info;
/* "size" is used by INT, ENUM, STRUCT and UNION.
@@ -52,6 +54,7 @@ struct btf_type {
#define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
#define BTF_INFO_VLEN(info) ((info) & 0xffff)
+#define BTF_INFO_KFLAG(info) ((info) >> 31)
#define BTF_KIND_UNKN 0 /* Unknown */
#define BTF_KIND_INT 1 /* Integer */
@@ -110,9 +113,17 @@ struct btf_array {
struct btf_member {
__u32 name_off;
__u32 type;
- __u32 offset; /* offset in bits */
+ __u32 offset; /* [bitfield_size and] offset in bits */
};
+/* If the type info kind_flag set, the btf_member.offset
+ * contains both member bit offset and bitfield size, and
+ * bitfield size will set for struct/union bitfield members.
+ * Otherwise, it contains only bit offset.
+ */
+#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
+#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
+
/* BTF_KIND_FUNC_PROTO is followed by multiple "struct btf_param".
* The exact number of btf_param is stored in the vlen (of the
* info in "struct btf_type").
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 72caa799e82f..ec3f80d3bef6 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -164,7 +164,7 @@
#define BITS_ROUNDUP_BYTES(bits) \
(BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
-#define BTF_INFO_MASK 0x0f00ffff
+#define BTF_INFO_MASK 0x8f00ffff
#define BTF_INT_MASK 0x0fffffff
#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
@@ -274,6 +274,10 @@ struct btf_kind_operations {
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type);
+ int (*check_kflag_member)(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type);
void (*log_details)(struct btf_verifier_env *env,
const struct btf_type *t);
void (*seq_show)(const struct btf *btf, const struct btf_type *t,
@@ -419,6 +423,25 @@ static u16 btf_type_vlen(const struct btf_type *t)
return BTF_INFO_VLEN(t->info);
}
+static bool btf_type_kflag(const struct btf_type *t)
+{
+ return BTF_INFO_KFLAG(t->info);
+}
+
+static u32 btf_member_bit_offset(const struct btf_type *struct_type,
+ const struct btf_member *member)
+{
+ return btf_type_kflag(struct_type) ? BTF_MEMBER_BIT_OFFSET(member->offset)
+ : member->offset;
+}
+
+static u32 btf_member_bitfield_size(const struct btf_type *struct_type,
+ const struct btf_member *member)
+{
+ return btf_type_kflag(struct_type) ? BTF_MEMBER_BITFIELD_SIZE(member->offset)
+ : 0;
+}
+
static u32 btf_type_int(const struct btf_type *t)
{
return *(u32 *)(t + 1);
@@ -627,9 +650,17 @@ static void btf_verifier_log_member(struct btf_verifier_env *env,
if (env->phase != CHECK_META)
btf_verifier_log_type(env, struct_type, NULL);
- __btf_verifier_log(log, "\t%s type_id=%u bits_offset=%u",
- __btf_name_by_offset(btf, member->name_off),
- member->type, member->offset);
+ if (btf_type_kflag(struct_type))
+ __btf_verifier_log(log,
+ "\t%s type_id=%u bitfield_size=%u bits_offset=%u",
+ __btf_name_by_offset(btf, member->name_off),
+ member->type,
+ BTF_MEMBER_BITFIELD_SIZE(member->offset),
+ BTF_MEMBER_BIT_OFFSET(member->offset));
+ else
+ __btf_verifier_log(log, "\t%s type_id=%u bits_offset=%u",
+ __btf_name_by_offset(btf, member->name_off),
+ member->type, member->offset);
if (fmt && *fmt) {
__btf_verifier_log(log, " ");
@@ -945,6 +976,38 @@ static int btf_df_check_member(struct btf_verifier_env *env,
return -EINVAL;
}
+static int btf_df_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ btf_verifier_log_basic(env, struct_type,
+ "Unsupported check_kflag_member");
+ return -EINVAL;
+}
+
+/* Used for ptr, array and struct/union type members.
+ * int, enum and modifier types have their specific callback functions.
+ */
+static int btf_generic_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ if (BTF_MEMBER_BITFIELD_SIZE(member->offset)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ /* bitfield size is 0, so member->offset represents bit offset only.
+ * It is safe to call non kflag check_member variants.
+ */
+ return btf_type_ops(member_type)->check_member(env, struct_type,
+ member,
+ member_type);
+}
+
static int btf_df_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
@@ -997,6 +1060,62 @@ static int btf_int_check_member(struct btf_verifier_env *env,
return 0;
}
+static int btf_int_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off, nr_bits, nr_int_data_bits, bytes_offset;
+ u32 int_data = btf_type_int(member_type);
+ u32 struct_size = struct_type->size;
+ u32 nr_copy_bits;
+
+ /* a regular int type is required for the kflag int member */
+ if (!btf_type_int_is_regular(member_type)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member base type");
+ return -EINVAL;
+ }
+
+ /* check sanity of bitfield size */
+ nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
+ struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
+ nr_int_data_bits = BTF_INT_BITS(int_data);
+ if (!nr_bits) {
+ /* Not a bitfield member, member offset must be at byte
+ * boundary.
+ */
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member offset");
+ return -EINVAL;
+ }
+
+ nr_bits = nr_int_data_bits;
+ } else if (nr_bits > nr_int_data_bits) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ nr_copy_bits = nr_bits + BITS_PER_BYTE_MASKED(struct_bits_off);
+ if (nr_copy_bits > BITS_PER_U64) {
+ btf_verifier_log_member(env, struct_type, member,
+ "nr_copy_bits exceeds 64");
+ return -EINVAL;
+ }
+
+ if (struct_size < bytes_offset ||
+ struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static s32 btf_int_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
@@ -1016,6 +1135,11 @@ static s32 btf_int_check_meta(struct btf_verifier_env *env,
return -EINVAL;
}
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
int_data = btf_type_int(t);
if (int_data & ~BTF_INT_MASK) {
btf_verifier_log_basic(env, t, "Invalid int_data:%x",
@@ -1097,6 +1221,7 @@ static void btf_bitfield_seq_show(void *data, u8 bits_offset,
seq_printf(m, "0x%llx", print_num);
}
+
static void btf_int_bits_seq_show(const struct btf *btf,
const struct btf_type *t,
void *data, u8 bits_offset,
@@ -1163,6 +1288,7 @@ static const struct btf_kind_operations int_ops = {
.check_meta = btf_int_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_int_check_member,
+ .check_kflag_member = btf_int_check_kflag_member,
.log_details = btf_int_log,
.seq_show = btf_int_seq_show,
};
@@ -1192,6 +1318,31 @@ static int btf_modifier_check_member(struct btf_verifier_env *env,
resolved_type);
}
+static int btf_modifier_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id = member->type;
+ struct btf_member resolved_member;
+ struct btf *btf = env->btf;
+
+ resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
+ if (!resolved_type) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member");
+ return -EINVAL;
+ }
+
+ resolved_member = *member;
+ resolved_member.type = resolved_type_id;
+
+ return btf_type_ops(resolved_type)->check_kflag_member(env, struct_type,
+ &resolved_member,
+ resolved_type);
+}
+
static int btf_ptr_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
@@ -1227,6 +1378,11 @@ static int btf_ref_type_check_meta(struct btf_verifier_env *env,
return -EINVAL;
}
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
if (!BTF_TYPE_ID_VALID(t->type)) {
btf_verifier_log_type(env, t, "Invalid type_id");
return -EINVAL;
@@ -1380,6 +1536,7 @@ static struct btf_kind_operations modifier_ops = {
.check_meta = btf_ref_type_check_meta,
.resolve = btf_modifier_resolve,
.check_member = btf_modifier_check_member,
+ .check_kflag_member = btf_modifier_check_kflag_member,
.log_details = btf_ref_type_log,
.seq_show = btf_modifier_seq_show,
};
@@ -1388,6 +1545,7 @@ static struct btf_kind_operations ptr_ops = {
.check_meta = btf_ref_type_check_meta,
.resolve = btf_ptr_resolve,
.check_member = btf_ptr_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
.log_details = btf_ref_type_log,
.seq_show = btf_ptr_seq_show,
};
@@ -1422,6 +1580,7 @@ static struct btf_kind_operations fwd_ops = {
.check_meta = btf_fwd_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_ref_type_log,
.seq_show = btf_df_seq_show,
};
@@ -1480,6 +1639,11 @@ static s32 btf_array_check_meta(struct btf_verifier_env *env,
return -EINVAL;
}
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
if (t->size) {
btf_verifier_log_type(env, t, "size != 0");
return -EINVAL;
@@ -1603,6 +1767,7 @@ static struct btf_kind_operations array_ops = {
.check_meta = btf_array_check_meta,
.resolve = btf_array_resolve,
.check_member = btf_array_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
.log_details = btf_array_log,
.seq_show = btf_array_seq_show,
};
@@ -1641,6 +1806,7 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env,
u32 meta_needed, last_offset;
struct btf *btf = env->btf;
u32 struct_size = t->size;
+ u32 offset;
u16 i;
meta_needed = btf_type_vlen(t) * sizeof(*member);
@@ -1682,7 +1848,8 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env,
return -EINVAL;
}
- if (is_union && member->offset) {
+ offset = btf_member_bit_offset(t, member);
+ if (is_union && offset) {
btf_verifier_log_member(env, t, member,
"Invalid member bits_offset");
return -EINVAL;
@@ -1692,20 +1859,20 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env,
* ">" instead of ">=" because the last member could be
* "char a[0];"
*/
- if (last_offset > member->offset) {
+ if (last_offset > offset) {
btf_verifier_log_member(env, t, member,
"Invalid member bits_offset");
return -EINVAL;
}
- if (BITS_ROUNDUP_BYTES(member->offset) > struct_size) {
+ if (BITS_ROUNDUP_BYTES(offset) > struct_size) {
btf_verifier_log_member(env, t, member,
"Member bits_offset exceeds its struct size");
return -EINVAL;
}
btf_verifier_log_member(env, t, member, NULL);
- last_offset = member->offset;
+ last_offset = offset;
}
return meta_needed;
@@ -1735,9 +1902,14 @@ static int btf_struct_resolve(struct btf_verifier_env *env,
last_member_type = btf_type_by_id(env->btf,
last_member_type_id);
- err = btf_type_ops(last_member_type)->check_member(env, v->t,
- last_member,
- last_member_type);
+ if (btf_type_kflag(v->t))
+ err = btf_type_ops(last_member_type)->check_kflag_member(env, v->t,
+ last_member,
+ last_member_type);
+ else
+ err = btf_type_ops(last_member_type)->check_member(env, v->t,
+ last_member,
+ last_member_type);
if (err)
return err;
}
@@ -1759,9 +1931,14 @@ static int btf_struct_resolve(struct btf_verifier_env *env,
return env_stack_push(env, member_type, member_type_id);
}
- err = btf_type_ops(member_type)->check_member(env, v->t,
- member,
- member_type);
+ if (btf_type_kflag(v->t))
+ err = btf_type_ops(member_type)->check_kflag_member(env, v->t,
+ member,
+ member_type);
+ else
+ err = btf_type_ops(member_type)->check_member(env, v->t,
+ member,
+ member_type);
if (err)
return err;
}
@@ -1789,17 +1966,26 @@ static void btf_struct_seq_show(const struct btf *btf, const struct btf_type *t,
for_each_member(i, t, member) {
const struct btf_type *member_type = btf_type_by_id(btf,
member->type);
- u32 member_offset = member->offset;
- u32 bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
- u8 bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
const struct btf_kind_operations *ops;
+ u32 member_offset, bitfield_size;
+ u32 bytes_offset;
+ u8 bits8_offset;
if (i)
seq_puts(m, seq);
- ops = btf_type_ops(member_type);
- ops->seq_show(btf, member_type, member->type,
- data + bytes_offset, bits8_offset, m);
+ member_offset = btf_member_bit_offset(t, member);
+ bitfield_size = btf_member_bitfield_size(t, member);
+ if (bitfield_size) {
+ btf_bitfield_seq_show(data, member_offset,
+ bitfield_size, m);
+ } else {
+ bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
+ bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
+ ops = btf_type_ops(member_type);
+ ops->seq_show(btf, member_type, member->type,
+ data + bytes_offset, bits8_offset, m);
+ }
}
seq_puts(m, "}");
}
@@ -1808,6 +1994,7 @@ static struct btf_kind_operations struct_ops = {
.check_meta = btf_struct_check_meta,
.resolve = btf_struct_resolve,
.check_member = btf_struct_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
.log_details = btf_struct_log,
.seq_show = btf_struct_seq_show,
};
@@ -1837,6 +2024,35 @@ static int btf_enum_check_member(struct btf_verifier_env *env,
return 0;
}
+static int btf_enum_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off, nr_bits, bytes_end, struct_size;
+ u32 int_bitsize = sizeof(int) * BITS_PER_BYTE;
+
+ struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
+ nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
+ if (!nr_bits) {
+ nr_bits = int_bitsize;
+ } else if (nr_bits > int_bitsize) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ struct_size = struct_type->size;
+ bytes_end = BITS_ROUNDUP_BYTES(struct_bits_off + nr_bits);
+ if (struct_size < bytes_end) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static s32 btf_enum_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
@@ -1856,6 +2072,11 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env,
return -EINVAL;
}
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
if (t->size != sizeof(int)) {
btf_verifier_log_type(env, t, "Expected size:%zu",
sizeof(int));
@@ -1924,6 +2145,7 @@ static struct btf_kind_operations enum_ops = {
.check_meta = btf_enum_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_enum_check_member,
+ .check_kflag_member = btf_enum_check_kflag_member,
.log_details = btf_enum_log,
.seq_show = btf_enum_seq_show,
};
@@ -1946,6 +2168,11 @@ static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
return -EINVAL;
}
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
btf_verifier_log_type(env, t, NULL);
return meta_needed;
@@ -2005,6 +2232,7 @@ static struct btf_kind_operations func_proto_ops = {
* Hence, there is no btf_func_check_member().
*/
.check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_func_proto_log,
.seq_show = btf_df_seq_show,
};
@@ -2024,6 +2252,11 @@ static s32 btf_func_check_meta(struct btf_verifier_env *env,
return -EINVAL;
}
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
btf_verifier_log_type(env, t, NULL);
return 0;
@@ -2033,6 +2266,7 @@ static struct btf_kind_operations func_ops = {
.check_meta = btf_func_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_ref_type_log,
.seq_show = btf_df_seq_show,
};
--
2.17.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-14 23:34 ` [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
@ 2018-12-15 16:37 ` Martin Lau
2018-12-15 17:42 ` Yonghong Song
2018-12-15 17:44 ` Alexei Starovoitov
2018-12-15 22:37 ` Daniel Borkmann
1 sibling, 2 replies; 30+ messages in thread
From: Martin Lau @ 2018-12-15 16:37 UTC (permalink / raw)
To: Yonghong Song
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On Fri, Dec 14, 2018 at 03:34:27PM -0800, Yonghong Song wrote:
> This patch fixed two issues with BTF. One is related to
> struct/union bitfield encoding and the other is related to
> forward type.
>
> Issue #1 and solution:
> ======================
>
> Current btf encoding of bitfield follows what pahole generates.
> For each bitfield, pahole will duplicate the type chain and
> put the bitfield size at the final int or enum type.
> Since the BTF enum type cannot encode bit size,
> pahole workarounds the issue by generating
> an int type whenever the enum bit size is not 32.
>
> For example,
> -bash-4.4$ cat t.c
> typedef int ___int;
> enum A { A1, A2, A3 };
> struct t {
> int a[5];
> ___int b:4;
> volatile enum A c:4;
> } g;
> -bash-4.4$ gcc -c -O2 -g t.c
> The current kernel supports the following BTF encoding:
> $ pahole -JV t.o
> [1] TYPEDEF ___int type_id=2
> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> [3] ENUM A size=4 vlen=3
> A1 val=0
> A2 val=1
> A3 val=2
> [4] STRUCT t size=24 vlen=3
> a type_id=5 bits_offset=0
> b type_id=9 bits_offset=160
> c type_id=11 bits_offset=164
> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> [7] VOLATILE (anon) type_id=3
> [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
> [9] TYPEDEF ___int type_id=8
> [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
> [11] VOLATILE (anon) type_id=10
>
> Two issues are in the above:
> . by changing enum type to int, we lost the original
> type information and this will not be ideal later
> when we try to convert BTF to a header file.
> . the type duplication for bitfields will cause
> BTF bloat. Duplicated types cannot be deduplicated
> later if the bitfield size is different.
>
> To fix this issue, this patch implemented a compatible
> change for BTF struct type encoding:
> . the bit 31 of struct_type->info, previously reserved,
> now is used to indicate whether bitfield_size is
> encoded in btf_member or not.
> . if bit 31 of struct_type->info is set,
> btf_member->offset will encode like:
> bit 0 - 23: bit offset
> bit 24 - 31: bitfield size
> if bit 31 is not set, the old behavior is preserved:
> bit 0 - 31: bit offset
>
> So if the struct contains a bit field, the maximum bit offset
> will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
> bitfield size will be 256 which is enough for today as maximum
> bitfield in compiler can be 128 where int128 type is supported.
>
> This kernel patch intends to support the new BTF encoding:
> $ pahole -JV t.o
> [1] TYPEDEF ___int type_id=2
> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> [3] ENUM A size=4 vlen=3
> A1 val=0
> A2 val=1
> A3 val=2
> [4] STRUCT t kind_flag=1 size=24 vlen=3
> a type_id=5 bitfield_size=0 bits_offset=0
> b type_id=1 bitfield_size=4 bits_offset=160
> c type_id=7 bitfield_size=4 bits_offset=164
> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> [7] VOLATILE (anon) type_id=3
>
> Issue #2 and solution:
> ======================
>
> Current forward type in BTF does not specify whether the original
> type is struct or union. This will not work for type pretty print
> and BTF-to-header-file conversion as struct/union must be specified.
> $ cat tt.c
> struct t;
> union u;
> int foo(struct t *t, union u *u) { return 0; }
> $ gcc -c -g -O2 tt.c
> $ pahole -JV tt.o
> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> [2] FWD t type_id=0
> [3] PTR (anon) type_id=2
> [4] FWD u type_id=0
> [5] PTR (anon) type_id=4
>
> To fix this issue, similar to issue #1, type->info bit 31
> is used. If the bit is set, it is union type. Otherwise, it is
> a struct type.
>
> $ pahole -JV tt.o
> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> [2] FWD t kind_flag=0 type_id=0
> [3] PTR (anon) kind_flag=0 type_id=2
> [4] FWD u kind_flag=1 type_id=0
> [5] PTR (anon) kind_flag=0 type_id=4
>
> Pahole/LLVM change:
> ===================
>
> The new kind_flag functionality has been implemented in pahole
> and llvm:
> https://github.com/yonghong-song/pahole/tree/bitfield
> https://github.com/yonghong-song/llvm/tree/bitfield
>
> Note that pahole hasn't implemented func/func_proto kind
> and .BTF.ext. So to print function signature with bpftool,
> the llvm compiler should be used.
>
> Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
> Signed-off-by: Martin KaFai Lau <kafai@fb.com>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
> include/uapi/linux/btf.h | 15 ++-
> kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
> 2 files changed, 267 insertions(+), 22 deletions(-)
>
> diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
> index 14f66948fc95..34aba40ed926 100644
> --- a/include/uapi/linux/btf.h
> +++ b/include/uapi/linux/btf.h
> @@ -34,7 +34,9 @@ struct btf_type {
> * bits 0-15: vlen (e.g. # of struct's members)
> * bits 16-23: unused
> * bits 24-27: kind (e.g. int, ptr, array...etc)
> - * bits 28-31: unused
> + * bits 28-30: unused
> + * bit 31: kind_flag, currently used by
> + * struct, union and fwd
> */
> __u32 info;
> /* "size" is used by INT, ENUM, STRUCT and UNION.
> @@ -52,6 +54,7 @@ struct btf_type {
>
> #define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
> #define BTF_INFO_VLEN(info) ((info) & 0xffff)
> +#define BTF_INFO_KFLAG(info) ((info) >> 31)
>
> #define BTF_KIND_UNKN 0 /* Unknown */
> #define BTF_KIND_INT 1 /* Integer */
> @@ -110,9 +113,17 @@ struct btf_array {
> struct btf_member {
> __u32 name_off;
> __u32 type;
> - __u32 offset; /* offset in bits */
> + __u32 offset; /* [bitfield_size and] offset in bits */
> };
>
> +/* If the type info kind_flag set, the btf_member.offset
> + * contains both member bit offset and bitfield size, and
> + * bitfield size will set for struct/union bitfield members.
> + * Otherwise, it contains only bit offset.
> + */
nit. It may be better to move this comment to the btf_member.offset
above.
> +#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
> +#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
After re-thinking this setup again, I still think
having these macros in btf.h to also do the kflag checking
would be nice.
Unlike BTF_INFO_KIND() and BTF_INT_ENCODING() which don't
depend on other facts, the btf.h raw user must check kflag
anyway before calling BTF_MEMBER_BIT*().
Forcing a kflag check before the user can access these convenient
0xfffff and >>24 conversions may enforce this kflag check to
some extend.
Since it is in uapi, it will not be easy to change later.
The above concern could be overkill ;), just want to ensure
it has been thought through a bit more here.
It could be as easy as moving the new btf_member_bit*() from
btf.c to here and remove these two macros (or move them back to btf.c).
> +
> /* BTF_KIND_FUNC_PROTO is followed by multiple "struct btf_param".
> * The exact number of btf_param is stored in the vlen (of the
> * info in "struct btf_type").
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index 72caa799e82f..ec3f80d3bef6 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
> @@ -164,7 +164,7 @@
> #define BITS_ROUNDUP_BYTES(bits) \
> (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
>
> -#define BTF_INFO_MASK 0x0f00ffff
> +#define BTF_INFO_MASK 0x8f00ffff
> #define BTF_INT_MASK 0x0fffffff
> #define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
> #define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
> @@ -274,6 +274,10 @@ struct btf_kind_operations {
> const struct btf_type *struct_type,
> const struct btf_member *member,
> const struct btf_type *member_type);
> + int (*check_kflag_member)(struct btf_verifier_env *env,
> + const struct btf_type *struct_type,
> + const struct btf_member *member,
> + const struct btf_type *member_type);
> void (*log_details)(struct btf_verifier_env *env,
> const struct btf_type *t);
> void (*seq_show)(const struct btf *btf, const struct btf_type *t,
> @@ -419,6 +423,25 @@ static u16 btf_type_vlen(const struct btf_type *t)
> return BTF_INFO_VLEN(t->info);
> }
>
[ ... ]
> +static int btf_enum_check_kflag_member(struct btf_verifier_env *env,
> + const struct btf_type *struct_type,
> + const struct btf_member *member,
> + const struct btf_type *member_type)
> +{
> + u32 struct_bits_off, nr_bits, bytes_end, struct_size;
> + u32 int_bitsize = sizeof(int) * BITS_PER_BYTE;
> +
> + struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
> + nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
> + if (!nr_bits) {
> + nr_bits = int_bitsize;
For non bitfield enum (i.e. !nr_bits), does it have to check for
byte alignment also? i.e. BITS_PER_BYTE_MASKED(struct_bits_off).
Others look good.
> + } else if (nr_bits > int_bitsize) {
> + btf_verifier_log_member(env, struct_type, member,
> + "Invalid member bitfield_size");
> + return -EINVAL;
> + }
> +
> + struct_size = struct_type->size;
> + bytes_end = BITS_ROUNDUP_BYTES(struct_bits_off + nr_bits);
> + if (struct_size < bytes_end) {
> + btf_verifier_log_member(env, struct_type, member,
> + "Member exceeds struct_size");
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 16:37 ` Martin Lau
@ 2018-12-15 17:42 ` Yonghong Song
2018-12-15 17:44 ` Alexei Starovoitov
1 sibling, 0 replies; 30+ messages in thread
From: Yonghong Song @ 2018-12-15 17:42 UTC (permalink / raw)
To: Martin Lau
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On 12/15/18 8:37 AM, Martin Lau wrote:
> On Fri, Dec 14, 2018 at 03:34:27PM -0800, Yonghong Song wrote:
>> This patch fixed two issues with BTF. One is related to
>> struct/union bitfield encoding and the other is related to
>> forward type.
>>
>> Issue #1 and solution:
>> ======================
>>
>> Current btf encoding of bitfield follows what pahole generates.
>> For each bitfield, pahole will duplicate the type chain and
>> put the bitfield size at the final int or enum type.
>> Since the BTF enum type cannot encode bit size,
>> pahole workarounds the issue by generating
>> an int type whenever the enum bit size is not 32.
>>
>> For example,
>> -bash-4.4$ cat t.c
>> typedef int ___int;
>> enum A { A1, A2, A3 };
>> struct t {
>> int a[5];
>> ___int b:4;
>> volatile enum A c:4;
>> } g;
>> -bash-4.4$ gcc -c -O2 -g t.c
>> The current kernel supports the following BTF encoding:
>> $ pahole -JV t.o
>> [1] TYPEDEF ___int type_id=2
>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>> [3] ENUM A size=4 vlen=3
>> A1 val=0
>> A2 val=1
>> A3 val=2
>> [4] STRUCT t size=24 vlen=3
>> a type_id=5 bits_offset=0
>> b type_id=9 bits_offset=160
>> c type_id=11 bits_offset=164
>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>> [7] VOLATILE (anon) type_id=3
>> [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
>> [9] TYPEDEF ___int type_id=8
>> [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
>> [11] VOLATILE (anon) type_id=10
>>
>> Two issues are in the above:
>> . by changing enum type to int, we lost the original
>> type information and this will not be ideal later
>> when we try to convert BTF to a header file.
>> . the type duplication for bitfields will cause
>> BTF bloat. Duplicated types cannot be deduplicated
>> later if the bitfield size is different.
>>
>> To fix this issue, this patch implemented a compatible
>> change for BTF struct type encoding:
>> . the bit 31 of struct_type->info, previously reserved,
>> now is used to indicate whether bitfield_size is
>> encoded in btf_member or not.
>> . if bit 31 of struct_type->info is set,
>> btf_member->offset will encode like:
>> bit 0 - 23: bit offset
>> bit 24 - 31: bitfield size
>> if bit 31 is not set, the old behavior is preserved:
>> bit 0 - 31: bit offset
>>
>> So if the struct contains a bit field, the maximum bit offset
>> will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
>> bitfield size will be 256 which is enough for today as maximum
>> bitfield in compiler can be 128 where int128 type is supported.
>>
>> This kernel patch intends to support the new BTF encoding:
>> $ pahole -JV t.o
>> [1] TYPEDEF ___int type_id=2
>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>> [3] ENUM A size=4 vlen=3
>> A1 val=0
>> A2 val=1
>> A3 val=2
>> [4] STRUCT t kind_flag=1 size=24 vlen=3
>> a type_id=5 bitfield_size=0 bits_offset=0
>> b type_id=1 bitfield_size=4 bits_offset=160
>> c type_id=7 bitfield_size=4 bits_offset=164
>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>> [7] VOLATILE (anon) type_id=3
>>
>> Issue #2 and solution:
>> ======================
>>
>> Current forward type in BTF does not specify whether the original
>> type is struct or union. This will not work for type pretty print
>> and BTF-to-header-file conversion as struct/union must be specified.
>> $ cat tt.c
>> struct t;
>> union u;
>> int foo(struct t *t, union u *u) { return 0; }
>> $ gcc -c -g -O2 tt.c
>> $ pahole -JV tt.o
>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>> [2] FWD t type_id=0
>> [3] PTR (anon) type_id=2
>> [4] FWD u type_id=0
>> [5] PTR (anon) type_id=4
>>
>> To fix this issue, similar to issue #1, type->info bit 31
>> is used. If the bit is set, it is union type. Otherwise, it is
>> a struct type.
>>
>> $ pahole -JV tt.o
>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>> [2] FWD t kind_flag=0 type_id=0
>> [3] PTR (anon) kind_flag=0 type_id=2
>> [4] FWD u kind_flag=1 type_id=0
>> [5] PTR (anon) kind_flag=0 type_id=4
>>
>> Pahole/LLVM change:
>> ===================
>>
>> The new kind_flag functionality has been implemented in pahole
>> and llvm:
>> https://github.com/yonghong-song/pahole/tree/bitfield
>> https://github.com/yonghong-song/llvm/tree/bitfield
>>
>> Note that pahole hasn't implemented func/func_proto kind
>> and .BTF.ext. So to print function signature with bpftool,
>> the llvm compiler should be used.
>>
>> Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
>> Signed-off-by: Martin KaFai Lau <kafai@fb.com>
>> Signed-off-by: Yonghong Song <yhs@fb.com>
>> ---
>> include/uapi/linux/btf.h | 15 ++-
>> kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
>> 2 files changed, 267 insertions(+), 22 deletions(-)
>>
>> diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
>> index 14f66948fc95..34aba40ed926 100644
>> --- a/include/uapi/linux/btf.h
>> +++ b/include/uapi/linux/btf.h
>> @@ -34,7 +34,9 @@ struct btf_type {
>> * bits 0-15: vlen (e.g. # of struct's members)
>> * bits 16-23: unused
>> * bits 24-27: kind (e.g. int, ptr, array...etc)
>> - * bits 28-31: unused
>> + * bits 28-30: unused
>> + * bit 31: kind_flag, currently used by
>> + * struct, union and fwd
>> */
>> __u32 info;
>> /* "size" is used by INT, ENUM, STRUCT and UNION.
>> @@ -52,6 +54,7 @@ struct btf_type {
>>
>> #define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
>> #define BTF_INFO_VLEN(info) ((info) & 0xffff)
>> +#define BTF_INFO_KFLAG(info) ((info) >> 31)
>>
>> #define BTF_KIND_UNKN 0 /* Unknown */
>> #define BTF_KIND_INT 1 /* Integer */
>> @@ -110,9 +113,17 @@ struct btf_array {
>> struct btf_member {
>> __u32 name_off;
>> __u32 type;
>> - __u32 offset; /* offset in bits */
>> + __u32 offset; /* [bitfield_size and] offset in bits */
>> };
>>
>> +/* If the type info kind_flag set, the btf_member.offset
>> + * contains both member bit offset and bitfield size, and
>> + * bitfield size will set for struct/union bitfield members.
>> + * Otherwise, it contains only bit offset.
>> + */
> nit. It may be better to move this comment to the btf_member.offset
> above.
Make sense. Will have smaller comments as well before the below two
macros to indicate kind_flag needs to be set for the below two
macros make sense.
>
>> +#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
>> +#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
> After re-thinking this setup again, I still think
> having these macros in btf.h to also do the kflag checking
> would be nice.
>
> Unlike BTF_INFO_KIND() and BTF_INT_ENCODING() which don't
> depend on other facts, the btf.h raw user must check kflag
> anyway before calling BTF_MEMBER_BIT*().
> Forcing a kflag check before the user can access these convenient
> 0xfffff and >>24 conversions may enforce this kflag check to
> some extend.
>
> Since it is in uapi, it will not be easy to change later.
> The above concern could be overkill ;), just want to ensure
> it has been thought through a bit more here.
>
> It could be as easy as moving the new btf_member_bit*() from
> btf.c to here and remove these two macros (or move them back to btf.c).
Not an expert in uapi design. I did not put static inline function
(check kflag and get bitfield_size/offset) there with following reasons.
(1). The user/kernel may want to check kflag either to simplify their
code or with certain guaranteed properties, e.g., extra
offset/regular_int checking. So putting macros in the header file will
be good and necessary as we do not want end user (kernel/usr) to have
"shift" or "and" in their codes.
(2). putting static inline functions there feels a little bit redundant
and could be misleading here if used blindly. First, we have macros here
which you can already get information. Second, if user just uses
btf_member_bitfield_size() and assumes that bitfield_size = 0, that will
be wrong since bitfield_size = 0 only if kflag is set, user needs to
trace to base int to find real bitfield size if kflag = 0. So user has
to be aware of kflag even if he calls btf_member_bitfield_size().
>
>> +
>> /* BTF_KIND_FUNC_PROTO is followed by multiple "struct btf_param".
>> * The exact number of btf_param is stored in the vlen (of the
>> * info in "struct btf_type").
>> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
>> index 72caa799e82f..ec3f80d3bef6 100644
>> --- a/kernel/bpf/btf.c
>> +++ b/kernel/bpf/btf.c
>> @@ -164,7 +164,7 @@
>> #define BITS_ROUNDUP_BYTES(bits) \
>> (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
>>
>> -#define BTF_INFO_MASK 0x0f00ffff
>> +#define BTF_INFO_MASK 0x8f00ffff
>> #define BTF_INT_MASK 0x0fffffff
>> #define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
>> #define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
>> @@ -274,6 +274,10 @@ struct btf_kind_operations {
>> const struct btf_type *struct_type,
>> const struct btf_member *member,
>> const struct btf_type *member_type);
>> + int (*check_kflag_member)(struct btf_verifier_env *env,
>> + const struct btf_type *struct_type,
>> + const struct btf_member *member,
>> + const struct btf_type *member_type);
>> void (*log_details)(struct btf_verifier_env *env,
>> const struct btf_type *t);
>> void (*seq_show)(const struct btf *btf, const struct btf_type *t,
>> @@ -419,6 +423,25 @@ static u16 btf_type_vlen(const struct btf_type *t)
>> return BTF_INFO_VLEN(t->info);
>> }
>>
>
> [ ... ]
>
>> +static int btf_enum_check_kflag_member(struct btf_verifier_env *env,
>> + const struct btf_type *struct_type,
>> + const struct btf_member *member,
>> + const struct btf_type *member_type)
>> +{
>> + u32 struct_bits_off, nr_bits, bytes_end, struct_size;
>> + u32 int_bitsize = sizeof(int) * BITS_PER_BYTE;
>> +
>> + struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
>> + nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
>> + if (!nr_bits) {
>> + nr_bits = int_bitsize;
> For non bitfield enum (i.e. !nr_bits), does it have to check for
> byte alignment also? i.e. BITS_PER_BYTE_MASKED(struct_bits_off).
Good catch. Will add this in next revision.
>
> Others look good.
>
>> + } else if (nr_bits > int_bitsize) {
>> + btf_verifier_log_member(env, struct_type, member,
>> + "Invalid member bitfield_size");
>> + return -EINVAL;
>> + }
>> +
>> + struct_size = struct_type->size;
>> + bytes_end = BITS_ROUNDUP_BYTES(struct_bits_off + nr_bits);
>> + if (struct_size < bytes_end) {
>> + btf_verifier_log_member(env, struct_type, member,
>> + "Member exceeds struct_size");
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
>> +
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 16:37 ` Martin Lau
2018-12-15 17:42 ` Yonghong Song
@ 2018-12-15 17:44 ` Alexei Starovoitov
2018-12-15 22:10 ` Martin Lau
2018-12-16 6:18 ` Yonghong Song
1 sibling, 2 replies; 30+ messages in thread
From: Alexei Starovoitov @ 2018-12-15 17:44 UTC (permalink / raw)
To: Martin Lau
Cc: Yonghong Song, netdev@vger.kernel.org, Alexei Starovoitov,
Daniel Borkmann, Kernel Team
On Sat, Dec 15, 2018 at 04:37:06PM +0000, Martin Lau wrote:
> On Fri, Dec 14, 2018 at 03:34:27PM -0800, Yonghong Song wrote:
> > This patch fixed two issues with BTF. One is related to
> > struct/union bitfield encoding and the other is related to
> > forward type.
> >
> > Issue #1 and solution:
> > ======================
> >
> > Current btf encoding of bitfield follows what pahole generates.
> > For each bitfield, pahole will duplicate the type chain and
> > put the bitfield size at the final int or enum type.
> > Since the BTF enum type cannot encode bit size,
> > pahole workarounds the issue by generating
> > an int type whenever the enum bit size is not 32.
> >
> > For example,
> > -bash-4.4$ cat t.c
> > typedef int ___int;
> > enum A { A1, A2, A3 };
> > struct t {
> > int a[5];
> > ___int b:4;
> > volatile enum A c:4;
> > } g;
> > -bash-4.4$ gcc -c -O2 -g t.c
> > The current kernel supports the following BTF encoding:
> > $ pahole -JV t.o
> > [1] TYPEDEF ___int type_id=2
> > [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> > [3] ENUM A size=4 vlen=3
> > A1 val=0
> > A2 val=1
> > A3 val=2
> > [4] STRUCT t size=24 vlen=3
> > a type_id=5 bits_offset=0
> > b type_id=9 bits_offset=160
> > c type_id=11 bits_offset=164
> > [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> > [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> > [7] VOLATILE (anon) type_id=3
> > [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
> > [9] TYPEDEF ___int type_id=8
> > [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
> > [11] VOLATILE (anon) type_id=10
> >
> > Two issues are in the above:
> > . by changing enum type to int, we lost the original
> > type information and this will not be ideal later
> > when we try to convert BTF to a header file.
> > . the type duplication for bitfields will cause
> > BTF bloat. Duplicated types cannot be deduplicated
> > later if the bitfield size is different.
> >
> > To fix this issue, this patch implemented a compatible
> > change for BTF struct type encoding:
> > . the bit 31 of struct_type->info, previously reserved,
> > now is used to indicate whether bitfield_size is
> > encoded in btf_member or not.
> > . if bit 31 of struct_type->info is set,
> > btf_member->offset will encode like:
> > bit 0 - 23: bit offset
> > bit 24 - 31: bitfield size
> > if bit 31 is not set, the old behavior is preserved:
> > bit 0 - 31: bit offset
> >
> > So if the struct contains a bit field, the maximum bit offset
> > will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
> > bitfield size will be 256 which is enough for today as maximum
> > bitfield in compiler can be 128 where int128 type is supported.
> >
> > This kernel patch intends to support the new BTF encoding:
> > $ pahole -JV t.o
> > [1] TYPEDEF ___int type_id=2
> > [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> > [3] ENUM A size=4 vlen=3
> > A1 val=0
> > A2 val=1
> > A3 val=2
> > [4] STRUCT t kind_flag=1 size=24 vlen=3
> > a type_id=5 bitfield_size=0 bits_offset=0
> > b type_id=1 bitfield_size=4 bits_offset=160
> > c type_id=7 bitfield_size=4 bits_offset=164
> > [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> > [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> > [7] VOLATILE (anon) type_id=3
> >
> > Issue #2 and solution:
> > ======================
> >
> > Current forward type in BTF does not specify whether the original
> > type is struct or union. This will not work for type pretty print
> > and BTF-to-header-file conversion as struct/union must be specified.
> > $ cat tt.c
> > struct t;
> > union u;
> > int foo(struct t *t, union u *u) { return 0; }
> > $ gcc -c -g -O2 tt.c
> > $ pahole -JV tt.o
> > [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> > [2] FWD t type_id=0
> > [3] PTR (anon) type_id=2
> > [4] FWD u type_id=0
> > [5] PTR (anon) type_id=4
> >
> > To fix this issue, similar to issue #1, type->info bit 31
> > is used. If the bit is set, it is union type. Otherwise, it is
> > a struct type.
> >
> > $ pahole -JV tt.o
> > [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> > [2] FWD t kind_flag=0 type_id=0
> > [3] PTR (anon) kind_flag=0 type_id=2
> > [4] FWD u kind_flag=1 type_id=0
> > [5] PTR (anon) kind_flag=0 type_id=4
> >
> > Pahole/LLVM change:
> > ===================
> >
> > The new kind_flag functionality has been implemented in pahole
> > and llvm:
> > https://github.com/yonghong-song/pahole/tree/bitfield
> > https://github.com/yonghong-song/llvm/tree/bitfield
> >
> > Note that pahole hasn't implemented func/func_proto kind
> > and .BTF.ext. So to print function signature with bpftool,
> > the llvm compiler should be used.
> >
> > Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
> > Signed-off-by: Martin KaFai Lau <kafai@fb.com>
> > Signed-off-by: Yonghong Song <yhs@fb.com>
> > ---
> > include/uapi/linux/btf.h | 15 ++-
> > kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
> > 2 files changed, 267 insertions(+), 22 deletions(-)
> >
> > diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
> > index 14f66948fc95..34aba40ed926 100644
> > --- a/include/uapi/linux/btf.h
> > +++ b/include/uapi/linux/btf.h
> > @@ -34,7 +34,9 @@ struct btf_type {
> > * bits 0-15: vlen (e.g. # of struct's members)
> > * bits 16-23: unused
> > * bits 24-27: kind (e.g. int, ptr, array...etc)
> > - * bits 28-31: unused
> > + * bits 28-30: unused
> > + * bit 31: kind_flag, currently used by
> > + * struct, union and fwd
> > */
> > __u32 info;
> > /* "size" is used by INT, ENUM, STRUCT and UNION.
> > @@ -52,6 +54,7 @@ struct btf_type {
> >
> > #define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
> > #define BTF_INFO_VLEN(info) ((info) & 0xffff)
> > +#define BTF_INFO_KFLAG(info) ((info) >> 31)
> >
> > #define BTF_KIND_UNKN 0 /* Unknown */
> > #define BTF_KIND_INT 1 /* Integer */
> > @@ -110,9 +113,17 @@ struct btf_array {
> > struct btf_member {
> > __u32 name_off;
> > __u32 type;
> > - __u32 offset; /* offset in bits */
> > + __u32 offset; /* [bitfield_size and] offset in bits */
> > };
> >
> > +/* If the type info kind_flag set, the btf_member.offset
> > + * contains both member bit offset and bitfield size, and
> > + * bitfield size will set for struct/union bitfield members.
> > + * Otherwise, it contains only bit offset.
> > + */
> nit. It may be better to move this comment to the btf_member.offset
> above.
>
> > +#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
> > +#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
> After re-thinking this setup again, I still think
> having these macros in btf.h to also do the kflag checking
> would be nice.
>
> Unlike BTF_INFO_KIND() and BTF_INT_ENCODING() which don't
> depend on other facts, the btf.h raw user must check kflag
> anyway before calling BTF_MEMBER_BIT*().
> Forcing a kflag check before the user can access these convenient
> 0xfffff and >>24 conversions may enforce this kflag check to
> some extend.
>
> Since it is in uapi, it will not be easy to change later.
> The above concern could be overkill ;), just want to ensure
> it has been thought through a bit more here.
>
> It could be as easy as moving the new btf_member_bit*() from
> btf.c to here and remove these two macros (or move them back to btf.c).
I think moving:
+static u32 btf_member_bitfield_size(const struct btf_type *struct_type,
+ const struct btf_member *member)
+{
+ return btf_type_kflag(struct_type) ? BTF_MEMBER_BITFIELD_SIZE(member->offset)
+ : 0;
+}
into uapi/btf.h may or may not be useful for btf uapi users.
What are the chances that these static inline helpers will be
reused by BTF logic in libbpf or other libs?
At this point we don't know.
So I would keep btf.h minimal.
I agree that BTF_MEMBER_BIT_OFFSET() shouldn't be reused blindly.
The users have to do BTF_INFO_KFLAG() check first.
But this is the case for pretty much all of BTF data structures.
In that sense BTF_MEMBER_BITFIELD_SIZE/BTF_MEMBER_BIT_OFFSET
serve as a documentation and fit the style of btf.h header.
imo having these two macros in uapi/btf.h is better than
replacing them with comment only.
After this set the whole BTF really needs to be documented.
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 17:44 ` Alexei Starovoitov
@ 2018-12-15 22:10 ` Martin Lau
2018-12-15 22:26 ` Yonghong Song
2018-12-16 6:18 ` Yonghong Song
1 sibling, 1 reply; 30+ messages in thread
From: Martin Lau @ 2018-12-15 22:10 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Yonghong Song, netdev@vger.kernel.org, Alexei Starovoitov,
Daniel Borkmann, Kernel Team
On Sat, Dec 15, 2018 at 09:44:44AM -0800, Alexei Starovoitov wrote:
> On Sat, Dec 15, 2018 at 04:37:06PM +0000, Martin Lau wrote:
> > On Fri, Dec 14, 2018 at 03:34:27PM -0800, Yonghong Song wrote:
> > > This patch fixed two issues with BTF. One is related to
> > > struct/union bitfield encoding and the other is related to
> > > forward type.
> > >
> > > Issue #1 and solution:
> > > ======================
> > >
> > > Current btf encoding of bitfield follows what pahole generates.
> > > For each bitfield, pahole will duplicate the type chain and
> > > put the bitfield size at the final int or enum type.
> > > Since the BTF enum type cannot encode bit size,
> > > pahole workarounds the issue by generating
> > > an int type whenever the enum bit size is not 32.
> > >
> > > For example,
> > > -bash-4.4$ cat t.c
> > > typedef int ___int;
> > > enum A { A1, A2, A3 };
> > > struct t {
> > > int a[5];
> > > ___int b:4;
> > > volatile enum A c:4;
> > > } g;
> > > -bash-4.4$ gcc -c -O2 -g t.c
> > > The current kernel supports the following BTF encoding:
> > > $ pahole -JV t.o
> > > [1] TYPEDEF ___int type_id=2
> > > [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> > > [3] ENUM A size=4 vlen=3
> > > A1 val=0
> > > A2 val=1
> > > A3 val=2
> > > [4] STRUCT t size=24 vlen=3
> > > a type_id=5 bits_offset=0
> > > b type_id=9 bits_offset=160
> > > c type_id=11 bits_offset=164
> > > [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> > > [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> > > [7] VOLATILE (anon) type_id=3
> > > [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
> > > [9] TYPEDEF ___int type_id=8
> > > [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
> > > [11] VOLATILE (anon) type_id=10
> > >
> > > Two issues are in the above:
> > > . by changing enum type to int, we lost the original
> > > type information and this will not be ideal later
> > > when we try to convert BTF to a header file.
> > > . the type duplication for bitfields will cause
> > > BTF bloat. Duplicated types cannot be deduplicated
> > > later if the bitfield size is different.
> > >
> > > To fix this issue, this patch implemented a compatible
> > > change for BTF struct type encoding:
> > > . the bit 31 of struct_type->info, previously reserved,
> > > now is used to indicate whether bitfield_size is
> > > encoded in btf_member or not.
> > > . if bit 31 of struct_type->info is set,
> > > btf_member->offset will encode like:
> > > bit 0 - 23: bit offset
> > > bit 24 - 31: bitfield size
> > > if bit 31 is not set, the old behavior is preserved:
> > > bit 0 - 31: bit offset
> > >
> > > So if the struct contains a bit field, the maximum bit offset
> > > will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
> > > bitfield size will be 256 which is enough for today as maximum
> > > bitfield in compiler can be 128 where int128 type is supported.
> > >
> > > This kernel patch intends to support the new BTF encoding:
> > > $ pahole -JV t.o
> > > [1] TYPEDEF ___int type_id=2
> > > [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> > > [3] ENUM A size=4 vlen=3
> > > A1 val=0
> > > A2 val=1
> > > A3 val=2
> > > [4] STRUCT t kind_flag=1 size=24 vlen=3
> > > a type_id=5 bitfield_size=0 bits_offset=0
> > > b type_id=1 bitfield_size=4 bits_offset=160
> > > c type_id=7 bitfield_size=4 bits_offset=164
> > > [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> > > [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> > > [7] VOLATILE (anon) type_id=3
> > >
> > > Issue #2 and solution:
> > > ======================
> > >
> > > Current forward type in BTF does not specify whether the original
> > > type is struct or union. This will not work for type pretty print
> > > and BTF-to-header-file conversion as struct/union must be specified.
> > > $ cat tt.c
> > > struct t;
> > > union u;
> > > int foo(struct t *t, union u *u) { return 0; }
> > > $ gcc -c -g -O2 tt.c
> > > $ pahole -JV tt.o
> > > [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> > > [2] FWD t type_id=0
> > > [3] PTR (anon) type_id=2
> > > [4] FWD u type_id=0
> > > [5] PTR (anon) type_id=4
> > >
> > > To fix this issue, similar to issue #1, type->info bit 31
> > > is used. If the bit is set, it is union type. Otherwise, it is
> > > a struct type.
> > >
> > > $ pahole -JV tt.o
> > > [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> > > [2] FWD t kind_flag=0 type_id=0
> > > [3] PTR (anon) kind_flag=0 type_id=2
> > > [4] FWD u kind_flag=1 type_id=0
> > > [5] PTR (anon) kind_flag=0 type_id=4
> > >
> > > Pahole/LLVM change:
> > > ===================
> > >
> > > The new kind_flag functionality has been implemented in pahole
> > > and llvm:
> > > https://github.com/yonghong-song/pahole/tree/bitfield
> > > https://github.com/yonghong-song/llvm/tree/bitfield
> > >
> > > Note that pahole hasn't implemented func/func_proto kind
> > > and .BTF.ext. So to print function signature with bpftool,
> > > the llvm compiler should be used.
> > >
> > > Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
> > > Signed-off-by: Martin KaFai Lau <kafai@fb.com>
> > > Signed-off-by: Yonghong Song <yhs@fb.com>
> > > ---
> > > include/uapi/linux/btf.h | 15 ++-
> > > kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
> > > 2 files changed, 267 insertions(+), 22 deletions(-)
> > >
> > > diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
> > > index 14f66948fc95..34aba40ed926 100644
> > > --- a/include/uapi/linux/btf.h
> > > +++ b/include/uapi/linux/btf.h
> > > @@ -34,7 +34,9 @@ struct btf_type {
> > > * bits 0-15: vlen (e.g. # of struct's members)
> > > * bits 16-23: unused
> > > * bits 24-27: kind (e.g. int, ptr, array...etc)
> > > - * bits 28-31: unused
> > > + * bits 28-30: unused
> > > + * bit 31: kind_flag, currently used by
> > > + * struct, union and fwd
> > > */
> > > __u32 info;
> > > /* "size" is used by INT, ENUM, STRUCT and UNION.
> > > @@ -52,6 +54,7 @@ struct btf_type {
> > >
> > > #define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
> > > #define BTF_INFO_VLEN(info) ((info) & 0xffff)
> > > +#define BTF_INFO_KFLAG(info) ((info) >> 31)
> > >
> > > #define BTF_KIND_UNKN 0 /* Unknown */
> > > #define BTF_KIND_INT 1 /* Integer */
> > > @@ -110,9 +113,17 @@ struct btf_array {
> > > struct btf_member {
> > > __u32 name_off;
> > > __u32 type;
> > > - __u32 offset; /* offset in bits */
> > > + __u32 offset; /* [bitfield_size and] offset in bits */
> > > };
> > >
> > > +/* If the type info kind_flag set, the btf_member.offset
> > > + * contains both member bit offset and bitfield size, and
> > > + * bitfield size will set for struct/union bitfield members.
> > > + * Otherwise, it contains only bit offset.
> > > + */
> > nit. It may be better to move this comment to the btf_member.offset
> > above.
> >
> > > +#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
> > > +#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
> > After re-thinking this setup again, I still think
> > having these macros in btf.h to also do the kflag checking
> > would be nice.
> >
> > Unlike BTF_INFO_KIND() and BTF_INT_ENCODING() which don't
> > depend on other facts, the btf.h raw user must check kflag
> > anyway before calling BTF_MEMBER_BIT*().
> > Forcing a kflag check before the user can access these convenient
> > 0xfffff and >>24 conversions may enforce this kflag check to
> > some extend.
> >
> > Since it is in uapi, it will not be easy to change later.
> > The above concern could be overkill ;), just want to ensure
> > it has been thought through a bit more here.
> >
> > It could be as easy as moving the new btf_member_bit*() from
> > btf.c to here and remove these two macros (or move them back to btf.c).
>
> I think moving:
> +static u32 btf_member_bitfield_size(const struct btf_type *struct_type,
> + const struct btf_member *member)
> +{
> + return btf_type_kflag(struct_type) ? BTF_MEMBER_BITFIELD_SIZE(member->offset)
> + : 0;
> +}
>
> into uapi/btf.h may or may not be useful for btf uapi users.
> What are the chances that these static inline helpers will be
> reused by BTF logic in libbpf or other libs?
> At this point we don't know.
> So I would keep btf.h minimal.
ok. Make sense
> I agree that BTF_MEMBER_BIT_OFFSET() shouldn't be reused blindly.
> The users have to do BTF_INFO_KFLAG() check first.
> But this is the case for pretty much all of BTF data structures.
Other similar situation in btf.h (i.e. a single u32 field can be
interpreted differently) has at least an union as an indication
(e.g. size and type in btf_type)
Here we cannot add the union (bitfield_offset:24 and bitfield_size:8)
and we cannot change the name "offset" also. I am worry about
m->offset will directly be used without checking the BTF_INFO_KFLAG().
may be a "union { __u32 offset; __u32 bitsize_offset; };"......
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 22:10 ` Martin Lau
@ 2018-12-15 22:26 ` Yonghong Song
2018-12-15 23:04 ` Martin Lau
0 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-15 22:26 UTC (permalink / raw)
To: Martin Lau, Alexei Starovoitov
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On 12/15/18 2:10 PM, Martin Lau wrote:
> On Sat, Dec 15, 2018 at 09:44:44AM -0800, Alexei Starovoitov wrote:
>> On Sat, Dec 15, 2018 at 04:37:06PM +0000, Martin Lau wrote:
>>> On Fri, Dec 14, 2018 at 03:34:27PM -0800, Yonghong Song wrote:
>>>> This patch fixed two issues with BTF. One is related to
>>>> struct/union bitfield encoding and the other is related to
>>>> forward type.
>>>>
>>>> Issue #1 and solution:
>>>> ======================
>>>>
>>>> Current btf encoding of bitfield follows what pahole generates.
>>>> For each bitfield, pahole will duplicate the type chain and
>>>> put the bitfield size at the final int or enum type.
>>>> Since the BTF enum type cannot encode bit size,
>>>> pahole workarounds the issue by generating
>>>> an int type whenever the enum bit size is not 32.
>>>>
>>>> For example,
>>>> -bash-4.4$ cat t.c
>>>> typedef int ___int;
>>>> enum A { A1, A2, A3 };
>>>> struct t {
>>>> int a[5];
>>>> ___int b:4;
>>>> volatile enum A c:4;
>>>> } g;
>>>> -bash-4.4$ gcc -c -O2 -g t.c
>>>> The current kernel supports the following BTF encoding:
>>>> $ pahole -JV t.o
>>>> [1] TYPEDEF ___int type_id=2
>>>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>>> [3] ENUM A size=4 vlen=3
>>>> A1 val=0
>>>> A2 val=1
>>>> A3 val=2
>>>> [4] STRUCT t size=24 vlen=3
>>>> a type_id=5 bits_offset=0
>>>> b type_id=9 bits_offset=160
>>>> c type_id=11 bits_offset=164
>>>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>>>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>>>> [7] VOLATILE (anon) type_id=3
>>>> [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
>>>> [9] TYPEDEF ___int type_id=8
>>>> [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
>>>> [11] VOLATILE (anon) type_id=10
>>>>
>>>> Two issues are in the above:
>>>> . by changing enum type to int, we lost the original
>>>> type information and this will not be ideal later
>>>> when we try to convert BTF to a header file.
>>>> . the type duplication for bitfields will cause
>>>> BTF bloat. Duplicated types cannot be deduplicated
>>>> later if the bitfield size is different.
>>>>
>>>> To fix this issue, this patch implemented a compatible
>>>> change for BTF struct type encoding:
>>>> . the bit 31 of struct_type->info, previously reserved,
>>>> now is used to indicate whether bitfield_size is
>>>> encoded in btf_member or not.
>>>> . if bit 31 of struct_type->info is set,
>>>> btf_member->offset will encode like:
>>>> bit 0 - 23: bit offset
>>>> bit 24 - 31: bitfield size
>>>> if bit 31 is not set, the old behavior is preserved:
>>>> bit 0 - 31: bit offset
>>>>
>>>> So if the struct contains a bit field, the maximum bit offset
>>>> will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
>>>> bitfield size will be 256 which is enough for today as maximum
>>>> bitfield in compiler can be 128 where int128 type is supported.
>>>>
>>>> This kernel patch intends to support the new BTF encoding:
>>>> $ pahole -JV t.o
>>>> [1] TYPEDEF ___int type_id=2
>>>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>>> [3] ENUM A size=4 vlen=3
>>>> A1 val=0
>>>> A2 val=1
>>>> A3 val=2
>>>> [4] STRUCT t kind_flag=1 size=24 vlen=3
>>>> a type_id=5 bitfield_size=0 bits_offset=0
>>>> b type_id=1 bitfield_size=4 bits_offset=160
>>>> c type_id=7 bitfield_size=4 bits_offset=164
>>>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>>>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>>>> [7] VOLATILE (anon) type_id=3
>>>>
>>>> Issue #2 and solution:
>>>> ======================
>>>>
>>>> Current forward type in BTF does not specify whether the original
>>>> type is struct or union. This will not work for type pretty print
>>>> and BTF-to-header-file conversion as struct/union must be specified.
>>>> $ cat tt.c
>>>> struct t;
>>>> union u;
>>>> int foo(struct t *t, union u *u) { return 0; }
>>>> $ gcc -c -g -O2 tt.c
>>>> $ pahole -JV tt.o
>>>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>>> [2] FWD t type_id=0
>>>> [3] PTR (anon) type_id=2
>>>> [4] FWD u type_id=0
>>>> [5] PTR (anon) type_id=4
>>>>
>>>> To fix this issue, similar to issue #1, type->info bit 31
>>>> is used. If the bit is set, it is union type. Otherwise, it is
>>>> a struct type.
>>>>
>>>> $ pahole -JV tt.o
>>>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>>> [2] FWD t kind_flag=0 type_id=0
>>>> [3] PTR (anon) kind_flag=0 type_id=2
>>>> [4] FWD u kind_flag=1 type_id=0
>>>> [5] PTR (anon) kind_flag=0 type_id=4
>>>>
>>>> Pahole/LLVM change:
>>>> ===================
>>>>
>>>> The new kind_flag functionality has been implemented in pahole
>>>> and llvm:
>>>> https://github.com/yonghong-song/pahole/tree/bitfield
>>>> https://github.com/yonghong-song/llvm/tree/bitfield
>>>>
>>>> Note that pahole hasn't implemented func/func_proto kind
>>>> and .BTF.ext. So to print function signature with bpftool,
>>>> the llvm compiler should be used.
>>>>
>>>> Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
>>>> Signed-off-by: Martin KaFai Lau <kafai@fb.com>
>>>> Signed-off-by: Yonghong Song <yhs@fb.com>
>>>> ---
>>>> include/uapi/linux/btf.h | 15 ++-
>>>> kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
>>>> 2 files changed, 267 insertions(+), 22 deletions(-)
>>>>
>>>> diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
>>>> index 14f66948fc95..34aba40ed926 100644
>>>> --- a/include/uapi/linux/btf.h
>>>> +++ b/include/uapi/linux/btf.h
>>>> @@ -34,7 +34,9 @@ struct btf_type {
>>>> * bits 0-15: vlen (e.g. # of struct's members)
>>>> * bits 16-23: unused
>>>> * bits 24-27: kind (e.g. int, ptr, array...etc)
>>>> - * bits 28-31: unused
>>>> + * bits 28-30: unused
>>>> + * bit 31: kind_flag, currently used by
>>>> + * struct, union and fwd
>>>> */
>>>> __u32 info;
>>>> /* "size" is used by INT, ENUM, STRUCT and UNION.
>>>> @@ -52,6 +54,7 @@ struct btf_type {
>>>>
>>>> #define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
>>>> #define BTF_INFO_VLEN(info) ((info) & 0xffff)
>>>> +#define BTF_INFO_KFLAG(info) ((info) >> 31)
>>>>
>>>> #define BTF_KIND_UNKN 0 /* Unknown */
>>>> #define BTF_KIND_INT 1 /* Integer */
>>>> @@ -110,9 +113,17 @@ struct btf_array {
>>>> struct btf_member {
>>>> __u32 name_off;
>>>> __u32 type;
>>>> - __u32 offset; /* offset in bits */
>>>> + __u32 offset; /* [bitfield_size and] offset in bits */
>>>> };
>>>>
>>>> +/* If the type info kind_flag set, the btf_member.offset
>>>> + * contains both member bit offset and bitfield size, and
>>>> + * bitfield size will set for struct/union bitfield members.
>>>> + * Otherwise, it contains only bit offset.
>>>> + */
>>> nit. It may be better to move this comment to the btf_member.offset
>>> above.
>>>
>>>> +#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
>>>> +#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
>>> After re-thinking this setup again, I still think
>>> having these macros in btf.h to also do the kflag checking
>>> would be nice.
>>>
>>> Unlike BTF_INFO_KIND() and BTF_INT_ENCODING() which don't
>>> depend on other facts, the btf.h raw user must check kflag
>>> anyway before calling BTF_MEMBER_BIT*().
>>> Forcing a kflag check before the user can access these convenient
>>> 0xfffff and >>24 conversions may enforce this kflag check to
>>> some extend.
>>>
>>> Since it is in uapi, it will not be easy to change later.
>>> The above concern could be overkill ;), just want to ensure
>>> it has been thought through a bit more here.
>>>
>>> It could be as easy as moving the new btf_member_bit*() from
>>> btf.c to here and remove these two macros (or move them back to btf.c).
>>
>> I think moving:
>> +static u32 btf_member_bitfield_size(const struct btf_type *struct_type,
>> + const struct btf_member *member)
>> +{
>> + return btf_type_kflag(struct_type) ? BTF_MEMBER_BITFIELD_SIZE(member->offset)
>> + : 0;
>> +}
>>
>> into uapi/btf.h may or may not be useful for btf uapi users.
>> What are the chances that these static inline helpers will be
>> reused by BTF logic in libbpf or other libs?
>> At this point we don't know.
>
>> So I would keep btf.h minimal.
> ok. Make sense
>
>> I agree that BTF_MEMBER_BIT_OFFSET() shouldn't be reused blindly.
>> The users have to do BTF_INFO_KFLAG() check first.
>> But this is the case for pretty much all of BTF data structures.
> Other similar situation in btf.h (i.e. a single u32 field can be
> interpreted differently) has at least an union as an indication
> (e.g. size and type in btf_type)
>
> Here we cannot add the union (bitfield_offset:24 and bitfield_size:8)
> and we cannot change the name "offset" also. I am worry about
> m->offset will directly be used without checking the BTF_INFO_KFLAG().
>
> may be a "union { __u32 offset; __u32 bitsize_offset; };"......
The union with two __u32 is great idea. Maybe the
bitsize_offset becomes "bitfield_size_offset" to reflect
its real intention?
>
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 22:26 ` Yonghong Song
@ 2018-12-15 23:04 ` Martin Lau
2018-12-15 23:13 ` Yonghong Song
0 siblings, 1 reply; 30+ messages in thread
From: Martin Lau @ 2018-12-15 23:04 UTC (permalink / raw)
To: Yonghong Song
Cc: Alexei Starovoitov, netdev@vger.kernel.org, Alexei Starovoitov,
Daniel Borkmann, Kernel Team
On Sat, Dec 15, 2018 at 02:26:44PM -0800, Yonghong Song wrote:
>
>
> On 12/15/18 2:10 PM, Martin Lau wrote:
> > On Sat, Dec 15, 2018 at 09:44:44AM -0800, Alexei Starovoitov wrote:
> >> On Sat, Dec 15, 2018 at 04:37:06PM +0000, Martin Lau wrote:
> >>> On Fri, Dec 14, 2018 at 03:34:27PM -0800, Yonghong Song wrote:
> >>>> This patch fixed two issues with BTF. One is related to
> >>>> struct/union bitfield encoding and the other is related to
> >>>> forward type.
> >>>>
> >>>> Issue #1 and solution:
> >>>> ======================
> >>>>
> >>>> Current btf encoding of bitfield follows what pahole generates.
> >>>> For each bitfield, pahole will duplicate the type chain and
> >>>> put the bitfield size at the final int or enum type.
> >>>> Since the BTF enum type cannot encode bit size,
> >>>> pahole workarounds the issue by generating
> >>>> an int type whenever the enum bit size is not 32.
> >>>>
> >>>> For example,
> >>>> -bash-4.4$ cat t.c
> >>>> typedef int ___int;
> >>>> enum A { A1, A2, A3 };
> >>>> struct t {
> >>>> int a[5];
> >>>> ___int b:4;
> >>>> volatile enum A c:4;
> >>>> } g;
> >>>> -bash-4.4$ gcc -c -O2 -g t.c
> >>>> The current kernel supports the following BTF encoding:
> >>>> $ pahole -JV t.o
> >>>> [1] TYPEDEF ___int type_id=2
> >>>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> >>>> [3] ENUM A size=4 vlen=3
> >>>> A1 val=0
> >>>> A2 val=1
> >>>> A3 val=2
> >>>> [4] STRUCT t size=24 vlen=3
> >>>> a type_id=5 bits_offset=0
> >>>> b type_id=9 bits_offset=160
> >>>> c type_id=11 bits_offset=164
> >>>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> >>>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> >>>> [7] VOLATILE (anon) type_id=3
> >>>> [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
> >>>> [9] TYPEDEF ___int type_id=8
> >>>> [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
> >>>> [11] VOLATILE (anon) type_id=10
> >>>>
> >>>> Two issues are in the above:
> >>>> . by changing enum type to int, we lost the original
> >>>> type information and this will not be ideal later
> >>>> when we try to convert BTF to a header file.
> >>>> . the type duplication for bitfields will cause
> >>>> BTF bloat. Duplicated types cannot be deduplicated
> >>>> later if the bitfield size is different.
> >>>>
> >>>> To fix this issue, this patch implemented a compatible
> >>>> change for BTF struct type encoding:
> >>>> . the bit 31 of struct_type->info, previously reserved,
> >>>> now is used to indicate whether bitfield_size is
> >>>> encoded in btf_member or not.
> >>>> . if bit 31 of struct_type->info is set,
> >>>> btf_member->offset will encode like:
> >>>> bit 0 - 23: bit offset
> >>>> bit 24 - 31: bitfield size
> >>>> if bit 31 is not set, the old behavior is preserved:
> >>>> bit 0 - 31: bit offset
> >>>>
> >>>> So if the struct contains a bit field, the maximum bit offset
> >>>> will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
> >>>> bitfield size will be 256 which is enough for today as maximum
> >>>> bitfield in compiler can be 128 where int128 type is supported.
> >>>>
> >>>> This kernel patch intends to support the new BTF encoding:
> >>>> $ pahole -JV t.o
> >>>> [1] TYPEDEF ___int type_id=2
> >>>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> >>>> [3] ENUM A size=4 vlen=3
> >>>> A1 val=0
> >>>> A2 val=1
> >>>> A3 val=2
> >>>> [4] STRUCT t kind_flag=1 size=24 vlen=3
> >>>> a type_id=5 bitfield_size=0 bits_offset=0
> >>>> b type_id=1 bitfield_size=4 bits_offset=160
> >>>> c type_id=7 bitfield_size=4 bits_offset=164
> >>>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> >>>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> >>>> [7] VOLATILE (anon) type_id=3
> >>>>
> >>>> Issue #2 and solution:
> >>>> ======================
> >>>>
> >>>> Current forward type in BTF does not specify whether the original
> >>>> type is struct or union. This will not work for type pretty print
> >>>> and BTF-to-header-file conversion as struct/union must be specified.
> >>>> $ cat tt.c
> >>>> struct t;
> >>>> union u;
> >>>> int foo(struct t *t, union u *u) { return 0; }
> >>>> $ gcc -c -g -O2 tt.c
> >>>> $ pahole -JV tt.o
> >>>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> >>>> [2] FWD t type_id=0
> >>>> [3] PTR (anon) type_id=2
> >>>> [4] FWD u type_id=0
> >>>> [5] PTR (anon) type_id=4
> >>>>
> >>>> To fix this issue, similar to issue #1, type->info bit 31
> >>>> is used. If the bit is set, it is union type. Otherwise, it is
> >>>> a struct type.
> >>>>
> >>>> $ pahole -JV tt.o
> >>>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> >>>> [2] FWD t kind_flag=0 type_id=0
> >>>> [3] PTR (anon) kind_flag=0 type_id=2
> >>>> [4] FWD u kind_flag=1 type_id=0
> >>>> [5] PTR (anon) kind_flag=0 type_id=4
> >>>>
> >>>> Pahole/LLVM change:
> >>>> ===================
> >>>>
> >>>> The new kind_flag functionality has been implemented in pahole
> >>>> and llvm:
> >>>> https://github.com/yonghong-song/pahole/tree/bitfield
> >>>> https://github.com/yonghong-song/llvm/tree/bitfield
> >>>>
> >>>> Note that pahole hasn't implemented func/func_proto kind
> >>>> and .BTF.ext. So to print function signature with bpftool,
> >>>> the llvm compiler should be used.
> >>>>
> >>>> Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
> >>>> Signed-off-by: Martin KaFai Lau <kafai@fb.com>
> >>>> Signed-off-by: Yonghong Song <yhs@fb.com>
> >>>> ---
> >>>> include/uapi/linux/btf.h | 15 ++-
> >>>> kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
> >>>> 2 files changed, 267 insertions(+), 22 deletions(-)
> >>>>
> >>>> diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
> >>>> index 14f66948fc95..34aba40ed926 100644
> >>>> --- a/include/uapi/linux/btf.h
> >>>> +++ b/include/uapi/linux/btf.h
> >>>> @@ -34,7 +34,9 @@ struct btf_type {
> >>>> * bits 0-15: vlen (e.g. # of struct's members)
> >>>> * bits 16-23: unused
> >>>> * bits 24-27: kind (e.g. int, ptr, array...etc)
> >>>> - * bits 28-31: unused
> >>>> + * bits 28-30: unused
> >>>> + * bit 31: kind_flag, currently used by
> >>>> + * struct, union and fwd
> >>>> */
> >>>> __u32 info;
> >>>> /* "size" is used by INT, ENUM, STRUCT and UNION.
> >>>> @@ -52,6 +54,7 @@ struct btf_type {
> >>>>
> >>>> #define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
> >>>> #define BTF_INFO_VLEN(info) ((info) & 0xffff)
> >>>> +#define BTF_INFO_KFLAG(info) ((info) >> 31)
> >>>>
> >>>> #define BTF_KIND_UNKN 0 /* Unknown */
> >>>> #define BTF_KIND_INT 1 /* Integer */
> >>>> @@ -110,9 +113,17 @@ struct btf_array {
> >>>> struct btf_member {
> >>>> __u32 name_off;
> >>>> __u32 type;
> >>>> - __u32 offset; /* offset in bits */
> >>>> + __u32 offset; /* [bitfield_size and] offset in bits */
> >>>> };
> >>>>
> >>>> +/* If the type info kind_flag set, the btf_member.offset
> >>>> + * contains both member bit offset and bitfield size, and
> >>>> + * bitfield size will set for struct/union bitfield members.
> >>>> + * Otherwise, it contains only bit offset.
> >>>> + */
> >>> nit. It may be better to move this comment to the btf_member.offset
> >>> above.
> >>>
> >>>> +#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
> >>>> +#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
> >>> After re-thinking this setup again, I still think
> >>> having these macros in btf.h to also do the kflag checking
> >>> would be nice.
> >>>
> >>> Unlike BTF_INFO_KIND() and BTF_INT_ENCODING() which don't
> >>> depend on other facts, the btf.h raw user must check kflag
> >>> anyway before calling BTF_MEMBER_BIT*().
> >>> Forcing a kflag check before the user can access these convenient
> >>> 0xfffff and >>24 conversions may enforce this kflag check to
> >>> some extend.
> >>>
> >>> Since it is in uapi, it will not be easy to change later.
> >>> The above concern could be overkill ;), just want to ensure
> >>> it has been thought through a bit more here.
> >>>
> >>> It could be as easy as moving the new btf_member_bit*() from
> >>> btf.c to here and remove these two macros (or move them back to btf.c).
> >>
> >> I think moving:
> >> +static u32 btf_member_bitfield_size(const struct btf_type *struct_type,
> >> + const struct btf_member *member)
> >> +{
> >> + return btf_type_kflag(struct_type) ? BTF_MEMBER_BITFIELD_SIZE(member->offset)
> >> + : 0;
> >> +}
> >>
> >> into uapi/btf.h may or may not be useful for btf uapi users.
> >> What are the chances that these static inline helpers will be
> >> reused by BTF logic in libbpf or other libs?
> >> At this point we don't know.
> >
> >> So I would keep btf.h minimal.
> > ok. Make sense
> >
> >> I agree that BTF_MEMBER_BIT_OFFSET() shouldn't be reused blindly.
> >> The users have to do BTF_INFO_KFLAG() check first.
> >> But this is the case for pretty much all of BTF data structures.
> > Other similar situation in btf.h (i.e. a single u32 field can be
> > interpreted differently) has at least an union as an indication
> > (e.g. size and type in btf_type)
> >
> > Here we cannot add the union (bitfield_offset:24 and bitfield_size:8)
> > and we cannot change the name "offset" also. I am worry about
> > m->offset will directly be used without checking the BTF_INFO_KFLAG().
> >
> > may be a "union { __u32 offset; __u32 bitsize_offset; };"......
>
> The union with two __u32 is great idea. Maybe the
> bitsize_offset becomes "bitfield_size_offset" to reflect
> its real intention?
SGTM. Probably also spell out when to use "offset"
and when to use "bitfield_size_offset" like the
union in "struct btf_type". The BTF_MEMBER_BIT*() macro
may also need to adjust to access the bitfield_size_offset
instead to make the case clearer.
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 23:04 ` Martin Lau
@ 2018-12-15 23:13 ` Yonghong Song
2018-12-15 23:19 ` Alexei Starovoitov
0 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-15 23:13 UTC (permalink / raw)
To: Martin Lau
Cc: Alexei Starovoitov, netdev@vger.kernel.org, Alexei Starovoitov,
Daniel Borkmann, Kernel Team
On 12/15/18 3:04 PM, Martin Lau wrote:
> On Sat, Dec 15, 2018 at 02:26:44PM -0800, Yonghong Song wrote:
>>
>>
>> On 12/15/18 2:10 PM, Martin Lau wrote:
>>> On Sat, Dec 15, 2018 at 09:44:44AM -0800, Alexei Starovoitov wrote:
>>>> On Sat, Dec 15, 2018 at 04:37:06PM +0000, Martin Lau wrote:
>>>>> On Fri, Dec 14, 2018 at 03:34:27PM -0800, Yonghong Song wrote:
>>>>>> This patch fixed two issues with BTF. One is related to
>>>>>> struct/union bitfield encoding and the other is related to
>>>>>> forward type.
>>>>>>
>>>>>> Issue #1 and solution:
>>>>>> ======================
>>>>>>
>>>>>> Current btf encoding of bitfield follows what pahole generates.
>>>>>> For each bitfield, pahole will duplicate the type chain and
>>>>>> put the bitfield size at the final int or enum type.
>>>>>> Since the BTF enum type cannot encode bit size,
>>>>>> pahole workarounds the issue by generating
>>>>>> an int type whenever the enum bit size is not 32.
>>>>>>
>>>>>> For example,
>>>>>> -bash-4.4$ cat t.c
>>>>>> typedef int ___int;
>>>>>> enum A { A1, A2, A3 };
>>>>>> struct t {
>>>>>> int a[5];
>>>>>> ___int b:4;
>>>>>> volatile enum A c:4;
>>>>>> } g;
>>>>>> -bash-4.4$ gcc -c -O2 -g t.c
>>>>>> The current kernel supports the following BTF encoding:
>>>>>> $ pahole -JV t.o
>>>>>> [1] TYPEDEF ___int type_id=2
>>>>>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>>>>> [3] ENUM A size=4 vlen=3
>>>>>> A1 val=0
>>>>>> A2 val=1
>>>>>> A3 val=2
>>>>>> [4] STRUCT t size=24 vlen=3
>>>>>> a type_id=5 bits_offset=0
>>>>>> b type_id=9 bits_offset=160
>>>>>> c type_id=11 bits_offset=164
>>>>>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>>>>>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>>>>>> [7] VOLATILE (anon) type_id=3
>>>>>> [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
>>>>>> [9] TYPEDEF ___int type_id=8
>>>>>> [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
>>>>>> [11] VOLATILE (anon) type_id=10
>>>>>>
>>>>>> Two issues are in the above:
>>>>>> . by changing enum type to int, we lost the original
>>>>>> type information and this will not be ideal later
>>>>>> when we try to convert BTF to a header file.
>>>>>> . the type duplication for bitfields will cause
>>>>>> BTF bloat. Duplicated types cannot be deduplicated
>>>>>> later if the bitfield size is different.
>>>>>>
>>>>>> To fix this issue, this patch implemented a compatible
>>>>>> change for BTF struct type encoding:
>>>>>> . the bit 31 of struct_type->info, previously reserved,
>>>>>> now is used to indicate whether bitfield_size is
>>>>>> encoded in btf_member or not.
>>>>>> . if bit 31 of struct_type->info is set,
>>>>>> btf_member->offset will encode like:
>>>>>> bit 0 - 23: bit offset
>>>>>> bit 24 - 31: bitfield size
>>>>>> if bit 31 is not set, the old behavior is preserved:
>>>>>> bit 0 - 31: bit offset
>>>>>>
>>>>>> So if the struct contains a bit field, the maximum bit offset
>>>>>> will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
>>>>>> bitfield size will be 256 which is enough for today as maximum
>>>>>> bitfield in compiler can be 128 where int128 type is supported.
>>>>>>
>>>>>> This kernel patch intends to support the new BTF encoding:
>>>>>> $ pahole -JV t.o
>>>>>> [1] TYPEDEF ___int type_id=2
>>>>>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>>>>> [3] ENUM A size=4 vlen=3
>>>>>> A1 val=0
>>>>>> A2 val=1
>>>>>> A3 val=2
>>>>>> [4] STRUCT t kind_flag=1 size=24 vlen=3
>>>>>> a type_id=5 bitfield_size=0 bits_offset=0
>>>>>> b type_id=1 bitfield_size=4 bits_offset=160
>>>>>> c type_id=7 bitfield_size=4 bits_offset=164
>>>>>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>>>>>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>>>>>> [7] VOLATILE (anon) type_id=3
>>>>>>
>>>>>> Issue #2 and solution:
>>>>>> ======================
>>>>>>
>>>>>> Current forward type in BTF does not specify whether the original
>>>>>> type is struct or union. This will not work for type pretty print
>>>>>> and BTF-to-header-file conversion as struct/union must be specified.
>>>>>> $ cat tt.c
>>>>>> struct t;
>>>>>> union u;
>>>>>> int foo(struct t *t, union u *u) { return 0; }
>>>>>> $ gcc -c -g -O2 tt.c
>>>>>> $ pahole -JV tt.o
>>>>>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>>>>> [2] FWD t type_id=0
>>>>>> [3] PTR (anon) type_id=2
>>>>>> [4] FWD u type_id=0
>>>>>> [5] PTR (anon) type_id=4
>>>>>>
>>>>>> To fix this issue, similar to issue #1, type->info bit 31
>>>>>> is used. If the bit is set, it is union type. Otherwise, it is
>>>>>> a struct type.
>>>>>>
>>>>>> $ pahole -JV tt.o
>>>>>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>>>>> [2] FWD t kind_flag=0 type_id=0
>>>>>> [3] PTR (anon) kind_flag=0 type_id=2
>>>>>> [4] FWD u kind_flag=1 type_id=0
>>>>>> [5] PTR (anon) kind_flag=0 type_id=4
>>>>>>
>>>>>> Pahole/LLVM change:
>>>>>> ===================
>>>>>>
>>>>>> The new kind_flag functionality has been implemented in pahole
>>>>>> and llvm:
>>>>>> https://github.com/yonghong-song/pahole/tree/bitfield
>>>>>> https://github.com/yonghong-song/llvm/tree/bitfield
>>>>>>
>>>>>> Note that pahole hasn't implemented func/func_proto kind
>>>>>> and .BTF.ext. So to print function signature with bpftool,
>>>>>> the llvm compiler should be used.
>>>>>>
>>>>>> Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
>>>>>> Signed-off-by: Martin KaFai Lau <kafai@fb.com>
>>>>>> Signed-off-by: Yonghong Song <yhs@fb.com>
>>>>>> ---
>>>>>> include/uapi/linux/btf.h | 15 ++-
>>>>>> kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
>>>>>> 2 files changed, 267 insertions(+), 22 deletions(-)
>>>>>>
>>>>>> diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
>>>>>> index 14f66948fc95..34aba40ed926 100644
>>>>>> --- a/include/uapi/linux/btf.h
>>>>>> +++ b/include/uapi/linux/btf.h
>>>>>> @@ -34,7 +34,9 @@ struct btf_type {
>>>>>> * bits 0-15: vlen (e.g. # of struct's members)
>>>>>> * bits 16-23: unused
>>>>>> * bits 24-27: kind (e.g. int, ptr, array...etc)
>>>>>> - * bits 28-31: unused
>>>>>> + * bits 28-30: unused
>>>>>> + * bit 31: kind_flag, currently used by
>>>>>> + * struct, union and fwd
>>>>>> */
>>>>>> __u32 info;
>>>>>> /* "size" is used by INT, ENUM, STRUCT and UNION.
>>>>>> @@ -52,6 +54,7 @@ struct btf_type {
>>>>>>
>>>>>> #define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
>>>>>> #define BTF_INFO_VLEN(info) ((info) & 0xffff)
>>>>>> +#define BTF_INFO_KFLAG(info) ((info) >> 31)
>>>>>>
>>>>>> #define BTF_KIND_UNKN 0 /* Unknown */
>>>>>> #define BTF_KIND_INT 1 /* Integer */
>>>>>> @@ -110,9 +113,17 @@ struct btf_array {
>>>>>> struct btf_member {
>>>>>> __u32 name_off;
>>>>>> __u32 type;
>>>>>> - __u32 offset; /* offset in bits */
>>>>>> + __u32 offset; /* [bitfield_size and] offset in bits */
>>>>>> };
>>>>>>
>>>>>> +/* If the type info kind_flag set, the btf_member.offset
>>>>>> + * contains both member bit offset and bitfield size, and
>>>>>> + * bitfield size will set for struct/union bitfield members.
>>>>>> + * Otherwise, it contains only bit offset.
>>>>>> + */
>>>>> nit. It may be better to move this comment to the btf_member.offset
>>>>> above.
>>>>>
>>>>>> +#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
>>>>>> +#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
>>>>> After re-thinking this setup again, I still think
>>>>> having these macros in btf.h to also do the kflag checking
>>>>> would be nice.
>>>>>
>>>>> Unlike BTF_INFO_KIND() and BTF_INT_ENCODING() which don't
>>>>> depend on other facts, the btf.h raw user must check kflag
>>>>> anyway before calling BTF_MEMBER_BIT*().
>>>>> Forcing a kflag check before the user can access these convenient
>>>>> 0xfffff and >>24 conversions may enforce this kflag check to
>>>>> some extend.
>>>>>
>>>>> Since it is in uapi, it will not be easy to change later.
>>>>> The above concern could be overkill ;), just want to ensure
>>>>> it has been thought through a bit more here.
>>>>>
>>>>> It could be as easy as moving the new btf_member_bit*() from
>>>>> btf.c to here and remove these two macros (or move them back to btf.c).
>>>>
>>>> I think moving:
>>>> +static u32 btf_member_bitfield_size(const struct btf_type *struct_type,
>>>> + const struct btf_member *member)
>>>> +{
>>>> + return btf_type_kflag(struct_type) ? BTF_MEMBER_BITFIELD_SIZE(member->offset)
>>>> + : 0;
>>>> +}
>>>>
>>>> into uapi/btf.h may or may not be useful for btf uapi users.
>>>> What are the chances that these static inline helpers will be
>>>> reused by BTF logic in libbpf or other libs?
>>>> At this point we don't know.
>>>
>>>> So I would keep btf.h minimal.
>>> ok. Make sense
>>>
>>>> I agree that BTF_MEMBER_BIT_OFFSET() shouldn't be reused blindly.
>>>> The users have to do BTF_INFO_KFLAG() check first.
>>>> But this is the case for pretty much all of BTF data structures.
>>> Other similar situation in btf.h (i.e. a single u32 field can be
>>> interpreted differently) has at least an union as an indication
>>> (e.g. size and type in btf_type)
>>>
>>> Here we cannot add the union (bitfield_offset:24 and bitfield_size:8)
>>> and we cannot change the name "offset" also. I am worry about
>>> m->offset will directly be used without checking the BTF_INFO_KFLAG().
>>>
>>> may be a "union { __u32 offset; __u32 bitsize_offset; };"......
>>
>> The union with two __u32 is great idea. Maybe the
>> bitsize_offset becomes "bitfield_size_offset" to reflect
>> its real intention?
> SGTM. Probably also spell out when to use "offset"
> and when to use "bitfield_size_offset" like the
> union in "struct btf_type". The BTF_MEMBER_BIT*() macro
> may also need to adjust to access the bitfield_size_offset
> instead to make the case clearer.
Sounds good. This is my plan to do as well.
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 23:13 ` Yonghong Song
@ 2018-12-15 23:19 ` Alexei Starovoitov
0 siblings, 0 replies; 30+ messages in thread
From: Alexei Starovoitov @ 2018-12-15 23:19 UTC (permalink / raw)
To: Yonghong Song, Martin Lau
Cc: Alexei Starovoitov, netdev@vger.kernel.org, Daniel Borkmann,
Kernel Team
On 12/15/18 3:13 PM, Yonghong Song wrote:
>> may be a "union { __u32 offset; __u32 bitsize_offset; };"......
> The union with two __u32 is great idea. Maybe the
> bitsize_offset becomes "bitfield_size_offset" to reflect
> its real intention?
I don't think union and verbose name will help.
imo it's add confusion.
I prefer to keep it as-is with simple 'offset' name.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 17:44 ` Alexei Starovoitov
2018-12-15 22:10 ` Martin Lau
@ 2018-12-16 6:18 ` Yonghong Song
1 sibling, 0 replies; 30+ messages in thread
From: Yonghong Song @ 2018-12-16 6:18 UTC (permalink / raw)
To: Alexei Starovoitov, Martin Lau
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On 12/15/18 9:44 AM, Alexei Starovoitov wrote:
> On Sat, Dec 15, 2018 at 04:37:06PM +0000, Martin Lau wrote:
>> On Fri, Dec 14, 2018 at 03:34:27PM -0800, Yonghong Song wrote:
>>> This patch fixed two issues with BTF. One is related to
>>> struct/union bitfield encoding and the other is related to
>>> forward type.
>>>
>>> Issue #1 and solution:
>>> ======================
>>>
>>> Current btf encoding of bitfield follows what pahole generates.
>>> For each bitfield, pahole will duplicate the type chain and
>>> put the bitfield size at the final int or enum type.
>>> Since the BTF enum type cannot encode bit size,
>>> pahole workarounds the issue by generating
>>> an int type whenever the enum bit size is not 32.
>>>
>>> For example,
>>> -bash-4.4$ cat t.c
>>> typedef int ___int;
>>> enum A { A1, A2, A3 };
>>> struct t {
>>> int a[5];
>>> ___int b:4;
>>> volatile enum A c:4;
>>> } g;
>>> -bash-4.4$ gcc -c -O2 -g t.c
>>> The current kernel supports the following BTF encoding:
>>> $ pahole -JV t.o
>>> [1] TYPEDEF ___int type_id=2
>>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>> [3] ENUM A size=4 vlen=3
>>> A1 val=0
>>> A2 val=1
>>> A3 val=2
>>> [4] STRUCT t size=24 vlen=3
>>> a type_id=5 bits_offset=0
>>> b type_id=9 bits_offset=160
>>> c type_id=11 bits_offset=164
>>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>>> [7] VOLATILE (anon) type_id=3
>>> [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
>>> [9] TYPEDEF ___int type_id=8
>>> [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
>>> [11] VOLATILE (anon) type_id=10
>>>
>>> Two issues are in the above:
>>> . by changing enum type to int, we lost the original
>>> type information and this will not be ideal later
>>> when we try to convert BTF to a header file.
>>> . the type duplication for bitfields will cause
>>> BTF bloat. Duplicated types cannot be deduplicated
>>> later if the bitfield size is different.
>>>
>>> To fix this issue, this patch implemented a compatible
>>> change for BTF struct type encoding:
>>> . the bit 31 of struct_type->info, previously reserved,
>>> now is used to indicate whether bitfield_size is
>>> encoded in btf_member or not.
>>> . if bit 31 of struct_type->info is set,
>>> btf_member->offset will encode like:
>>> bit 0 - 23: bit offset
>>> bit 24 - 31: bitfield size
>>> if bit 31 is not set, the old behavior is preserved:
>>> bit 0 - 31: bit offset
>>>
>>> So if the struct contains a bit field, the maximum bit offset
>>> will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
>>> bitfield size will be 256 which is enough for today as maximum
>>> bitfield in compiler can be 128 where int128 type is supported.
>>>
>>> This kernel patch intends to support the new BTF encoding:
>>> $ pahole -JV t.o
>>> [1] TYPEDEF ___int type_id=2
>>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>> [3] ENUM A size=4 vlen=3
>>> A1 val=0
>>> A2 val=1
>>> A3 val=2
>>> [4] STRUCT t kind_flag=1 size=24 vlen=3
>>> a type_id=5 bitfield_size=0 bits_offset=0
>>> b type_id=1 bitfield_size=4 bits_offset=160
>>> c type_id=7 bitfield_size=4 bits_offset=164
>>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>>> [7] VOLATILE (anon) type_id=3
>>>
>>> Issue #2 and solution:
>>> ======================
>>>
>>> Current forward type in BTF does not specify whether the original
>>> type is struct or union. This will not work for type pretty print
>>> and BTF-to-header-file conversion as struct/union must be specified.
>>> $ cat tt.c
>>> struct t;
>>> union u;
>>> int foo(struct t *t, union u *u) { return 0; }
>>> $ gcc -c -g -O2 tt.c
>>> $ pahole -JV tt.o
>>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>> [2] FWD t type_id=0
>>> [3] PTR (anon) type_id=2
>>> [4] FWD u type_id=0
>>> [5] PTR (anon) type_id=4
>>>
>>> To fix this issue, similar to issue #1, type->info bit 31
>>> is used. If the bit is set, it is union type. Otherwise, it is
>>> a struct type.
>>>
>>> $ pahole -JV tt.o
>>> [1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>>> [2] FWD t kind_flag=0 type_id=0
>>> [3] PTR (anon) kind_flag=0 type_id=2
>>> [4] FWD u kind_flag=1 type_id=0
>>> [5] PTR (anon) kind_flag=0 type_id=4
>>>
>>> Pahole/LLVM change:
>>> ===================
>>>
>>> The new kind_flag functionality has been implemented in pahole
>>> and llvm:
>>> https://github.com/yonghong-song/pahole/tree/bitfield
>>> https://github.com/yonghong-song/llvm/tree/bitfield
>>>
>>> Note that pahole hasn't implemented func/func_proto kind
>>> and .BTF.ext. So to print function signature with bpftool,
>>> the llvm compiler should be used.
>>>
>>> Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
>>> Signed-off-by: Martin KaFai Lau <kafai@fb.com>
>>> Signed-off-by: Yonghong Song <yhs@fb.com>
>>> ---
>>> include/uapi/linux/btf.h | 15 ++-
>>> kernel/bpf/btf.c | 274 ++++++++++++++++++++++++++++++++++++---
>>> 2 files changed, 267 insertions(+), 22 deletions(-)
>>>
>>> diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
>>> index 14f66948fc95..34aba40ed926 100644
>>> --- a/include/uapi/linux/btf.h
>>> +++ b/include/uapi/linux/btf.h
>>> @@ -34,7 +34,9 @@ struct btf_type {
>>> * bits 0-15: vlen (e.g. # of struct's members)
>>> * bits 16-23: unused
>>> * bits 24-27: kind (e.g. int, ptr, array...etc)
>>> - * bits 28-31: unused
>>> + * bits 28-30: unused
>>> + * bit 31: kind_flag, currently used by
>>> + * struct, union and fwd
>>> */
>>> __u32 info;
>>> /* "size" is used by INT, ENUM, STRUCT and UNION.
>>> @@ -52,6 +54,7 @@ struct btf_type {
>>>
>>> #define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
>>> #define BTF_INFO_VLEN(info) ((info) & 0xffff)
>>> +#define BTF_INFO_KFLAG(info) ((info) >> 31)
>>>
>>> #define BTF_KIND_UNKN 0 /* Unknown */
>>> #define BTF_KIND_INT 1 /* Integer */
>>> @@ -110,9 +113,17 @@ struct btf_array {
>>> struct btf_member {
>>> __u32 name_off;
>>> __u32 type;
>>> - __u32 offset; /* offset in bits */
>>> + __u32 offset; /* [bitfield_size and] offset in bits */
>>> };
>>>
>>> +/* If the type info kind_flag set, the btf_member.offset
>>> + * contains both member bit offset and bitfield size, and
>>> + * bitfield size will set for struct/union bitfield members.
>>> + * Otherwise, it contains only bit offset.
>>> + */
>> nit. It may be better to move this comment to the btf_member.offset
>> above.
>>
>>> +#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
>>> +#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
>> After re-thinking this setup again, I still think
>> having these macros in btf.h to also do the kflag checking
>> would be nice.
>>
>> Unlike BTF_INFO_KIND() and BTF_INT_ENCODING() which don't
>> depend on other facts, the btf.h raw user must check kflag
>> anyway before calling BTF_MEMBER_BIT*().
>> Forcing a kflag check before the user can access these convenient
>> 0xfffff and >>24 conversions may enforce this kflag check to
>> some extend.
>>
>> Since it is in uapi, it will not be easy to change later.
>> The above concern could be overkill ;), just want to ensure
>> it has been thought through a bit more here.
>>
>> It could be as easy as moving the new btf_member_bit*() from
>> btf.c to here and remove these two macros (or move them back to btf.c).
>
> I think moving:
> +static u32 btf_member_bitfield_size(const struct btf_type *struct_type,
> + const struct btf_member *member)
> +{
> + return btf_type_kflag(struct_type) ? BTF_MEMBER_BITFIELD_SIZE(member->offset)
> + : 0;
> +}
>
> into uapi/btf.h may or may not be useful for btf uapi users.
> What are the chances that these static inline helpers will be
> reused by BTF logic in libbpf or other libs?
> At this point we don't know.
> So I would keep btf.h minimal.
> I agree that BTF_MEMBER_BIT_OFFSET() shouldn't be reused blindly.
> The users have to do BTF_INFO_KFLAG() check first.
> But this is the case for pretty much all of BTF data structures.
> In that sense BTF_MEMBER_BITFIELD_SIZE/BTF_MEMBER_BIT_OFFSET
> serve as a documentation and fit the style of btf.h header.
> imo having these two macros in uapi/btf.h is better than
> replacing them with comment only.
>
> After this set the whole BTF really needs to be documented.
Yes，Martin and I will work on the documentation very soon.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-14 23:34 ` [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
2018-12-15 16:37 ` Martin Lau
@ 2018-12-15 22:37 ` Daniel Borkmann
2018-12-15 23:12 ` Yonghong Song
1 sibling, 1 reply; 30+ messages in thread
From: Daniel Borkmann @ 2018-12-15 22:37 UTC (permalink / raw)
To: Yonghong Song, netdev; +Cc: Alexei Starovoitov, kernel-team, Martin KaFai Lau
On 12/15/2018 12:34 AM, Yonghong Song wrote:
> This patch fixed two issues with BTF. One is related to
> struct/union bitfield encoding and the other is related to
> forward type.
>
> Issue #1 and solution:
> ======================
>
> Current btf encoding of bitfield follows what pahole generates.
> For each bitfield, pahole will duplicate the type chain and
> put the bitfield size at the final int or enum type.
> Since the BTF enum type cannot encode bit size,
> pahole workarounds the issue by generating
> an int type whenever the enum bit size is not 32.
>
> For example,
> -bash-4.4$ cat t.c
> typedef int ___int;
> enum A { A1, A2, A3 };
> struct t {
> int a[5];
> ___int b:4;
> volatile enum A c:4;
> } g;
> -bash-4.4$ gcc -c -O2 -g t.c
> The current kernel supports the following BTF encoding:
> $ pahole -JV t.o
> [1] TYPEDEF ___int type_id=2
> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> [3] ENUM A size=4 vlen=3
> A1 val=0
> A2 val=1
> A3 val=2
> [4] STRUCT t size=24 vlen=3
> a type_id=5 bits_offset=0
> b type_id=9 bits_offset=160
> c type_id=11 bits_offset=164
> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> [7] VOLATILE (anon) type_id=3
> [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
> [9] TYPEDEF ___int type_id=8
> [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
> [11] VOLATILE (anon) type_id=10
>
> Two issues are in the above:
> . by changing enum type to int, we lost the original
> type information and this will not be ideal later
> when we try to convert BTF to a header file.
> . the type duplication for bitfields will cause
> BTF bloat. Duplicated types cannot be deduplicated
> later if the bitfield size is different.
>
> To fix this issue, this patch implemented a compatible
> change for BTF struct type encoding:
> . the bit 31 of struct_type->info, previously reserved,
> now is used to indicate whether bitfield_size is
> encoded in btf_member or not.
> . if bit 31 of struct_type->info is set,
> btf_member->offset will encode like:
> bit 0 - 23: bit offset
> bit 24 - 31: bitfield size
> if bit 31 is not set, the old behavior is preserved:
> bit 0 - 31: bit offset
>
> So if the struct contains a bit field, the maximum bit offset
> will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
> bitfield size will be 256 which is enough for today as maximum
> bitfield in compiler can be 128 where int128 type is supported.
Looks good in general, just small nit below.
> This kernel patch intends to support the new BTF encoding:
> $ pahole -JV t.o
> [1] TYPEDEF ___int type_id=2
> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
> [3] ENUM A size=4 vlen=3
> A1 val=0
> A2 val=1
> A3 val=2
> [4] STRUCT t kind_flag=1 size=24 vlen=3
> a type_id=5 bitfield_size=0 bits_offset=0
> b type_id=1 bitfield_size=4 bits_offset=160
> c type_id=7 bitfield_size=4 bits_offset=164
> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
> [7] VOLATILE (anon) type_id=3
[...]
> +static int btf_int_check_kflag_member(struct btf_verifier_env *env,
> + const struct btf_type *struct_type,
> + const struct btf_member *member,
> + const struct btf_type *member_type)
> +{
> + u32 struct_bits_off, nr_bits, nr_int_data_bits, bytes_offset;
> + u32 int_data = btf_type_int(member_type);
> + u32 struct_size = struct_type->size;
> + u32 nr_copy_bits;
> +
> + /* a regular int type is required for the kflag int member */
> + if (!btf_type_int_is_regular(member_type)) {
> + btf_verifier_log_member(env, struct_type, member,
> + "Invalid member base type");
> + return -EINVAL;
> + }
> +
> + /* check sanity of bitfield size */
> + nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
> + struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
> + nr_int_data_bits = BTF_INT_BITS(int_data);
> + if (!nr_bits) {
> + /* Not a bitfield member, member offset must be at byte
> + * boundary.
> + */
> + if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
> + btf_verifier_log_member(env, struct_type, member,
> + "Invalid member offset");
> + return -EINVAL;
> + }
> +
> + nr_bits = nr_int_data_bits;
> + } else if (nr_bits > nr_int_data_bits) {
Should the test here not include the bit offset as well aka nr_copy_bits?
Thus test would be e.g. (nr_copy_bits > nr_int_data_bits || nr_copy_bits >
BITS_PER_U64) ...
Wrt to future 256 bit support, doesn't UAPI on BTF_INT_BITS() only support
up to max 255 bits?
> + btf_verifier_log_member(env, struct_type, member,
> + "Invalid member bitfield_size");
> + return -EINVAL;
> + }
> +
> + bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
> + nr_copy_bits = nr_bits + BITS_PER_BYTE_MASKED(struct_bits_off);
> + if (nr_copy_bits > BITS_PER_U64) {
> + btf_verifier_log_member(env, struct_type, member,
> + "nr_copy_bits exceeds 64");
> + return -EINVAL;
> + }
> +
> + if (struct_size < bytes_offset ||
> + struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
> + btf_verifier_log_member(env, struct_type, member,
> + "Member exceeds struct_size");
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag
2018-12-15 22:37 ` Daniel Borkmann
@ 2018-12-15 23:12 ` Yonghong Song
0 siblings, 0 replies; 30+ messages in thread
From: Yonghong Song @ 2018-12-15 23:12 UTC (permalink / raw)
To: Daniel Borkmann, netdev@vger.kernel.org
Cc: Alexei Starovoitov, Kernel Team, Martin Lau
On 12/15/18 2:37 PM, Daniel Borkmann wrote:
> On 12/15/2018 12:34 AM, Yonghong Song wrote:
>> This patch fixed two issues with BTF. One is related to
>> struct/union bitfield encoding and the other is related to
>> forward type.
>>
>> Issue #1 and solution:
>> ======================
>>
>> Current btf encoding of bitfield follows what pahole generates.
>> For each bitfield, pahole will duplicate the type chain and
>> put the bitfield size at the final int or enum type.
>> Since the BTF enum type cannot encode bit size,
>> pahole workarounds the issue by generating
>> an int type whenever the enum bit size is not 32.
>>
>> For example,
>> -bash-4.4$ cat t.c
>> typedef int ___int;
>> enum A { A1, A2, A3 };
>> struct t {
>> int a[5];
>> ___int b:4;
>> volatile enum A c:4;
>> } g;
>> -bash-4.4$ gcc -c -O2 -g t.c
>> The current kernel supports the following BTF encoding:
>> $ pahole -JV t.o
>> [1] TYPEDEF ___int type_id=2
>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>> [3] ENUM A size=4 vlen=3
>> A1 val=0
>> A2 val=1
>> A3 val=2
>> [4] STRUCT t size=24 vlen=3
>> a type_id=5 bits_offset=0
>> b type_id=9 bits_offset=160
>> c type_id=11 bits_offset=164
>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>> [7] VOLATILE (anon) type_id=3
>> [8] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
>> [9] TYPEDEF ___int type_id=8
>> [10] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
>> [11] VOLATILE (anon) type_id=10
>>
>> Two issues are in the above:
>> . by changing enum type to int, we lost the original
>> type information and this will not be ideal later
>> when we try to convert BTF to a header file.
>> . the type duplication for bitfields will cause
>> BTF bloat. Duplicated types cannot be deduplicated
>> later if the bitfield size is different.
>>
>> To fix this issue, this patch implemented a compatible
>> change for BTF struct type encoding:
>> . the bit 31 of struct_type->info, previously reserved,
>> now is used to indicate whether bitfield_size is
>> encoded in btf_member or not.
>> . if bit 31 of struct_type->info is set,
>> btf_member->offset will encode like:
>> bit 0 - 23: bit offset
>> bit 24 - 31: bitfield size
>> if bit 31 is not set, the old behavior is preserved:
>> bit 0 - 31: bit offset
>>
>> So if the struct contains a bit field, the maximum bit offset
>> will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
>> bitfield size will be 256 which is enough for today as maximum
>> bitfield in compiler can be 128 where int128 type is supported.
>
> Looks good in general, just small nit below.
>
>> This kernel patch intends to support the new BTF encoding:
>> $ pahole -JV t.o
>> [1] TYPEDEF ___int type_id=2
>> [2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
>> [3] ENUM A size=4 vlen=3
>> A1 val=0
>> A2 val=1
>> A3 val=2
>> [4] STRUCT t kind_flag=1 size=24 vlen=3
>> a type_id=5 bitfield_size=0 bits_offset=0
>> b type_id=1 bitfield_size=4 bits_offset=160
>> c type_id=7 bitfield_size=4 bits_offset=164
>> [5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
>> [6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
>> [7] VOLATILE (anon) type_id=3
> [...]
>> +static int btf_int_check_kflag_member(struct btf_verifier_env *env,
>> + const struct btf_type *struct_type,
>> + const struct btf_member *member,
>> + const struct btf_type *member_type)
>> +{
>> + u32 struct_bits_off, nr_bits, nr_int_data_bits, bytes_offset;
>> + u32 int_data = btf_type_int(member_type);
>> + u32 struct_size = struct_type->size;
>> + u32 nr_copy_bits;
>> +
>> + /* a regular int type is required for the kflag int member */
>> + if (!btf_type_int_is_regular(member_type)) {
>> + btf_verifier_log_member(env, struct_type, member,
>> + "Invalid member base type");
>> + return -EINVAL;
>> + }
>> +
>> + /* check sanity of bitfield size */
>> + nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
>> + struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
>> + nr_int_data_bits = BTF_INT_BITS(int_data);
>> + if (!nr_bits) {
>> + /* Not a bitfield member, member offset must be at byte
>> + * boundary.
>> + */
>> + if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
>> + btf_verifier_log_member(env, struct_type, member,
>> + "Invalid member offset");
>> + return -EINVAL;
>> + }
>> +
>> + nr_bits = nr_int_data_bits;
>> + } else if (nr_bits > nr_int_data_bits) {
>
> Should the test here not include the bit offset as well aka nr_copy_bits?
> Thus test would be e.g. (nr_copy_bits > nr_int_data_bits || nr_copy_bits >
> BITS_PER_U64) ...
The test here is strictly checking whether the bitfield size is covered
by underlying int type.
struct t {
int a:1;
char b:8;
...
}
In this case, for member "b", bitsize = 8, nr_copy_bits = 9,
nr_int_data_bits = 8. nr_copy_bits > nr_int_data_bits, but it is legal.
> Wrt to future 256 bit support, doesn't UAPI on BTF_INT_BITS() only support
> up to max 255 bits?
This is a good question. BTF_INT_BITS() can be extended to maximum
0xffff. There are bits available there.
When the kind_flag is set, the bitfield size can range from 1 to 255,
the same as today BTF_INT_BITS range. If user writes
struct t {
...
int256 m:256;
...
};
The bitfield size information will not get encoded and the member "m"
will just refer to a base type int256, the same as
struct t {
...
int256 m;
...
};
Did you any problem with this?
>
>> + btf_verifier_log_member(env, struct_type, member,
>> + "Invalid member bitfield_size");
>> + return -EINVAL;
>> + }
>> +
>> + bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
>> + nr_copy_bits = nr_bits + BITS_PER_BYTE_MASKED(struct_bits_off);
>> + if (nr_copy_bits > BITS_PER_U64) {
>> + btf_verifier_log_member(env, struct_type, member,
>> + "nr_copy_bits exceeds 64");
>> + return -EINVAL;
>> + }
>> +
>> + if (struct_size < bytes_offset ||
>> + struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
>> + btf_verifier_log_member(env, struct_type, member,
>> + "Member exceeds struct_size");
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH bpf-next v2 3/8] bpf: enable cgroup local storage map pretty print with kind_flag
2018-12-14 23:34 [PATCH bpf-next v2 0/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
2018-12-14 23:34 ` [PATCH bpf-next v2 1/8] bpf: btf: refactor btf_int_bits_seq_show() Yonghong Song
2018-12-14 23:34 ` [PATCH bpf-next v2 2/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
@ 2018-12-14 23:34 ` Yonghong Song
2018-12-15 16:43 ` Martin Lau
2018-12-14 23:34 ` [PATCH bpf-next v2 4/8] tools/bpf: sync btf.h header from kernel to tools Yonghong Song
` (4 subsequent siblings)
7 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-14 23:34 UTC (permalink / raw)
To: netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team,
Martin KaFai Lau
Commit 970289fc0a83 ("bpf: add bpffs pretty print for cgroup
local storage maps") added bpffs pretty print for cgroup
local storage maps. The commit worked for struct without kind_flag
set.
This patch refactored and made pretty print also work
with kind_flag set for the struct.
Signed-off-by: Yonghong Song <yhs@fb.com>
---
include/linux/btf.h | 5 ++++-
kernel/bpf/btf.c | 37 ++++++++++++++++++++++++++++---------
kernel/bpf/local_storage.c | 17 ++++-------------
3 files changed, 36 insertions(+), 23 deletions(-)
diff --git a/include/linux/btf.h b/include/linux/btf.h
index 58000d7e06e3..12502e25e767 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -7,6 +7,7 @@
#include <linux/types.h>
struct btf;
+struct btf_member;
struct btf_type;
union bpf_attr;
@@ -46,7 +47,9 @@ void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
struct seq_file *m);
int btf_get_fd_by_id(u32 id);
u32 btf_id(const struct btf *btf);
-bool btf_type_is_reg_int(const struct btf_type *t, u32 expected_size);
+bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
+ const struct btf_member *m,
+ u32 expected_offset, u32 expected_size);
#ifdef CONFIG_BPF_SYSCALL
const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index ec3f80d3bef6..30028fa187df 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -546,22 +546,41 @@ static bool btf_type_int_is_regular(const struct btf_type *t)
}
/*
- * Check that given type is a regular int and has the expected size.
+ * Check that given struct member is a regular int with expected
+ * offset and size.
*/
-bool btf_type_is_reg_int(const struct btf_type *t, u32 expected_size)
+bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
+ const struct btf_member *m,
+ u32 expected_offset, u32 expected_size)
{
- u8 nr_bits, nr_bytes;
- u32 int_data;
+ const struct btf_type *t;
+ u32 id, int_data;
+ u8 nr_bits;
- if (!btf_type_is_int(t))
+ id = m->type;
+ t = btf_type_id_size(btf, &id, NULL);
+ if (!t || !btf_type_is_int(t))
return false;
int_data = btf_type_int(t);
nr_bits = BTF_INT_BITS(int_data);
- nr_bytes = BITS_ROUNDUP_BYTES(nr_bits);
- if (BITS_PER_BYTE_MASKED(nr_bits) ||
- BTF_INT_OFFSET(int_data) ||
- nr_bytes != expected_size)
+ if (btf_type_kflag(s)) {
+ u32 bitfield_size = BTF_MEMBER_BITFIELD_SIZE(m->offset);
+ u32 bit_offset = BTF_MEMBER_BIT_OFFSET(m->offset);
+
+ /* if kflag set, int should be a regular int and
+ * bit offset should be at byte boundary.
+ */
+ return !bitfield_size &&
+ BITS_ROUNDUP_BYTES(bit_offset) == expected_offset &&
+ BITS_ROUNDUP_BYTES(nr_bits) == expected_size;
+ }
+
+ if (BTF_INT_OFFSET(int_data) ||
+ BITS_PER_BYTE_MASKED(m->offset) ||
+ BITS_ROUNDUP_BYTES(m->offset) != expected_offset ||
+ BITS_PER_BYTE_MASKED(nr_bits) ||
+ BITS_ROUNDUP_BYTES(nr_bits) != expected_size)
return false;
return true;
diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c
index 5eca03da0989..07a34ef562a0 100644
--- a/kernel/bpf/local_storage.c
+++ b/kernel/bpf/local_storage.c
@@ -315,9 +315,8 @@ static int cgroup_storage_check_btf(const struct bpf_map *map,
const struct btf_type *key_type,
const struct btf_type *value_type)
{
- const struct btf_type *t;
struct btf_member *m;
- u32 id, size;
+ u32 offset, size;
/* Key is expected to be of struct bpf_cgroup_storage_key type,
* which is:
@@ -338,25 +337,17 @@ static int cgroup_storage_check_btf(const struct bpf_map *map,
* The first field must be a 64 bit integer at 0 offset.
*/
m = (struct btf_member *)(key_type + 1);
- if (m->offset)
- return -EINVAL;
- id = m->type;
- t = btf_type_id_size(btf, &id, NULL);
size = FIELD_SIZEOF(struct bpf_cgroup_storage_key, cgroup_inode_id);
- if (!t || !btf_type_is_reg_int(t, size))
+ if (!btf_member_is_reg_int(btf, key_type, m, 0, size))
return -EINVAL;
/*
* The second field must be a 32 bit integer at 64 bit offset.
*/
m++;
- if (m->offset != offsetof(struct bpf_cgroup_storage_key, attach_type) *
- BITS_PER_BYTE)
- return -EINVAL;
- id = m->type;
- t = btf_type_id_size(btf, &id, NULL);
+ offset = offsetof(struct bpf_cgroup_storage_key, attach_type);
size = FIELD_SIZEOF(struct bpf_cgroup_storage_key, attach_type);
- if (!t || !btf_type_is_reg_int(t, size))
+ if (!btf_member_is_reg_int(btf, key_type, m, offset, size))
return -EINVAL;
return 0;
--
2.17.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 3/8] bpf: enable cgroup local storage map pretty print with kind_flag
2018-12-14 23:34 ` [PATCH bpf-next v2 3/8] bpf: enable cgroup local storage map pretty print " Yonghong Song
@ 2018-12-15 16:43 ` Martin Lau
0 siblings, 0 replies; 30+ messages in thread
From: Martin Lau @ 2018-12-15 16:43 UTC (permalink / raw)
To: Yonghong Song
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On Fri, Dec 14, 2018 at 03:34:28PM -0800, Yonghong Song wrote:
> Commit 970289fc0a83 ("bpf: add bpffs pretty print for cgroup
> local storage maps") added bpffs pretty print for cgroup
> local storage maps. The commit worked for struct without kind_flag
> set.
>
> This patch refactored and made pretty print also work
> with kind_flag set for the struct.
>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
> include/linux/btf.h | 5 ++++-
> kernel/bpf/btf.c | 37 ++++++++++++++++++++++++++++---------
> kernel/bpf/local_storage.c | 17 ++++-------------
> 3 files changed, 36 insertions(+), 23 deletions(-)
>
> diff --git a/include/linux/btf.h b/include/linux/btf.h
> index 58000d7e06e3..12502e25e767 100644
> --- a/include/linux/btf.h
> +++ b/include/linux/btf.h
> @@ -7,6 +7,7 @@
> #include <linux/types.h>
>
> struct btf;
> +struct btf_member;
> struct btf_type;
> union bpf_attr;
>
> @@ -46,7 +47,9 @@ void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
> struct seq_file *m);
> int btf_get_fd_by_id(u32 id);
> u32 btf_id(const struct btf *btf);
> -bool btf_type_is_reg_int(const struct btf_type *t, u32 expected_size);
> +bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
> + const struct btf_member *m,
> + u32 expected_offset, u32 expected_size);
>
> #ifdef CONFIG_BPF_SYSCALL
> const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index ec3f80d3bef6..30028fa187df 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
> @@ -546,22 +546,41 @@ static bool btf_type_int_is_regular(const struct btf_type *t)
> }
>
> /*
> - * Check that given type is a regular int and has the expected size.
> + * Check that given struct member is a regular int with expected
> + * offset and size.
> */
> -bool btf_type_is_reg_int(const struct btf_type *t, u32 expected_size)
> +bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
> + const struct btf_member *m,
> + u32 expected_offset, u32 expected_size)
> {
> - u8 nr_bits, nr_bytes;
> - u32 int_data;
> + const struct btf_type *t;
> + u32 id, int_data;
> + u8 nr_bits;
>
> - if (!btf_type_is_int(t))
> + id = m->type;
> + t = btf_type_id_size(btf, &id, NULL);
> + if (!t || !btf_type_is_int(t))
> return false;
>
> int_data = btf_type_int(t);
> nr_bits = BTF_INT_BITS(int_data);
> - nr_bytes = BITS_ROUNDUP_BYTES(nr_bits);
> - if (BITS_PER_BYTE_MASKED(nr_bits) ||
> - BTF_INT_OFFSET(int_data) ||
> - nr_bytes != expected_size)
> + if (btf_type_kflag(s)) {
> + u32 bitfield_size = BTF_MEMBER_BITFIELD_SIZE(m->offset);
> + u32 bit_offset = BTF_MEMBER_BIT_OFFSET(m->offset);
> +
> + /* if kflag set, int should be a regular int and
> + * bit offset should be at byte boundary.
> + */
> + return !bitfield_size &&
Make sense. When kflag == true, by design, the member is a reg int only if
bitfield_size == 0. The BTF verifier has already ensured everything else.
Acked-by: Martin KaFai Lau <kafai@fb.com>
> + BITS_ROUNDUP_BYTES(bit_offset) == expected_offset &&
> + BITS_ROUNDUP_BYTES(nr_bits) == expected_size;
> + }
> +
> + if (BTF_INT_OFFSET(int_data) ||
> + BITS_PER_BYTE_MASKED(m->offset) ||
> + BITS_ROUNDUP_BYTES(m->offset) != expected_offset ||
> + BITS_PER_BYTE_MASKED(nr_bits) ||
> + BITS_ROUNDUP_BYTES(nr_bits) != expected_size)
> return false;
>
> return true;
> diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c
> index 5eca03da0989..07a34ef562a0 100644
> --- a/kernel/bpf/local_storage.c
> +++ b/kernel/bpf/local_storage.c
> @@ -315,9 +315,8 @@ static int cgroup_storage_check_btf(const struct bpf_map *map,
> const struct btf_type *key_type,
> const struct btf_type *value_type)
> {
> - const struct btf_type *t;
> struct btf_member *m;
> - u32 id, size;
> + u32 offset, size;
>
> /* Key is expected to be of struct bpf_cgroup_storage_key type,
> * which is:
> @@ -338,25 +337,17 @@ static int cgroup_storage_check_btf(const struct bpf_map *map,
> * The first field must be a 64 bit integer at 0 offset.
> */
> m = (struct btf_member *)(key_type + 1);
> - if (m->offset)
> - return -EINVAL;
> - id = m->type;
> - t = btf_type_id_size(btf, &id, NULL);
> size = FIELD_SIZEOF(struct bpf_cgroup_storage_key, cgroup_inode_id);
> - if (!t || !btf_type_is_reg_int(t, size))
> + if (!btf_member_is_reg_int(btf, key_type, m, 0, size))
> return -EINVAL;
>
> /*
> * The second field must be a 32 bit integer at 64 bit offset.
> */
> m++;
> - if (m->offset != offsetof(struct bpf_cgroup_storage_key, attach_type) *
> - BITS_PER_BYTE)
> - return -EINVAL;
> - id = m->type;
> - t = btf_type_id_size(btf, &id, NULL);
> + offset = offsetof(struct bpf_cgroup_storage_key, attach_type);
> size = FIELD_SIZEOF(struct bpf_cgroup_storage_key, attach_type);
> - if (!t || !btf_type_is_reg_int(t, size))
> + if (!btf_member_is_reg_int(btf, key_type, m, offset, size))
> return -EINVAL;
>
> return 0;
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH bpf-next v2 4/8] tools/bpf: sync btf.h header from kernel to tools
2018-12-14 23:34 [PATCH bpf-next v2 0/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
` (2 preceding siblings ...)
2018-12-14 23:34 ` [PATCH bpf-next v2 3/8] bpf: enable cgroup local storage map pretty print " Yonghong Song
@ 2018-12-14 23:34 ` Yonghong Song
2018-12-15 20:56 ` Martin Lau
2018-12-14 23:34 ` [PATCH bpf-next v2 5/8] tools/bpf: add test_btf unit tests for kind_flag Yonghong Song
` (3 subsequent siblings)
7 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-14 23:34 UTC (permalink / raw)
To: netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team,
Martin KaFai Lau
Sync include/uapi/linux/btf.h to tools/include/uapi/linux/btf.h.
Signed-off-by: Yonghong Song <yhs@fb.com>
---
tools/include/uapi/linux/btf.h | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/tools/include/uapi/linux/btf.h b/tools/include/uapi/linux/btf.h
index 14f66948fc95..34aba40ed926 100644
--- a/tools/include/uapi/linux/btf.h
+++ b/tools/include/uapi/linux/btf.h
@@ -34,7 +34,9 @@ struct btf_type {
* bits 0-15: vlen (e.g. # of struct's members)
* bits 16-23: unused
* bits 24-27: kind (e.g. int, ptr, array...etc)
- * bits 28-31: unused
+ * bits 28-30: unused
+ * bit 31: kind_flag, currently used by
+ * struct, union and fwd
*/
__u32 info;
/* "size" is used by INT, ENUM, STRUCT and UNION.
@@ -52,6 +54,7 @@ struct btf_type {
#define BTF_INFO_KIND(info) (((info) >> 24) & 0x0f)
#define BTF_INFO_VLEN(info) ((info) & 0xffff)
+#define BTF_INFO_KFLAG(info) ((info) >> 31)
#define BTF_KIND_UNKN 0 /* Unknown */
#define BTF_KIND_INT 1 /* Integer */
@@ -110,9 +113,17 @@ struct btf_array {
struct btf_member {
__u32 name_off;
__u32 type;
- __u32 offset; /* offset in bits */
+ __u32 offset; /* [bitfield_size and] offset in bits */
};
+/* If the type info kind_flag set, the btf_member.offset
+ * contains both member bit offset and bitfield size, and
+ * bitfield size will set for struct/union bitfield members.
+ * Otherwise, it contains only bit offset.
+ */
+#define BTF_MEMBER_BITFIELD_SIZE(val) ((val) >> 24)
+#define BTF_MEMBER_BIT_OFFSET(val) ((val) & 0xffffff)
+
/* BTF_KIND_FUNC_PROTO is followed by multiple "struct btf_param".
* The exact number of btf_param is stored in the vlen (of the
* info in "struct btf_type").
--
2.17.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* [PATCH bpf-next v2 5/8] tools/bpf: add test_btf unit tests for kind_flag
2018-12-14 23:34 [PATCH bpf-next v2 0/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
` (3 preceding siblings ...)
2018-12-14 23:34 ` [PATCH bpf-next v2 4/8] tools/bpf: sync btf.h header from kernel to tools Yonghong Song
@ 2018-12-14 23:34 ` Yonghong Song
2018-12-15 21:03 ` Martin Lau
2018-12-14 23:34 ` [PATCH bpf-next v2 6/8] tools/bpf: test kernel bpffs map pretty print with struct kind_flag Yonghong Song
` (2 subsequent siblings)
7 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-14 23:34 UTC (permalink / raw)
To: netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team,
Martin KaFai Lau
This patch added unit tests for different types handling
type->info.kind_flag. The following new tests are added:
$ test_btf
...
BTF raw test[82] (invalid int kind_flag): OK
BTF raw test[83] (invalid ptr kind_flag): OK
BTF raw test[84] (invalid array kind_flag): OK
BTF raw test[85] (invalid enum kind_flag): OK
BTF raw test[86] (valid fwd kind_flag): OK
BTF raw test[87] (invalid typedef kind_flag): OK
BTF raw test[88] (invalid volatile kind_flag): OK
BTF raw test[89] (invalid const kind_flag): OK
BTF raw test[90] (invalid restrict kind_flag): OK
BTF raw test[91] (invalid func kind_flag): OK
BTF raw test[92] (invalid func_proto kind_flag): OK
BTF raw test[93] (valid struct kind_flag, bitfield_size = 0): OK
BTF raw test[94] (valid struct kind_flag, int member, bitfield_size != 0): OK
BTF raw test[95] (valid union kind_flag, int member, bitfield_size != 0): OK
BTF raw test[96] (valid struct kind_flag, enum member, bitfield_size != 0): OK
BTF raw test[97] (valid union kind_flag, enum member, bitfield_size != 0): OK
BTF raw test[98] (valid struct kind_flag, typedef member, bitfield_size != 0): OK
BTF raw test[99] (valid union kind_flag, typedef member, bitfield_size != 0): OK
BTF raw test[100] (invalid struct type, bitfield_size greater than struct size): OK
BTF raw test[101] (invalid struct type, kind_flag bitfield base_type int not regular): OK
BTF raw test[102] (invalid struct type, kind_flag base_type int not regular): OK
BTF raw test[103] (invalid union type, bitfield_size greater than struct size): OK
...
PASS:122 SKIP:0 FAIL:0
The second parameter name of macro
BTF_INFO_ENC(kind, root, vlen)
in selftests test_btf.c is also renamed from "root" to "kind_flag".
Note that before this patch "root" is not used and always 0.
Signed-off-by: Yonghong Song <yhs@fb.com>
---
tools/testing/selftests/bpf/test_btf.c | 450 ++++++++++++++++++++++++-
1 file changed, 448 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/bpf/test_btf.c b/tools/testing/selftests/bpf/test_btf.c
index 8478316aaf9a..0108b7f8a184 100644
--- a/tools/testing/selftests/bpf/test_btf.c
+++ b/tools/testing/selftests/bpf/test_btf.c
@@ -65,8 +65,8 @@ static int __base_pr(const char *format, ...)
return err;
}
-#define BTF_INFO_ENC(kind, root, vlen) \
- ((!!(root) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
+#define BTF_INFO_ENC(kind, kind_flag, vlen) \
+ ((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
#define BTF_TYPE_ENC(name, info, size_or_type) \
(name), (info), (size_or_type)
@@ -86,6 +86,8 @@ static int __base_pr(const char *format, ...)
#define BTF_MEMBER_ENC(name, type, bits_offset) \
(name), (type), (bits_offset)
#define BTF_ENUM_ENC(name, val) (name), (val)
+#define BTF_MEMBER_OFFSET(bitfield_size, bits_offset) \
+ ((bitfield_size) << 24 | (bits_offset))
#define BTF_TYPEDEF_ENC(name, type) \
BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_TYPEDEF, 0, 0), type)
@@ -2215,6 +2217,450 @@ static struct btf_raw_test raw_tests[] = {
.err_str = "Invalid type_id",
},
+{
+ .descr = "invalid int kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_INT, 1, 0), 4), /* [2] */
+ BTF_INT_ENC(0, 0, 32),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC(""),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "int_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "invalid ptr kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_PTR, 1, 0), 1), /* [2] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC(""),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "ptr_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "invalid array kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ARRAY, 1, 0), 0), /* [2] */
+ BTF_ARRAY_ENC(1, 1, 1),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC(""),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "array_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "invalid enum kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ENUM, 1, 1), 4), /* [2] */
+ BTF_ENUM_ENC(NAME_TBD, 0),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "enum_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "valid fwd kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(NAME_TBD,
+ BTF_INFO_ENC(BTF_KIND_FWD, 1, 0), 0), /* [2] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "fwd_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+},
+
+{
+ .descr = "invalid typedef kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(NAME_TBD,
+ BTF_INFO_ENC(BTF_KIND_TYPEDEF, 1, 0), 1), /* [2] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "typedef_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "invalid volatile kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_VOLATILE, 1, 0), 1), /* [2] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC(""),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "volatile_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "invalid const kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_CONST, 1, 0), 1), /* [2] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC(""),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "const_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "invalid restrict kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_RESTRICT, 1, 0), 1), /* [2] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC(""),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "restrict_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "invalid func kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_FUNC_PROTO, 0, 0), 0), /* [2] */
+ BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_FUNC, 1, 0), 2), /* [3] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "func_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "invalid func_proto kind_flag",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_FUNC_PROTO, 1, 0), 0), /* [2] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC(""),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "func_proto_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid btf_info kind_flag",
+},
+
+{
+ .descr = "valid struct kind_flag, bitfield_size = 0",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 2), 8), /* [2] */
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(0, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(0, 32)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "struct_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+},
+
+{
+ .descr = "valid struct kind_flag, int member, bitfield_size != 0",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 2), 4), /* [2] */
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(4, 4)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "struct_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+},
+
+{
+ .descr = "valid union kind_flag, int member, bitfield_size != 0",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_UNION, 1, 2), 4), /* [2] */
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "union_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+},
+
+{
+ .descr = "valid struct kind_flag, enum member, bitfield_size != 0",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ENUM, 0, 1), 4), /* [2] */
+ BTF_ENUM_ENC(NAME_TBD, 0),
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 2), 4),/* [3] */
+ BTF_MEMBER_ENC(NAME_TBD, 2, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 2, BTF_MEMBER_OFFSET(4, 4)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B\0C"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "struct_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+},
+
+{
+ .descr = "valid union kind_flag, enum member, bitfield_size != 0",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ENUM, 0, 1), 4), /* [2] */
+ BTF_ENUM_ENC(NAME_TBD, 0),
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_UNION, 1, 2), 4), /* [3] */
+ BTF_MEMBER_ENC(NAME_TBD, 2, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 2, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B\0C"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "union_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+},
+
+{
+ .descr = "valid struct kind_flag, typedef member, bitfield_size != 0",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ENUM, 0, 1), 4), /* [2] */
+ BTF_ENUM_ENC(NAME_TBD, 0),
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 2), 4),/* [3] */
+ BTF_MEMBER_ENC(NAME_TBD, 4, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 5, BTF_MEMBER_OFFSET(4, 4)),
+ BTF_TYPEDEF_ENC(NAME_TBD, 1), /* [4] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 2), /* [5] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B\0C\0D\0E"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "struct_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+},
+
+{
+ .descr = "valid union kind_flag, typedef member, bitfield_size != 0",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ENUM, 0, 1), 4), /* [2] */
+ BTF_ENUM_ENC(NAME_TBD, 0),
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_UNION, 1, 2), 4), /* [3] */
+ BTF_MEMBER_ENC(NAME_TBD, 4, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 5, BTF_MEMBER_OFFSET(4, 0)),
+ BTF_TYPEDEF_ENC(NAME_TBD, 1), /* [4] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 2), /* [5] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B\0C\0D\0E"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "union_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+},
+
+{
+ .descr = "invalid struct type, bitfield_size greater than struct size",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 2), 4), /* [2] */
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(20, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(20, 20)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "struct_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Member exceeds struct_size",
+},
+
+{
+ .descr = "invalid struct type, kind_flag bitfield base_type int not regular",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 20, 4), /* [2] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 2), 4), /* [3] */
+ BTF_MEMBER_ENC(NAME_TBD, 2, BTF_MEMBER_OFFSET(20, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 2, BTF_MEMBER_OFFSET(20, 20)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "struct_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid member base type",
+},
+
+{
+ .descr = "invalid struct type, kind_flag base_type int not regular",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 12, 4), /* [2] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 2), 4), /* [3] */
+ BTF_MEMBER_ENC(NAME_TBD, 2, BTF_MEMBER_OFFSET(8, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 2, BTF_MEMBER_OFFSET(8, 8)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "struct_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Invalid member base type",
+},
+
+{
+ .descr = "invalid union type, bitfield_size greater than struct size",
+ .raw_types = {
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_UNION, 1, 2), 2), /* [2] */
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(8, 0)),
+ BTF_MEMBER_ENC(NAME_TBD, 1, BTF_MEMBER_OFFSET(20, 0)),
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0A\0B"),
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .map_name = "union_type_check_btf",
+ .key_size = sizeof(int),
+ .value_size = sizeof(int),
+ .key_type_id = 1,
+ .value_type_id = 1,
+ .max_entries = 4,
+ .btf_load_err = true,
+ .err_str = "Member exceeds struct_size",
+},
+
}; /* struct btf_raw_test raw_tests[] */
static const char *get_next_str(const char *start, const char *end)
--
2.17.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 5/8] tools/bpf: add test_btf unit tests for kind_flag
2018-12-14 23:34 ` [PATCH bpf-next v2 5/8] tools/bpf: add test_btf unit tests for kind_flag Yonghong Song
@ 2018-12-15 21:03 ` Martin Lau
2018-12-16 4:10 ` Yonghong Song
0 siblings, 1 reply; 30+ messages in thread
From: Martin Lau @ 2018-12-15 21:03 UTC (permalink / raw)
To: Yonghong Song
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On Fri, Dec 14, 2018 at 03:34:30PM -0800, Yonghong Song wrote:
> This patch added unit tests for different types handling
> type->info.kind_flag. The following new tests are added:
> $ test_btf
> ...
> BTF raw test[82] (invalid int kind_flag): OK
> BTF raw test[83] (invalid ptr kind_flag): OK
> BTF raw test[84] (invalid array kind_flag): OK
> BTF raw test[85] (invalid enum kind_flag): OK
> BTF raw test[86] (valid fwd kind_flag): OK
> BTF raw test[87] (invalid typedef kind_flag): OK
> BTF raw test[88] (invalid volatile kind_flag): OK
> BTF raw test[89] (invalid const kind_flag): OK
> BTF raw test[90] (invalid restrict kind_flag): OK
> BTF raw test[91] (invalid func kind_flag): OK
> BTF raw test[92] (invalid func_proto kind_flag): OK
> BTF raw test[93] (valid struct kind_flag, bitfield_size = 0): OK
> BTF raw test[94] (valid struct kind_flag, int member, bitfield_size != 0): OK
> BTF raw test[95] (valid union kind_flag, int member, bitfield_size != 0): OK
> BTF raw test[96] (valid struct kind_flag, enum member, bitfield_size != 0): OK
> BTF raw test[97] (valid union kind_flag, enum member, bitfield_size != 0): OK
> BTF raw test[98] (valid struct kind_flag, typedef member, bitfield_size != 0): OK
> BTF raw test[99] (valid union kind_flag, typedef member, bitfield_size != 0): OK
> BTF raw test[100] (invalid struct type, bitfield_size greater than struct size): OK
> BTF raw test[101] (invalid struct type, kind_flag bitfield base_type int not regular): OK
> BTF raw test[102] (invalid struct type, kind_flag base_type int not regular): OK
> BTF raw test[103] (invalid union type, bitfield_size greater than struct size): OK
Would it be useful to add some
"struct kind_flag, (int|enum) member, bitfield_size == 0, wrong byte alignment"
tests to catch the "!nr_bits && BITS_PER_BYTE_MASKED(struct_bits_off)"
in patch 2. The extra tests could be a follow up though.
Or the above tests have covered that already?
> ...
> PASS:122 SKIP:0 FAIL:0
>
> The second parameter name of macro
> BTF_INFO_ENC(kind, root, vlen)
> in selftests test_btf.c is also renamed from "root" to "kind_flag".
> Note that before this patch "root" is not used and always 0.
For the tests:
Acked-by: Martin KaFai Lau <kafai@fb.com>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH bpf-next v2 5/8] tools/bpf: add test_btf unit tests for kind_flag
2018-12-15 21:03 ` Martin Lau
@ 2018-12-16 4:10 ` Yonghong Song
0 siblings, 0 replies; 30+ messages in thread
From: Yonghong Song @ 2018-12-16 4:10 UTC (permalink / raw)
To: Martin Lau
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On 12/15/18 1:03 PM, Martin Lau wrote:
> On Fri, Dec 14, 2018 at 03:34:30PM -0800, Yonghong Song wrote:
>> This patch added unit tests for different types handling
>> type->info.kind_flag. The following new tests are added:
>> $ test_btf
>> ...
>> BTF raw test[82] (invalid int kind_flag): OK
>> BTF raw test[83] (invalid ptr kind_flag): OK
>> BTF raw test[84] (invalid array kind_flag): OK
>> BTF raw test[85] (invalid enum kind_flag): OK
>> BTF raw test[86] (valid fwd kind_flag): OK
>> BTF raw test[87] (invalid typedef kind_flag): OK
>> BTF raw test[88] (invalid volatile kind_flag): OK
>> BTF raw test[89] (invalid const kind_flag): OK
>> BTF raw test[90] (invalid restrict kind_flag): OK
>> BTF raw test[91] (invalid func kind_flag): OK
>> BTF raw test[92] (invalid func_proto kind_flag): OK
>> BTF raw test[93] (valid struct kind_flag, bitfield_size = 0): OK
>> BTF raw test[94] (valid struct kind_flag, int member, bitfield_size != 0): OK
>> BTF raw test[95] (valid union kind_flag, int member, bitfield_size != 0): OK
>> BTF raw test[96] (valid struct kind_flag, enum member, bitfield_size != 0): OK
>> BTF raw test[97] (valid union kind_flag, enum member, bitfield_size != 0): OK
>> BTF raw test[98] (valid struct kind_flag, typedef member, bitfield_size != 0): OK
>> BTF raw test[99] (valid union kind_flag, typedef member, bitfield_size != 0): OK
>> BTF raw test[100] (invalid struct type, bitfield_size greater than struct size): OK
>> BTF raw test[101] (invalid struct type, kind_flag bitfield base_type int not regular): OK
>> BTF raw test[102] (invalid struct type, kind_flag base_type int not regular): OK
>> BTF raw test[103] (invalid union type, bitfield_size greater than struct size): OK
> Would it be useful to add some
> "struct kind_flag, (int|enum) member, bitfield_size == 0, wrong byte alignment"
> tests to catch the "!nr_bits && BITS_PER_BYTE_MASKED(struct_bits_off)"
> in patch 2. The extra tests could be a follow up though.
>
> Or the above tests have covered that already?
No. They are not covered. I will add tests for these two cases.
>
>> ...
>> PASS:122 SKIP:0 FAIL:0
>>
>> The second parameter name of macro
>> BTF_INFO_ENC(kind, root, vlen)
>> in selftests test_btf.c is also renamed from "root" to "kind_flag".
>> Note that before this patch "root" is not used and always 0.
> For the tests:
>
> Acked-by: Martin KaFai Lau <kafai@fb.com>
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH bpf-next v2 6/8] tools/bpf: test kernel bpffs map pretty print with struct kind_flag
2018-12-14 23:34 [PATCH bpf-next v2 0/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
` (4 preceding siblings ...)
2018-12-14 23:34 ` [PATCH bpf-next v2 5/8] tools/bpf: add test_btf unit tests for kind_flag Yonghong Song
@ 2018-12-14 23:34 ` Yonghong Song
2018-12-15 21:18 ` Martin Lau
2018-12-14 23:34 ` [PATCH bpf-next v2 7/8] tools: bpftool: refactor btf_dumper_int_bits() Yonghong Song
2018-12-14 23:34 ` [PATCH bpf-next v2 8/8] tools: bpftool: support pretty print with kind_flag set Yonghong Song
7 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-14 23:34 UTC (permalink / raw)
To: netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team,
Martin KaFai Lau
The new tests are added to test bpffs map pretty print in kernel with kind_flag
for structure type.
$ test_btf -p
......
BTF pretty print array(#1)......OK
BTF pretty print array(#2)......OK
PASS:8 SKIP:0 FAIL:0
Signed-off-by: Yonghong Song <yhs@fb.com>
---
tools/testing/selftests/bpf/test_btf.c | 168 ++++++++++++++++++++++---
1 file changed, 154 insertions(+), 14 deletions(-)
diff --git a/tools/testing/selftests/bpf/test_btf.c b/tools/testing/selftests/bpf/test_btf.c
index 0108b7f8a184..51f53d5dbd55 100644
--- a/tools/testing/selftests/bpf/test_btf.c
+++ b/tools/testing/selftests/bpf/test_btf.c
@@ -3482,7 +3482,8 @@ struct pprint_mapv {
} aenum;
};
-static struct btf_raw_test pprint_test_template = {
+static struct btf_raw_test pprint_test_template[] = {
+{
.raw_types = {
/* unsighed char */ /* [1] */
BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 8, 1),
@@ -3532,13 +3533,140 @@ static struct btf_raw_test pprint_test_template = {
BTF_MEMBER_ENC(NAME_TBD, 15, 192), /* aenum */
BTF_END_RAW,
},
- .str_sec = "\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum",
- .str_sec_size = sizeof("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum"),
+ BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum"),
+ .key_size = sizeof(unsigned int),
+ .value_size = sizeof(struct pprint_mapv),
+ .key_type_id = 3, /* unsigned int */
+ .value_type_id = 16, /* struct pprint_mapv */
+ .max_entries = 128 * 1024,
+},
+
+{
+ /* this type will have the same type as the
+ * first .raw_types definition, but struct type will
+ * be encoded with kind_flag set.
+ */
+ .raw_types = {
+ /* unsighed char */ /* [1] */
+ BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 8, 1),
+ /* unsigned short */ /* [2] */
+ BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 16, 2),
+ /* unsigned int */ /* [3] */
+ BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 32, 4),
+ /* int */ /* [4] */
+ BTF_TYPE_INT_ENC(NAME_TBD, BTF_INT_SIGNED, 0, 32, 4),
+ /* unsigned long long */ /* [5] */
+ BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 64, 8),
+ BTF_TYPE_INT_ENC(0, 0, 0, 32, 4), /* [6] */
+ BTF_TYPE_INT_ENC(0, 0, 0, 32, 4), /* [7] */
+ /* uint8_t[8] */ /* [8] */
+ BTF_TYPE_ARRAY_ENC(9, 1, 8),
+ /* typedef unsigned char uint8_t */ /* [9] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 1),
+ /* typedef unsigned short uint16_t */ /* [10] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 2),
+ /* typedef unsigned int uint32_t */ /* [11] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 3),
+ /* typedef int int32_t */ /* [12] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 4),
+ /* typedef unsigned long long uint64_t *//* [13] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 5),
+ /* union (anon) */ /* [14] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_UNION, 0, 2), 8),
+ BTF_MEMBER_ENC(NAME_TBD, 13, 0),/* uint64_t ui64; */
+ BTF_MEMBER_ENC(NAME_TBD, 8, 0), /* uint8_t ui8a[8]; */
+ /* enum (anon) */ /* [15] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ENUM, 0, 4), 4),
+ BTF_ENUM_ENC(NAME_TBD, 0),
+ BTF_ENUM_ENC(NAME_TBD, 1),
+ BTF_ENUM_ENC(NAME_TBD, 2),
+ BTF_ENUM_ENC(NAME_TBD, 3),
+ /* struct pprint_mapv */ /* [16] */
+ BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 8), 32),
+ BTF_MEMBER_ENC(NAME_TBD, 11, BTF_MEMBER_OFFSET(0, 0)), /* uint32_t ui32 */
+ BTF_MEMBER_ENC(NAME_TBD, 10, BTF_MEMBER_OFFSET(0, 32)), /* uint16_t ui16 */
+ BTF_MEMBER_ENC(NAME_TBD, 12, BTF_MEMBER_OFFSET(0, 64)), /* int32_t si32 */
+ BTF_MEMBER_ENC(NAME_TBD, 6, BTF_MEMBER_OFFSET(2, 96)), /* unused_bits2a */
+ BTF_MEMBER_ENC(NAME_TBD, 7, BTF_MEMBER_OFFSET(28, 98)), /* bits28 */
+ BTF_MEMBER_ENC(NAME_TBD, 6, BTF_MEMBER_OFFSET(2, 126)), /* unused_bits2b */
+ BTF_MEMBER_ENC(0, 14, BTF_MEMBER_OFFSET(0, 128)), /* union (anon) */
+ BTF_MEMBER_ENC(NAME_TBD, 15, BTF_MEMBER_OFFSET(0, 192)), /* aenum */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum"),
.key_size = sizeof(unsigned int),
.value_size = sizeof(struct pprint_mapv),
.key_type_id = 3, /* unsigned int */
.value_type_id = 16, /* struct pprint_mapv */
.max_entries = 128 * 1024,
+},
+
+{
+ /* this type will have the same layout as the
+ * first .raw_types definition. The struct type will
+ * be encoded with kind_flag set, bitfield members
+ * are added typedef/const/volatile, and bitfield members
+ * will have both int and enum types.
+ */
+ .raw_types = {
+ /* unsighed char */ /* [1] */
+ BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 8, 1),
+ /* unsigned short */ /* [2] */
+ BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 16, 2),
+ /* unsigned int */ /* [3] */
+ BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 32, 4),
+ /* int */ /* [4] */
+ BTF_TYPE_INT_ENC(NAME_TBD, BTF_INT_SIGNED, 0, 32, 4),
+ /* unsigned long long */ /* [5] */
+ BTF_TYPE_INT_ENC(NAME_TBD, 0, 0, 64, 8),
+ BTF_TYPE_INT_ENC(0, 0, 0, 32, 4), /* [6] */
+ BTF_TYPE_INT_ENC(0, 0, 0, 32, 4), /* [7] */
+ /* uint8_t[8] */ /* [8] */
+ BTF_TYPE_ARRAY_ENC(9, 1, 8),
+ /* typedef unsigned char uint8_t */ /* [9] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 1),
+ /* typedef unsigned short uint16_t */ /* [10] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 2),
+ /* typedef unsigned int uint32_t */ /* [11] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 3),
+ /* typedef int int32_t */ /* [12] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 4),
+ /* typedef unsigned long long uint64_t *//* [13] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 5),
+ /* union (anon) */ /* [14] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_UNION, 0, 2), 8),
+ BTF_MEMBER_ENC(NAME_TBD, 13, 0),/* uint64_t ui64; */
+ BTF_MEMBER_ENC(NAME_TBD, 8, 0), /* uint8_t ui8a[8]; */
+ /* enum (anon) */ /* [15] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ENUM, 0, 4), 4),
+ BTF_ENUM_ENC(NAME_TBD, 0),
+ BTF_ENUM_ENC(NAME_TBD, 1),
+ BTF_ENUM_ENC(NAME_TBD, 2),
+ BTF_ENUM_ENC(NAME_TBD, 3),
+ /* struct pprint_mapv */ /* [16] */
+ BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 8), 32),
+ BTF_MEMBER_ENC(NAME_TBD, 11, BTF_MEMBER_OFFSET(0, 0)), /* uint32_t ui32 */
+ BTF_MEMBER_ENC(NAME_TBD, 10, BTF_MEMBER_OFFSET(0, 32)), /* uint16_t ui16 */
+ BTF_MEMBER_ENC(NAME_TBD, 12, BTF_MEMBER_OFFSET(0, 64)), /* int32_t si32 */
+ BTF_MEMBER_ENC(NAME_TBD, 17, BTF_MEMBER_OFFSET(2, 96)), /* unused_bits2a */
+ BTF_MEMBER_ENC(NAME_TBD, 7, BTF_MEMBER_OFFSET(28, 98)), /* bits28 */
+ BTF_MEMBER_ENC(NAME_TBD, 19, BTF_MEMBER_OFFSET(2, 126)),/* unused_bits2b */
+ BTF_MEMBER_ENC(0, 14, BTF_MEMBER_OFFSET(0, 128)), /* union (anon) */
+ BTF_MEMBER_ENC(NAME_TBD, 15, BTF_MEMBER_OFFSET(0, 192)), /* aenum */
+ /* typedef unsigned int ___int */ /* [17] */
+ BTF_TYPEDEF_ENC(NAME_TBD, 18),
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_VOLATILE, 0, 0), 6), /* [18] */
+ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_CONST, 0, 0), 15), /* [19] */
+ BTF_END_RAW,
+ },
+ BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum\0___int"),
+ .key_size = sizeof(unsigned int),
+ .value_size = sizeof(struct pprint_mapv),
+ .key_type_id = 3, /* unsigned int */
+ .value_type_id = 16, /* struct pprint_mapv */
+ .max_entries = 128 * 1024,
+},
+
};
static struct btf_pprint_test_meta {
@@ -3641,9 +3769,9 @@ static int check_line(const char *expected_line, int nexpected_line,
}
-static int do_test_pprint(void)
+static int do_test_pprint(int test_num)
{
- const struct btf_raw_test *test = &pprint_test_template;
+ const struct btf_raw_test *test = &pprint_test_template[test_num];
struct bpf_create_map_attr create_attr = {};
bool ordered_map, lossless_map, percpu_map;
int err, ret, num_cpus, rounded_value_size;
@@ -3659,7 +3787,7 @@ static int do_test_pprint(void)
uint8_t *raw_btf;
ssize_t nread;
- fprintf(stderr, "%s......", test->descr);
+ fprintf(stderr, "%s(#%d)......", test->descr, test_num);
raw_btf = btf_raw_create(&hdr_tmpl, test->raw_types,
test->str_sec, test->str_sec_size,
&raw_btf_size, NULL);
@@ -3852,15 +3980,27 @@ static int test_pprint(void)
unsigned int i;
int err = 0;
+ /* test various maps with the first test template */
for (i = 0; i < ARRAY_SIZE(pprint_tests_meta); i++) {
- pprint_test_template.descr = pprint_tests_meta[i].descr;
- pprint_test_template.map_type = pprint_tests_meta[i].map_type;
- pprint_test_template.map_name = pprint_tests_meta[i].map_name;
- pprint_test_template.ordered_map = pprint_tests_meta[i].ordered_map;
- pprint_test_template.lossless_map = pprint_tests_meta[i].lossless_map;
- pprint_test_template.percpu_map = pprint_tests_meta[i].percpu_map;
-
- err |= count_result(do_test_pprint());
+ pprint_test_template[0].descr = pprint_tests_meta[i].descr;
+ pprint_test_template[0].map_type = pprint_tests_meta[i].map_type;
+ pprint_test_template[0].map_name = pprint_tests_meta[i].map_name;
+ pprint_test_template[0].ordered_map = pprint_tests_meta[i].ordered_map;
+ pprint_test_template[0].lossless_map = pprint_tests_meta[i].lossless_map;
+ pprint_test_template[0].percpu_map = pprint_tests_meta[i].percpu_map;
+
+ err |= count_result(do_test_pprint(0));
+ }
+
+ /* test rest test templates with the first map */
+ for (i = 1; i < ARRAY_SIZE(pprint_test_template); i++) {
+ pprint_test_template[i].descr = pprint_tests_meta[0].descr;
+ pprint_test_template[i].map_type = pprint_tests_meta[0].map_type;
+ pprint_test_template[i].map_name = pprint_tests_meta[0].map_name;
+ pprint_test_template[i].ordered_map = pprint_tests_meta[0].ordered_map;
+ pprint_test_template[i].lossless_map = pprint_tests_meta[0].lossless_map;
+ pprint_test_template[i].percpu_map = pprint_tests_meta[0].percpu_map;
+ err |= count_result(do_test_pprint(i));
}
return err;
--
2.17.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* [PATCH bpf-next v2 7/8] tools: bpftool: refactor btf_dumper_int_bits()
2018-12-14 23:34 [PATCH bpf-next v2 0/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
` (5 preceding siblings ...)
2018-12-14 23:34 ` [PATCH bpf-next v2 6/8] tools/bpf: test kernel bpffs map pretty print with struct kind_flag Yonghong Song
@ 2018-12-14 23:34 ` Yonghong Song
2018-12-15 21:26 ` Martin Lau
2018-12-14 23:34 ` [PATCH bpf-next v2 8/8] tools: bpftool: support pretty print with kind_flag set Yonghong Song
7 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-14 23:34 UTC (permalink / raw)
To: netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team,
Martin KaFai Lau
The core dump funcitonality in btf_dumper_int_bits() is
refactored into a separate function btf_dumper_bitfield()
which will be used by the next patch.
Signed-off-by: Yonghong Song <yhs@fb.com>
---
tools/bpf/bpftool/btf_dumper.c | 22 ++++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c
index 5cdb2ef8b6f1..ec1da87c3d65 100644
--- a/tools/bpf/bpftool/btf_dumper.c
+++ b/tools/bpf/bpftool/btf_dumper.c
@@ -73,20 +73,17 @@ static int btf_dumper_array(const struct btf_dumper *d, __u32 type_id,
return ret;
}
-static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
+static void btf_dumper_bitfield(__u32 nr_bits, __u8 bit_offset,
const void *data, json_writer_t *jw,
bool is_plain_text)
{
int left_shift_bits, right_shift_bits;
- int nr_bits = BTF_INT_BITS(int_type);
- int total_bits_offset;
int bytes_to_copy;
int bits_to_copy;
__u64 print_num;
- total_bits_offset = bit_offset + BTF_INT_OFFSET(int_type);
- data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
- bit_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
+ data += BITS_ROUNDDOWN_BYTES(bit_offset);
+ bit_offset = BITS_PER_BYTE_MASKED(bit_offset);
bits_to_copy = bit_offset + nr_bits;
bytes_to_copy = BITS_ROUNDUP_BYTES(bits_to_copy);
@@ -109,6 +106,19 @@ static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
jsonw_printf(jw, "%llu", print_num);
}
+
+static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
+ const void *data, json_writer_t *jw,
+ bool is_plain_text)
+{
+ int nr_bits = BTF_INT_BITS(int_type);
+ int total_bits_offset;
+
+ total_bits_offset = bit_offset + BTF_INT_OFFSET(int_type);
+ btf_dumper_bitfield(nr_bits, total_bits_offset, data, jw,
+ is_plain_text);
+}
+
static int btf_dumper_int(const struct btf_type *t, __u8 bit_offset,
const void *data, json_writer_t *jw,
bool is_plain_text)
--
2.17.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 7/8] tools: bpftool: refactor btf_dumper_int_bits()
2018-12-14 23:34 ` [PATCH bpf-next v2 7/8] tools: bpftool: refactor btf_dumper_int_bits() Yonghong Song
@ 2018-12-15 21:26 ` Martin Lau
2018-12-16 5:31 ` Yonghong Song
0 siblings, 1 reply; 30+ messages in thread
From: Martin Lau @ 2018-12-15 21:26 UTC (permalink / raw)
To: Yonghong Song
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On Fri, Dec 14, 2018 at 03:34:33PM -0800, Yonghong Song wrote:
> The core dump funcitonality in btf_dumper_int_bits() is
> refactored into a separate function btf_dumper_bitfield()
> which will be used by the next patch.
>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
> tools/bpf/bpftool/btf_dumper.c | 22 ++++++++++++++++------
> 1 file changed, 16 insertions(+), 6 deletions(-)
>
> diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c
> index 5cdb2ef8b6f1..ec1da87c3d65 100644
> --- a/tools/bpf/bpftool/btf_dumper.c
> +++ b/tools/bpf/bpftool/btf_dumper.c
> @@ -73,20 +73,17 @@ static int btf_dumper_array(const struct btf_dumper *d, __u32 type_id,
> return ret;
> }
>
> -static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
> +static void btf_dumper_bitfield(__u32 nr_bits, __u8 bit_offset,
> const void *data, json_writer_t *jw,
> bool is_plain_text)
> {
> int left_shift_bits, right_shift_bits;
> - int nr_bits = BTF_INT_BITS(int_type);
> - int total_bits_offset;
> int bytes_to_copy;
> int bits_to_copy;
> __u64 print_num;
>
> - total_bits_offset = bit_offset + BTF_INT_OFFSET(int_type);
> - data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
> - bit_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
> + data += BITS_ROUNDDOWN_BYTES(bit_offset);
> + bit_offset = BITS_PER_BYTE_MASKED(bit_offset);
> bits_to_copy = bit_offset + nr_bits;
> bytes_to_copy = BITS_ROUNDUP_BYTES(bits_to_copy);
>
> @@ -109,6 +106,19 @@ static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
> jsonw_printf(jw, "%llu", print_num);
> }
>
> +
> +static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
> + const void *data, json_writer_t *jw,
> + bool is_plain_text)
> +{
> + int nr_bits = BTF_INT_BITS(int_type);
> + int total_bits_offset;
> +
> + total_bits_offset = bit_offset + BTF_INT_OFFSET(int_type);
> + btf_dumper_bitfield(nr_bits, total_bits_offset, data, jw,
The 2nd arg is "__u8 bit_offset". Can you check if total_bits_offset
is fine here, considering BTF_INT_OFFSET() is 8 bits itself.
A comment would help if it is safe.
Other than that,
Acked-by: Martin KaFai Lau <kafai@fb.com>
> + is_plain_text);
> +}
> +
> static int btf_dumper_int(const struct btf_type *t, __u8 bit_offset,
> const void *data, json_writer_t *jw,
> bool is_plain_text)
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 7/8] tools: bpftool: refactor btf_dumper_int_bits()
2018-12-15 21:26 ` Martin Lau
@ 2018-12-16 5:31 ` Yonghong Song
0 siblings, 0 replies; 30+ messages in thread
From: Yonghong Song @ 2018-12-16 5:31 UTC (permalink / raw)
To: Martin Lau
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On 12/15/18 1:26 PM, Martin Lau wrote:
> On Fri, Dec 14, 2018 at 03:34:33PM -0800, Yonghong Song wrote:
>> The core dump funcitonality in btf_dumper_int_bits() is
>> refactored into a separate function btf_dumper_bitfield()
>> which will be used by the next patch.
>>
>> Signed-off-by: Yonghong Song <yhs@fb.com>
>> ---
>> tools/bpf/bpftool/btf_dumper.c | 22 ++++++++++++++++------
>> 1 file changed, 16 insertions(+), 6 deletions(-)
>>
>> diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c
>> index 5cdb2ef8b6f1..ec1da87c3d65 100644
>> --- a/tools/bpf/bpftool/btf_dumper.c
>> +++ b/tools/bpf/bpftool/btf_dumper.c
>> @@ -73,20 +73,17 @@ static int btf_dumper_array(const struct btf_dumper *d, __u32 type_id,
>> return ret;
>> }
>>
>> -static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
>> +static void btf_dumper_bitfield(__u32 nr_bits, __u8 bit_offset,
>> const void *data, json_writer_t *jw,
>> bool is_plain_text)
>> {
>> int left_shift_bits, right_shift_bits;
>> - int nr_bits = BTF_INT_BITS(int_type);
>> - int total_bits_offset;
>> int bytes_to_copy;
>> int bits_to_copy;
>> __u64 print_num;
>>
>> - total_bits_offset = bit_offset + BTF_INT_OFFSET(int_type);
>> - data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
>> - bit_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
>> + data += BITS_ROUNDDOWN_BYTES(bit_offset);
>> + bit_offset = BITS_PER_BYTE_MASKED(bit_offset);
>> bits_to_copy = bit_offset + nr_bits;
>> bytes_to_copy = BITS_ROUNDUP_BYTES(bits_to_copy);
>>
>> @@ -109,6 +106,19 @@ static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
>> jsonw_printf(jw, "%llu", print_num);
>> }
>>
>> +
>> +static void btf_dumper_int_bits(__u32 int_type, __u8 bit_offset,
>> + const void *data, json_writer_t *jw,
>> + bool is_plain_text)
>> +{
>> + int nr_bits = BTF_INT_BITS(int_type);
>> + int total_bits_offset;
>> +
>> + total_bits_offset = bit_offset + BTF_INT_OFFSET(int_type);
>> + btf_dumper_bitfield(nr_bits, total_bits_offset, data, jw,
> The 2nd arg is "__u8 bit_offset". Can you check if total_bits_offset
> is fine here, considering BTF_INT_OFFSET() is 8 bits itself.
> A comment would help if it is safe.
This should be okay.
In kernel btf_int_check_meta, we have
nr_bits = BTF_INT_BITS(int_data) + BTF_INT_OFFSET(int_data);
if (nr_bits > BITS_PER_U64) {
btf_verifier_log_type(env, t, "nr_bits exceeds %zu",
BITS_PER_U64);
return -EINVAL;
}
So BTF_INT_OFFSET(int_data) at most BITS_PER_U64.
bit_offset at most 7, so total_bits_offset still within range.
kernel has similar implementation.
I will add a comment here.
>
> Other than that,
> Acked-by: Martin KaFai Lau <kafai@fb.com>
>
>> + is_plain_text);
>> +}
>> +
>> static int btf_dumper_int(const struct btf_type *t, __u8 bit_offset,
>> const void *data, json_writer_t *jw,
>> bool is_plain_text)
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH bpf-next v2 8/8] tools: bpftool: support pretty print with kind_flag set
2018-12-14 23:34 [PATCH bpf-next v2 0/8] bpf: btf: fix struct/union/fwd types with kind_flag Yonghong Song
` (6 preceding siblings ...)
2018-12-14 23:34 ` [PATCH bpf-next v2 7/8] tools: bpftool: refactor btf_dumper_int_bits() Yonghong Song
@ 2018-12-14 23:34 ` Yonghong Song
2018-12-15 21:49 ` Martin Lau
7 siblings, 1 reply; 30+ messages in thread
From: Yonghong Song @ 2018-12-14 23:34 UTC (permalink / raw)
To: netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team,
Martin KaFai Lau
The following example shows map pretty print with structures
which include bitfield members.
enum A { A1, A2, A3, A4, A5 };
typedef enum A ___A;
struct tmp_t {
char a1:4;
int a2:4;
int :4;
__u32 a3:4;
int b;
___A b1:4;
enum A b2:4;
};
struct bpf_map_def SEC("maps") tmpmap = {
.type = BPF_MAP_TYPE_ARRAY,
.key_size = sizeof(__u32),
.value_size = sizeof(struct tmp_t),
.max_entries = 1,
};
BPF_ANNOTATE_KV_PAIR(tmpmap, int, struct tmp_t);
and the following map update in the bpf program:
key = 0;
struct tmp_t t = {};
t.a1 = 2;
t.a2 = 4;
t.a3 = 6;
t.b = 7;
t.b1 = 8;
t.b2 = 10;
bpf_map_update_elem(&tmpmap, &key, &t, 0);
With this patch, I am able to print out the map values
correctly with this patch:
bpftool map dump id 187
[{
"key": 0,
"value": {
"a1": 0x2,
"a2": 0x4,
"a3": 0x6,
"b": 7,
"b1": 0x8,
"b2": 0xa
}
}
]
The following example shows forward type can be properly
printed in function prototype with modified tst_btf_haskv.c.
struct t;
union u;
__attribute__((noinline))
static int test_long_fname_1(struct dummy_tracepoint_args *arg,
struct t *p1, union u *p2,
__u32 unused)
...
int _dummy_tracepoint(struct dummy_tracepoint_args *arg) {
return test_long_fname_1(arg, 0, 0);
}
$ bpftool p d xlated id 24
...
int test_long_fname_1(struct dummy_tracepoint_args * arg,
struct t * p1, union u * p2,
__u32 unused)
...
Signed-off-by: Yonghong Song <yhs@fb.com>
---
tools/bpf/bpftool/btf_dumper.c | 36 +++++++++++++++++++++++++---------
1 file changed, 27 insertions(+), 9 deletions(-)
diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c
index ec1da87c3d65..23fc6a84d82b 100644
--- a/tools/bpf/bpftool/btf_dumper.c
+++ b/tools/bpf/bpftool/btf_dumper.c
@@ -190,6 +190,7 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id,
const struct btf_type *t;
struct btf_member *m;
const void *data_off;
+ int kind_flag;
int ret = 0;
int i, vlen;
@@ -197,18 +198,32 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id,
if (!t)
return -EINVAL;
+ kind_flag = BTF_INFO_KFLAG(t->info);
vlen = BTF_INFO_VLEN(t->info);
jsonw_start_object(d->jw);
m = (struct btf_member *)(t + 1);
for (i = 0; i < vlen; i++) {
- data_off = data + BITS_ROUNDDOWN_BYTES(m[i].offset);
+ __u32 bit_offset = m[i].offset;
+ __u32 bitfield_size = 0;
+
+ if (kind_flag) {
+ bitfield_size = BTF_MEMBER_BITFIELD_SIZE(bit_offset);
+ bit_offset = BTF_MEMBER_BIT_OFFSET(bit_offset);
+ }
+
jsonw_name(d->jw, btf__name_by_offset(d->btf, m[i].name_off));
- ret = btf_dumper_do_type(d, m[i].type,
- BITS_PER_BYTE_MASKED(m[i].offset),
- data_off);
- if (ret)
- break;
+ if (bitfield_size) {
+ btf_dumper_bitfield(bitfield_size, bit_offset,
+ data, d->jw, d->is_plain_text);
+ } else {
+ data_off = data + BITS_ROUNDDOWN_BYTES(bit_offset);
+ ret = btf_dumper_do_type(d, m[i].type,
+ BITS_PER_BYTE_MASKED(bit_offset),
+ data_off);
+ if (ret)
+ break;
+ }
}
jsonw_end_object(d->jw);
@@ -295,6 +310,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
switch (BTF_INFO_KIND(t->info)) {
case BTF_KIND_INT:
+ case BTF_KIND_TYPEDEF:
BTF_PRINT_ARG("%s ", btf__name_by_offset(btf, t->name_off));
break;
case BTF_KIND_STRUCT:
@@ -318,10 +334,11 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
BTF_PRINT_TYPE(t->type);
BTF_PRINT_ARG("* ");
break;
- case BTF_KIND_UNKN:
case BTF_KIND_FWD:
- case BTF_KIND_TYPEDEF:
- return -1;
+ BTF_PRINT_ARG("%s %s ",
+ BTF_INFO_KFLAG(t->info) ? "union" : "struct",
+ btf__name_by_offset(btf, t->name_off));
+ break;
case BTF_KIND_VOLATILE:
BTF_PRINT_ARG("volatile ");
BTF_PRINT_TYPE(t->type);
@@ -345,6 +362,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
if (pos == -1)
return -1;
break;
+ case BTF_KIND_UNKN:
default:
return -1;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 8/8] tools: bpftool: support pretty print with kind_flag set
2018-12-14 23:34 ` [PATCH bpf-next v2 8/8] tools: bpftool: support pretty print with kind_flag set Yonghong Song
@ 2018-12-15 21:49 ` Martin Lau
2018-12-16 5:36 ` Yonghong Song
0 siblings, 1 reply; 30+ messages in thread
From: Martin Lau @ 2018-12-15 21:49 UTC (permalink / raw)
To: Yonghong Song
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On Fri, Dec 14, 2018 at 03:34:34PM -0800, Yonghong Song wrote:
> The following example shows map pretty print with structures
> which include bitfield members.
>
> enum A { A1, A2, A3, A4, A5 };
> typedef enum A ___A;
> struct tmp_t {
> char a1:4;
> int a2:4;
> int :4;
> __u32 a3:4;
> int b;
> ___A b1:4;
> enum A b2:4;
> };
> struct bpf_map_def SEC("maps") tmpmap = {
> .type = BPF_MAP_TYPE_ARRAY,
> .key_size = sizeof(__u32),
> .value_size = sizeof(struct tmp_t),
> .max_entries = 1,
> };
> BPF_ANNOTATE_KV_PAIR(tmpmap, int, struct tmp_t);
>
> and the following map update in the bpf program:
>
> key = 0;
> struct tmp_t t = {};
> t.a1 = 2;
> t.a2 = 4;
> t.a3 = 6;
> t.b = 7;
> t.b1 = 8;
> t.b2 = 10;
> bpf_map_update_elem(&tmpmap, &key, &t, 0);
>
> With this patch, I am able to print out the map values
> correctly with this patch:
> bpftool map dump id 187
> [{
> "key": 0,
> "value": {
> "a1": 0x2,
> "a2": 0x4,
> "a3": 0x6,
> "b": 7,
> "b1": 0x8,
> "b2": 0xa
> }
> }
> ]
>
> The following example shows forward type can be properly
> printed in function prototype with modified tst_btf_haskv.c.
>
> struct t;
> union u;
>
> __attribute__((noinline))
> static int test_long_fname_1(struct dummy_tracepoint_args *arg,
> struct t *p1, union u *p2,
> __u32 unused)
> ...
> int _dummy_tracepoint(struct dummy_tracepoint_args *arg) {
> return test_long_fname_1(arg, 0, 0);
> }
>
> $ bpftool p d xlated id 24
> ...
> int test_long_fname_1(struct dummy_tracepoint_args * arg,
> struct t * p1, union u * p2,
> __u32 unused)
> ...
>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
> tools/bpf/bpftool/btf_dumper.c | 36 +++++++++++++++++++++++++---------
> 1 file changed, 27 insertions(+), 9 deletions(-)
>
> diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c
> index ec1da87c3d65..23fc6a84d82b 100644
> --- a/tools/bpf/bpftool/btf_dumper.c
> +++ b/tools/bpf/bpftool/btf_dumper.c
> @@ -190,6 +190,7 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id,
> const struct btf_type *t;
> struct btf_member *m;
> const void *data_off;
> + int kind_flag;
> int ret = 0;
> int i, vlen;
>
> @@ -197,18 +198,32 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id,
> if (!t)
> return -EINVAL;
>
> + kind_flag = BTF_INFO_KFLAG(t->info);
> vlen = BTF_INFO_VLEN(t->info);
> jsonw_start_object(d->jw);
> m = (struct btf_member *)(t + 1);
>
> for (i = 0; i < vlen; i++) {
> - data_off = data + BITS_ROUNDDOWN_BYTES(m[i].offset);
> + __u32 bit_offset = m[i].offset;
Without always checking the kind_flag first, here is an example
that accessing the .offset looks incorrect at the first glance.
> + __u32 bitfield_size = 0;
> +
> + if (kind_flag) {
but then I noticed it is fine here ;)
> + bitfield_size = BTF_MEMBER_BITFIELD_SIZE(bit_offset);
> + bit_offset = BTF_MEMBER_BIT_OFFSET(bit_offset);
> + }
> +
> jsonw_name(d->jw, btf__name_by_offset(d->btf, m[i].name_off));
> - ret = btf_dumper_do_type(d, m[i].type,
> - BITS_PER_BYTE_MASKED(m[i].offset),
> - data_off);
> - if (ret)
> - break;
> + if (bitfield_size) {
> + btf_dumper_bitfield(bitfield_size, bit_offset,
> + data, d->jw, d->is_plain_text);
> + } else {
> + data_off = data + BITS_ROUNDDOWN_BYTES(bit_offset);
> + ret = btf_dumper_do_type(d, m[i].type,
> + BITS_PER_BYTE_MASKED(bit_offset),
> + data_off);
> + if (ret)
> + break;
> + }
> }
>
> jsonw_end_object(d->jw);
> @@ -295,6 +310,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
>
> switch (BTF_INFO_KIND(t->info)) {
> case BTF_KIND_INT:
> + case BTF_KIND_TYPEDEF:
> BTF_PRINT_ARG("%s ", btf__name_by_offset(btf, t->name_off));
> break;
> case BTF_KIND_STRUCT:
> @@ -318,10 +334,11 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
> BTF_PRINT_TYPE(t->type);
> BTF_PRINT_ARG("* ");
> break;
> - case BTF_KIND_UNKN:
> case BTF_KIND_FWD:
> - case BTF_KIND_TYPEDEF:
hmm.... Is this TYPEDEF change related to this patch?
If not, a comment in the commit message would help.
Other than that,
Acked-by: Martin KaFai Lau <kafai@fb.com>
> - return -1;
> + BTF_PRINT_ARG("%s %s ",
> + BTF_INFO_KFLAG(t->info) ? "union" : "struct",
> + btf__name_by_offset(btf, t->name_off));
> + break;
> case BTF_KIND_VOLATILE:
> BTF_PRINT_ARG("volatile ");
> BTF_PRINT_TYPE(t->type);
> @@ -345,6 +362,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
> if (pos == -1)
> return -1;
> break;
> + case BTF_KIND_UNKN:
> default:
> return -1;
> }
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH bpf-next v2 8/8] tools: bpftool: support pretty print with kind_flag set
2018-12-15 21:49 ` Martin Lau
@ 2018-12-16 5:36 ` Yonghong Song
0 siblings, 0 replies; 30+ messages in thread
From: Yonghong Song @ 2018-12-16 5:36 UTC (permalink / raw)
To: Martin Lau
Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann,
Kernel Team
On 12/15/18 1:49 PM, Martin Lau wrote:
> On Fri, Dec 14, 2018 at 03:34:34PM -0800, Yonghong Song wrote:
>> The following example shows map pretty print with structures
>> which include bitfield members.
>>
>> enum A { A1, A2, A3, A4, A5 };
>> typedef enum A ___A;
>> struct tmp_t {
>> char a1:4;
>> int a2:4;
>> int :4;
>> __u32 a3:4;
>> int b;
>> ___A b1:4;
>> enum A b2:4;
>> };
>> struct bpf_map_def SEC("maps") tmpmap = {
>> .type = BPF_MAP_TYPE_ARRAY,
>> .key_size = sizeof(__u32),
>> .value_size = sizeof(struct tmp_t),
>> .max_entries = 1,
>> };
>> BPF_ANNOTATE_KV_PAIR(tmpmap, int, struct tmp_t);
>>
>> and the following map update in the bpf program:
>>
>> key = 0;
>> struct tmp_t t = {};
>> t.a1 = 2;
>> t.a2 = 4;
>> t.a3 = 6;
>> t.b = 7;
>> t.b1 = 8;
>> t.b2 = 10;
>> bpf_map_update_elem(&tmpmap, &key, &t, 0);
>>
>> With this patch, I am able to print out the map values
>> correctly with this patch:
>> bpftool map dump id 187
>> [{
>> "key": 0,
>> "value": {
>> "a1": 0x2,
>> "a2": 0x4,
>> "a3": 0x6,
>> "b": 7,
>> "b1": 0x8,
>> "b2": 0xa
>> }
>> }
>> ]
>>
>> The following example shows forward type can be properly
>> printed in function prototype with modified tst_btf_haskv.c.
>>
>> struct t;
>> union u;
>>
>> __attribute__((noinline))
>> static int test_long_fname_1(struct dummy_tracepoint_args *arg,
>> struct t *p1, union u *p2,
>> __u32 unused)
>> ...
>> int _dummy_tracepoint(struct dummy_tracepoint_args *arg) {
>> return test_long_fname_1(arg, 0, 0);
>> }
>>
>> $ bpftool p d xlated id 24
>> ...
>> int test_long_fname_1(struct dummy_tracepoint_args * arg,
>> struct t * p1, union u * p2,
>> __u32 unused)
>> ...
>>
>> Signed-off-by: Yonghong Song <yhs@fb.com>
>> ---
>> tools/bpf/bpftool/btf_dumper.c | 36 +++++++++++++++++++++++++---------
>> 1 file changed, 27 insertions(+), 9 deletions(-)
>>
>> diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c
>> index ec1da87c3d65..23fc6a84d82b 100644
>> --- a/tools/bpf/bpftool/btf_dumper.c
>> +++ b/tools/bpf/bpftool/btf_dumper.c
>> @@ -190,6 +190,7 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id,
>> const struct btf_type *t;
>> struct btf_member *m;
>> const void *data_off;
>> + int kind_flag;
>> int ret = 0;
>> int i, vlen;
>>
>> @@ -197,18 +198,32 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id,
>> if (!t)
>> return -EINVAL;
>>
>> + kind_flag = BTF_INFO_KFLAG(t->info);
>> vlen = BTF_INFO_VLEN(t->info);
>> jsonw_start_object(d->jw);
>> m = (struct btf_member *)(t + 1);
>>
>> for (i = 0; i < vlen; i++) {
>> - data_off = data + BITS_ROUNDDOWN_BYTES(m[i].offset);
>
>
>> + __u32 bit_offset = m[i].offset;
> Without always checking the kind_flag first, here is an example
> that accessing the .offset looks incorrect at the first glance.
>
>> + __u32 bitfield_size = 0;
>> +
>> + if (kind_flag) {
> but then I noticed it is fine here ;)
>
>> + bitfield_size = BTF_MEMBER_BITFIELD_SIZE(bit_offset);
>> + bit_offset = BTF_MEMBER_BIT_OFFSET(bit_offset);
>> + }
>> +
>> jsonw_name(d->jw, btf__name_by_offset(d->btf, m[i].name_off));
>> - ret = btf_dumper_do_type(d, m[i].type,
>> - BITS_PER_BYTE_MASKED(m[i].offset),
>> - data_off);
>> - if (ret)
>> - break;
>> + if (bitfield_size) {
>> + btf_dumper_bitfield(bitfield_size, bit_offset,
>> + data, d->jw, d->is_plain_text);
>> + } else {
>> + data_off = data + BITS_ROUNDDOWN_BYTES(bit_offset);
>> + ret = btf_dumper_do_type(d, m[i].type,
>> + BITS_PER_BYTE_MASKED(bit_offset),
>> + data_off);
>> + if (ret)
>> + break;
>> + }
>> }
>>
>> jsonw_end_object(d->jw);
>> @@ -295,6 +310,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
>>
>> switch (BTF_INFO_KIND(t->info)) {
>> case BTF_KIND_INT:
>> + case BTF_KIND_TYPEDEF:
>> BTF_PRINT_ARG("%s ", btf__name_by_offset(btf, t->name_off));
>> break;
>> case BTF_KIND_STRUCT:
>> @@ -318,10 +334,11 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
>> BTF_PRINT_TYPE(t->type);
>> BTF_PRINT_ARG("* ");
>> break;
>> - case BTF_KIND_UNKN:
>> case BTF_KIND_FWD:
>> - case BTF_KIND_TYPEDEF:
> hmm.... Is this TYPEDEF change related to this patch?
> If not, a comment in the commit message would help.
TYPEDEF is not related to kind_flag.
I will add a comment in commit message.
>
> Other than that,
>
> Acked-by: Martin KaFai Lau <kafai@fb.com>
>
>> - return -1;
>> + BTF_PRINT_ARG("%s %s ",
>> + BTF_INFO_KFLAG(t->info) ? "union" : "struct",
>> + btf__name_by_offset(btf, t->name_off));
>> + break;
>> case BTF_KIND_VOLATILE:
>> BTF_PRINT_ARG("volatile ");
>> BTF_PRINT_TYPE(t->type);
>> @@ -345,6 +362,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
>> if (pos == -1)
>> return -1;
>> break;
>> + case BTF_KIND_UNKN:
>> default:
>> return -1;
>> }
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 30+ messages in thread