Error: bug: failed to retrieve CAP_BPF status: Invalid argument

Error: bug: failed to retrieve CAP_BPF status: Invalid argument

[Vincent Li]

Hi,

I compile and run kernel 5.18.0 in Centos 8 from bpf-next in my dev
machine, I also compiled bpftool from bpf-next on same machine, when
run bpftool on same machine, I got :

./bpftool feature probe

Error: bug: failed to retrieve CAP_BPF status: Invalid argument

where bpftool to retrieve CAP_BPF ? from running kernel or from somewhere else?

strace show:

capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0

capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0},
{effective=1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_FSETID|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_LINUX_IMMUTABLE|1<<CAP_NET_BIND_SERVICE|1<<CAP_NET_BROADCAST|1<<CAP_NET_ADMIN|1<<CAP_NET_RAW|1<<CAP_IPC_LOCK|1<<CAP_IPC_OWNER|1<<CAP_SYS_MODULE|1<<CAP_SYS_RAWIO|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_PACCT|1<<CAP_SYS_ADMIN|1<<CAP_SYS_BOOT|1<<CAP_SYS_NICE|1<<CAP_SYS_RESOURCE|1<<CAP_SYS_TIME|1<<CAP_SYS_TTY_CONFIG|1<<CAP_MKNOD|1<<CAP_LEASE|1<<CAP_AUDIT_WRITE|1<<CAP_AUDIT_CONTROL|1<<CAP_SETFCAP|1<<CAP_MAC_OVERRIDE|1<<CAP_MAC_ADMIN|1<<CAP_SYSLOG|1<<CAP_WAKE_ALARM|1<<CAP_BLOCK_SUSPEND|1<<CAP_AUDIT_READ|0x1c0,
permitted=1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_FSETID|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_LINUX_IMMUTABLE|1<<CAP_NET_BIND_SERVICE|1<<CAP_NET_BROADCAST|1<<CAP_NET_ADMIN|1<<CAP_NET_RAW|1<<CAP_IPC_LOCK|1<<CAP_IPC_OWNER|1<<CAP_SYS_MODULE|1<<CAP_SYS_RAWIO|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_PACCT|1<<CAP_SYS_ADMIN|1<<CAP_SYS_BOOT|1<<CAP_SYS_NICE|1<<CAP_SYS_RESOURCE|1<<CAP_SYS_TIME|1<<CAP_SYS_TTY_CONFIG|1<<CAP_MKNOD|1<<CAP_LEASE|1<<CAP_AUDIT_WRITE|1<<CAP_AUDIT_CONTROL|1<<CAP_SETFCAP|1<<CAP_MAC_OVERRIDE|1<<CAP_MAC_ADMIN|1<<CAP_SYSLOG|1<<CAP_WAKE_ALARM|1<<CAP_BLOCK_SUSPEND|1<<CAP_AUDIT_READ|0x1c0,
inheritable=0}) = 0

Re: Error: bug: failed to retrieve CAP_BPF status: Invalid argument

[Quentin Monnet]

Hi Vincent,

On Mon, 15 Aug 2022 at 18:46, Vincent Li <vincent.mc.li@gmail.com> wrote:
>
> Hi,
>
> I compile and run kernel 5.18.0 in Centos 8 from bpf-next in my dev
> machine, I also compiled bpftool from bpf-next on same machine, when
> run bpftool on same machine, I got :
>
> ./bpftool feature probe
>
> Error: bug: failed to retrieve CAP_BPF status: Invalid argument
>
> where bpftool to retrieve CAP_BPF ? from running kernel or from somewhere else?

Yes, bpftool calls cap_get_proc() to get the capabilities of the
current process. From what I understand of your output, it looks like
capget() returns CAP_BPF: I believe the "0x1c0" value at the end is
(1<<(CAP_CHECKPOINT_RESTORE-32)) + (1<<(CAP_BPF-32)) +
(1<<(CAP_PERFMON-32)). You could probably check this with a more
recent version of strace.

Then assuming you do retrieve CAP_BPF from capget(), I don't know why
cap_get_flag() in bpftool fails to retrieve the capability state. It
would be worth running bpftool in GDB to check what happens. The check
in libcap is here [0] but I don't see where we would fail to provide
valid arguments. Just in case, could you please let me know what
version of libcap you're using when compiling bpftool?

Thanks,
Quentin

[0] https://git.kernel.org/pub/scm/libs/libcap/libcap.git/tree/libcap/cap_flag.c?h=libcap-2.65#n12

Re: Error: bug: failed to retrieve CAP_BPF status: Invalid argument

[Vincent Li]

On Mon, Aug 15, 2022 at 3:18 PM Quentin Monnet <quentin@isovalent.com> wrote:
>
> Hi Vincent,
>
> On Mon, 15 Aug 2022 at 18:46, Vincent Li <vincent.mc.li@gmail.com> wrote:
> >
> > Hi,
> >
> > I compile and run kernel 5.18.0 in Centos 8 from bpf-next in my dev
> > machine, I also compiled bpftool from bpf-next on same machine, when
> > run bpftool on same machine, I got :
> >
> > ./bpftool feature probe
> >
> > Error: bug: failed to retrieve CAP_BPF status: Invalid argument
> >
> > where bpftool to retrieve CAP_BPF ? from running kernel or from somewhere else?
>
> Yes, bpftool calls cap_get_proc() to get the capabilities of the
> current process. From what I understand of your output, it looks like
> capget() returns CAP_BPF: I believe the "0x1c0" value at the end is
> (1<<(CAP_CHECKPOINT_RESTORE-32)) + (1<<(CAP_BPF-32)) +
> (1<<(CAP_PERFMON-32)). You could probably check this with a more
> recent version of strace.
>
> Then assuming you do retrieve CAP_BPF from capget(), I don't know why
> cap_get_flag() in bpftool fails to retrieve the capability state. It
> would be worth running bpftool in GDB to check what happens. The check
> in libcap is here [0] but I don't see where we would fail to provide
> valid arguments. Just in case, could you please let me know what
> version of libcap you're using when compiling bpftool?

I think I installed libcap through centos distro

[root@centos-dev ~]# rpm -qi libcap.x86_64

Name        : libcap

Version     : 2.26

Release     : 4.el8

Architecture: x86_64

Install Date: Mon 14 Jun 2021 02:35:08 PM EDT

Group       : System Environment/Libraries

Size        : 129682

License     : GPLv2

Signature   : RSA/SHA256, Mon 15 Jun 2020 06:31:12 PM EDT, Key ID
05b555b38483c65d

Source RPM  : libcap-2.26-4.el8.src.rpm

Build Date  : Mon 15 Jun 2020 06:28:10 PM EDT

Build Host  : x86-02.mbox.centos.org

Relocations : (not relocatable)

Packager    : CentOS Buildsys <bugs@centos.org>

Vendor      : CentOS

URL         : https://sites.google.com/site/fullycapable/

Summary     : Library for getting and setting POSIX.1e capabilities

Description :

libcap is a library for getting and setting POSIX.1e (formerly POSIX 6)

draft 15 capabilities.

>
> Thanks,
> Quentin
>
> [0] https://git.kernel.org/pub/scm/libs/libcap/libcap.git/tree/libcap/cap_flag.c?h=libcap-2.65#n12

Re: Error: bug: failed to retrieve CAP_BPF status: Invalid argument

[Quentin Monnet]

On Mon, 15 Aug 2022 at 23:26, Vincent Li <vincent.mc.li@gmail.com> wrote:
>
> On Mon, Aug 15, 2022 at 3:18 PM Quentin Monnet <quentin@isovalent.com> wrote:
> >
> > Hi Vincent,
> >
> > On Mon, 15 Aug 2022 at 18:46, Vincent Li <vincent.mc.li@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I compile and run kernel 5.18.0 in Centos 8 from bpf-next in my dev
> > > machine, I also compiled bpftool from bpf-next on same machine, when
> > > run bpftool on same machine, I got :
> > >
> > > ./bpftool feature probe
> > >
> > > Error: bug: failed to retrieve CAP_BPF status: Invalid argument
> > >
> > > where bpftool to retrieve CAP_BPF ? from running kernel or from somewhere else?
> >
> > Yes, bpftool calls cap_get_proc() to get the capabilities of the
> > current process. From what I understand of your output, it looks like
> > capget() returns CAP_BPF: I believe the "0x1c0" value at the end is
> > (1<<(CAP_CHECKPOINT_RESTORE-32)) + (1<<(CAP_BPF-32)) +
> > (1<<(CAP_PERFMON-32)). You could probably check this with a more
> > recent version of strace.
> >
> > Then assuming you do retrieve CAP_BPF from capget(), I don't know why
> > cap_get_flag() in bpftool fails to retrieve the capability state. It
> > would be worth running bpftool in GDB to check what happens. The check
> > in libcap is here [0] but I don't see where we would fail to provide
> > valid arguments. Just in case, could you please let me know what
> > version of libcap you're using when compiling bpftool?
>
> I think I installed libcap through centos distro
>
> [root@centos-dev ~]# rpm -qi libcap.x86_64
>
> Name        : libcap
>
> Version     : 2.26

So we investigated this on Slack. The issue is related to libcap (and
to how libcap is built on CentOS); it is fixed in libcap 2.30 and
older.

For the record, this is the commit that fixed it:
https://git.kernel.org/pub/scm/libs/libcap/libcap.git/commit/?id=f1f62a748d7c67361e91e32d26abafbfb03eeee4

Before this, cap_get_flag() would compare its argument "value" (in our
case, CAP_BPF == 39) with __CAP_BITS. This __CAP_BITS constant is
defined in libcap/cap_names.h, generated by libcap/_makenames.c from
the list in libcap/cap_names.list.h. The latter header file is itself
generated in libcap/Makefile from the UAPI header at
$(KERNEL_HEADERS)/linux/capability.h, which defaults to the local
libcap/include/uapi/linux/capability.h.

On your CentOS, the libcap version may have been compiled without
setting KERNEL_HEADERS to make it point to the correct system UAPI
header (or the header could be too old, but looking at it, it seems
that it does have CAP_BPF), in which case it defaulted to libcap's
version of the header, which in 2.26 stops at CAP_AUDIT (37). In that
case, __CAP_BITS is worth 37 and is lower than CAP_BPF, the check in
cap_get_flag() fails and we get -EINVAL.

The commit referenced above changed the comparison for libcap 2.30+ to
compare "value" with __CAP_MAXBITS == 64 instead, which works
correctly.

Thanks for the report and the shared debug session!
Quentin

Re: Error: bug: failed to retrieve CAP_BPF status: Invalid argument

[Quentin Monnet]

On Tue, 16 Aug 2022 at 01:05, Quentin Monnet <quentin@isovalent.com> wrote:
>
> On Mon, 15 Aug 2022 at 23:26, Vincent Li <vincent.mc.li@gmail.com> wrote:
> >
> > On Mon, Aug 15, 2022 at 3:18 PM Quentin Monnet <quentin@isovalent.com> wrote:

> So we investigated this on Slack. The issue is related to libcap (and
> to how libcap is built on CentOS); it is fixed in libcap 2.30 and
> older.

2.30 and newer, of course.

Re: Error: bug: failed to retrieve CAP_BPF status: Invalid argument

[Vincent Li]

On Mon, Aug 15, 2022 at 5:05 PM Quentin Monnet <quentin@isovalent.com> wrote:
>
> On Mon, 15 Aug 2022 at 23:26, Vincent Li <vincent.mc.li@gmail.com> wrote:
> >
> > On Mon, Aug 15, 2022 at 3:18 PM Quentin Monnet <quentin@isovalent.com> wrote:
> > >
> > > Hi Vincent,
> > >
> > > On Mon, 15 Aug 2022 at 18:46, Vincent Li <vincent.mc.li@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I compile and run kernel 5.18.0 in Centos 8 from bpf-next in my dev
> > > > machine, I also compiled bpftool from bpf-next on same machine, when
> > > > run bpftool on same machine, I got :
> > > >
> > > > ./bpftool feature probe
> > > >
> > > > Error: bug: failed to retrieve CAP_BPF status: Invalid argument
> > > >
> > > > where bpftool to retrieve CAP_BPF ? from running kernel or from somewhere else?
> > >
> > > Yes, bpftool calls cap_get_proc() to get the capabilities of the
> > > current process. From what I understand of your output, it looks like
> > > capget() returns CAP_BPF: I believe the "0x1c0" value at the end is
> > > (1<<(CAP_CHECKPOINT_RESTORE-32)) + (1<<(CAP_BPF-32)) +
> > > (1<<(CAP_PERFMON-32)). You could probably check this with a more
> > > recent version of strace.
> > >
> > > Then assuming you do retrieve CAP_BPF from capget(), I don't know why
> > > cap_get_flag() in bpftool fails to retrieve the capability state. It
> > > would be worth running bpftool in GDB to check what happens. The check
> > > in libcap is here [0] but I don't see where we would fail to provide
> > > valid arguments. Just in case, could you please let me know what
> > > version of libcap you're using when compiling bpftool?
> >
> > I think I installed libcap through centos distro
> >
> > [root@centos-dev ~]# rpm -qi libcap.x86_64
> >
> > Name        : libcap
> >
> > Version     : 2.26
>
> So we investigated this on Slack. The issue is related to libcap (and
> to how libcap is built on CentOS); it is fixed in libcap 2.30 and
> older.
>
> For the record, this is the commit that fixed it:
> https://git.kernel.org/pub/scm/libs/libcap/libcap.git/commit/?id=f1f62a748d7c67361e91e32d26abafbfb03eeee4
>
> Before this, cap_get_flag() would compare its argument "value" (in our
> case, CAP_BPF == 39) with __CAP_BITS. This __CAP_BITS constant is
> defined in libcap/cap_names.h, generated by libcap/_makenames.c from
> the list in libcap/cap_names.list.h. The latter header file is itself
> generated in libcap/Makefile from the UAPI header at
> $(KERNEL_HEADERS)/linux/capability.h, which defaults to the local
> libcap/include/uapi/linux/capability.h.
>
> On your CentOS, the libcap version may have been compiled without
> setting KERNEL_HEADERS to make it point to the correct system UAPI
> header (or the header could be too old, but looking at it, it seems
> that it does have CAP_BPF), in which case it defaulted to libcap's
> version of the header, which in 2.26 stops at CAP_AUDIT (37). In that
> case, __CAP_BITS is worth 37 and is lower than CAP_BPF, the check in
> cap_get_flag() fails and we get -EINVAL.
>
> The commit referenced above changed the comparison for libcap 2.30+ to
> compare "value" with __CAP_MAXBITS == 64 instead, which works
> correctly.
>
> Thanks for the report and the shared debug session!
> Quentin

Thanks Quentin for your quick response and analysis :)

Re: Error: bug: failed to retrieve CAP_BPF status: Invalid argument

[Toke Høiland-Jørgensen]

>Vincent Li <vincent.mc.li@gmail.com> writes:

> On Mon, Aug 15, 2022 at 5:05 PM Quentin Monnet <quentin@isovalent.com> wrote:
>>
>> On Mon, 15 Aug 2022 at 23:26, Vincent Li <vincent.mc.li@gmail.com> wrote:
>> >
>> > On Mon, Aug 15, 2022 at 3:18 PM Quentin Monnet <quentin@isovalent.com> wrote:
>> > >
>> > > Hi Vincent,
>> > >
>> > > On Mon, 15 Aug 2022 at 18:46, Vincent Li <vincent.mc.li@gmail.com> wrote:
>> > > >
>> > > > Hi,
>> > > >
>> > > > I compile and run kernel 5.18.0 in Centos 8 from bpf-next in my dev
>> > > > machine, I also compiled bpftool from bpf-next on same machine, when
>> > > > run bpftool on same machine, I got :
>> > > >
>> > > > ./bpftool feature probe
>> > > >
>> > > > Error: bug: failed to retrieve CAP_BPF status: Invalid argument
>> > > >
>> > > > where bpftool to retrieve CAP_BPF ? from running kernel or from somewhere else?
>> > >
>> > > Yes, bpftool calls cap_get_proc() to get the capabilities of the
>> > > current process. From what I understand of your output, it looks like
>> > > capget() returns CAP_BPF: I believe the "0x1c0" value at the end is
>> > > (1<<(CAP_CHECKPOINT_RESTORE-32)) + (1<<(CAP_BPF-32)) +
>> > > (1<<(CAP_PERFMON-32)). You could probably check this with a more
>> > > recent version of strace.
>> > >
>> > > Then assuming you do retrieve CAP_BPF from capget(), I don't know why
>> > > cap_get_flag() in bpftool fails to retrieve the capability state. It
>> > > would be worth running bpftool in GDB to check what happens. The check
>> > > in libcap is here [0] but I don't see where we would fail to provide
>> > > valid arguments. Just in case, could you please let me know what
>> > > version of libcap you're using when compiling bpftool?
>> >
>> > I think I installed libcap through centos distro
>> >
>> > [root@centos-dev ~]# rpm -qi libcap.x86_64
>> >
>> > Name        : libcap
>> >
>> > Version     : 2.26
>>
>> So we investigated this on Slack. The issue is related to libcap (and
>> to how libcap is built on CentOS); it is fixed in libcap 2.30 and
>> older.
>>
>> For the record, this is the commit that fixed it:
>> https://git.kernel.org/pub/scm/libs/libcap/libcap.git/commit/?id=f1f62a748d7c67361e91e32d26abafbfb03eeee4
>>
>> Before this, cap_get_flag() would compare its argument "value" (in our
>> case, CAP_BPF == 39) with __CAP_BITS. This __CAP_BITS constant is
>> defined in libcap/cap_names.h, generated by libcap/_makenames.c from
>> the list in libcap/cap_names.list.h. The latter header file is itself
>> generated in libcap/Makefile from the UAPI header at
>> $(KERNEL_HEADERS)/linux/capability.h, which defaults to the local
>> libcap/include/uapi/linux/capability.h.
>>
>> On your CentOS, the libcap version may have been compiled without
>> setting KERNEL_HEADERS to make it point to the correct system UAPI
>> header (or the header could be too old, but looking at it, it seems
>> that it does have CAP_BPF), in which case it defaulted to libcap's
>> version of the header, which in 2.26 stops at CAP_AUDIT (37). In that
>> case, __CAP_BITS is worth 37 and is lower than CAP_BPF, the check in
>> cap_get_flag() fails and we get -EINVAL.
>>
>> The commit referenced above changed the comparison for libcap 2.30+ to
>> compare "value" with __CAP_MAXBITS == 64 instead, which works
>> correctly.
>>
>> Thanks for the report and the shared debug session!
>> Quentin
>
> Thanks Quentin for your quick response and analysis :)

FYI, CAP_BPF should also be fixed in the version of libcap shipped with
RHEL8.5 (version libcap-2.26-5.el8). This should be available in CentOS
Stream as well, so just updating the package should be enough...

-Toke

Re: Error: bug: failed to retrieve CAP_BPF status: Invalid argument

[Vincent Li]

On Tue, Aug 16, 2022 at 3:52 AM Toke Høiland-Jørgensen <toke@kernel.org> wrote:
>
> >Vincent Li <vincent.mc.li@gmail.com> writes:
>
> > On Mon, Aug 15, 2022 at 5:05 PM Quentin Monnet <quentin@isovalent.com> wrote:
> >>
> >> On Mon, 15 Aug 2022 at 23:26, Vincent Li <vincent.mc.li@gmail.com> wrote:
> >> >
> >> > On Mon, Aug 15, 2022 at 3:18 PM Quentin Monnet <quentin@isovalent.com> wrote:
> >> > >
> >> > > Hi Vincent,
> >> > >
> >> > > On Mon, 15 Aug 2022 at 18:46, Vincent Li <vincent.mc.li@gmail.com> wrote:
> >> > > >
> >> > > > Hi,
> >> > > >
> >> > > > I compile and run kernel 5.18.0 in Centos 8 from bpf-next in my dev
> >> > > > machine, I also compiled bpftool from bpf-next on same machine, when
> >> > > > run bpftool on same machine, I got :
> >> > > >
> >> > > > ./bpftool feature probe
> >> > > >
> >> > > > Error: bug: failed to retrieve CAP_BPF status: Invalid argument
> >> > > >
> >> > > > where bpftool to retrieve CAP_BPF ? from running kernel or from somewhere else?
> >> > >
> >> > > Yes, bpftool calls cap_get_proc() to get the capabilities of the
> >> > > current process. From what I understand of your output, it looks like
> >> > > capget() returns CAP_BPF: I believe the "0x1c0" value at the end is
> >> > > (1<<(CAP_CHECKPOINT_RESTORE-32)) + (1<<(CAP_BPF-32)) +
> >> > > (1<<(CAP_PERFMON-32)). You could probably check this with a more
> >> > > recent version of strace.
> >> > >
> >> > > Then assuming you do retrieve CAP_BPF from capget(), I don't know why
> >> > > cap_get_flag() in bpftool fails to retrieve the capability state. It
> >> > > would be worth running bpftool in GDB to check what happens. The check
> >> > > in libcap is here [0] but I don't see where we would fail to provide
> >> > > valid arguments. Just in case, could you please let me know what
> >> > > version of libcap you're using when compiling bpftool?
> >> >
> >> > I think I installed libcap through centos distro
> >> >
> >> > [root@centos-dev ~]# rpm -qi libcap.x86_64
> >> >
> >> > Name        : libcap
> >> >
> >> > Version     : 2.26
> >>
> >> So we investigated this on Slack. The issue is related to libcap (and
> >> to how libcap is built on CentOS); it is fixed in libcap 2.30 and
> >> older.
> >>
> >> For the record, this is the commit that fixed it:
> >> https://git.kernel.org/pub/scm/libs/libcap/libcap.git/commit/?id=f1f62a748d7c67361e91e32d26abafbfb03eeee4
> >>
> >> Before this, cap_get_flag() would compare its argument "value" (in our
> >> case, CAP_BPF == 39) with __CAP_BITS. This __CAP_BITS constant is
> >> defined in libcap/cap_names.h, generated by libcap/_makenames.c from
> >> the list in libcap/cap_names.list.h. The latter header file is itself
> >> generated in libcap/Makefile from the UAPI header at
> >> $(KERNEL_HEADERS)/linux/capability.h, which defaults to the local
> >> libcap/include/uapi/linux/capability.h.
> >>
> >> On your CentOS, the libcap version may have been compiled without
> >> setting KERNEL_HEADERS to make it point to the correct system UAPI
> >> header (or the header could be too old, but looking at it, it seems
> >> that it does have CAP_BPF), in which case it defaulted to libcap's
> >> version of the header, which in 2.26 stops at CAP_AUDIT (37). In that
> >> case, __CAP_BITS is worth 37 and is lower than CAP_BPF, the check in
> >> cap_get_flag() fails and we get -EINVAL.
> >>
> >> The commit referenced above changed the comparison for libcap 2.30+ to
> >> compare "value" with __CAP_MAXBITS == 64 instead, which works
> >> correctly.
> >>
> >> Thanks for the report and the shared debug session!
> >> Quentin
> >
> > Thanks Quentin for your quick response and analysis :)
>
> FYI, CAP_BPF should also be fixed in the version of libcap shipped with
> RHEL8.5 (version libcap-2.26-5.el8). This should be available in CentOS
> Stream as well, so just updating the package should be enough...
>
> -Toke

Thanks! good to know.