Occasionally hanging sockopts selftests

Occasionally hanging sockopts selftests

[Andrii Nakryiko]

Hey Stanislav,

I've noticed that on 5.5 kernel libbpf Travis CI test runs
occasionally fail due to selftests hanging indefinitely. It seems like
it always happens after sockopt tests succeed, and while
sockopt_inherit test is running. Doesn't seem like the latest kernel
is affected (I haven't found hangs for the latest kernel in recent
history).

This is the latest version of selftests, but running on an older (5.5)
version of kernel. So whatever fixes went into selftests are there
already. So I wonder if there were any kernel bugs that were fixed
already but could cause hangs on 5.5?

I can disable that specific test for 5.5, but though I should bring
this up first, just in case there are still some bugs in selftests.

Thanks for checking!

Two most recent failures (not that they are helpful, because there is
no output until tests completes, but still):

  - https://travis-ci.com/github/libbpf/libbpf/jobs/359067538
  - https://travis-ci.com/github/libbpf/libbpf/jobs/359784775

Re: Occasionally hanging sockopts selftests

[sdf]

On 07/10, Andrii Nakryiko wrote:
> Hey Stanislav,

> I've noticed that on 5.5 kernel libbpf Travis CI test runs
> occasionally fail due to selftests hanging indefinitely. It seems like
> it always happens after sockopt tests succeed, and while
> sockopt_inherit test is running. Doesn't seem like the latest kernel
> is affected (I haven't found hangs for the latest kernel in recent
> history).

> This is the latest version of selftests, but running on an older (5.5)
> version of kernel. So whatever fixes went into selftests are there
> already. So I wonder if there were any kernel bugs that were fixed
> already but could cause hangs on 5.5?

> I can disable that specific test for 5.5, but though I should bring
> this up first, just in case there are still some bugs in selftests.

> Thanks for checking!

> Two most recent failures (not that they are helpful, because there is
> no output until tests completes, but still):

>    - https://travis-ci.com/github/libbpf/libbpf/jobs/359067538
>    - https://travis-ci.com/github/libbpf/libbpf/jobs/359784775
Nothing pops up, I don't think we did any fixes to address any
occasional failles like that.

The only fixes we did are:
d8fe449a9c51 - bpf: Don't return EINVAL from {get,set}sockopt when optlen >  
PAGE_SIZE
9babe825da76 - bpf: always allocate at least 16 bytes for setsockopt hook

And I don't see how this test can hang without any of those (should
either always pass or fail).

Let's maybe try to disable it, as you suggested, and see if that's
indeed this sockopt_inherit test that's misbehaving?
I can try to find some time next week to reproduce.

Re: Occasionally hanging sockopts selftests

[Andrii Nakryiko]

On Fri, Jul 10, 2020 at 1:20 PM <sdf@google.com> wrote:
>
> On 07/10, Andrii Nakryiko wrote:
> > Hey Stanislav,
>
> > I've noticed that on 5.5 kernel libbpf Travis CI test runs
> > occasionally fail due to selftests hanging indefinitely. It seems like
> > it always happens after sockopt tests succeed, and while
> > sockopt_inherit test is running. Doesn't seem like the latest kernel
> > is affected (I haven't found hangs for the latest kernel in recent
> > history).
>
> > This is the latest version of selftests, but running on an older (5.5)
> > version of kernel. So whatever fixes went into selftests are there
> > already. So I wonder if there were any kernel bugs that were fixed
> > already but could cause hangs on 5.5?
>
> > I can disable that specific test for 5.5, but though I should bring
> > this up first, just in case there are still some bugs in selftests.
>
> > Thanks for checking!
>
> > Two most recent failures (not that they are helpful, because there is
> > no output until tests completes, but still):
>
> >    - https://travis-ci.com/github/libbpf/libbpf/jobs/359067538
> >    - https://travis-ci.com/github/libbpf/libbpf/jobs/359784775
> Nothing pops up, I don't think we did any fixes to address any
> occasional failles like that.
>
> The only fixes we did are:
> d8fe449a9c51 - bpf: Don't return EINVAL from {get,set}sockopt when optlen >
> PAGE_SIZE
> 9babe825da76 - bpf: always allocate at least 16 bytes for setsockopt hook
>
> And I don't see how this test can hang without any of those (should
> either always pass or fail).
>
> Let's maybe try to disable it, as you suggested, and see if that's

I'm pretty sure it's sockopt_inherit, because I compared what's the
next test after sockopt in the normal case. So there is no doubt at
which test it hangs, but why is a different matter. I'll leave it
enabled for now, it doesn't fail very often, so it's ok.

> indeed this sockopt_inherit test that's misbehaving?
> I can try to find some time next week to reproduce.

Thanks!