From mboxrd@z Thu Jan  1 00:00:00 1970
From: Yonghong Song <yhs@fb.com>
Subject: selftests/bpf test_sockmap failure
Date: Tue, 24 Jul 2018 08:45:44 -0700
Message-ID: <fa8e6a0c-beee-a38b-dac9-d41c712f7345@fb.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
Cc: Yonghong Song <yhs@fb.com>, Martin Lau <kafai@fb.com>
To: John Fastabend <john.fastabend@gmail.com>,
        netdev <netdev@vger.kernel.org>
Return-path: <netdev-owner@vger.kernel.org>
Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:35294 "EHLO
        mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL)
        by vger.kernel.org with ESMTP id S2388371AbeGXQx2 (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 24 Jul 2018 12:53:28 -0400
Content-Language: en-US
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

In one of our production machines, tools/testing/selftests/bpf
test_sockmap failed randomly like below:

...
[TEST 78]: (512, 1, 1, sendmsg, pass,apply 1,): rx thread exited with 
err 1. FAILED
...

...
[TEST 80]: (2, 1024, 256, sendmsg, pass,apply 1,): rx thread exited with 
err 1. FAILED
...

...
[TEST 83]: (100, 1, 5, sendpage, pass,apply 1,): rx thread exited with 
err 1. FAILED
...

...
[TEST 79]: (512, 1, 1, sendpage, pass,apply 1,): rx thread exited with 
err 1. FAILED
...

The command line is just `test_sockmap`. The machine has 80 cpus, 256G 
memory. The kernel is based on 4.16 but backported with latest bpf-next 
bpf changes.

The failed test number (78, 79, 80, or 83) is random. But they all share
similar characteristics:
    . the option rate is greater than one, i.e., more than one
      sendmsg/sendpage in the sender forked process.
    . The txmsg_apply is not 0

I debugged a little bit. It happens in msg_loop() function below
"unexpected timeout" path.

...
                         slct = select(max_fd + 1, &w, NULL, NULL, 
&timeout);
                         if (slct == -1) {
                                 perror("select()");
                                 clock_gettime(CLOCK_MONOTONIC, &s->end);
                                 goto out_errno;
                         } else if (!slct) {
                                 if (opt->verbose)
                                         fprintf(stderr, "unexpected 
timeout\n");
                                 errno = -EIO;
                                 clock_gettime(CLOCK_MONOTONIC, &s->end);
                                 goto out_errno;
                         }
...

It appears that when the error happens, the receive process does not 
receive all bytes sent from the send process and eventually times out.

Has anybody seen this issue as well?
John, any comments on this failure?

Thanks,

Yonghong