From: Olga Kornievskaia <aglo@umich.edu>
To: Timothy Pearson <tpearson@raptorengineering.com>
Cc: linux-nfs <linux-nfs@vger.kernel.org>
Subject: Re: Callback slot table overflowed
Date: Fri, 6 Aug 2021 15:53:19 -0400 [thread overview]
Message-ID: <CAN-5tyHmiEE-vw=s6t_7UmWgHo2_U7zJOSwTPESY_NQA27ZsPQ@mail.gmail.com> (raw)
In-Reply-To: <361337129.54635.1628123839436.JavaMail.zimbra@raptorengineeringinc.com>
On Thu, Aug 5, 2021 at 12:15 AM Timothy Pearson
<tpearson@raptorengineering.com> wrote:
>
> On further investigation, the working server had already been rolled back to 4.19.0. Apparently the issue was insurmountable in 5.x.
>
> It should be simple enough to set up a test environment out of production for 5.x, if you have any debug tips / would like to see any debug options compiled in.
>
> Thanks!
>
> ----- Original Message -----
> > From: "Timothy Pearson" <tpearson@raptorengineering.com>
> > To: "linux-nfs" <linux-nfs@vger.kernel.org>
> > Sent: Wednesday, August 4, 2021 7:04:16 PM
> > Subject: Re: Callback slot table overflowed
>
> > Other information that may be helpful:
> >
> > All clients are using TCP
> > arm64 clients are unaffected by the bug
> > The armel clients use very small (4k) rsize/wsize buffers
> > Prior to the upgrade from Debian Stretch, everything was working perfectly
> >
> > ----- Original Message -----
> >> From: "Timothy Pearson" <tpearson@raptorengineering.com>
> >> To: "linux-nfs" <linux-nfs@vger.kernel.org>
> >> Sent: Wednesday, August 4, 2021 7:00:20 PM
> >> Subject: Callback slot table overflowed
> >
> >> All,
> >>
> >> We've hit an odd issue after upgrading a main NFS server from Debian Stretch to
> >> Debian Buster. In both cases the 5.13.4 kernel was used, however after the
> >> upgrade none of our ARM thin clients can mount their root filesystems -- early
> >> in the boot process I/O errors are returned immediately following "Callback
> >> slot table overflowed" in the client dmesg.
> >>
> >> I am unable to find any useful information on this "Callback slot table
> >> overflowed" message, and have no idea why it is only impacting our ARM (armel)
> >> clients. Both 4.14 and 5.3 on the client side show the issue, other client
> >> kernel versions were not tested.
> >>
> >> Curiously, increasing the rsize/wsize values to 65536 or higher reduces (but
> >> does not eliminate) the number of callback overflow messages.
> >>
> >> The server is a ppc64el 64k page host, and none of our pcc64el or amd64 thin
> >> clients are experiencing any problems. Nothing of interest appears in the
> >> server message log.
> >>
> >> Any troubleshooting hints would be most welcome.
A network trace would be useful.
5.3 should have this patch "SUNRPC: Fix up backchannel slot table
accounting". I believe "callback slot table overflowed" is hit when
the server sent more reqs than client can handle (ie doesn't have a
free slot to handle the request). A network trace would show that.
However you said this happens when the client is trying to mount and
besides cb_null requests I'm not sure what could be happening.
> >>
> > > Thank you!
next prev parent reply other threads:[~2021-08-06 19:53 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-05 0:00 Callback slot table overflowed Timothy Pearson
2021-08-05 0:04 ` Timothy Pearson
2021-08-05 0:37 ` Timothy Pearson
2021-08-06 19:53 ` Olga Kornievskaia [this message]
2021-08-06 20:37 ` Timothy Pearson
-- strict thread matches above, loose matches on Subject: below --
2011-01-20 3:07 Jim Rees
2011-01-20 14:01 ` Halevy, Benny
2011-01-20 15:19 ` Andy Adamson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAN-5tyHmiEE-vw=s6t_7UmWgHo2_U7zJOSwTPESY_NQA27ZsPQ@mail.gmail.com' \
--to=aglo@umich.edu \
--cc=linux-nfs@vger.kernel.org \
--cc=tpearson@raptorengineering.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).