Re: NFS over RDMA issues on Linux 5.4

[Leon Romanovsky <leon@kernel.org>]

On Mon, Aug 03, 2020 at 12:24:21PM -0400, Chuck Lever wrote:
> Hi Timo-
>
> > On Aug 3, 2020, at 11:05 AM, Timo Rothenpieler <timo@rothenpieler.org> wrote:
> >
> > Hello,
> >
> > I have just deployed a new system with Mellanox ConnectX-4 VPI EDR IB cards and wanted to setup NFS over RDMA on it.
> >
> > However, while mounting the FS over RDMA works fine, actually using it results in the following messages absolutely hammering dmesg on both client and server:
> >
> >> https://gist.github.com/BtbN/9582e597b6581f552fa15982b0285b80#file-server-log
> >
> > The spam only stops once I forcibly reboot the client. The filesystem gets nowhere during all this. The retrans counter in nfsstat just keeps going up, nothing actually gets done.
> >
> > This is on Linux 5.4.54, using nfs-utils 2.4.3.
> > The mlx5 driver had enhanced-mode disabled in order to enable IPoIB connected mode with an MTU of 65520.
> >
> > Normal NFS 4.2 over tcp works perfectly fine on this setup, it's only when I mount via rdma that things go wrong.
> >
> > Is this an issue on my end, or did I run into a bug somewhere here?
> > Any pointers, patches and solutions to test are welcome.
>
> I haven't seen that failure mode here, so best I can recommend is
> keep investigating. I've copied linux-rdma in case they have any
> advice.

The mentioning of IPoIB is a slightly confusing in the context of NFS-over-RDMA.
Are you running NFS over IPoIB?

From brief look on CQE error syndrome (local length error), the client sends wrong WQE.

Thanks

>
> --
> Chuck Lever
>
>
>