From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A4CCC433DF for ; Tue, 4 Aug 2020 15:56:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 555CA2177B for ; Tue, 4 Aug 2020 15:56:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596556570; bh=JwByKtuKkgVsE2mmR5XBVyd2OPKZ4OuZl1WgOiRQTiU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-ID:From; b=Jqxatrr5bwzJscR4KCTag8epW213E03txvzKa4xZZ4VKNUwrEclOmxAGe9BjiCNaz q4hJU2rkEFijQ5c1vbSs+aJu5TTNJRDmDrxWwgdZhcMt+y0Rz2Pg1btVOo22s0NBts yyUvw022xOPFoEn1gJLDHJeLYP1zLtOhUf3fFkqE= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728157AbgHDP4B (ORCPT ); Tue, 4 Aug 2020 11:56:01 -0400 Received: from mail.kernel.org ([198.145.29.99]:56260 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727997AbgHDPz6 (ORCPT ); Tue, 4 Aug 2020 11:55:58 -0400 Received: from localhost (unknown [213.57.247.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 85CCF208A9; Tue, 4 Aug 2020 15:55:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596556558; bh=JwByKtuKkgVsE2mmR5XBVyd2OPKZ4OuZl1WgOiRQTiU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=aZW+mdAjwooYJ2UI2kYt2aKqnAwDsTnnOunVMWwZtOOPFHh4tn9mnXdOEt42wxKFd oXbapRUUUkaOtH4rAmA+pEG/s4OjfF/ygX0ur8yLE7L5gCfYjFkxt+hQAJJXelVP/n ENwbGhzBEpnpUyaXDG9C6BQSvEN8+BrR1OMCTfpk= Date: Tue, 4 Aug 2020 18:55:54 +0300 From: Leon Romanovsky To: Chuck Lever Cc: Timo Rothenpieler , Linux NFS Mailing List , linux-rdma Subject: Re: NFS over RDMA issues on Linux 5.4 Message-ID: <20200804155554.GD4432@unreal> References: <20200804093635.GA4432@unreal> <92a5a932-b843-eed3-555e-7557ccc1f308@rothenpieler.org> <20200804122557.GB4432@unreal> <20200804134642.GC4432@unreal> <45BA86D8-52A3-407E-83BE-27343C0182C5@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Tue, Aug 04, 2020 at 11:34:05AM -0400, Chuck Lever wrote: > > > > On Aug 4, 2020, at 9:53 AM, Chuck Lever wrote: > > > > > > > >> On Aug 4, 2020, at 9:46 AM, Leon Romanovsky wrote: > >> > >> On Tue, Aug 04, 2020 at 09:12:55AM -0400, Chuck Lever wrote: > >>> > >>> > >>>> On Aug 4, 2020, at 9:08 AM, Timo Rothenpieler wrote: > >>>> > >>>> On 04.08.2020 14:49, Chuck Lever wrote: > >>>>> Timo, I tend to think this is not a configuration issue. > >>>>> Do you know of a known working kernel? > >>>> > >>>> This is a brand new system, it's never been running with any kernel older than 5.4, and downgrading it to 4.19 or something else while in operation is unfortunately not easily possible. For a client it would definitely not be out of the question, but the main nfs server I cannot easily downgrade. > >>>> > >>>> Also keep in mind that the dmesg spam happens on both server and client simultaneously. > >>> > >>> Let's start with the client only, since restarting it seems to clear the problem. > >> > >> It is client because according to the server CQE errors, it is Remote_Invalid_Request_Error > >> with "9.7.5.2.2 NAK CODES" from IBTA. > > > > Thanks! OK, then let's use ftrace. > > > > Timo, can you install trace-cmd on your client? Then: > > > > 1. # trace-cmd record -e rpcrdma -e sunrpc > > > > 2. Trigger the problem > > > > 3. Control-C the trace-cmd, and copy the trace.dat file to another system > > > > 4. reboot your client > > > > Then send me your trace.dat. You don't have to cc the mailing lists. > > I see a LOC_LEN_ERR on a Receive. Leon, doesn't that mean the server's > Send was too large? 1. We have local_length_error counter, it can help to run it on server and clients. [leonro@vm ~]$ cat /sys/class/infiniband/ibp0s9/ports/1/hw_counters/resp_local_length_error 0 resp_local_length_error - "Number of times responder detected local length errors." 2. LOC_LEN_ERR supports that is written in CQE error on the client. This is what is written in our HW document: IB compliant completion with error syndrome 0x1: Local_Length_Error 3. >From IBTA, 11.6.2 COMPLETION RETURN STATUS Local Length Error - Generated for a Work Request posted to the local Send Queue when the sum of the Data Segment lengths exceeds the message length for the channel adapter port. Generated for a Work Request posted to the local Receive Queue when the sum of the Data Segment lengths is too small to receive a valid incoming message or the length of the incoming message is greater than the maximum message size supported by the HCA port that received the message. So if "1" works :), we will be able to distinguish if client sends too large WR or recieves too large. Thanks > > Timo, what filesystem are you sharing on your NFS server? The thing that > comes to mind is https://bugzilla.kernel.org/show_bug.cgi?id=198053 > > > -- > Chuck Lever > > >