From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757168Ab0CaOXM (ORCPT ); Wed, 31 Mar 2010 10:23:12 -0400 Received: from acsinet11.oracle.com ([141.146.126.233]:23438 "EHLO acsinet11.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753765Ab0CaOXJ (ORCPT ); Wed, 31 Mar 2010 10:23:09 -0400 Message-ID: <4BB35A53.5000003@oracle.com> Date: Wed, 31 Mar 2010 10:21:07 -0400 From: Chuck Lever User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.8) Gecko/20100301 Fedora/3.0.3-1.fc12 Lightning/1.0b2pre Thunderbird/3.0.3 MIME-Version: 1.0 To: Daniel J Blueman CC: Trond Myklebust , Al Viro , linux-nfs@vger.kernel.org, Linux Kernel Subject: Re: [2.6.34-rc2 NFS4 oops] open error path failure... References: <6278d2221003291136p6481fe8emfb039403343c082@mail.gmail.com> <20100329190307.GJ30031@ZenIV.linux.org.uk> <1269897740.15895.82.camel@localhost.localdomain> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A0B0205.4BB35AC4.0095:SCFMA4539814,ss=1,fgs=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/31/2010 07:20 AM, Daniel J Blueman wrote: > Talking of expensive, I see latencytop show>16000ms latency for > writing pages when I have a workload that does large buffered I/O to > an otherwise uncongested server. The gigabit network is saturated, and > reads often stall for 1000-4000ms (!). Client has the default 16 TCP > request slots, and server has 8 nfsds - the server is far from disk or > processor-saturated. I'll see if there is any useful debugging I can > get about this. That latency is pretty much guaranteed to be due to a long RPC backlog queue on the client. Bumping the size of the slot table to 128 and increasing the number of NFSD threads may help. -- chuck[dot]lever[at]oracle[dot]com