From: bfields@fieldses.org (J. Bruce Fields)
To: krichy@tvnetwork.hu
Cc: linux-nfs@vger.kernel.org
Subject: Re: nfs lockup
Date: Fri, 23 Oct 2015 14:10:01 -0400 [thread overview]
Message-ID: <20151023181001.GA15564@fieldses.org> (raw)
In-Reply-To: <alpine.DEB.2.20.1510211715430.5353@krichy.tvnetwork.hu>
On Wed, Oct 21, 2015 at 05:25:53PM +0200, krichy@tvnetwork.hu wrote:
> Dear devs,
>
> We have an nfs lockup issue. We run a ganeti cluster consisting of 7
> debian linux nodes and 1 freenas for hosting the vm images. The
> images are exported via nfsv3. The problem is that randomly we end
> in a livelock on one of our nodes.
>
> That means the nfs share is alive, we can list directories, files,
> even can read files (very slow, see later). And even can write to
> files, but the file close operation does not return, it gets
> blocked.
>
> The read is slow in that way that while copying a file from the
> share to /tmp, the data arrives very fast to the node, but in /tmp
> it accumulates slowly.
I don't understand what you mean by that. Do you have some measurements
to help quantify "very fast" and "slowly"?
--b.
>
> I've also opened a debian bug report on it, but I think it is not
> related to debian
> (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=801924).
>
> The only way is to reboot machine, with all the vm's running on it
> getting interrupted.
>
> I've captured each tasks' stack trace, hopefully it helps someone to
> find out the issue.
>
> Meanwhile the other 6 nodes can access the nfs share right, so I
> think this is not a networking or server issue. Restarting the nfs
> server on the server side still does not have any effect, not
> recovering. The nfs tcp connection is established, listing files
> works again, but writes not.
>
> Some information of the nodes:
> # uname -a
> Linux host 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u4
> (2015-09-19) x86_64 GNU/Linux
>
> They have 1.5G ram allocated to dom0, that should be enough.
>
> I know this information is little information, give me advice what
> to look for next time. Unfortunately I dont know how to reproduce
> it.
>
> Thanks in advance,
>
> Kojedzinszky Richard
> Euronet Magyarorszag Informatika Zrt.
next prev parent reply other threads:[~2015-10-23 18:10 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-21 15:25 nfs lockup krichy
2015-10-21 19:05 ` Benjamin Coddington
2015-10-21 20:09 ` krichy
2015-10-22 11:17 ` Benjamin Coddington
2015-10-23 18:10 ` J. Bruce Fields [this message]
2015-10-26 7:38 ` krichy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20151023181001.GA15564@fieldses.org \
--to=bfields@fieldses.org \
--cc=krichy@tvnetwork.hu \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox