Re: [admin] [Pkg-xen-devel] [BUG] task jbd2/xvda4-8:174 blocked for more than 120 seconds.

From: Samuel Thibault <admin@listes.aquilenet.fr>
To: admin@listes.aquilenet.fr, Dongli Zhang <dongli.zhang@oracle.com>
Cc: Hans van Kranenburg <hans@knorrie.org>,
	xen-devel <xen-devel@lists.xenproject.org>,
	Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>
Subject: Re: [admin] [Pkg-xen-devel] [BUG] task jbd2/xvda4-8:174 blocked for more than 120 seconds.
Date: Wed, 20 Feb 2019 00:49:46 +0100	[thread overview]
Message-ID: <20190219234946.wsjvht2fdbfnfdf7@function> (raw)
In-Reply-To: <29cb64d5-1d7f-30d5-d294-c3556b3e02e1@oracle.com>

Dongli Zhang, le lun. 18 févr. 2019 08:09:56 +0800, a ecrit:
> 
> 
> On 2/18/19 5:29 AM, Samuel Thibault wrote:
> > Hello,
> > 
> > Dongli Zhang, le mar. 12 févr. 2019 12:11:20 +0800, a ecrit:
> >> On 02/12/2019 06:10 AM, Samuel Thibault wrote:
> >>> Hans van Kranenburg, le lun. 11 févr. 2019 22:59:11 +0100, a ecrit:
> >>>> On 2/11/19 2:37 AM, Dongli Zhang wrote:
> >>>>>
> >>>>> On 2/10/19 12:35 AM, Samuel Thibault wrote:
> >>>>>>
> >>>>>> Hans van Kranenburg, le sam. 09 févr. 2019 17:01:55 +0100, a ecrit:
> >>>>>>>> I have forwarded the original mail: all VM I/O get stuck, and thus the
> >>>>>>>> VM becomes unusable.
> >>>>>>>
> >>>>>>> These are in many cases the symptoms of running out of "grant frames".
> >>>>>>
> >>>>>> Oh!  That could be it indeed.  I'm wondering what could be monopolizing
> >>>>>> them, though, and why +deb9u11 is affected while +deb9u10 is not.  I'm
> >>>>>> afraid increasing the gnttab max size to 32 might just defer filling it
> >>>>>> up.
> >>>>>>
> >>>>>>>   -# ./xen-diag  gnttab_query_size 5
> >>>>>>>   domid=5: nr_frames=11, max_nr_frames=32
> >>>>>>
> >>>>>> The current value is 31 over max 32 indeed.
> >>>>>
> >>>>> Assuming this is grant v1, there are still 4096/8=512 grant references available
> >>>>> (32-31=1 frame available). I do not think the I/O hang can be affected by the
> >>>>> lack of grant entry.
> >>>>
> >>>> I suspect that 31 measurement was taken when the domU was not hanging yet.
> >>>
> >>> Indeed, I didn't have the hanging VM offhand.  I have looked again, it's
> >>> now at 33. We'll have to monitor to check that it doesn't continue just
> >>> increasing.
> >>
> >> If the max used to be 32 and the current is already 33, this indicates the grant
> >> entries might be used up in the past before the max_nr_frames is tuned.
> > 
> > The number seems to be going up by about one every day. So probably a
> > grant entry leak somewhere :/
> 
> This might not be a grant leak. The block pv driver would hold the persistent
> grant for a long time.
> 
> Juergen has introduced the feature to reclaim the stale grants.
> 
> blkfront since a46b53672b2c2e3770b38a4abf90d16364d2584b
> 
> blkback since 973e5405f2f67ddbb2bf07b3ffc71908a37fea8e

Ok, that hasn't reached Debian Stretch yet :/

Let's keep monitoring for now.

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel