From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= Subject: Re: [Hackathon minutes] PV block improvements Date: Thu, 27 Jun 2013 17:12:37 +0200 Message-ID: <51CC5665.3070904@citrix.com> References: <519F81E1.8030203@citrix.com> <51C48923.1040808@citrix.com> <20130621180721.GA32653@u109add4315675089e695.ant.amazon.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20130621180721.GA32653@u109add4315675089e695.ant.amazon.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Matt Wilson Cc: Konrad Rzeszutek Wilk , xen-devel List-Id: xen-devel@lists.xenproject.org On 21/06/13 20:07, Matt Wilson wrote: > On Fri, Jun 21, 2013 at 07:10:59PM +0200, Roger Pau Monn=E9 wrote: >> Hello, >> >> While working on further block improvements I've found an issue with >> persistent grants in blkfront. >> >> Persistent grants basically allocate grants and then they are never >> released, so both blkfront and blkback keep using the same memory pages >> for all the transactions. >> >> This is not a problem in blkback, because we can dynamically choose how >> many grants we want to map. On the other hand, blkfront cannot remove >> the access to those grants at any point, because blkfront doesn't know >> if blkback has this grants mapped persistently or not. >> >> So if for example we start expanding the number of segments in indirect >> requests, to a value like 512 segments per requests, blkfront will >> probably try to persistently map 512*32+512 =3D 16896 grants per device, >> that's much more grants that the current default, which is 32*256 =3D 81= 92 >> (if using grant tables v2). This can cause serious problems to other >> interfaces inside the DomU, since blkfront basically starts hoarding all >> possible grants, leaving other interfaces completely locked. > = > Yikes. > = >> I've been thinking about different ways to solve this, but so far I >> haven't been able to found a nice solution: >> >> 1. Limit the number of persistent grants a blkfront instance can use, >> let's say that only the first X used grants will be persistently mapped >> by both blkfront and blkback, and if more grants are needed the previous >> map/unmap will be used. > = > I'm not thrilled with this option. It would likely introduce some > significant performance variability, wouldn't it? > = >> 2. Switch to grant copy in blkback, and get rid of persistent grants (I >> have not benchmarked this solution, but I'm quite sure it will involve a >> performance regression, specially when scaling to a high number of domai= ns). > = > Why do you think so? I've hacked a prototype blkback using grant_copy instead of persistent grants, and removed the persistent grants support in blkfront and indeed the performance of grant_copy is lower than persistent grants, and it seems to scale much worse. I've run several fio read/write benchmarks, using 512 segments per request on a ramdisk, and the output is the following: http://xenbits.xen.org/people/royger/grant_copy/ Roger.