Re: [Hackathon minutes] PV block improvements

["Roger Pau Monné" <roger.pau@citrix.com>]

On 21/06/13 20:07, Matt Wilson wrote:
> On Fri, Jun 21, 2013 at 07:10:59PM +0200, Roger Pau Monné wrote:
>> Hello,
>>
>> While working on further block improvements I've found an issue with
>> persistent grants in blkfront.
>>
>> Persistent grants basically allocate grants and then they are never
>> released, so both blkfront and blkback keep using the same memory pages
>> for all the transactions.
>>
>> This is not a problem in blkback, because we can dynamically choose how
>> many grants we want to map. On the other hand, blkfront cannot remove
>> the access to those grants at any point, because blkfront doesn't know
>> if blkback has this grants mapped persistently or not.
>>
>> So if for example we start expanding the number of segments in indirect
>> requests, to a value like 512 segments per requests, blkfront will
>> probably try to persistently map 512*32+512 = 16896 grants per device,
>> that's much more grants that the current default, which is 32*256 = 8192
>> (if using grant tables v2). This can cause serious problems to other
>> interfaces inside the DomU, since blkfront basically starts hoarding all
>> possible grants, leaving other interfaces completely locked.
> 
> Yikes.
> 
>> I've been thinking about different ways to solve this, but so far I
>> haven't been able to found a nice solution:
>>
>> 1. Limit the number of persistent grants a blkfront instance can use,
>> let's say that only the first X used grants will be persistently mapped
>> by both blkfront and blkback, and if more grants are needed the previous
>> map/unmap will be used.
> 
> I'm not thrilled with this option. It would likely introduce some
> significant performance variability, wouldn't it?
> 
>> 2. Switch to grant copy in blkback, and get rid of persistent grants (I
>> have not benchmarked this solution, but I'm quite sure it will involve a
>> performance regression, specially when scaling to a high number of domains).
> 
> Why do you think so?

I've hacked a prototype blkback using grant_copy instead of persistent
grants, and removed the persistent grants support in blkfront and indeed
the performance of grant_copy is lower than persistent grants, and it
seems to scale much worse. I've run several fio read/write benchmarks,
using 512 segments per request on a ramdisk, and the output is the
following:

http://xenbits.xen.org/people/royger/grant_copy/

Roger.