Re: [Hackathon minutes] PV block improvements

["Roger Pau Monné" <roger.pau@citrix.com>]

On 21/06/13 22:16, Konrad Rzeszutek Wilk wrote:
> On Fri, Jun 21, 2013 at 07:10:59PM +0200, Roger Pau Monné wrote:
>> Hello,
>>
>> While working on further block improvements I've found an issue with
>> persistent grants in blkfront.
>>
>> Persistent grants basically allocate grants and then they are never
>> released, so both blkfront and blkback keep using the same memory pages
>> for all the transactions.
>>
>> This is not a problem in blkback, because we can dynamically choose how
>> many grants we want to map. On the other hand, blkfront cannot remove
>> the access to those grants at any point, because blkfront doesn't know
>> if blkback has this grants mapped persistently or not.
>>
>> So if for example we start expanding the number of segments in indirect
>> requests, to a value like 512 segments per requests, blkfront will
>> probably try to persistently map 512*32+512 = 16896 grants per device,
>> that's much more grants that the current default, which is 32*256 = 8192
>> (if using grant tables v2). This can cause serious problems to other
>> interfaces inside the DomU, since blkfront basically starts hoarding all
>> possible grants, leaving other interfaces completely locked.
>>
>> I've been thinking about different ways to solve this, but so far I
>> haven't been able to found a nice solution:
>>
>> 1. Limit the number of persistent grants a blkfront instance can use,
>> let's say that only the first X used grants will be persistently mapped
>> by both blkfront and blkback, and if more grants are needed the previous
>> map/unmap will be used.
>>
>> 2. Switch to grant copy in blkback, and get rid of persistent grants (I
>> have not benchmarked this solution, but I'm quite sure it will involve a
>> performance regression, specially when scaling to a high number of domains).
>>
>> 3. Increase the size of the grant_table or the size of a single grant
>> (from 4k to 2M) (this is from Stefano Stabellini).
>>
>> 4. Introduce a new request type that we can use to request blkback to
>> unmap certain grefs so we can free them in blkfront.
> 
> 
> 5). Lift the limit of grant pages a domain can have.
> 
> 6). Have an outstanding of grant pools that are mapped to a guest and
> recycle them? That way both netfront and blkfront could use them as needed?

If all the backends run in the same guest that could be a viable option,
but if we have backends running in different domains we will end up with
several different pools for each backend domain, and thus the scenario
is going to be quite similar to what we have now (a pool can hoard all
available grants and leave the others starving).