Re: Create a iSCSI DomU with disks in another DomU running on the same Dom0

["Roger Pau Monné" <roger.pau@citrix.com>]

On 11/01/13 22:09, Konrad Rzeszutek Wilk wrote:
> On Fri, Jan 11, 2013 at 08:29:12PM +0100, Roger Pau Monné wrote:
>> On 11/01/13 19:51, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Jan 11, 2013 at 04:57:52PM +0100, Roger Pau Monné wrote:
>>>> Hello Konrad,
>>>>
>>>> I've found the problem, blkback is adding granted pages to the bio that 
>>>> is then passed to the underlying block device. When using a iscsi 
>>>> target running on another DomU in the same h/w this bios end up in 
>>>> netback, and then when performing the gnttab copy operation, it 
>>>> complains because the passed mfn belongs to a different domain.
>>>
>>> OK, so my original theory was sound. The m2p override "sticks".
>>>>
>>>> I've checked this by applying the appended patch to blkback, which 
>>>> allocates a buffer to pass to the bio instead of using the granted 
>>>> page. Of course this should not applied, since it implies additional 
>>>> memcpys.
>>>>
>>>> I think the right way to solve this would be to change netback to 
>>>> use gnttab_map and memcpy instead of gnttab_copy, but I guess this 
>>>> will imply a performance degradation (haven't benchmarked it, but I 
>>>> assume gnttab_copy is used in netback because it is faster than 
>>>> gnttab_map + memcpy + gnttab_unmap).
>>>
>>> Or blkback is altered to use grant_copy.
>>
>> This would not work with the persistent-grants extension, and also when
>> scaling to a large number of guests will probably have a degraded
>> performance due to the grant table lock (compared to using persistent
>> grants).
>>
>>> Or perhaps m2p_override
>>> can do multiple PAGE_FOREIGN? (So if it detects a collision it will
>>> do something smart.. like allocate a new page or update the 
>>> kmap_op with extra information).
>>
>> What we could do is add extra information to m2p_override, containing
>> the grant_ref_t and domid, so when a FOREIGN_FRAME is detected in
>> grant_copy (or netback) the grant_ref_t and domid of the passed mfn is
>> used instead of the mfn (provided that grant_copy can perform a copy
>> between two grant references of different domains).

Since the issue I'm having is not common I'm not sure if this solution
is worth it, it will imply storing a pointer to a struct in the page
private data, that stores the mfn, grant reference and domid (now we are
only storing the mfn in the page private data).

>>>
>>>
>>> And yes, grant_map in netback is much much slower that grant_copy
>>> (I tested 2.6.32 vs 3.7 using a Xen 4.1.3 with the grant_copy fixes
>>> that Jan came up with).
>>
>> Yes, I see there's no way we are going to use grant_map instead of
>> grant_copy. I guess this will no longer be true once netback/front
>> starts using the persistent grants extension.
> 
> Hm? Annie posted patches for the persistent grants on netback/netfront
> and they did not show much improvement (as you are already doing grant_copy).
> 
> Or are you saying change netback to use grant_map and utilize the
> skb->deconstructor to keep track of it? And then do persistent grant
> extensions on it?

I'm not really familiar with the net code, but I've had a quick look at
the grant_copy operation in Xen and it is indeed using the grant lock to
protect some parts of the code.

Using persistent grants should provide better performance in the long
run because once the grant is mapped we don't have to issue any more
grant operations, thus avoiding grant lock contention (maybe I'm missing
something here). skb deconstructor should probably be used in netfront,
to return the persistently mapped grant to the list of free grants, but
I'm not sure if we will need to use it in netback.