Re: Create a iSCSI DomU with disks in another DomU running on the same Dom0

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Fri, Jan 11, 2013 at 08:29:12PM +0100, Roger Pau Monné wrote:
> On 11/01/13 19:51, Konrad Rzeszutek Wilk wrote:
> > On Fri, Jan 11, 2013 at 04:57:52PM +0100, Roger Pau Monné wrote:
> >> Hello Konrad,
> >>
> >> I've found the problem, blkback is adding granted pages to the bio that 
> >> is then passed to the underlying block device. When using a iscsi 
> >> target running on another DomU in the same h/w this bios end up in 
> >> netback, and then when performing the gnttab copy operation, it 
> >> complains because the passed mfn belongs to a different domain.
> > 
> > OK, so my original theory was sound. The m2p override "sticks".
> >>
> >> I've checked this by applying the appended patch to blkback, which 
> >> allocates a buffer to pass to the bio instead of using the granted 
> >> page. Of course this should not applied, since it implies additional 
> >> memcpys.
> >>
> >> I think the right way to solve this would be to change netback to 
> >> use gnttab_map and memcpy instead of gnttab_copy, but I guess this 
> >> will imply a performance degradation (haven't benchmarked it, but I 
> >> assume gnttab_copy is used in netback because it is faster than 
> >> gnttab_map + memcpy + gnttab_unmap).
> > 
> > Or blkback is altered to use grant_copy.
> 
> This would not work with the persistent-grants extension, and also when
> scaling to a large number of guests will probably have a degraded
> performance due to the grant table lock (compared to using persistent
> grants).
> 
> > Or perhaps m2p_override
> > can do multiple PAGE_FOREIGN? (So if it detects a collision it will
> > do something smart.. like allocate a new page or update the 
> > kmap_op with extra information).
> 
> What we could do is add extra information to m2p_override, containing
> the grant_ref_t and domid, so when a FOREIGN_FRAME is detected in
> grant_copy (or netback) the grant_ref_t and domid of the passed mfn is
> used instead of the mfn (provided that grant_copy can perform a copy
> between two grant references of different domains).
> 
> > 
> > 
> > And yes, grant_map in netback is much much slower that grant_copy
> > (I tested 2.6.32 vs 3.7 using a Xen 4.1.3 with the grant_copy fixes
> > that Jan came up with).
> 
> Yes, I see there's no way we are going to use grant_map instead of
> grant_copy. I guess this will no longer be true once netback/front
> starts using the persistent grants extension.

Hm? Annie posted patches for the persistent grants on netback/netfront
and they did not show much improvement (as you are already doing grant_copy).

Or are you saying change netback to use grant_map and utilize the
skb->deconstructor to keep track of it? And then do persistent grant
extensions on it?