Sub page Grant Table mappings

Sub page Grant Table mappings

[Mike Belopuhov]

Hi,

I'm trying to get grant table sub page mappings working on Xen 4.5.
I know there have been some changes in the trunk regarding moving src/
dst checks closer together, but I can't test this easily atm.  Please
bear with me for a moment.  Or tell me that it might have been broken
previously.

What I'm trying to do is to map in a 2k cluster from the networking
stack (we call it an mbuf cluster, you might call it skb something)
onto the Netfront Rx ring.  2 clusters fit one page.  Therefore for
one frame address which is a (PA >> 12) I might have 2 entries one
after the other, both with sub_page.length = 2048 but one with a
sub_page.page_off = 2048 and the other with a 0 offset.  Both come
with (GTF_permit_access | GTF_sub_page) flags.

When I do that, Xen stops liking me and says:

(XEN) grant_table.c:2162:d0v1 copy dest out of bounds: 0 < 2048 || 90 > 2048
(XEN) grant_table.c:2162:d0v1 copy dest out of bounds: 0 < 2048 || 119 > 2048
(XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 70 > 2048
(XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 119 > 2048
(XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 119 > 2048
...

The relevant code does this:

    if ( dest_is_gref )
    {
        unsigned dest_off, dest_len;
        rc = __acquire_grant_for_copy(dd, op->dest.u.ref,
                                      current->domain->domain_id, 0,
                                      &d_frame, &d_pg, &dest_off, &dest_len, 1);
        if ( rc != GNTST_okay )
            goto error_out;
        have_d_grant = 1;
        if ( op->dest.offset < dest_off ||
             op->len > dest_len )
            PIN_FAIL(error_out, GNTST_general_error,
                     "copy dest out of bounds: %d < %d || %d > %d\n",
                     op->dest.offset, dest_off,
                     op->len, dest_len);
    }

I fail to understand what am I doing wrong in this case.  Any clues
will be greatly appreciated.

4k clusters mapped in as full_page mappings work as expected.

Cheers,
Mike

Re: Sub page Grant Table mappings

[Andrew Cooper]

On 22/12/15 20:59, Mike Belopuhov wrote:
> Hi,
>
> I'm trying to get grant table sub page mappings working on Xen 4.5.
> I know there have been some changes in the trunk regarding moving src/
> dst checks closer together, but I can't test this easily atm.  Please
> bear with me for a moment.  Or tell me that it might have been broken
> previously.
>
> What I'm trying to do is to map in a 2k cluster from the networking
> stack (we call it an mbuf cluster, you might call it skb something)
> onto the Netfront Rx ring.  2 clusters fit one page.  Therefore for
> one frame address which is a (PA >> 12) I might have 2 entries one
> after the other, both with sub_page.length = 2048 but one with a
> sub_page.page_off = 2048 and the other with a 0 offset.  Both come
> with (GTF_permit_access | GTF_sub_page) flags.
>
> When I do that, Xen stops liking me and says:
>
> (XEN) grant_table.c:2162:d0v1 copy dest out of bounds: 0 < 2048 || 90 > 2048
> (XEN) grant_table.c:2162:d0v1 copy dest out of bounds: 0 < 2048 || 119 > 2048
> (XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 70 > 2048
> (XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 119 > 2048
> (XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 119 > 2048
> ...
>
> The relevant code does this:
>
>     if ( dest_is_gref )
>     {
>         unsigned dest_off, dest_len;
>         rc = __acquire_grant_for_copy(dd, op->dest.u.ref,
>                                       current->domain->domain_id, 0,
>                                       &d_frame, &d_pg, &dest_off, &dest_len, 1);
>         if ( rc != GNTST_okay )
>             goto error_out;
>         have_d_grant = 1;
>         if ( op->dest.offset < dest_off ||
>              op->len > dest_len )
>             PIN_FAIL(error_out, GNTST_general_error,
>                      "copy dest out of bounds: %d < %d || %d > %d\n",
>                      op->dest.offset, dest_off,
>                      op->len, dest_len);
>     }
>
> I fail to understand what am I doing wrong in this case.  Any clues
> will be greatly appreciated.
>
> 4k clusters mapped in as full_page mappings work as expected.
>
> Cheers,
> Mike

I presume you are on x86 here.

Architecturally, a 4k page is the minimum granularity which permissions
can be applied to.  It is not possible to make a 2k block accessible,
but an adjacent 2k block inaccessible.

As a result, grant mapping of a subpage grant is prohibited, but you are
(or at least should be) able to grant copy it.

Can you describe precisely which domains are doing what in your example?

~Andrew

Re: Sub page Grant Table mappings

[Mike Belopuhov]

> On Tue, Dec 22, 2015 at 21:59 +0100, Mike Belopuhov wrote:
> > Hi,
> > 
> > I'm trying to get grant table sub page mappings working on Xen 4.5.
> > I know there have been some changes in the trunk regarding moving src/
> > dst checks closer together, but I can't test this easily atm.  Please
> > bear with me for a moment.  Or tell me that it might have been broken
> > previously.
> > 
> > What I'm trying to do is to map in a 2k cluster from the networking
> > stack (we call it an mbuf cluster, you might call it skb something)
> > onto the Netfront Rx ring.  2 clusters fit one page.  Therefore for
> > one frame address which is a (PA >> 12) I might have 2 entries one
> > after the other, both with sub_page.length = 2048 but one with a
> > sub_page.page_off = 2048 and the other with a 0 offset.  Both come
> > with (GTF_permit_access | GTF_sub_page) flags.
> > 
> > When I do that, Xen stops liking me and says:
> > 
> > (XEN) grant_table.c:2162:d0v1 copy dest out of bounds: 0 < 2048 || 90 > 2048
> > (XEN) grant_table.c:2162:d0v1 copy dest out of bounds: 0 < 2048 || 119 > 2048
> > (XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 70 > 2048
> > (XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 119 > 2048
> > (XEN) grant_table.c:2162:d0v2 copy dest out of bounds: 0 < 2048 || 119 > 2048
> > ...
> > 
> > The relevant code does this:
> > 
> >     if ( dest_is_gref )
> >     {
> >         unsigned dest_off, dest_len;
> >         rc = __acquire_grant_for_copy(dd, op->dest.u.ref,
> >                                       current->domain->domain_id, 0,
> >                                       &d_frame, &d_pg, &dest_off, &dest_len, 1);
> >         if ( rc != GNTST_okay )
> >             goto error_out;
> >         have_d_grant = 1;
> >         if ( op->dest.offset < dest_off ||
> >              op->len > dest_len )
> >             PIN_FAIL(error_out, GNTST_general_error,
> >                      "copy dest out of bounds: %d < %d || %d > %d\n",
> >                      op->dest.offset, dest_off,
> >                      op->len, dest_len);
> >     }
> > 
> > I fail to understand what am I doing wrong in this case.  Any clues
> > will be greatly appreciated.
> > 
> > 4k clusters mapped in as full_page mappings work as expected.
> > 
> > Cheers,
> > Mike


Hi Andrew!

I'm sorry I have to fake up this reply.  I'm not subscribe to the list
and either greylisting services or what not prevents me from receiving
your mail verbatim.

Thanks for your prompt response!

> I presume you are on x86 here.
>

Ack.

> Architecturally, a 4k page is the minimum granularity which permissions
> can be applied to.  It is not possible to make a 2k block accessible,
> but an adjacent 2k block inaccessible.
>

Sure.

What I probably didn't realise is that this sub page mapping exists for
situations when one region of the page needs to be granted access to,
but not the whole page.  I'm clearly trying to grant different regions
of the same page twice.

I would have figured this quickly out if I was dealing with different
page protections, but in this case page protections are supposed to be
the same.

> As a result, grant mapping of a subpage grant is prohibited, but you are
> (or at least should be) able to grant copy it.
>
> Can you describe precisely which domains are doing what in your example?
>

Just a domU trying to configure its Netfront's Rx ring.

> ~Andrew

Thanks a lot for your comments!

Cheers,
Mike

Re: Sub page Grant Table mappings

[David Vrabel]

On 22/12/15 20:59, Mike Belopuhov wrote:
> Hi,
> 
> I'm trying to get grant table sub page mappings working on Xen 4.5.
> I know there have been some changes in the trunk regarding moving src/
> dst checks closer together, but I can't test this easily atm.  Please
> bear with me for a moment.  Or tell me that it might have been broken
> previously.
> 
> What I'm trying to do is to map in a 2k cluster from the networking
> stack (we call it an mbuf cluster, you might call it skb something)
> onto the Netfront Rx ring.  2 clusters fit one page.  Therefore for
> one frame address which is a (PA >> 12) I might have 2 entries one
> after the other, both with sub_page.length = 2048 but one with a
> sub_page.page_off = 2048 and the other with a 0 offset.  Both come
> with (GTF_permit_access | GTF_sub_page) flags.

The netif protocol doesn't allow subpages for Rx requests -- the backend
requires a full page to be granted.  Note how struct netif_rx_req only
has id and gref fields.

David