Queries on RGW Multitenancy

Queries on RGW Multitenancy

[Abhishek Lekshmanan]

A few questions on URLs behaviour after enabling multitenancy.

Are there any changes in the way bucket and object urls are used by
clients after creating a user with a tenant? My understanding so far has
been that the urls look the same for clients, and based on the
credentials passed in, we determine the tenant for requests. Is this
correct?

Also on making a bucket/object public the urls follow a format like:
`<host>/<tenant>:<bucket>`

Is there an equivalent s3 vhost style url for the above? Trying any of
tenant.bucket.host combos mostly gave  404s

And similarly for swift making a bucket public, the only url that seemed
accessible was the above mentioned `<host>/<tenant>:<bucket>` format and
there is no swift url of the format
`<host>/swift/v1/<tenant>/<container>` etc? Is this also the expected
behaviour?

Thanks
Abhishek
--
Abhishek Lekshmanan
SUSE Linux GmbH,
GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Queries on RGW Multitenancy

[Pete Zaitcev]

On Wed, 20 Apr 2016 18:14:36 +0200
Abhishek Lekshmanan <abhishek@suse.com> wrote:

> Are there any changes in the way bucket and object urls are used by
> clients after creating a user with a tenant? My understanding so far has
> been that the urls look the same for clients, and based on the
> credentials passed in, we determine the tenant for requests. Is this
> correct?

Yes.

> Also on making a bucket/object public the urls follow a format like:
> `<host>/<tenant>:<bucket>`
> 
> Is there an equivalent s3 vhost style url for the above? Trying any of
> tenant.bucket.host combos mostly gave  404s

No, sadly there is no such syntax. This is because a bucket name of
"buc.ket" is valid, so there's no way to tell a cross-tenant reference
in your proposal from a valid bucket name within a tenant. One has to
continue using the old access method, like when using extended names
of Virgina region buckets.

> And similarly for swift making a bucket public, the only url that seemed
> accessible was the above mentioned `<host>/<tenant>:<bucket>` format and
> there is no swift url of the format
> `<host>/swift/v1/<tenant>/<container>` etc? Is this also the expected
> behaviour?

Historically, we did not have the ability to do this, because, unlike
traditional Swift, our URLs did not have a tenant in them. E.g. it was
just /swift/v1/<container>. In Jewel, the tenant is included. It was
made fully backwards compatible by saving the bit to signify the
tenant-included URL into the token (the "rgwts" token). So now the
cross-tenant access is done in the exactly the same way as in OpenStack
Swift.

One problem remains, as you can guess: cross-tenant access while
authenticated with Keystone. This is simply not implemented. We considered
a couple of approaches, such as
 - a special syntax, such as backslash "tenant\bucket"
 - a new header "X-RGW-Tenant"
 - a user attribute in Keystone (requires cooperation from Keystone)
 - a separate endpoint in the Keystone catalog, possibly using regions

Thanks for raising this issue. Honestly I gave up on finding an elegant
solution for now, although tinkering with endpoints seems like a winner.
I think Radoslaw liked it too. If you have ideas, by all means please
propose them.

-- Pete

Re: Queries on RGW Multitenancy

[Pete Zaitcev]

> On Wed, 20 Apr 2016 18:14:36 +0200
> Abhishek Lekshmanan <abhishek@suse.com> wrote:
> 
> > Are there any changes in the way bucket and object urls are used by
> > clients after creating a user with a tenant? My understanding so far has
> > been that the urls look the same for clients, and based on the
> > credentials passed in, we determine the tenant for requests. Is this
> > correct?  
> 
> Yes.

Sorry to reply to myself, but I need to elaborate. It's yes for the
case of normal access within the tenant: everything stays as before.
But, if you want to break out of the tenant, we have additional
ways to do that which involve trick URLs or other breakouts,
as discussed in the reply.

-- Pete

Re: Queries on RGW Multitenancy

[Abhishek Lekshmanan]

Pete Zaitcev writes:

> On Wed, 20 Apr 2016 18:14:36 +0200
> Abhishek Lekshmanan <abhishek@suse.com> wrote:
>
>> Are there any changes in the way bucket and object urls are used by
>> clients after creating a user with a tenant? My understanding so far has
>> been that the urls look the same for clients, and based on the
>> credentials passed in, we determine the tenant for requests. Is this
>> correct?
>
> Yes.
>
>> Also on making a bucket/object public the urls follow a format like:
>> `<host>/<tenant>:<bucket>`
>>
>> Is there an equivalent s3 vhost style url for the above? Trying any of
>> tenant.bucket.host combos mostly gave  404s
>
> No, sadly there is no such syntax. This is because a bucket name of
> "buc.ket" is valid, so there's no way to tell a cross-tenant reference
> in your proposal from a valid bucket name within a tenant. One has to
> continue using the old access method, like when using extended names
> of Virgina region buckets.

Thanks for clarifying, yeah I expected it would be difficult since `.`
was allowed to be used in bucket names and you can't construct domain
names with most other seperators, not a deal breaker :)
>
>> And similarly for swift making a bucket public, the only url that seemed
>> accessible was the above mentioned `<host>/<tenant>:<bucket>` format and
>> there is no swift url of the format
>> `<host>/swift/v1/<tenant>/<container>` etc? Is this also the expected
>> behaviour?
>
> Historically, we did not have the ability to do this, because, unlike
> traditional Swift, our URLs did not have a tenant in them. E.g. it was
> just /swift/v1/<container>. In Jewel, the tenant is included. It was
> made fully backwards compatible by saving the bit to signify the
> tenant-included URL into the token (the "rgwts" token).
> So now the cross-tenant access is done in the exactly the same way as in OpenStack
> Swift.

Ah ok, the information is encoded in the token, so on making a
bucket/container public we'll still access it at
<host>/tenant:container/... location, rather than a /swift/tenant sort
of url
>
> One problem remains, as you can guess: cross-tenant access while
> authenticated with Keystone. This is simply not implemented. We considered
> a couple of approaches, such as
>  - a special syntax, such as backslash "tenant\bucket"
>  - a new header "X-RGW-Tenant"
>  - a user attribute in Keystone (requires cooperation from Keystone)
>  - a separate endpoint in the Keystone catalog, possibly using regions
>
> Thanks for raising this issue. Honestly I gave up on finding an elegant
> solution for now, although tinkering with endpoints seems like a winner.
> I think Radoslaw liked it too. If you have ideas, by all means please
> propose them.

I might be understanding this wrong, but since endpoints are an admin only
thing (to create), are you suggesting something like a different region
per tenant or so?

Doesn't keystone returns the tenantid while authenticating (I could be
wrong), in which case we somehow enforce that the tenant-id for a user
created in rgw[1] if you're using keystone should match the ones used at
openstack and check for the value of tenant-id returned by keystone when
authenticating?

[1]: this is still tricky, as we create users at runtime after first
authentication with keystone iirc
>
> -- Pete


--
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Queries on RGW Multitenancy

[Pete Zaitcev]

On Fri, 22 Apr 2016 18:32:01 +0200
Abhishek Lekshmanan <abhishek@suse.com> wrote:

> > One problem remains, as you can guess: cross-tenant access while
> > authenticated with Keystone. This is simply not implemented. We considered
> > a couple of approaches, such as
> >  - a special syntax, such as backslash "tenant\bucket"
> >  - a new header "X-RGW-Tenant"
> >  - a user attribute in Keystone (requires cooperation from Keystone)
> >  - a separate endpoint in the Keystone catalog, possibly using regions
> >
> > Thanks for raising this issue. Honestly I gave up on finding an elegant
> > solution for now, although tinkering with endpoints seems like a winner.
> > I think Radoslaw liked it too. If you have ideas, by all means please
> > propose them.  
> 
> I might be understanding this wrong, but since endpoints are an admin only
> thing (to create), are you suggesting something like a different region
> per tenant or so?

I suggest having two endpoints: one traditional with /swift/v1 and another
with /swift/v1/%(tenant_id)s. The problem here is, client has to be told
to use one region or ther other, and the default is the old way.

> Doesn't keystone returns the tenantid while authenticating (I could be
> wrong), in which case we somehow enforce that the tenant-id for a user
> created in rgw[1] if you're using keystone should match the ones used at
> openstack and check for the value of tenant-id returned by keystone when
> authenticating?

I'll have to get back to you on this. When I considered using Keystone
tenant IDs, I was going to put dollars into them. It works, but but you
have to educate administartors to create users with a specific tenant ID.
None of the current tools support this. They always let Keystone to select
the ID, and it uses a random UUID. Having them not matching was very useful
therefore. An operator just lets Keystone do its thing, then we use the
"name" instead of their ID, and use it in RGW as ID (clear as mud, right?).

-- Pete

Re: Queries on RGW Multitenancy

[Abhishek L]

Pete Zaitcev writes:

> On Fri, 22 Apr 2016 18:32:01 +0200
> Abhishek Lekshmanan <abhishek@suse.com> wrote:
>
>> > One problem remains, as you can guess: cross-tenant access while
>> > authenticated with Keystone. This is simply not implemented. We considered
>> > a couple of approaches, such as
>> >  - a special syntax, such as backslash "tenant\bucket"
>> >  - a new header "X-RGW-Tenant"
>> >  - a user attribute in Keystone (requires cooperation from Keystone)
>> >  - a separate endpoint in the Keystone catalog, possibly using regions
>> >
>> > Thanks for raising this issue. Honestly I gave up on finding an elegant
>> > solution for now, although tinkering with endpoints seems like a winner.
>> > I think Radoslaw liked it too. If you have ideas, by all means please
>> > propose them.  
>> 
>> I might be understanding this wrong, but since endpoints are an admin only
>> thing (to create), are you suggesting something like a different region
>> per tenant or so?
>
> I suggest having two endpoints: one traditional with /swift/v1 and another
> with /swift/v1/%(tenant_id)s. The problem here is, client has to be told
> to use one region or ther other, and the default is the old way.

Ah ok,
>
>> Doesn't keystone returns the tenantid while authenticating (I could be
>> wrong), in which case we somehow enforce that the tenant-id for a user
>> created in rgw[1] if you're using keystone should match the ones used at
>> openstack and check for the value of tenant-id returned by keystone when
>> authenticating?
>
> I'll have to get back to you on this. When I considered using Keystone
> tenant IDs, I was going to put dollars into them. It works, but but you
> have to educate administartors to create users with a specific tenant ID.
> None of the current tools support this.
I was meaning more like when we create users in RGW, we specify the
--tenant with whatever keystone uuid was assigned for the user..but I
forgot about the the whole keystone-tenant -> rgw-uid mapping thing, so
the keystone tenant id is already stored as a RGW user id, we could
potentially check the response against the value of uid, but then again
creating the user for the first time in RGW will still be sort of a
challenge.
> They always let Keystone to select
> the ID, and it uses a random UUID. Having them not matching was very useful
> therefore. An operator just lets Keystone do its thing, then we use the
> "name" instead of their ID, and use it in RGW as ID (clear as mud, right?).

Yeah sure this could also work. 
>
> -- Pete


-- 
Abhishek