Bucket name restrictions in RGW

Bucket name restrictions in RGW

[Harshal Gupta]

Hi,
I was looking into the bucket creation and found out that we are able
to create buckets with names which are not DNS compliant. One such
example is names ending with a non-alphanumeric character. There are
other rules which make bucket name restrictions in RGW more lenient
than what is recommended for DNS compliant names as well.

In case we plan to support website hosting in future on RGW, we will
need to make bucket names DNS compliant. Keeping that in mind, I am
thinking about modifying the bucket name rules and applying more
restrictions to make them more towards DNS compliant.

Please share your opinion about this.

Thanks,
Harshal Gupta

Re: Bucket name restrictions in RGW

[Wido den Hollander]

On 06/12/2015 05:28 PM, Harshal Gupta wrote:
> Hi,
> I was looking into the bucket creation and found out that we are able
> to create buckets with names which are not DNS compliant. One such
> example is names ending with a non-alphanumeric character. There are
> other rules which make bucket name restrictions in RGW more lenient
> than what is recommended for DNS compliant names as well.
> 
> In case we plan to support website hosting in future on RGW, we will
> need to make bucket names DNS compliant. Keeping that in mind, I am
> thinking about modifying the bucket name rules and applying more
> restrictions to make them more towards DNS compliant.
> 
> Please share your opinion about this.
> 

I'm in favor. I would even like more strict bucket names, eg a setting
where you can force all names to lowercase or refuse names with
uppercase in it. This sometimes gives conflicts with DNS names when
using lower and uppercase mixed.

> Thanks,
> Harshal Gupta
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: Bucket name restrictions in RGW

[Robin H. Johnson]

On Fri, Jun 12, 2015 at 08:44:05PM +0200,  Wido den Hollander wrote:
> > In case we plan to support website hosting in future on RGW,
Yes, I'm working on this presently. You can find the work in my fork on
Github, and the scratchpad documentation here:
https://github.com/robbat2/temp-ceph-rgw-static-website-wiki/wiki

> > need to make bucket names DNS compliant. Keeping that in mind, I am
> > thinking about modifying the bucket name rules and applying more
> > restrictions to make them more towards DNS compliant.
> > 
> > Please share your opinion about this.
> I'm in favor. I would even like more strict bucket names, eg a setting
> where you can force all names to lowercase or refuse names with
> uppercase in it. This sometimes gives conflicts with DNS names when
> using lower and uppercase mixed.
Some of the below will be known to the existing posters on this thread,
but as it's probably novel to some people on the list, I include it
anyway. Feel free to jump to the 'Questions' section for the moment.

Background:
-----------
AmazonS3 bucket names were originally used in path style, as the first
argument in the path. Ability to use them in as prefix in the hostname
came later.

In the US-Standard region, you can still create such names even, you
just can't use them with hostname-style access. In other regions, they
are much stricter than the Ceph S3 code presently:
http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

Path-style access is still valid, and expected for some cases, esp when
SSL is being used.

Details:
--------
The existing RGW code has 3 sets of validation:
RGWHandler_ObjStore::validate_bucket_name
- length 3..255

RGWHandler_ObjStore_SWIFT::validate_bucket_name
- RGWHandler_ObjStore::validate_bucket_name AND
- does not start with '.' AND
- is valid UTF8 AND
- does not contain any 0xFF bytes

RGWHandler_ObjStore_S3::validate_bucket_name(relaxed_names = true)
- RGWHandler_ObjStore::validate_bucket_name AND
- starts with [a-z0-9] AND
- contains only [-._A-Za-z0-9] AND
- is not an IPv4 address [2]

RGWHandler_ObjStore_S3::validate_bucket_name(relaxed_names = false)
- RGWHandler_ObjStore::validate_bucket_name AND
- starts with [0_-9A-Za-z0-9] AND
- contains only [-._A-Za-z0-9] AND
- is not an IPv4 address [2]

Right now, the AmazonS3 constraints, for new buckets [1] in regions
other than US-Standard, or created via the management console are: (if
they already exist, you can continue to use them)
- length 3..63
- one or more labels, separated with exactly one single period '.'
  - each label must start and end with [a-z0-9]
  - must contain only [-a-z0-9]
- is not an IPv4 address [3]

Questions:
----------

Now that we've got both Ceph's current behavior and currently enforced
S3 constraints documented, we have some questions to decide on.
- Since we've allowed such flexibility in the past, how should we handle
  access to existing non-complaint buckets in future? Right now
  RGWHandler_ObjStore_S3 will reject ANY access to a non-compliant
  bucket name, even if it was created before the restrictions were so
  tight.
- This also applies to creating buckets with 'rgw relaxed s3 bucket
  names' enabled, then turning it off, and trying to access the bucket.
- What happens if you create a bucket via Swift, compliant with Swift
  constraints, and then try to access it via S3? What SHOULD happen?
  I don't use swift at all, so I'm really unsure about this.

Footnotes:
----------
1. I include them because I've seen the older versions, and there
doesn't seem to be way to easily see those again, archive.org doesn't
have it. 
2. looks_like_ip_address will actually reject anything that matches the
   regex of /^([0-9]+\.){3}[0-9]+$/, so if each element is larger than
   255, it will reject it as well.
3. They just say "Bucket names must not be formatted as an IP address
   (e.g., 192.168.5.4)"; they don't clarify edge cases.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: Bucket name restrictions in RGW

[Yehuda Sadeh-Weinraub]

Whatever we end up doing, we need to make it configurable, and also keep backward compatibility, so that buckets that were created prior to such a change will still remain accessible. Some setups would not need this limitation and will find it too restricting so I'm not sure that it's really that needed. In short, make it configurable.

Yehuda

----- Original Message -----
> From: "Robin H. Johnson" <robbat2@gentoo.org>
> To: "ceph-devel" <ceph-devel@vger.kernel.org>
> Sent: Friday, June 12, 2015 3:50:48 PM
> Subject: Re: Bucket name restrictions in RGW
> 
> On Fri, Jun 12, 2015 at 08:44:05PM +0200,  Wido den Hollander wrote:
> > > In case we plan to support website hosting in future on RGW,
> Yes, I'm working on this presently. You can find the work in my fork on
> Github, and the scratchpad documentation here:
> https://github.com/robbat2/temp-ceph-rgw-static-website-wiki/wiki
> 
> > > need to make bucket names DNS compliant. Keeping that in mind, I am
> > > thinking about modifying the bucket name rules and applying more
> > > restrictions to make them more towards DNS compliant.
> > > 
> > > Please share your opinion about this.
> > I'm in favor. I would even like more strict bucket names, eg a setting
> > where you can force all names to lowercase or refuse names with
> > uppercase in it. This sometimes gives conflicts with DNS names when
> > using lower and uppercase mixed.
> Some of the below will be known to the existing posters on this thread,
> but as it's probably novel to some people on the list, I include it
> anyway. Feel free to jump to the 'Questions' section for the moment.
> 
> Background:
> -----------
> AmazonS3 bucket names were originally used in path style, as the first
> argument in the path. Ability to use them in as prefix in the hostname
> came later.
> 
> In the US-Standard region, you can still create such names even, you
> just can't use them with hostname-style access. In other regions, they
> are much stricter than the Ceph S3 code presently:
> http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
> 
> Path-style access is still valid, and expected for some cases, esp when
> SSL is being used.
> 
> Details:
> --------
> The existing RGW code has 3 sets of validation:
> RGWHandler_ObjStore::validate_bucket_name
> - length 3..255
> 
> RGWHandler_ObjStore_SWIFT::validate_bucket_name
> - RGWHandler_ObjStore::validate_bucket_name AND
> - does not start with '.' AND
> - is valid UTF8 AND
> - does not contain any 0xFF bytes
> 
> RGWHandler_ObjStore_S3::validate_bucket_name(relaxed_names = true)
> - RGWHandler_ObjStore::validate_bucket_name AND
> - starts with [a-z0-9] AND
> - contains only [-._A-Za-z0-9] AND
> - is not an IPv4 address [2]
> 
> RGWHandler_ObjStore_S3::validate_bucket_name(relaxed_names = false)
> - RGWHandler_ObjStore::validate_bucket_name AND
> - starts with [0_-9A-Za-z0-9] AND
> - contains only [-._A-Za-z0-9] AND
> - is not an IPv4 address [2]
> 
> Right now, the AmazonS3 constraints, for new buckets [1] in regions
> other than US-Standard, or created via the management console are: (if
> they already exist, you can continue to use them)
> - length 3..63
> - one or more labels, separated with exactly one single period '.'
>   - each label must start and end with [a-z0-9]
>   - must contain only [-a-z0-9]
> - is not an IPv4 address [3]
> 
> Questions:
> ----------
> 
> Now that we've got both Ceph's current behavior and currently enforced
> S3 constraints documented, we have some questions to decide on.
> - Since we've allowed such flexibility in the past, how should we handle
>   access to existing non-complaint buckets in future? Right now
>   RGWHandler_ObjStore_S3 will reject ANY access to a non-compliant
>   bucket name, even if it was created before the restrictions were so
>   tight.
> - This also applies to creating buckets with 'rgw relaxed s3 bucket
>   names' enabled, then turning it off, and trying to access the bucket.
> - What happens if you create a bucket via Swift, compliant with Swift
>   constraints, and then try to access it via S3? What SHOULD happen?
>   I don't use swift at all, so I'm really unsure about this.
> 
> Footnotes:
> ----------
> 1. I include them because I've seen the older versions, and there
> doesn't seem to be way to easily see those again, archive.org doesn't
> have it.
> 2. looks_like_ip_address will actually reject anything that matches the
>    regex of /^([0-9]+\.){3}[0-9]+$/, so if each element is larger than
>    255, it will reject it as well.
> 3. They just say "Bucket names must not be formatted as an IP address
>    (e.g., 192.168.5.4)"; they don't clarify edge cases.
> 
> --
> Robin Hugh Johnson
> Gentoo Linux: Developer, Infrastructure Lead
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Bucket name restrictions in RGW

[Robin H. Johnson]

On Fri, Jun 12, 2015 at 07:13:48PM -0400,  Yehuda Sadeh-Weinraub wrote:
> Whatever we end up doing, we need to make it configurable, and also
> keep backward compatibility, so that buckets that were created prior
> to such a change will still remain accessible. Some setups would not
> need this limitation and will find it too restricting so I'm not sure
> that it's really that needed. In short, make it configurable.
Configurable:
- Can we obsolete 'rgw relaxed s3 bucket names', and convert it to a new
  option: 'rgw s3 bucket name create strictness'
  Value '0' => existing 'rgw relaxed s3 bucket names = true' logic
  Value '1' => existing 'rgw relaxed s3 bucket names = false' logic
  Value '2' => compliance with AmazonS3 DNS rules

Backwards-Compatibility:
- Make a new option 'rgw s3 bucket name access strictness'
  Same values as above, but used to access buckets, not create new ones.
- Proposed default values:
  rgw s3 bucket name create strictness = 2
  rgw s3 bucket name access strictness = 1

So you can only create DNS-compliant buckets, but still access your
existing non-compliant buckets. Maybe also have keywords of major
releases and 'relaxed' supported in addition to the integer values.

I don't like the names of the config keys, but I'm coming up blank on
something that is shorter while still being immediately clear.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: Bucket name restrictions in RGW

[Wido den Hollander]

On 06/13/2015 01:29 AM, Robin H. Johnson wrote:
> On Fri, Jun 12, 2015 at 07:13:48PM -0400,  Yehuda Sadeh-Weinraub wrote:
>> Whatever we end up doing, we need to make it configurable, and also
>> keep backward compatibility, so that buckets that were created prior
>> to such a change will still remain accessible. Some setups would not
>> need this limitation and will find it too restricting so I'm not sure
>> that it's really that needed. In short, make it configurable.
> Configurable:
> - Can we obsolete 'rgw relaxed s3 bucket names', and convert it to a new
>   option: 'rgw s3 bucket name create strictness'
>   Value '0' => existing 'rgw relaxed s3 bucket names = true' logic
>   Value '1' => existing 'rgw relaxed s3 bucket names = false' logic
>   Value '2' => compliance with AmazonS3 DNS rules
> 
> Backwards-Compatibility:
> - Make a new option 'rgw s3 bucket name access strictness'
>   Same values as above, but used to access buckets, not create new ones.
> - Proposed default values:
>   rgw s3 bucket name create strictness = 2
>   rgw s3 bucket name access strictness = 1
> 
> So you can only create DNS-compliant buckets, but still access your
> existing non-compliant buckets. Maybe also have keywords of major
> releases and 'relaxed' supported in addition to the integer values.
> 
> I don't like the names of the config keys, but I'm coming up blank on
> something that is shorter while still being immediately clear.
> 

Seems like a good plan to me. I would like to restrict them as much as
possible, but we shouldn't break anything which is online now.

-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: Bucket name restrictions in RGW

[Abhishek L]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]


Wido den Hollander writes:

> On 06/13/2015 01:29 AM, Robin H. Johnson wrote:
>> On Fri, Jun 12, 2015 at 07:13:48PM -0400,  Yehuda Sadeh-Weinraub wrote:
>>> Whatever we end up doing, we need to make it configurable, and also
>>> keep backward compatibility, so that buckets that were created prior
>>> to such a change will still remain accessible. Some setups would not
>>> need this limitation and will find it too restricting so I'm not sure
>>> that it's really that needed. In short, make it configurable.
>> Configurable:
>> - Can we obsolete 'rgw relaxed s3 bucket names', and convert it to a new
>>   option: 'rgw s3 bucket name create strictness'
>>   Value '0' => existing 'rgw relaxed s3 bucket names = true' logic
>>   Value '1' => existing 'rgw relaxed s3 bucket names = false' logic
>>   Value '2' => compliance with AmazonS3 DNS rules

Alternatively we could just add an opt `rgw s3 bucket name create
strictness` (maybe `rgw s3 bucket dns comply` instead) as a bool which
would set your value '2' logic when both `relaxed s3 bucket names` and
this value are set, so that we dont break backwards compatibility?
>> 
>> Backwards-Compatibility:
>> - Make a new option 'rgw s3 bucket name access strictness'
>>   Same values as above, but used to access buckets, not create new ones.
>> - Proposed default values:
>>   rgw s3 bucket name create strictness = 2
>>   rgw s3 bucket name access strictness = 1
>> 
>> So you can only create DNS-compliant buckets, but still access your
>> existing non-compliant buckets. Maybe also have keywords of major
>> releases and 'relaxed' supported in addition to the integer values.
>> 
>> I don't like the names of the config keys, but I'm coming up blank on
>> something that is shorter while still being immediately clear.
>> 
>
> Seems like a good plan to me. I would like to restrict them as much as
> possible, but we shouldn't break anything which is online now.
>
> -- 
> Wido den Hollander
> 42on B.V.
> Ceph trainer and consultant
>
> Phone: +31 (0)20 700 9902
> Skype: contact42on

-- 
Abhishek

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]

Re: Bucket name restrictions in RGW

[Harshal Gupta]

Hi, according to the inputs from the thread, I have included following
changes for adding more restrictions to the RGW bucket name :

1. removed previous boolean flag for specifying bucket naming rule and
added a int option with values 0,1,2

-OPTION(rgw_relaxed_s3_bucket_names, OPT_BOOL, false) // enable
relaxed bucket name rules for US region buckets
+
+/*
+ * RGW Bucket name restriction option. values can be {0,1,2). Default
value is 1.
+ * Setting value as 0 : enable relaxed bucket name rules for US region buckets.
+ * Setting as 1 : name can't start with a non-alphanumeric. rest same
as value =1.
+ * Setting as 2 : further restricts name so as to follow AWS S3
bucket naming conventions for non-standard regions.
+ */
+OPTION(rgw_s3_bucket_names_strictness, OPT_INT, 1)

As per my understanding existing users can change the options in
configuration file anytime, so just by giving an update in
documentation for this, we can enable old users to continue as it is.
Please correct me if I am wrong.

2. As suggested by Robin, added two functions which will take care of
validating bucket name according to the above 3 values, to avoid
repetitive code in
RGWHandler_ObjStore_S3::validate_bucket_name(const string& bucket, int
name_strictness)

You can find the changes here :

https://github.com/ceph/ceph/commit/ae33e144a10d170823641bdf750628857912f656?diff=split

Please review it and if you find it OK, I will go ahead and raise a
pull request.

@Abhishek, I think instead of having two boolean flags with almost
same function, it is better to have only an integer one. With an int
Option, we can further configure more rules in future, if needed. What
is your opinion about it.

Thanks,

On Sun, Jun 14, 2015 at 11:00 AM, Abhishek L
<abhishek.lekshmanan@gmail.com> wrote:
>
> Wido den Hollander writes:
>
>> On 06/13/2015 01:29 AM, Robin H. Johnson wrote:
>>> On Fri, Jun 12, 2015 at 07:13:48PM -0400,  Yehuda Sadeh-Weinraub wrote:
>>>> Whatever we end up doing, we need to make it configurable, and also
>>>> keep backward compatibility, so that buckets that were created prior
>>>> to such a change will still remain accessible. Some setups would not
>>>> need this limitation and will find it too restricting so I'm not sure
>>>> that it's really that needed. In short, make it configurable.
>>> Configurable:
>>> - Can we obsolete 'rgw relaxed s3 bucket names', and convert it to a new
>>>   option: 'rgw s3 bucket name create strictness'
>>>   Value '0' => existing 'rgw relaxed s3 bucket names = true' logic
>>>   Value '1' => existing 'rgw relaxed s3 bucket names = false' logic
>>>   Value '2' => compliance with AmazonS3 DNS rules
>
> Alternatively we could just add an opt `rgw s3 bucket name create
> strictness` (maybe `rgw s3 bucket dns comply` instead) as a bool which
> would set your value '2' logic when both `relaxed s3 bucket names` and
> this value are set, so that we dont break backwards compatibility?
>>>
>>> Backwards-Compatibility:
>>> - Make a new option 'rgw s3 bucket name access strictness'
>>>   Same values as above, but used to access buckets, not create new ones.
>>> - Proposed default values:
>>>   rgw s3 bucket name create strictness = 2
>>>   rgw s3 bucket name access strictness = 1
>>>
>>> So you can only create DNS-compliant buckets, but still access your
>>> existing non-compliant buckets. Maybe also have keywords of major
>>> releases and 'relaxed' supported in addition to the integer values.
>>>
>>> I don't like the names of the config keys, but I'm coming up blank on
>>> something that is shorter while still being immediately clear.
>>>
>>
>> Seems like a good plan to me. I would like to restrict them as much as
>> possible, but we shouldn't break anything which is online now.
>>
>> --
>> Wido den Hollander
>> 42on B.V.
>> Ceph trainer and consultant
>>
>> Phone: +31 (0)20 700 9902
>> Skype: contact42on
>
> --
> Abhishek



-- 
HARSHAL GUPTA
Software Engineer
KIWI Inc.

Re: Bucket name restrictions in RGW

[Harshal Gupta]

Hi,
I collaborated with Robin on this and finally came with the
implementation which will provide backward compatibility for the
existing users and is configurable for new users.

Also the strictness levels can be further extended if we wish to
further impose more restrictions over naming convention of buckets.
The current solutions is as follows:

1. Created two configurable options.
a. rgw_s3_bucket_name_access_strictness b. rgw_s3_bucket_name_create_strictness
Both can have values 0, 1 and 2 which signifies :
0 - Follows same validation rules as already present with flag
relaxed_name=true.
1 - Follows same validation rules as already present with flag
relaxed_names=false.
2 - Follows new naming rules as per DNS compliance. (similar to what S3 follows)

2. Users can configure both flags independently so as to access
already created buckets and to create new buckets, with creation
always more or equally stricter than access. and can pass those
options in existing
RGWHandler_ObjStore_s3::validate_bucket_name(bucket, relaxed_names) in
place of 'relaxed_names' and appropriate checks will be then applied
to bucket name depending upon the value of strictness option.

3. Implemented a new method call to
RGWHandler_ObjStore_s3::validate_bucket_name(bucket, name_strictness)
from RGWCreateBucket::pre_exec() so as to avoid interference with
already running functionality. This call will pass create strictness
option for further validation of bucket name specifically for
creation, based on the configured create strictness level

4. Added virtual method validate_bucket_name(const string&, int) in
RGWHandler for polymorphism and compile time check.

You can find the changes here :

https://github.com/ceph/ceph/compare/master...harshalgupta:wip-bucket_name_restrictions

Please review it and let me know your concerns and suggestions. I am
thinking about raising a pull request by EOD.

Thanks,

On Mon, Jun 15, 2015 at 10:56 PM, Harshal Gupta
<harshal.gupta001@gmail.com> wrote:
> Hi, according to the inputs from the thread, I have included following
> changes for adding more restrictions to the RGW bucket name :
>
> 1. removed previous boolean flag for specifying bucket naming rule and
> added a int option with values 0,1,2
>
> -OPTION(rgw_relaxed_s3_bucket_names, OPT_BOOL, false) // enable
> relaxed bucket name rules for US region buckets
> +
> +/*
> + * RGW Bucket name restriction option. values can be {0,1,2). Default
> value is 1.
> + * Setting value as 0 : enable relaxed bucket name rules for US region buckets.
> + * Setting as 1 : name can't start with a non-alphanumeric. rest same
> as value =1.
> + * Setting as 2 : further restricts name so as to follow AWS S3
> bucket naming conventions for non-standard regions.
> + */
> +OPTION(rgw_s3_bucket_names_strictness, OPT_INT, 1)
>
> As per my understanding existing users can change the options in
> configuration file anytime, so just by giving an update in
> documentation for this, we can enable old users to continue as it is.
> Please correct me if I am wrong.
>
> 2. As suggested by Robin, added two functions which will take care of
> validating bucket name according to the above 3 values, to avoid
> repetitive code in
> RGWHandler_ObjStore_S3::validate_bucket_name(const string& bucket, int
> name_strictness)
>
> You can find the changes here :
>
> https://github.com/ceph/ceph/commit/ae33e144a10d170823641bdf750628857912f656?diff=split
>
> Please review it and if you find it OK, I will go ahead and raise a
> pull request.
>
> @Abhishek, I think instead of having two boolean flags with almost
> same function, it is better to have only an integer one. With an int
> Option, we can further configure more rules in future, if needed. What
> is your opinion about it.
>
> Thanks,
>
> On Sun, Jun 14, 2015 at 11:00 AM, Abhishek L
> <abhishek.lekshmanan@gmail.com> wrote:
>>
>> Wido den Hollander writes:
>>
>>> On 06/13/2015 01:29 AM, Robin H. Johnson wrote:
>>>> On Fri, Jun 12, 2015 at 07:13:48PM -0400,  Yehuda Sadeh-Weinraub wrote:
>>>>> Whatever we end up doing, we need to make it configurable, and also
>>>>> keep backward compatibility, so that buckets that were created prior
>>>>> to such a change will still remain accessible. Some setups would not
>>>>> need this limitation and will find it too restricting so I'm not sure
>>>>> that it's really that needed. In short, make it configurable.
>>>> Configurable:
>>>> - Can we obsolete 'rgw relaxed s3 bucket names', and convert it to a new
>>>>   option: 'rgw s3 bucket name create strictness'
>>>>   Value '0' => existing 'rgw relaxed s3 bucket names = true' logic
>>>>   Value '1' => existing 'rgw relaxed s3 bucket names = false' logic
>>>>   Value '2' => compliance with AmazonS3 DNS rules
>>
>> Alternatively we could just add an opt `rgw s3 bucket name create
>> strictness` (maybe `rgw s3 bucket dns comply` instead) as a bool which
>> would set your value '2' logic when both `relaxed s3 bucket names` and
>> this value are set, so that we dont break backwards compatibility?
>>>>
>>>> Backwards-Compatibility:
>>>> - Make a new option 'rgw s3 bucket name access strictness'
>>>>   Same values as above, but used to access buckets, not create new ones.
>>>> - Proposed default values:
>>>>   rgw s3 bucket name create strictness = 2
>>>>   rgw s3 bucket name access strictness = 1
>>>>
>>>> So you can only create DNS-compliant buckets, but still access your
>>>> existing non-compliant buckets. Maybe also have keywords of major
>>>> releases and 'relaxed' supported in addition to the integer values.
>>>>
>>>> I don't like the names of the config keys, but I'm coming up blank on
>>>> something that is shorter while still being immediately clear.
>>>>
>>>
>>> Seems like a good plan to me. I would like to restrict them as much as
>>> possible, but we shouldn't break anything which is online now.
>>>
>>> --
>>> Wido den Hollander
>>> 42on B.V.
>>> Ceph trainer and consultant
>>>
>>> Phone: +31 (0)20 700 9902
>>> Skype: contact42on
>>
>> --
>> Abhishek
>
>
>
> --
> HARSHAL GUPTA
> Software Engineer
> KIWI Inc.



-- 
HARSHAL GUPTA
Software Engineer
KIWI Inc.

Re: Bucket name restrictions in RGW

[Harshal Gupta]

Hi,
I will be raising a pull request. Please merge the code if you feel
everything fine.

On Thu, Jun 18, 2015 at 11:28 PM, Harshal Gupta
<harshal.gupta001@gmail.com> wrote:
> Hi,
> I collaborated with Robin on this and finally came with the
> implementation which will provide backward compatibility for the
> existing users and is configurable for new users.
>
> Also the strictness levels can be further extended if we wish to
> further impose more restrictions over naming convention of buckets.
> The current solutions is as follows:
>
> 1. Created two configurable options.
> a. rgw_s3_bucket_name_access_strictness b. rgw_s3_bucket_name_create_strictness
> Both can have values 0, 1 and 2 which signifies :
> 0 - Follows same validation rules as already present with flag
> relaxed_name=true.
> 1 - Follows same validation rules as already present with flag
> relaxed_names=false.
> 2 - Follows new naming rules as per DNS compliance. (similar to what S3 follows)
>
> 2. Users can configure both flags independently so as to access
> already created buckets and to create new buckets, with creation
> always more or equally stricter than access. and can pass those
> options in existing
> RGWHandler_ObjStore_s3::validate_bucket_name(bucket, relaxed_names) in
> place of 'relaxed_names' and appropriate checks will be then applied
> to bucket name depending upon the value of strictness option.
>
> 3. Implemented a new method call to
> RGWHandler_ObjStore_s3::validate_bucket_name(bucket, name_strictness)
> from RGWCreateBucket::pre_exec() so as to avoid interference with
> already running functionality. This call will pass create strictness
> option for further validation of bucket name specifically for
> creation, based on the configured create strictness level
>
> 4. Added virtual method validate_bucket_name(const string&, int) in
> RGWHandler for polymorphism and compile time check.
>
> You can find the changes here :
>
> https://github.com/ceph/ceph/compare/master...harshalgupta:wip-bucket_name_restrictions
>
> Please review it and let me know your concerns and suggestions. I am
> thinking about raising a pull request by EOD.
>
> Thanks,
>
> On Mon, Jun 15, 2015 at 10:56 PM, Harshal Gupta
> <harshal.gupta001@gmail.com> wrote:
>> Hi, according to the inputs from the thread, I have included following
>> changes for adding more restrictions to the RGW bucket name :
>>
>> 1. removed previous boolean flag for specifying bucket naming rule and
>> added a int option with values 0,1,2
>>
>> -OPTION(rgw_relaxed_s3_bucket_names, OPT_BOOL, false) // enable
>> relaxed bucket name rules for US region buckets
>> +
>> +/*
>> + * RGW Bucket name restriction option. values can be {0,1,2). Default
>> value is 1.
>> + * Setting value as 0 : enable relaxed bucket name rules for US region buckets.
>> + * Setting as 1 : name can't start with a non-alphanumeric. rest same
>> as value =1.
>> + * Setting as 2 : further restricts name so as to follow AWS S3
>> bucket naming conventions for non-standard regions.
>> + */
>> +OPTION(rgw_s3_bucket_names_strictness, OPT_INT, 1)
>>
>> As per my understanding existing users can change the options in
>> configuration file anytime, so just by giving an update in
>> documentation for this, we can enable old users to continue as it is.
>> Please correct me if I am wrong.
>>
>> 2. As suggested by Robin, added two functions which will take care of
>> validating bucket name according to the above 3 values, to avoid
>> repetitive code in
>> RGWHandler_ObjStore_S3::validate_bucket_name(const string& bucket, int
>> name_strictness)
>>
>> You can find the changes here :
>>
>> https://github.com/ceph/ceph/commit/ae33e144a10d170823641bdf750628857912f656?diff=split
>>
>> Please review it and if you find it OK, I will go ahead and raise a
>> pull request.
>>
>> @Abhishek, I think instead of having two boolean flags with almost
>> same function, it is better to have only an integer one. With an int
>> Option, we can further configure more rules in future, if needed. What
>> is your opinion about it.
>>
>> Thanks,
>>
>> On Sun, Jun 14, 2015 at 11:00 AM, Abhishek L
>> <abhishek.lekshmanan@gmail.com> wrote:
>>>
>>> Wido den Hollander writes:
>>>
>>>> On 06/13/2015 01:29 AM, Robin H. Johnson wrote:
>>>>> On Fri, Jun 12, 2015 at 07:13:48PM -0400,  Yehuda Sadeh-Weinraub wrote:
>>>>>> Whatever we end up doing, we need to make it configurable, and also
>>>>>> keep backward compatibility, so that buckets that were created prior
>>>>>> to such a change will still remain accessible. Some setups would not
>>>>>> need this limitation and will find it too restricting so I'm not sure
>>>>>> that it's really that needed. In short, make it configurable.
>>>>> Configurable:
>>>>> - Can we obsolete 'rgw relaxed s3 bucket names', and convert it to a new
>>>>>   option: 'rgw s3 bucket name create strictness'
>>>>>   Value '0' => existing 'rgw relaxed s3 bucket names = true' logic
>>>>>   Value '1' => existing 'rgw relaxed s3 bucket names = false' logic
>>>>>   Value '2' => compliance with AmazonS3 DNS rules
>>>
>>> Alternatively we could just add an opt `rgw s3 bucket name create
>>> strictness` (maybe `rgw s3 bucket dns comply` instead) as a bool which
>>> would set your value '2' logic when both `relaxed s3 bucket names` and
>>> this value are set, so that we dont break backwards compatibility?
>>>>>
>>>>> Backwards-Compatibility:
>>>>> - Make a new option 'rgw s3 bucket name access strictness'
>>>>>   Same values as above, but used to access buckets, not create new ones.
>>>>> - Proposed default values:
>>>>>   rgw s3 bucket name create strictness = 2
>>>>>   rgw s3 bucket name access strictness = 1
>>>>>
>>>>> So you can only create DNS-compliant buckets, but still access your
>>>>> existing non-compliant buckets. Maybe also have keywords of major
>>>>> releases and 'relaxed' supported in addition to the integer values.
>>>>>
>>>>> I don't like the names of the config keys, but I'm coming up blank on
>>>>> something that is shorter while still being immediately clear.
>>>>>
>>>>
>>>> Seems like a good plan to me. I would like to restrict them as much as
>>>> possible, but we shouldn't break anything which is online now.
>>>>
>>>> --
>>>> Wido den Hollander
>>>> 42on B.V.
>>>> Ceph trainer and consultant
>>>>
>>>> Phone: +31 (0)20 700 9902
>>>> Skype: contact42on
>>>
>>> --
>>> Abhishek
>>
>>
>>
>> --
>> HARSHAL GUPTA
>> Software Engineer
>> KIWI Inc.
>
>
>
> --
> HARSHAL GUPTA
> Software Engineer
> KIWI Inc.



-- 
HARSHAL GUPTA
Software Engineer
KIWI Inc.

Re: Bucket name restrictions in RGW

[Robin H. Johnson]

On Sun, Jun 14, 2015 at 11:00:23AM +0530,  Abhishek L wrote:
> 
> Wido den Hollander writes:
> 
> > On 06/13/2015 01:29 AM, Robin H. Johnson wrote:
> >> On Fri, Jun 12, 2015 at 07:13:48PM -0400,  Yehuda Sadeh-Weinraub wrote:
> >>> Whatever we end up doing, we need to make it configurable, and also
> >>> keep backward compatibility, so that buckets that were created prior
> >>> to such a change will still remain accessible. Some setups would not
> >>> need this limitation and will find it too restricting so I'm not sure
> >>> that it's really that needed. In short, make it configurable.
> >> Configurable:
> >> - Can we obsolete 'rgw relaxed s3 bucket names', and convert it to a new
> >>   option: 'rgw s3 bucket name create strictness'
> >>   Value '0' => existing 'rgw relaxed s3 bucket names = true' logic
> >>   Value '1' => existing 'rgw relaxed s3 bucket names = false' logic
> >>   Value '2' => compliance with AmazonS3 DNS rules
> 
> Alternatively we could just add an opt `rgw s3 bucket name create
> strictness` (maybe `rgw s3 bucket dns comply` instead) as a bool which
> would set your value '2' logic when both `relaxed s3 bucket names` and
> this value are set, so that we dont break backwards compatibility?
If the NEW option names are unset, then:
default 'create strictness' = 2
default 'access strictness' = (relaxed s3 bucket names) ? 0 : 1

That gets us the new strictness on create AND compatibility with
existing configurations.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85