RGW Beast frontend and ipv6 options

RGW Beast frontend and ipv6 options

[Abhishek Lekshmanan]

Currently RGW's beast frontend supports ipv6 via the endpoint
configurable. The port option will bind to ipv4 _only_.

http://docs.ceph.com/docs/master/radosgw/frontends/#options

Since many Linux systems may default the sysconfig net.ipv6.bindv6only
flag to true, it usually means that specifying a v6 endpoint will bind
to both v4 and v6. But this also means that deployment systems must be
aware of this while configuring depending on whether both v4 and v6
endpoints need to work or not. Specifying both a v4 and v6 endpoint or a
port (v4) and endpoint with the same v6 port will currently lead to a
failure as the system would've already bound the v6 port to both v4 and
v6. This leaves us with a few options.

1. Keep the implicit behaviour as it is, document this, as systems are
already aware of sysconfig flags and will expect that at a v6 endpoint
will bind to both v4 and v6.

2. Be explicit with endpoints & configuration, Beast itself overrides
the socket option to bind both v4 and v6, which means that v6 endpoint
will bind to v6 *only* and binding to a v4 will need an explicit
specification. (there is a pr in progress for this:
https://github.com/ceph/ceph/pull/27270)

Any more suggestions on how systems handle this are also welcome.

--
Abhishek 

Re: RGW Beast frontend and ipv6 options

[Daniel Gryniewicz]

Based on past experience with this issue in other projects, I would 
propose this:

1. By default (rgw frontends=beast), we should bind to both IPv4 and 
IPv6, if available.

2. Just specifying port (rgw frontends=beast port=8000) should apply to 
both IPv4 and IPv6, if available.

3. If the user provides endpoint config, we should use only that 
endpoint config.  For example, if they provide only v4 addresses, we 
should only bind to v4.

This should all be independent of the bindv6only setting; that is, we 
should specifically bind our v4 and v6 addresses, and not depend on the 
system to automatically bind v4 when binding v6.

In the case of 1 or 2, if the system has disabled either v4 or v6, this 
should not be an error, as long as one of the two binds works.  In the 
case of 3, we should error out if any configured endpoint cannot be bound.

This should allow an orchestrator to confidently install a system, 
knowing what will happen, without needing to know or manipulate the 
bindv6only flag.

As for what happens if you specify an endpoint and a port, I don't have 
a strong opinion.  I see 2 reasonable possibilites:

a. Make it an error

b. Treat a port in this case as an endpoint of 0.0.0.0:port (v4-only)

Daniel

On 4/26/19 4:49 AM, Abhishek Lekshmanan wrote:
> 
> Currently RGW's beast frontend supports ipv6 via the endpoint
> configurable. The port option will bind to ipv4 _only_.
> 
> http://docs.ceph.com/docs/master/radosgw/frontends/#options
> 
> Since many Linux systems may default the sysconfig net.ipv6.bindv6only
> flag to true, it usually means that specifying a v6 endpoint will bind
> to both v4 and v6. But this also means that deployment systems must be
> aware of this while configuring depending on whether both v4 and v6
> endpoints need to work or not. Specifying both a v4 and v6 endpoint or a
> port (v4) and endpoint with the same v6 port will currently lead to a
> failure as the system would've already bound the v6 port to both v4 and
> v6. This leaves us with a few options.
> 
> 1. Keep the implicit behaviour as it is, document this, as systems are
> already aware of sysconfig flags and will expect that at a v6 endpoint
> will bind to both v4 and v6.
> 
> 2. Be explicit with endpoints & configuration, Beast itself overrides
> the socket option to bind both v4 and v6, which means that v6 endpoint
> will bind to v6 *only* and binding to a v4 will need an explicit
> specification. (there is a pr in progress for this:
> https://github.com/ceph/ceph/pull/27270)
> 
> Any more suggestions on how systems handle this are also welcome.
> 
> --
> Abhishek
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

Re: RGW Beast frontend and ipv6 options

[Daniel Gryniewicz]

After discussing with Casey, I'd like to propose some clarifications to 
this.

First, we do not treat EAFNOSUPPORT as a non-fatal error.  Any other 
error binding is fatal, but that one we warn and continue.

Second, we treat "port=<port>" as expanding to "endpoint=0.0.0.0:<port>, 
endpoint=[::<port>]".

Then, we just process the set of endpoints properly.  This should, I 
believe, result in simple, straight-forward code, and easily 
understandable semantics, and should make it simple for orchetrators.

This would make 1 and 2 below fallout naturally.  3 is modified so that 
we only use configured endpoints, but port= is now implicit endpoint 
configuration.

Daniel

On 5/2/19 10:08 AM, Daniel Gryniewicz wrote:
> Based on past experience with this issue in other projects, I would 
> propose this:
> 
> 1. By default (rgw frontends=beast), we should bind to both IPv4 and 
> IPv6, if available.
> 
> 2. Just specifying port (rgw frontends=beast port=8000) should apply to 
> both IPv4 and IPv6, if available.
> 
> 3. If the user provides endpoint config, we should use only that 
> endpoint config.  For example, if they provide only v4 addresses, we 
> should only bind to v4.
> 
> This should all be independent of the bindv6only setting; that is, we 
> should specifically bind our v4 and v6 addresses, and not depend on the 
> system to automatically bind v4 when binding v6.
> 
> In the case of 1 or 2, if the system has disabled either v4 or v6, this 
> should not be an error, as long as one of the two binds works.  In the 
> case of 3, we should error out if any configured endpoint cannot be bound.
> 
> This should allow an orchestrator to confidently install a system, 
> knowing what will happen, without needing to know or manipulate the 
> bindv6only flag.
> 
> As for what happens if you specify an endpoint and a port, I don't have 
> a strong opinion.  I see 2 reasonable possibilites:
> 
> a. Make it an error
> 
> b. Treat a port in this case as an endpoint of 0.0.0.0:port (v4-only)
> 
> Daniel
> 
> On 4/26/19 4:49 AM, Abhishek Lekshmanan wrote:
>>
>> Currently RGW's beast frontend supports ipv6 via the endpoint
>> configurable. The port option will bind to ipv4 _only_.
>>
>> http://docs.ceph.com/docs/master/radosgw/frontends/#options
>>
>> Since many Linux systems may default the sysconfig net.ipv6.bindv6only
>> flag to true, it usually means that specifying a v6 endpoint will bind
>> to both v4 and v6. But this also means that deployment systems must be
>> aware of this while configuring depending on whether both v4 and v6
>> endpoints need to work or not. Specifying both a v4 and v6 endpoint or a
>> port (v4) and endpoint with the same v6 port will currently lead to a
>> failure as the system would've already bound the v6 port to both v4 and
>> v6. This leaves us with a few options.
>>
>> 1. Keep the implicit behaviour as it is, document this, as systems are
>> already aware of sysconfig flags and will expect that at a v6 endpoint
>> will bind to both v4 and v6.
>>
>> 2. Be explicit with endpoints & configuration, Beast itself overrides
>> the socket option to bind both v4 and v6, which means that v6 endpoint
>> will bind to v6 *only* and binding to a v4 will need an explicit
>> specification. (there is a pr in progress for this:
>> https://github.com/ceph/ceph/pull/27270)
>>
>> Any more suggestions on how systems handle this are also welcome.
>>
>> -- 
>> Abhishek
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> 

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: RGW Beast frontend and ipv6 options

[Abhishek Lekshmanan]

Daniel Gryniewicz <dang@redhat.com> writes:

> After discussing with Casey, I'd like to propose some clarifications to 
> this.
>
> First, we do not treat EAFNOSUPPORT as a non-fatal error.  Any other 
> error binding is fatal, but that one we warn and continue.
>
> Second, we treat "port=<port>" as expanding to "endpoint=0.0.0.0:<port>, 
> endpoint=[::<port>]".
>
> Then, we just process the set of endpoints properly.  This should, I 
> believe, result in simple, straight-forward code, and easily 
> understandable semantics, and should make it simple for orchetrators.

Agree, this makes a lot of sense. specifying both a port and a endpoint
is somewhat of a corner case and I guess for this particular case
failure to bind is acceptable with the documentation already mentioning
the port's implicit endpoint behaviour.
>
> This would make 1 and 2 below fallout naturally.  3 is modified so that 
> we only use configured endpoints, but port= is now implicit endpoint 
> configuration.



>
> Daniel
>
> On 5/2/19 10:08 AM, Daniel Gryniewicz wrote:
>> Based on past experience with this issue in other projects, I would 
>> propose this:
>> 
>> 1. By default (rgw frontends=beast), we should bind to both IPv4 and 
>> IPv6, if available.
>> 
>> 2. Just specifying port (rgw frontends=beast port=8000) should apply to 
>> both IPv4 and IPv6, if available.
>> 
>> 3. If the user provides endpoint config, we should use only that 
>> endpoint config.  For example, if they provide only v4 addresses, we 
>> should only bind to v4.
>> 
>> This should all be independent of the bindv6only setting; that is, we 
>> should specifically bind our v4 and v6 addresses, and not depend on the 
>> system to automatically bind v4 when binding v6.
>> 
>> In the case of 1 or 2, if the system has disabled either v4 or v6, this 
>> should not be an error, as long as one of the two binds works.  In the 
>> case of 3, we should error out if any configured endpoint cannot be bound.
>> 
>> This should allow an orchestrator to confidently install a system, 
>> knowing what will happen, without needing to know or manipulate the 
>> bindv6only flag.
>> 
>> As for what happens if you specify an endpoint and a port, I don't have 
>> a strong opinion.  I see 2 reasonable possibilites:
>> 
>> a. Make it an error
>> 
>> b. Treat a port in this case as an endpoint of 0.0.0.0:port (v4-only)
>> 
>> Daniel
>> 
>> On 4/26/19 4:49 AM, Abhishek Lekshmanan wrote:
>>>
>>> Currently RGW's beast frontend supports ipv6 via the endpoint
>>> configurable. The port option will bind to ipv4 _only_.
>>>
>>> http://docs.ceph.com/docs/master/radosgw/frontends/#options
>>>
>>> Since many Linux systems may default the sysconfig net.ipv6.bindv6only
>>> flag to true, it usually means that specifying a v6 endpoint will bind
>>> to both v4 and v6. But this also means that deployment systems must be
>>> aware of this while configuring depending on whether both v4 and v6
>>> endpoints need to work or not. Specifying both a v4 and v6 endpoint or a
>>> port (v4) and endpoint with the same v6 port will currently lead to a
>>> failure as the system would've already bound the v6 port to both v4 and
>>> v6. This leaves us with a few options.
>>>
>>> 1. Keep the implicit behaviour as it is, document this, as systems are
>>> already aware of sysconfig flags and will expect that at a v6 endpoint
>>> will bind to both v4 and v6.
>>>
>>> 2. Be explicit with endpoints & configuration, Beast itself overrides
>>> the socket option to bind both v4 and v6, which means that v6 endpoint
>>> will bind to v6 *only* and binding to a v4 will need an explicit
>>> specification. (there is a pr in progress for this:
>>> https://github.com/ceph/ceph/pull/27270)
>>>
>>> Any more suggestions on how systems handle this are also welcome.
>>>
>>> -- 
>>> Abhishek
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>> 
>
>

-- 
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: RGW Beast frontend and ipv6 options

[Wido den Hollander]

On 5/2/19 4:08 PM, Daniel Gryniewicz wrote:
> Based on past experience with this issue in other projects, I would
> propose this:
> 
> 1. By default (rgw frontends=beast), we should bind to both IPv4 and
> IPv6, if available.
> 
> 2. Just specifying port (rgw frontends=beast port=8000) should apply to
> both IPv4 and IPv6, if available.
> 
> 3. If the user provides endpoint config, we should use only that
> endpoint config.  For example, if they provide only v4 addresses, we
> should only bind to v4.
> 

I agree with these points above. The default should be v4 and v6 as
that's expected nowadays by most admins.

Only if you specify v4 or v6 explicitly it should bind to that family.

Wido

> This should all be independent of the bindv6only setting; that is, we
> should specifically bind our v4 and v6 addresses, and not depend on the
> system to automatically bind v4 when binding v6.
> 
> In the case of 1 or 2, if the system has disabled either v4 or v6, this
> should not be an error, as long as one of the two binds works.  In the
> case of 3, we should error out if any configured endpoint cannot be bound.
> 
> This should allow an orchestrator to confidently install a system,
> knowing what will happen, without needing to know or manipulate the
> bindv6only flag.
> 
> As for what happens if you specify an endpoint and a port, I don't have
> a strong opinion.  I see 2 reasonable possibilites:
> 
> a. Make it an error
> 
> b. Treat a port in this case as an endpoint of 0.0.0.0:port (v4-only)
> 
> Daniel
> 
> On 4/26/19 4:49 AM, Abhishek Lekshmanan wrote:
>>
>> Currently RGW's beast frontend supports ipv6 via the endpoint
>> configurable. The port option will bind to ipv4 _only_.
>>
>> http://docs.ceph.com/docs/master/radosgw/frontends/#options
>>
>> Since many Linux systems may default the sysconfig net.ipv6.bindv6only
>> flag to true, it usually means that specifying a v6 endpoint will bind
>> to both v4 and v6. But this also means that deployment systems must be
>> aware of this while configuring depending on whether both v4 and v6
>> endpoints need to work or not. Specifying both a v4 and v6 endpoint or a
>> port (v4) and endpoint with the same v6 port will currently lead to a
>> failure as the system would've already bound the v6 port to both v4 and
>> v6. This leaves us with a few options.
>>
>> 1. Keep the implicit behaviour as it is, document this, as systems are
>> already aware of sysconfig flags and will expect that at a v6 endpoint
>> will bind to both v4 and v6.
>>
>> 2. Be explicit with endpoints & configuration, Beast itself overrides
>> the socket option to bind both v4 and v6, which means that v6 endpoint
>> will bind to v6 *only* and binding to a v4 will need an explicit
>> specification. (there is a pr in progress for this:
>> https://github.com/ceph/ceph/pull/27270)
>>
>> Any more suggestions on how systems handle this are also welcome.
>>
>> -- 
>> Abhishek
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com