question about default values for per-namespace settings

question about default values for per-namespace settings

[Vasily Averin]

Dear Tejun,

how do you think, which defaults should be used for per-namespace settings in general case
and for per-netns sysctls especially? Do we have some common position about this or
perhaps we already have some setting that allows to select desired behavior?

I'm preparing patch that makes per-netns sysctls in br_netfilter,
to be able to enable/disable br-nf-call processing in each network namespace independently.

I've initialized sysctl values in each netns by system defaults, like it was done in similar cases.
However Bart pointed that "this does introduce a bit of backwards incompatibility":
currently all netns shares the br_netfilter sysctl settings applied in init_net.

From OpenVz point of view containers should be properly isolated,
should have predictable initial configuration
and should not depend on settings applied in another containers.
On the other hand independent containers is only one of possible usecases,
and I have no strong objections against Bart's proposal. Frankly speaking
initially I've planned to copy setting from init_net too.

To make possible both variants I can introduce one more setting,
it allows to specify desired behavior:
to use system defaults or to copy current settings from init_net.

However probably the same dilemma was observed in another subsystems?
Perhaps this selector already exists?

If isn't, how do you think, should I introduce some new global parameter,
or may be it should be some local bridge-only-related setting?

Thank you,
	Vasily Averin

you can find some more details about my patch in netfilter-devel@
[PATCH RFC v3 2/2] br_netfilter: per-netns copy of structure for sysctl flags
On 05/13/2014 11:28 PM, Bart De Schuymer wrote:
> Vasily Averin schreef op 12/05/2014 22:11:
>> On 05/12/2014 11:04 PM, Bart De Schuymer wrote:
>>> Vasily Averin schreef op 12/05/2014 18:32:
>>>> pernet_operations creates per-netns copy of common structure for sysctl flags
>>>> and initialize it values taken from init_brnf_net.
>>>>
>>>> Signed-off-by: Vasily Averin <vvs-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
>>>
>>>> +static int __net_init brnf_net_init(struct net *net)
>>>> +{
>>>> +    struct brnf_net *bn = brnf_net(net);
>>>> +
>>>> +    memcpy(bn, &init_brnf_net, sizeof(struct brnf_net));
>>>> +    bn->net = net;
>>>> +    return brnf_sysctl_net_register(bn);
>>>
>>> This does introduce a bit of backwards incompatibility (easily fixed
>>> by adapting scripts), but this is really unavoidable when
>>> transforming an existing global configuration to a per-netns
>>> configuration. I'm ok with it.
>>
>> Could you please explain, which backward incompatibility you mean here?
>> Nobody changes values init_brnf_net,
>> init_net have own copy, like any other network namespaces.
> 
> Well, init_brnf_net is never written to, so it keeps the default flags.
> If a new netns is created, a copy of the contents of init_brnf_net is made. So, whenever a netns is created, it starts with the default flags (e.g. brnf_call_iptables is always 1 for a newly created netns).
> 
> In the current kernel, when a new netns is created, the configuration of the main netns is used (the proc system doesn't even show the flags in the created netns): if brnf_call_iptables is 0 before the new netns is created, iptables won't see bridged IP traffic in the new netns.
> With your patch, this behaviour will change.
> 
> It's possible to alter your patch to keep the same behaviour as before at netns creation, but always starting from the same defaults is cleaner.

Re: question about default values for per-namespace settings

[Serge Hallyn]

Quoting Vasily Averin (vvs-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org):
> Dear Tejun,
> 
> how do you think, which defaults should be used for per-namespace settings in general case
> and for per-netns sysctls especially? Do we have some common position about this or
> perhaps we already have some setting that allows to select desired behavior?
> 
> I'm preparing patch that makes per-netns sysctls in br_netfilter,
> to be able to enable/disable br-nf-call processing in each network namespace independently.
> 
> I've initialized sysctl values in each netns by system defaults, like it was done in similar cases.
> However Bart pointed that "this does introduce a bit of backwards incompatibility":
> currently all netns shares the br_netfilter sysctl settings applied in init_net.
> 
> From OpenVz point of view containers should be properly isolated,
> should have predictable initial configuration
> and should not depend on settings applied in another containers.

In 'another container' no, but if you are starting a nested container, then
the from the parent container, yes.  Not from init_net.

> On the other hand independent containers is only one of possible usecases,
> and I have no strong objections against Bart's proposal. Frankly speaking
> initially I've planned to copy setting from init_net too.
> 
> To make possible both variants I can introduce one more setting,
> it allows to specify desired behavior:
> to use system defaults or to copy current settings from init_net.
> 
> However probably the same dilemma was observed in another subsystems?
> Perhaps this selector already exists?
> 
> If isn't, how do you think, should I introduce some new global parameter,
> or may be it should be some local bridge-only-related setting?
> 
> Thank you,
> 	Vasily Averin
> 
> you can find some more details about my patch in netfilter-devel@
> [PATCH RFC v3 2/2] br_netfilter: per-netns copy of structure for sysctl flags
> On 05/13/2014 11:28 PM, Bart De Schuymer wrote:
> > Vasily Averin schreef op 12/05/2014 22:11:
> >> On 05/12/2014 11:04 PM, Bart De Schuymer wrote:
> >>> Vasily Averin schreef op 12/05/2014 18:32:
> >>>> pernet_operations creates per-netns copy of common structure for sysctl flags
> >>>> and initialize it values taken from init_brnf_net.
> >>>>
> >>>> Signed-off-by: Vasily Averin <vvs-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> >>>
> >>>> +static int __net_init brnf_net_init(struct net *net)
> >>>> +{
> >>>> +    struct brnf_net *bn = brnf_net(net);
> >>>> +
> >>>> +    memcpy(bn, &init_brnf_net, sizeof(struct brnf_net));
> >>>> +    bn->net = net;
> >>>> +    return brnf_sysctl_net_register(bn);
> >>>
> >>> This does introduce a bit of backwards incompatibility (easily fixed
> >>> by adapting scripts), but this is really unavoidable when
> >>> transforming an existing global configuration to a per-netns
> >>> configuration. I'm ok with it.
> >>
> >> Could you please explain, which backward incompatibility you mean here?
> >> Nobody changes values init_brnf_net,
> >> init_net have own copy, like any other network namespaces.
> > 
> > Well, init_brnf_net is never written to, so it keeps the default flags.
> > If a new netns is created, a copy of the contents of init_brnf_net is made. So, whenever a netns is created, it starts with the default flags (e.g. brnf_call_iptables is always 1 for a newly created netns).
> > 
> > In the current kernel, when a new netns is created, the configuration of the main netns is used (the proc system doesn't even show the flags in the created netns): if brnf_call_iptables is 0 before the new netns is created, iptables won't see bridged IP traffic in the new netns.
> > With your patch, this behaviour will change.
> > 
> > It's possible to alter your patch to keep the same behaviour as before at netns creation, but always starting from the same defaults is cleaner.
> 
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers

Re: question about default values for per-namespace settings

[Tejun Heo]

Hello,

On Thu, May 15, 2014 at 01:01:52PM +0400, Vasily Averin wrote:
> how do you think, which defaults should be used for per-namespace settings in general case
> and for per-netns sysctls especially? Do we have some common position about this or
> perhaps we already have some setting that allows to select desired behavior?

Unfortunately, I don't have much idea on the subject.  Serge and
others on the containers list should know a lot better than me.

Thanks.

-- 
tejun

Re: question about default values for per-namespace settings

[Bart De Schuymer]

Vasily Averin schreef op 15/05/2014 11:01:
> Dear Tejun,
>
> how do you think, which defaults should be used for per-namespace settings in general case
> and for per-netns sysctls especially? Do we have some common position about this or
> perhaps we already have some setting that allows to select desired behavior?
>
> I'm preparing patch that makes per-netns sysctls in br_netfilter,
> to be able to enable/disable br-nf-call processing in each network namespace independently.
>
> I've initialized sysctl values in each netns by system defaults, like it was done in similar cases.
> However Bart pointed that "this does introduce a bit of backwards incompatibility":
> currently all netns shares the br_netfilter sysctl settings applied in init_net.
>
>>From OpenVz point of view containers should be properly isolated,
> should have predictable initial configuration
> and should not depend on settings applied in another containers.
> On the other hand independent containers is only one of possible usecases,
> and I have no strong objections against Bart's proposal. Frankly speaking
> initially I've planned to copy setting from init_net too.

You misread my mail. I stated that I'm ok with always starting from the 
defaults (as your patch does). As pointed out by Maciej, always starting 
from init_net isn't really an option in case of nested namespaces (start 
from the parent's namespace instead).
There'll always be pros and cons to whatever you choose here. Complete 
backwards compatibility isn't possible either way.
The only way to keep backwards compatibility is to introduce new proc 
file names and keep the old behavior for the old names (but I'm not in 
favor of that).

cheers,
Bart

Re: question about default values for per-namespace settings

[Maciej Żenczykowski]

Isn't pulling current settings from init_net an isolation violation if
init_net isn't the namespace you are in at the time you are creating
the new namespace?

The way I see it there are 2 possibilities:
(a) you use some kernel (probably compile time) defaults (ie. what
init_net gets when you boot machine)
(b) you inherit from current namespace
I'm not sure what the right choice is.

For something like 'iptables configuration' it seems (a) is correct
(come up with no firewall).
For something like 'tcp socket memory limits' or 'bindv6only' or
'v6.default_use_tempaddr) it does seem like (b) is possibly more
appropriate.

That said I think there are cases where (a) is clearly correct and (b)
is clearly not desirable (iptables conf being a prime example).
After all a new namespace doesn't inherit interfaces from the
namespace we're in when we create it.

I can't think of any cases where (b) is clearly correct and (a) is
clearly not desirable.
[I guess this is less than clear for settings which auto scale at boot
with available ram and/or number of cpus in the machine]

Based on that doing (a) for everything may be the right choice
(consistency trump...).
This would imply network namespace you are in should have no effect on
the new network namespace you are creating.

OTOH, if I want to change some tcp mem tuning sysctl (or something
like net.ipv6.conf.default.use_tempaddr = 2)
it would be annoying if /etc/sysctl.conf didn't apply to non-init
namespace.  But perhaps this is better solved in userspace
by loading some /etc/sysctls-for-new-network-namespaces.conf settings
in some network namespace creating libraries.

- Maciej

Re: question about default values for per-namespace settings

[Vasily Averin]

On 05/19/2014 11:30 PM, Bart De Schuymer wrote:
> As pointed out by Maciej, always
> starting from init_net isn't really an option in case of nested
> namespaces (start from the parent's namespace instead).

Dear Bart, Serge, Maciej
thank you very much for your feedback!

I've analyzed possibility to inherit settings from parent net-namespace,
discovered problems described below and finally decided to follow
Maciej's way (a) "use some kernel defaults", with adding an ability
to change pre-compiled kernel defaults.

Below you can found more detailed description of discovered problems.

1) there are no (easy) ways to find parent of given network namespace.

Network namespaces in kernel are not hierarchical but flat,
"struct net" have no reference to parent netns, and my collegians expect
that Eric Biederman will likely object to adding a parent netns pointer.

Without this reference I do not see any good ways to copy parents settings.

2) settings inheriting does not work if subsystem module is loaded after
creation of network namespace. 

In this case all namespaces get pre-compiled defaults settings, and seems
there are no ways to apply "adjusted" setting to all already existing netns.

Moreover there is curious situation: to apply required sysctl settings
during module loading, Red Hat recommends to force "sysctl -p" execution
via install command in modprobe.conf
https://bugzilla.redhat.com/show_bug.cgi?id=634735#c7

However if module is loaded from inside one of network namespaces
it does not work!

In this case sysctl is executed inside netns. 
If assigned sysctl key is not virtualized -- sysctl command can fail
if key is virtualized  -- setting  in current netns  will be adjusted,
but not -- in init_net, that looks unexpected for me.

I believe initial subsystem settings of newly created namespace should
not differ from initial settings of newly created subsystem in already
existing namespace. In case in-kernel setting inheriting this behavior
cannot be reached, additional subsystem tuning is required anyway.

Therefore Maceiej's variant (a) "use some kernel defaults" looks like
right choice for me. If parent wants to assign some adjusted settings
in child environments -- it can only force loading of required modules
and apply required settings directly.

At the same time I would like to have an ability to change pre-compiled
defaults somehow. In my patch I'm going to add new module options, that
allows node owner to specify wished "safe" settings before module loading,
and change them via sysfs after this.

Re: question about default values for per-namespace settings

[Eric W. Biederman]

Vasily Averin <vvs-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org> writes:

> On 05/19/2014 11:30 PM, Bart De Schuymer wrote:
>> As pointed out by Maciej, always
>> starting from init_net isn't really an option in case of nested
>> namespaces (start from the parent's namespace instead).
>
> Dear Bart, Serge, Maciej
> thank you very much for your feedback!

I am missing the context which makes raises this issue.

> I've analyzed possibility to inherit settings from parent net-namespace,
> discovered problems described below and finally decided to follow
> Maciej's way (a) "use some kernel defaults", with adding an ability
> to change pre-compiled kernel defaults.
>
> Below you can found more detailed description of discovered problems.
>
> 1) there are no (easy) ways to find parent of given network namespace.
>
> Network namespaces in kernel are not hierarchical but flat,
> "struct net" have no reference to parent netns, and my collegians expect
> that Eric Biederman will likely object to adding a parent netns pointer.
>
> Without this reference I do not see any good ways to copy parents
> settings.

Copying settings can easily happen at netowkr namespace creation time.
Copying at any other time is too weird to even think about.  So no you
don't need a parent network namespace pointer to enable copying.

> 2) settings inheriting does not work if subsystem module is loaded after
> creation of network namespace. 
>
> In this case all namespaces get pre-compiled defaults settings, and seems
> there are no ways to apply "adjusted" setting to all already existing
> netns.

setns.

> Moreover there is curious situation: to apply required sysctl settings
> during module loading, Red Hat recommends to force "sysctl -p" execution
> via install command in modprobe.conf
> https://bugzilla.redhat.com/show_bug.cgi?id=634735#c7
>
> However if module is loaded from inside one of network namespaces
> it does not work!

Why not?  The appropriate events should fire globally.  And note
in most use cases participants in a network namespace won't have
permissions to load modules.

> In this case sysctl is executed inside netns. 
> If assigned sysctl key is not virtualized -- sysctl command can fail
> if key is virtualized  -- setting  in current netns  will be adjusted,
> but not -- in init_net, that looks unexpected for me.

From what little you have said.  This sounds like a don't do that
then situation.  Certainly if a module is the kernel triggers request
module the module will be loaded in the initial set of namespaces.

> I believe initial subsystem settings of newly created namespace should
> not differ from initial settings of newly created subsystem in already
> existing namespace. In case in-kernel setting inheriting this behavior
> cannot be reached, additional subsystem tuning is required anyway.

You are arguing that creation of a network namespace should use the
kernel's default values for sysctls?  That is a fairly reasonable
position to take.

> Therefore Maceiej's variant (a) "use some kernel defaults" looks like
> right choice for me. If parent wants to assign some adjusted settings
> in child environments -- it can only force loading of required modules
> and apply required settings directly.
>
> At the same time I would like to have an ability to change pre-compiled
> defaults somehow. In my patch I'm going to add new module options, that
> allows node owner to specify wished "safe" settings before module loading,
> and change them via sysfs after this.

Why sysfs and not sysctl?

It is not clear to me what is going on, from the limited details I see
in this message it sounds like there may be a bit of overdesign and
tackling problems that do not matter in the real world going on.

For any kernel settings that apply to a network namespace we have
two very basic choices.
- Set them to default values when a namespace is initialized.
- Copy them from somewhere when the namespace is created.

Last I looked at that code we were copying sysctl values from
the initial network namespace instead of the creators network
namespace.  Which has always seemed a bit silly to me.

In general most people don't care and this does not cause an
issue for most folks, or we could not have gone 5+ years without
addressing it.

For most things any practical program at this point is going to
have to set the sysctls it cares about because it is going to
have to run on existing kernels.

Beyond that I don't have a strong opinion but we could either set values
to well expected defaults, or copy them from the creators previous
network namespace.  Both would give deterministic results without any
significant chance of breaking userspace today.

Is their a compelling use case in this conversation that could weigh the
decision of which semantics make the most sense?

Adding sysfs entries or module parameters to change the action of
sysctls sounds like there is something broken somewhere.  Unfortunately
it is not clear to me where that somewhere is.

Eric