Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Laurent DENIEL]

Jeff Garzik a écrit :
> 
> The answer is, like life, it's a balance.
> 
> As a general rule, we do prefer to move all code possible out of the
> Linux kernel.  We have even created "initramfs", which for 2.7, will be
> used as a vehicle to move code from the kernel to userspace, that
> previously had to be in the kernel only because it was a task that "had
> to be performed at boot time".
> 
> However, one must consider
> (1) does moving code to userspace create any security holes?
> (2) does moving code to userspace dramatically increase the number of
> context switches?
> (3) does moving code to userspace violate some atomicity that being
> inside the kernel guarantees?

You forgot one important aspect : 

  (4) does moving code to userspace break compatibility (or behavior) 
      with user land applications (or systems)

What can one do if say, kernel 2.[4|5] switches the NIC in 10 mseconds 
while kernel 2.7 with user land daemon switches in a few seconds ? 
nothing but stay with the previous version or fork the driver development ;-(

But I agree that it is interesting to do some stuff at user land, and if 
the bonding had an option to disable the automatic failover policy, 
this could be implemented with trigger towards user land application that 
could use an ioctl call to switch to the appropriate NIC according to 
the user lan configuration ...

But the fast and simple failover policy shall remain in kernel code.

Laurent

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Jeff Garzik]

Laurent DENIEL wrote:
> Jeff Garzik a écrit :
> 
>>The answer is, like life, it's a balance.
>>
>>As a general rule, we do prefer to move all code possible out of the
>>Linux kernel.  We have even created "initramfs", which for 2.7, will be
>>used as a vehicle to move code from the kernel to userspace, that
>>previously had to be in the kernel only because it was a task that "had
>>to be performed at boot time".
>>
>>However, one must consider
>>(1) does moving code to userspace create any security holes?
>>(2) does moving code to userspace dramatically increase the number of
>>context switches?
>>(3) does moving code to userspace violate some atomicity that being
>>inside the kernel guarantees?
> 
> 
> You forgot one important aspect : 
> 
>   (4) does moving code to userspace break compatibility (or behavior) 
>       with user land applications (or systems)

I agree... assuming these userland interfaces are fairly standard and 
widely deployed.


> What can one do if say, kernel 2.[4|5] switches the NIC in 10 mseconds 
> while kernel 2.7 with user land daemon switches in a few seconds ? 
> nothing but stay with the previous version or fork the driver development ;-(

This is a silly example.  If that happens in practice, then that is a 
bug in the configuration of the userland daemon, or a bug in the 
kernel<->userland ABI.


> But I agree that it is interesting to do some stuff at user land, and if 
> the bonding had an option to disable the automatic failover policy, 
> this could be implemented with trigger towards user land application that 
> could use an ioctl call to switch to the appropriate NIC according to 
> the user lan configuration ...

Remember, ioctls are bad.  :)  Unix design mistake.


> But the fast and simple failover policy shall remain in kernel code.

I would not make such absolute predictions, especially about policy :)

	Jeff

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[jamal]

On Mon, 2003-08-11 at 13:31, Laurent DENIEL wrote:

> But I agree that it is interesting to do some stuff at user land, and if 
> the bonding had an option to disable the automatic failover policy, 
> this could be implemented with trigger towards user land application that 
> could use an ioctl call to switch to the appropriate NIC according to

You spoilt otherwise sane text by mentioning ioctl;-> 

> But the fast and simple failover policy shall remain in kernel code.

nod from here. Simple failover policies should stay in the kernel.

cheers,
jamal

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Laurent DENIEL]

Jeff Garzik a écrit :
>
> > You forgot one important aspect :
> >
> >   (4) does moving code to userspace break compatibility (or behavior)
> >       with user land applications (or systems)
> 
> I agree... assuming these userland interfaces are fairly standard and
> widely deployed.
> 
> > What can one do if say, kernel 2.[4|5] switches the NIC in 10 mseconds
> > while kernel 2.7 with user land daemon switches in a few seconds ?
> > nothing but stay with the previous version or fork the driver development ;-(
> 
> This is a silly example.  If that happens in practice, then that is a
> bug in the configuration of the userland daemon, or a bug in the
> kernel<->userland ABI.

Not a silly example but a real case that happened to me with another
operating system and I'd hate if it happens also with Linux ...
 
> > But I agree that it is interesting to do some stuff at user land, and if
> > the bonding had an option to disable the automatic failover policy,
> > this could be implemented with trigger towards user land application that
> > could use an ioctl call to switch to the appropriate NIC according to
> > the user lan configuration ...
> 
> Remember, ioctls are bad.  :)  Unix design mistake.

ioctl (which already exist) or something else, this is not the point here.

> > what happens when the 
> > system is heavily loaded ? 
> 
> What happens now ? 
> 
> > What happens if the application dies for 
> > some reason ? 
> 
> What happens when the kernel oopses? ;->

Such silly responses make me think that it is no longer worth to argue ...

Laurent

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[jamal]

On Tue, 2003-08-12 at 02:31, Laurent DENIEL wrote:

 
> > > What happens if the application dies for 
> > > some reason ? 
> > 
> > What happens when the kernel oopses? ;->
> 
> Such silly responses make me think that it is no longer worth to argue ...
> 

You dont think asking "what if the application dies" is in the same
calibre as "what happens when the kernel oopses"?

cheers,
jamal

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[David S. Miller]

On 12 Aug 2003 08:59:17 -0400
jamal <hadi@cyberus.ca> wrote:

> You dont think asking "what if the application dies" is in the same
> calibre as "what happens when the kernel oopses"?

Don't sweat it Jamal, some people just don't get it :-)

Look, people, when userlevel routing daemon dies your system
effectively stops to route.

There is zero difference between that example and the ones
we are discussing here.

Policy belongs strictly at user space.

One of the great things about what Jamal spends his time working
on is finally a strict seperation of the control layer from everything
else.  And part of this is moving all of the control logic into userspace.
Once that is accomplished, I can have my toilet flush every time a TCP
packet is routed through my system and this won't crap up the kernel.

If you don't see the value in that, perhaps you shouldn't be partaking
in this discussion :-)

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Shmulik Hen]

On Tuesday 12 August 2003 04:08 pm, David S. Miller wrote:

> Policy belongs strictly at user space.

Regarding bonding, what policies are we talking about ?
run-time ?  config time ?  both ?  other ?
Our current aim is config time only.

> One of the great things about what Jamal spends his time working
> on is finally a strict seperation of the control layer from
> everything else.  And part of this is moving all of the control
> logic into userspace. Once that is accomplished, I can have my
> toilet flush every time a TCP packet is routed through my system
> and this won't crap up the kernel.

What scope are we talking about here ?
2.4 ?  2.6 ?  2.7 ?  other ?
Our current aim is 2.4 and 2.6.

Taking into account the two statements I made above:
Do you think that what we're doing right now might interfere with what 
Jamal is suggesting ?
Wouldn't it be possible to do things the old way first and than 
convert everything to the new way ?
Shouldn't all this be a part of a totally new project like "bonding2"

-- 
| Shmulik Hen   Advanced Network Services  |
| Israel Design Center, Jerusalem          |
| LAN Access Division, Platform Networking |
| Intel Communications Group, Intel corp.  |

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[David S. Miller]

On Tue, 12 Aug 2003 17:03:15 +0300
Shmulik Hen <shmulik.hen@intel.com> wrote:

> > One of the great things about what Jamal spends his time working
> > on is finally a strict seperation of the control layer from
> > everything else.  And part of this is moving all of the control
> > logic into userspace. Once that is accomplished, I can have my
> > toilet flush every time a TCP packet is routed through my system
> > and this won't crap up the kernel.
> 
> What scope are we talking about here ?
> 2.4 ?  2.6 ?  2.7 ?  other ?
> Our current aim is 2.4 and 2.6.

Jamal's work is longer term, I'll let him answer your other
questions.

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Laurent DENIEL]

"David S. Miller" a écrit :
> 
> On 12 Aug 2003 08:59:17 -0400
> jamal <hadi@cyberus.ca> wrote:
> 
> > You dont think asking "what if the application dies" is in the same
> > calibre as "what happens when the kernel oopses"?
> 
> Don't sweat it Jamal, some people just don't get it :-)
> 
> Look, people, when userlevel routing daemon dies your system
> effectively stops to route.

That's why in really *safe* systems, we do not use routing daemon
but only static routes ;-)

And there is a BIG difference : 

When user level daemon dies, you have to be sure that some stuff
exists to monitor and recover from that situation (either by 
restarting the faulty deamon (if it could recover in time which
I doubt with the bonding case), or by switching to a new machine
in a fault tolerant configuration). With kernel ooops, there is
NOTHING to do in such in such a fault tolerant systems, since the
machine is unusable (this is the same as a hardware failure).

But people does not understand the constraints of really safe
systems.

> Policy belongs strictly at user space.
> 
> One of the great things about what Jamal spends his time working
> on is finally a strict seperation of the control layer from everything
> else.  And part of this is moving all of the control logic into userspace.
> Once that is accomplished, I can have my toilet flush every time a TCP
> packet is routed through my system and this won't crap up the kernel.
> 
> If you don't see the value in that, perhaps you shouldn't be partaking
> in this discussion :-)

This is OK as long as your kernel and user space stuff remains suitable
for highly fault tolerant systems and which does not require big montains
of user stuff to do the same as a few line of code in the kernel. Remember
the aim of bonding : NIC fault tolerance or load balancing. I am not against
a user space configurable policy for more complex job but the initial aim of 
the bonding shall remain coded in the kernel (and is only usable in the above 
mentioned systems in that way).

Laurent

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[jamal]

On Tue, 2003-08-12 at 10:03, Shmulik Hen wrote:

> 
> What scope are we talking about here ?
> 2.4 ?  2.6 ?  2.7 ?  other ?
> Our current aim is 2.4 and 2.6.
> 

You should start thinking 2.7. 

> Taking into account the two statements I made above:
> Do you think that what we're doing right now might interfere with what 
> Jamal is suggesting ?
> Wouldn't it be possible to do things the old way first and than 
> convert everything to the new way ?
> Shouldn't all this be a part of a totally new project like "bonding2"

yes, this would be next gen - 2.7 or above target.

cheers,
jamal

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Laurent DENIEL]

jamal a écrit :
> 
> On Tue, 2003-08-12 at 10:10, Laurent DENIEL wrote:
> > "David S. Miller" a écrit :
> 
> > That's why in really *safe* systems, we do not use routing daemon
> > but only static routes ;-)
> >
> > And there is a BIG difference :
> >
> > When user level daemon dies, you have to be sure that some stuff
> > exists to monitor and recover from that situation (either by
> > restarting the faulty deamon (if it could recover in time which
> > I doubt with the bonding case), or by switching to a new machine
> > in a fault tolerant configuration). With kernel ooops, there is
> > NOTHING to do in such in such a fault tolerant systems, since the
> > machine is unusable (this is the same as a hardware failure).
> >
> > But people does not understand the constraints of really safe
> > systems.
> >
> 
> We have hardware watchdog timers to put the kernel into a known state by
> rebooting. If you were not aware of all these RAS efforts on Linux
> (projects like kexec for example) I suggest you start looking at them.

I am aware of this great stuff but see below.

> The kernel will oops and the app will die because of one thing: _A
> software bug_. It doesnt matter what causes the death of the kernel or
> app ( a misconfig for example causing a broadcast loop making the app
> die is a bug).
> If you want a safe system then you donot trust software neither do you
> trust hardware - You must have workarounds incase they go beserk. Heck
> the only entity you should trust is God and thats assuming you believe
> in God.

Hardware / software watchdogs are great but do not necessarily 
solve all problems especially where timing constraints are important.
I prefer to rely on the timing of the bonding kernel code to switch
NIC in milli seconds that to wait seconds or minutes that a user space
daemon have the hand to handle the problem (and yes, I am aware of 
real time class scheduling and so on, but you say don't trust the 
software, and I agree so I prefer a direct kernel hang than nothing 
or something too late (software watchdog will not help in that case).

Laurent

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[jamal]

On Tue, 2003-08-12 at 10:36, Laurent DENIEL wrote:


> Hardware / software watchdogs are great but do not necessarily 
> solve all problems especially where timing constraints are important.

I think we are going on a tangent; i could ask you next why you think it
is less likely to have bugs in the kernel than in user space. Please
dont respond because we'll get into long circular debates.

> I prefer to rely on the timing of the bonding kernel code to switch
> NIC in milli seconds that to wait seconds or minutes that a user space
> daemon have the hand to handle the problem (and yes, I am aware of 
> real time class scheduling and so on, but you say don't trust the 
> software, and I agree so I prefer a direct kernel hang than nothing 
> or something too late (software watchdog will not help in that case).

I dont think we have any disagreements that minimalistic kernel policy
should stay. I am not suggesting to move what the _original_ bonding
driver did out of the kernel - so i dont think there are issues with
"waiting for seconds". Are you basing this on experience?

The key is this: We should start looking at bonding as an enabler for 
availabilty not as _the solution_. Bonding provides link availability
for single hops;
 
cheers,
jamal