Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Shmulik Hen]

On Tuesday 12 August 2003 04:08 pm, David S. Miller wrote:

> Policy belongs strictly at user space.

Regarding bonding, what policies are we talking about ?
run-time ?  config time ?  both ?  other ?
Our current aim is config time only.

> One of the great things about what Jamal spends his time working
> on is finally a strict seperation of the control layer from
> everything else.  And part of this is moving all of the control
> logic into userspace. Once that is accomplished, I can have my
> toilet flush every time a TCP packet is routed through my system
> and this won't crap up the kernel.

What scope are we talking about here ?
2.4 ?  2.6 ?  2.7 ?  other ?
Our current aim is 2.4 and 2.6.

Taking into account the two statements I made above:
Do you think that what we're doing right now might interfere with what 
Jamal is suggesting ?
Wouldn't it be possible to do things the old way first and than 
convert everything to the new way ?
Shouldn't all this be a part of a totally new project like "bonding2"

-- 
| Shmulik Hen   Advanced Network Services  |
| Israel Design Center, Jerusalem          |
| LAN Access Division, Platform Networking |
| Intel Communications Group, Intel corp.  |

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[David S. Miller]

On Tue, 12 Aug 2003 17:03:15 +0300
Shmulik Hen <shmulik.hen@intel.com> wrote:

> > One of the great things about what Jamal spends his time working
> > on is finally a strict seperation of the control layer from
> > everything else.  And part of this is moving all of the control
> > logic into userspace. Once that is accomplished, I can have my
> > toilet flush every time a TCP packet is routed through my system
> > and this won't crap up the kernel.
> 
> What scope are we talking about here ?
> 2.4 ?  2.6 ?  2.7 ?  other ?
> Our current aim is 2.4 and 2.6.

Jamal's work is longer term, I'll let him answer your other
questions.

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[jamal]

On Tue, 2003-08-12 at 10:03, Shmulik Hen wrote:

> 
> What scope are we talking about here ?
> 2.4 ?  2.6 ?  2.7 ?  other ?
> Our current aim is 2.4 and 2.6.
> 

You should start thinking 2.7. 

> Taking into account the two statements I made above:
> Do you think that what we're doing right now might interfere with what 
> Jamal is suggesting ?
> Wouldn't it be possible to do things the old way first and than 
> convert everything to the new way ?
> Shouldn't all this be a part of a totally new project like "bonding2"

yes, this would be next gen - 2.7 or above target.

cheers,
jamal

RE: [SET 2][PATCH 2/8][bonding] Propagating master's settings to slaves

[Hen, Shmulik]

> -----Original Message-----
> From: jamal [mailto:hadi@cyberus.ca]
> Sent: Saturday, August 09, 2003 1:01 AM
> To: Hen, Shmulik
> Cc: bonding-devel@lists.sourceforge.net; netdev@oss.sgi.com
> Subject: Re: [SET 2][PATCH 2/8][bonding] Propagating master's settings
> to slaves
> 
> Shmulik,
> 
> Some of this bonding stuff is pretty scary. Lotsa policies in the 
> kernel and communication seems to be centred around /proc.
> Shouldnt policies on failover be really driven from user space?
> Also shouldnt communication be using something like netlink?
> 
> cheers,
> jamal
> 
> On Fri, 2003-08-08 at 10:44, Shmulik Hen wrote:
> > 2 - Change monitoring function use the new functionality.
> > 
> 

Not sure I fully understood the concerns above, but I'll try
to explain what the change was all about.

By monitoring, I meant the 3 timer function running in bonding
to monitor link changes and act once a link fail/recovery is
detected. The old code used to do all the activity related to
changing the current active slave separately in each timer
function and it seemed redundant since it was basically the
same thing repeated 3 times. Instead, we thought it would be
best if we put that into 3 new functions - reselect_active,
find_best_slave and change_active that does all the actual stuff
of swapping an old current with the new one.

The change we did in /proc was to reduce the amount of data
extarcted each time the proc entry is polled. Instead of dumping
all the data of all the bond devices that exist, each bond returns
just data that is relevant to itself.

In the lonf term, the drive is to move any *smart* code done in
the config application into the driver itself and be left with
the smallest, most compact application as possible. This is the
trend we've seen in the VLAN config app, and the bridge module.
All the "brain" is in the kernel module and very little should be
done in the application.


	Shmulik.

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master's settings toslaves

[Shmulik Hen]

On Monday 11 August 2003 05:07 pm, Laurent DENIEL wrote:
> HP/Compaq/Digital used to have the same approach with their Netrain
> implementation, and from one release of Tru64 UNIX to another, they
> could no longer support resolution ala milli-seconds but only
> seconds due to the move of such "richness" to user space (among
> other things). I am not saying that doing so on Linux will result
> to the same, but a minimal failover policy shall remain in the
> kernel for performance reason ... (or a user space facility could
> exist to *configure* such policy but without direct interaction
> with user space when the kernel has to decide).
>
> Laurent

That was my point. Thank you for putting it into better words.
If high availbilty and fast failovers are what's needed, why move it 
out of kernel space and put it in an application ? How fast could it 
work compared to a kernel module ? Why need an extra piece of code 
running in user space (daemon?) to monitor a module when the module 
can do that itself ?

If smarter behavior is needed (e.g. falling to eth4 instead of eth1 
when eth0 fails), we can add some priority mechanism to the driver to 
do that when it decides to swap. Otherwise, we'll be devleoping 
applications from now on, not the Linux kernel :)


	Shmulik.

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master's settings toslaves

[jamal]

On Mon, 2003-08-11 at 10:20, Shmulik Hen wrote:
> On Monday 11 August 2003 05:07 pm, Laurent DENIEL wrote:
> > HP/Compaq/Digital used to have the same approach with their Netrain
> > implementation, and from one release of Tru64 UNIX to another, they
> > could no longer support resolution ala milli-seconds but only
> > seconds due to the move of such "richness" to user space (among
> > other things). I am not saying that doing so on Linux will result
> > to the same, but a minimal failover policy shall remain in the
> > kernel for performance reason ... (or a user space facility could
> > exist to *configure* such policy but without direct interaction
> > with user space when the kernel has to decide).
> >
> > Laurent
> 
> That was my point. Thank you for putting it into better words.
> If high availbilty and fast failovers are what's needed, why move it 
> out of kernel space and put it in an application ? How fast could it 
> work compared to a kernel module ? Why need an extra piece of code 
> running in user space (daemon?) to monitor a module when the module 
> can do that itself ?
>
> If smarter behavior is needed (e.g. falling to eth4 instead of eth1 
> when eth0 fails), we can add some priority mechanism to the driver to 
> do that when it decides to swap. Otherwise, we'll be devleoping 
> applications from now on, not the Linux kernel :)
> 

So how many smart things are you going to add to the driver? ;->
Do you wanna add the qos policy changeover as well? What about route
changes, firewalling etc. What about sliceing bread and adding butter?
Where do you draw the line?
BTW, I dont understand why it would slow down failover; sure it will a
tiny bit because you have to cross user space to lookup the policy.
Maybe this is the part that i havent made clear, heres an example:
- User space gets notified link eth0 went down
- User space looks up a policy config on what to do when eth0 goes down
- user space executes commands which may include telling kernel to 
move activity to eth1.
 
Note: I agree on a minimal failover policy staying in the kernel; very
basic stuff like what bonding used to do (may still do, dont know).

cheers,
jamal

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master's settings toslaves

[Shmulik Hen]

On Monday 11 August 2003 05:34 pm, jamal wrote:
> So how many smart things are you going to add to the driver? ;->
> Do you wanna add the qos policy changeover as well? What about
> route changes, firewalling etc. What about sliceing bread and
> adding butter? Where do you draw the line?
> BTW, I dont understand why it would slow down failover; sure it
> will a tiny bit because you have to cross user space to lookup the
> policy. Maybe this is the part that i havent made clear, heres an
> example: - User space gets notified link eth0 went down
> - User space looks up a policy config on what to do when eth0 goes
> down - user space executes commands which may include telling
> kernel to move activity to eth1.
>
> Note: I agree on a minimal failover policy staying in the kernel;
> very basic stuff like what bonding used to do (may still do, dont
> know).
>
> cheers,
> jamal

Why have any kernel code other than device drivers in the first place 
?
Why not move all the TCP/IP stack out of kernel space and put it in an 
application ? Lets have the entire ARP mechanism in an appliaction 
and let it handle everything from routing tables management to arp 
negotiation while the kernel will only know how to create arp packets 
that it gets from that app and send them away ? It doesn't need to 
have the know how.
Say we do thing s your way and use the notification mechanism, how 
long do you think it's going to take for the whole operation to 
finish taking into consideration how the kernel runs user space 
applications in comparison with kernel code? what happens when the 
system is heavily loaded ? What happens if the application dies for 
some reason ? 
Why should the bonding driver even care about routes or firewalling ?
It's only meant to group several physical ethernet devices and group 
them under one logical device to handle teaming solutions.

-- 
| Shmulik Hen   Advanced Network Services  |
| Israel Design Center, Jerusalem          |
| LAN Access Division, Platform Networking |
| Intel Communications Group, Intel corp.  |

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master's settings toslaves

[Jeff Garzik]

The answer is, like life, it's a balance.

As a general rule, we do prefer to move all code possible out of the 
Linux kernel.  We have even created "initramfs", which for 2.7, will be 
used as a vehicle to move code from the kernel to userspace, that 
previously had to be in the kernel only because it was a task that "had 
to be performed at boot time".

However, one must consider
(1) does moving code to userspace create any security holes?
(2) does moving code to userspace dramatically increase the number of 
context switches?
(3) does moving code to userspace violate some atomicity that being 
inside the kernel guarantees?

In practice, #3 is the showstopper that occurs most often.

This is why I push for a "bonding-utils" package from Jay.... because of 
the general rule above:  put it into userspace, where possible.

	Jeff

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Laurent DENIEL]

Jeff Garzik a écrit :
> 
> The answer is, like life, it's a balance.
> 
> As a general rule, we do prefer to move all code possible out of the
> Linux kernel.  We have even created "initramfs", which for 2.7, will be
> used as a vehicle to move code from the kernel to userspace, that
> previously had to be in the kernel only because it was a task that "had
> to be performed at boot time".
> 
> However, one must consider
> (1) does moving code to userspace create any security holes?
> (2) does moving code to userspace dramatically increase the number of
> context switches?
> (3) does moving code to userspace violate some atomicity that being
> inside the kernel guarantees?

You forgot one important aspect : 

  (4) does moving code to userspace break compatibility (or behavior) 
      with user land applications (or systems)

What can one do if say, kernel 2.[4|5] switches the NIC in 10 mseconds 
while kernel 2.7 with user land daemon switches in a few seconds ? 
nothing but stay with the previous version or fork the driver development ;-(

But I agree that it is interesting to do some stuff at user land, and if 
the bonding had an option to disable the automatic failover policy, 
this could be implemented with trigger towards user land application that 
could use an ioctl call to switch to the appropriate NIC according to 
the user lan configuration ...

But the fast and simple failover policy shall remain in kernel code.

Laurent

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Jeff Garzik]

Laurent DENIEL wrote:
> Jeff Garzik a écrit :
> 
>>The answer is, like life, it's a balance.
>>
>>As a general rule, we do prefer to move all code possible out of the
>>Linux kernel.  We have even created "initramfs", which for 2.7, will be
>>used as a vehicle to move code from the kernel to userspace, that
>>previously had to be in the kernel only because it was a task that "had
>>to be performed at boot time".
>>
>>However, one must consider
>>(1) does moving code to userspace create any security holes?
>>(2) does moving code to userspace dramatically increase the number of
>>context switches?
>>(3) does moving code to userspace violate some atomicity that being
>>inside the kernel guarantees?
> 
> 
> You forgot one important aspect : 
> 
>   (4) does moving code to userspace break compatibility (or behavior) 
>       with user land applications (or systems)

I agree... assuming these userland interfaces are fairly standard and 
widely deployed.


> What can one do if say, kernel 2.[4|5] switches the NIC in 10 mseconds 
> while kernel 2.7 with user land daemon switches in a few seconds ? 
> nothing but stay with the previous version or fork the driver development ;-(

This is a silly example.  If that happens in practice, then that is a 
bug in the configuration of the userland daemon, or a bug in the 
kernel<->userland ABI.


> But I agree that it is interesting to do some stuff at user land, and if 
> the bonding had an option to disable the automatic failover policy, 
> this could be implemented with trigger towards user land application that 
> could use an ioctl call to switch to the appropriate NIC according to 
> the user lan configuration ...

Remember, ioctls are bad.  :)  Unix design mistake.


> But the fast and simple failover policy shall remain in kernel code.

I would not make such absolute predictions, especially about policy :)

	Jeff

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Laurent DENIEL]

Jeff Garzik a écrit :
>
> > You forgot one important aspect :
> >
> >   (4) does moving code to userspace break compatibility (or behavior)
> >       with user land applications (or systems)
> 
> I agree... assuming these userland interfaces are fairly standard and
> widely deployed.
> 
> > What can one do if say, kernel 2.[4|5] switches the NIC in 10 mseconds
> > while kernel 2.7 with user land daemon switches in a few seconds ?
> > nothing but stay with the previous version or fork the driver development ;-(
> 
> This is a silly example.  If that happens in practice, then that is a
> bug in the configuration of the userland daemon, or a bug in the
> kernel<->userland ABI.

Not a silly example but a real case that happened to me with another
operating system and I'd hate if it happens also with Linux ...
 
> > But I agree that it is interesting to do some stuff at user land, and if
> > the bonding had an option to disable the automatic failover policy,
> > this could be implemented with trigger towards user land application that
> > could use an ioctl call to switch to the appropriate NIC according to
> > the user lan configuration ...
> 
> Remember, ioctls are bad.  :)  Unix design mistake.

ioctl (which already exist) or something else, this is not the point here.

> > what happens when the 
> > system is heavily loaded ? 
> 
> What happens now ? 
> 
> > What happens if the application dies for 
> > some reason ? 
> 
> What happens when the kernel oopses? ;->

Such silly responses make me think that it is no longer worth to argue ...

Laurent

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[jamal]

On Tue, 2003-08-12 at 02:31, Laurent DENIEL wrote:

 
> > > What happens if the application dies for 
> > > some reason ? 
> > 
> > What happens when the kernel oopses? ;->
> 
> Such silly responses make me think that it is no longer worth to argue ...
> 

You dont think asking "what if the application dies" is in the same
calibre as "what happens when the kernel oopses"?

cheers,
jamal

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[David S. Miller]

On 12 Aug 2003 08:59:17 -0400
jamal <hadi@cyberus.ca> wrote:

> You dont think asking "what if the application dies" is in the same
> calibre as "what happens when the kernel oopses"?

Don't sweat it Jamal, some people just don't get it :-)

Look, people, when userlevel routing daemon dies your system
effectively stops to route.

There is zero difference between that example and the ones
we are discussing here.

Policy belongs strictly at user space.

One of the great things about what Jamal spends his time working
on is finally a strict seperation of the control layer from everything
else.  And part of this is moving all of the control logic into userspace.
Once that is accomplished, I can have my toilet flush every time a TCP
packet is routed through my system and this won't crap up the kernel.

If you don't see the value in that, perhaps you shouldn't be partaking
in this discussion :-)

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Laurent DENIEL]

"David S. Miller" a écrit :
> 
> On 12 Aug 2003 08:59:17 -0400
> jamal <hadi@cyberus.ca> wrote:
> 
> > You dont think asking "what if the application dies" is in the same
> > calibre as "what happens when the kernel oopses"?
> 
> Don't sweat it Jamal, some people just don't get it :-)
> 
> Look, people, when userlevel routing daemon dies your system
> effectively stops to route.

That's why in really *safe* systems, we do not use routing daemon
but only static routes ;-)

And there is a BIG difference : 

When user level daemon dies, you have to be sure that some stuff
exists to monitor and recover from that situation (either by 
restarting the faulty deamon (if it could recover in time which
I doubt with the bonding case), or by switching to a new machine
in a fault tolerant configuration). With kernel ooops, there is
NOTHING to do in such in such a fault tolerant systems, since the
machine is unusable (this is the same as a hardware failure).

But people does not understand the constraints of really safe
systems.

> Policy belongs strictly at user space.
> 
> One of the great things about what Jamal spends his time working
> on is finally a strict seperation of the control layer from everything
> else.  And part of this is moving all of the control logic into userspace.
> Once that is accomplished, I can have my toilet flush every time a TCP
> packet is routed through my system and this won't crap up the kernel.
> 
> If you don't see the value in that, perhaps you shouldn't be partaking
> in this discussion :-)

This is OK as long as your kernel and user space stuff remains suitable
for highly fault tolerant systems and which does not require big montains
of user stuff to do the same as a few line of code in the kernel. Remember
the aim of bonding : NIC fault tolerance or load balancing. I am not against
a user space configurable policy for more complex job but the initial aim of 
the bonding shall remain coded in the kernel (and is only usable in the above 
mentioned systems in that way).

Laurent

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[Laurent DENIEL]

jamal a écrit :
> 
> On Tue, 2003-08-12 at 10:10, Laurent DENIEL wrote:
> > "David S. Miller" a écrit :
> 
> > That's why in really *safe* systems, we do not use routing daemon
> > but only static routes ;-)
> >
> > And there is a BIG difference :
> >
> > When user level daemon dies, you have to be sure that some stuff
> > exists to monitor and recover from that situation (either by
> > restarting the faulty deamon (if it could recover in time which
> > I doubt with the bonding case), or by switching to a new machine
> > in a fault tolerant configuration). With kernel ooops, there is
> > NOTHING to do in such in such a fault tolerant systems, since the
> > machine is unusable (this is the same as a hardware failure).
> >
> > But people does not understand the constraints of really safe
> > systems.
> >
> 
> We have hardware watchdog timers to put the kernel into a known state by
> rebooting. If you were not aware of all these RAS efforts on Linux
> (projects like kexec for example) I suggest you start looking at them.

I am aware of this great stuff but see below.

> The kernel will oops and the app will die because of one thing: _A
> software bug_. It doesnt matter what causes the death of the kernel or
> app ( a misconfig for example causing a broadcast loop making the app
> die is a bug).
> If you want a safe system then you donot trust software neither do you
> trust hardware - You must have workarounds incase they go beserk. Heck
> the only entity you should trust is God and thats assuming you believe
> in God.

Hardware / software watchdogs are great but do not necessarily 
solve all problems especially where timing constraints are important.
I prefer to rely on the timing of the bonding kernel code to switch
NIC in milli seconds that to wait seconds or minutes that a user space
daemon have the hand to handle the problem (and yes, I am aware of 
real time class scheduling and so on, but you say don't trust the 
software, and I agree so I prefer a direct kernel hang than nothing 
or something too late (software watchdog will not help in that case).

Laurent

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[jamal]

On Tue, 2003-08-12 at 10:36, Laurent DENIEL wrote:


> Hardware / software watchdogs are great but do not necessarily 
> solve all problems especially where timing constraints are important.

I think we are going on a tangent; i could ask you next why you think it
is less likely to have bugs in the kernel than in user space. Please
dont respond because we'll get into long circular debates.

> I prefer to rely on the timing of the bonding kernel code to switch
> NIC in milli seconds that to wait seconds or minutes that a user space
> daemon have the hand to handle the problem (and yes, I am aware of 
> real time class scheduling and so on, but you say don't trust the 
> software, and I agree so I prefer a direct kernel hang than nothing 
> or something too late (software watchdog will not help in that case).

I dont think we have any disagreements that minimalistic kernel policy
should stay. I am not suggesting to move what the _original_ bonding
driver did out of the kernel - so i dont think there are issues with
"waiting for seconds". Are you basing this on experience?

The key is this: We should start looking at bonding as an enabler for 
availabilty not as _the solution_. Bonding provides link availability
for single hops;
 
cheers,
jamal

Re: [Bonding-devel] Re: [SET 2][PATCH 2/8][bonding] Propagating master'ssettings toslaves

[jamal]

On Mon, 2003-08-11 at 13:31, Laurent DENIEL wrote:

> But I agree that it is interesting to do some stuff at user land, and if 
> the bonding had an option to disable the automatic failover policy, 
> this could be implemented with trigger towards user land application that 
> could use an ioctl call to switch to the appropriate NIC according to

You spoilt otherwise sane text by mentioning ioctl;-> 

> But the fast and simple failover policy shall remain in kernel code.

nod from here. Simple failover policies should stay in the kernel.

cheers,
jamal