Re: Patch: Device operative state notification against 2.5.7

Re: Patch: Device operative state notification against 2.5.7

[Stefan Rompf]

Hi Jamal,

> in the future can you please just post to netdev on networking
> related issues? I responded to lk, but feel free to remove it
> from the list.

ok. Do all netdev people already know the patch or shall I repost it to
this list?

> -I am not sure the idea of using a kernel thread is the best. Maybe
> move the checks to both the tx or rx softirqs instead of its own scheduling.
> -In particular, it would be a better idea not to just go walking all the
> devices; only walk devices that have raised an netif_carrier_.

Is it possible to send (netlink) messages from a softirq? Anyway, in the
tx/rx softirqs the check would be executed for nearly every packet, so
the tradeoff of having a kernel thread that polls the devices on request
(so may be never as long as the network is fine) seems better to me,
even if it has to walk through the complete list.

> -A shared devices bitmask across SMP should be enough (i.e no need for
> per-CPU state)

Neither dev->flags nor dev->state are per-CPU. Am I missing something?

> -Another thing might be to double check that the condition that raised
> the state change is still valid example -
> in between the moment you say a link is down due to some bad hardware
> fault to the moment some device timer recovers it;

The patch does that. After the thread wakes up, it iterates over the
device list and sends only messages if it sees differences between
IFF_RUNNING and LINK_STATE_NO_CARRIER.

> -Also IFF_RUNNING seems to have inconsistent semantics in a lot of
> drivers. It should really stand for "operational status" whereas
> IFF_UP should stand for "admin status" -- anyone wanna shed historical
> wisdom here?

Ultimately, that's what I want IFF_RUNNING to become in linux.

There are 14 files left below linux/drivers that still play with
IFF_RUNNING instead of using the netif_carrier_o* functions. I'm willing
to supply patches for them as they are broken anyway, but I won't be
able to do more than a test if the modifications compile.

Cheers, Stefan

Re: Patch: Device operative state notification against 2.5.7

[Stefan Rompf]

Hi Jamal,

> in the future can you please just post to netdev on networking
> related issues? I responded to lk, but feel free to remove it
> from the list.

ok. Do all netdev people already know the patch or shall I repost it to
this list?

> -I am not sure the idea of using a kernel thread is the best. Maybe
> move the checks to both the tx or rx softirqs instead of its own scheduling.
> -In particular, it would be a better idea not to just go walking all the
> devices; only walk devices that have raised an netif_carrier_.

Is it possible to send (netlink) messages from a softirq? Anyway, in the
tx/rx softirqs the check would be executed for nearly every packet, so
the tradeoff of having a kernel thread that polls the devices on request
(so may be never as long as the network is fine) seems better to me,
even if it has to walk through the complete list.

> -A shared devices bitmask across SMP should be enough (i.e no need for
> per-CPU state)

Neither dev->flags nor dev->state are per-CPU. Am I missing something?

> -Another thing might be to double check that the condition that raised
> the state change is still valid example -
> in between the moment you say a link is down due to some bad hardware
> fault to the moment some device timer recovers it;

The patch does that. After the thread wakes up, it iterates over the
device list and sends only messages if it sees differences between
IFF_RUNNING and LINK_STATE_NO_CARRIER.

> -Also IFF_RUNNING seems to have inconsistent semantics in a lot of
> drivers. It should really stand for "operational status" whereas
> IFF_UP should stand for "admin status" -- anyone wanna shed historical
> wisdom here?

Ultimately, that's what I want IFF_RUNNING to become in linux.

There are 14 files left below linux/drivers that still play with
IFF_RUNNING instead of using the netif_carrier_o* functions. I'm willing
to supply patches for them as they are broken anyway, but I won't be
able to do more than a test if the modifications compile.

Cheers, Stefan

Re: Patch: Device operative state notification against 2.5.7

[jamal]

On Sun, 31 Mar 2002, Stefan Rompf wrote:

> Hi Jamal,
>
> > in the future can you please just post to netdev on networking
> > related issues? I responded to lk, but feel free to remove it
> > from the list.
>
> ok. Do all netdev people already know the patch or shall I repost it to
> this list?

For the uninitiated, here is where i saw it:
http://marc.theaimsgroup.com/?l=linux-net&m=101751062827998&w=1

(sorry, i am not subscribed to lk)

>
> > -I am not sure the idea of using a kernel thread is the best. Maybe
> > move the checks to both the tx or rx softirqs instead of its own scheduling.
> > -In particular, it would be a better idea not to just go walking all the
> > devices; only walk devices that have raised an netif_carrier_.
>
> Is it possible to send (netlink) messages from a softirq?

Why not?

> Anyway, in the
> tx/rx softirqs the check would be executed for nearly every packet, so
> the tradeoff of having a kernel thread that polls the devices on request
> (so may be never as long as the network is fine) seems better to me,
> even if it has to walk through the complete list.

Moving it to softirq gives it more privilege; however, you are right, the
tx/rx softirqs might not be the best place (i thought people will scream
at you if you went ahead and created a sofirq just for this). In any
case, this is a lesser concern for now, a few ms later is no biggie...

In regards to walking the whole dev list:
There is no point in polling when there is no work to be done (and
therefore no need for you to walk all the devices which have not shown
interest). A simple "registration" by devices raising or lowering carrier
would be the best way to do it (via netif_carrier_on/ff). The first
device on the list also wakes up the thread (or as you do right now
always wakes it up).
Keep rescheduling the thread until there are no more devices left (no
need for the timer) -- killing the thread might not be a bad idea either
since you would expect this not to happen that often.

>
> > -A shared devices bitmask across SMP should be enough (i.e no need for
> > per-CPU state)
>
> Neither dev->flags nor dev->state are per-CPU. Am I missing something?

I meant the structure that keeps track of changed carrier state ..
Example the simple linked list i mentioned above;

> > -Another thing might be to double check that the condition that raised
> > the state change is still valid example -
> > in between the moment you say a link is down due to some bad hardware
> > fault to the moment some device timer recovers it;
>
> The patch does that. After the thread wakes up, it iterates over the
> device list and sends only messages if it sees differences between
> IFF_RUNNING and LINK_STATE_NO_CARRIER.
>

I think you need to watch serialization issues with
netif_carrier_on/off between the driver and the top layer.
The difference between IFF_RUNNING and LINK_STATE_NO_CARRIER, IMO, is
insufficient check. This is because the semantics of IFF_RUNNING are not
well defined. Currently from scanning drivers i see IFF_RUNNING following
LINK_STATE_NO_CARRIER always i.e IFF_RUNNING is set when
LINK_STATE_NO_CARRIER is deasserted and vice-versa. Note this is not
always the case and it also does not say that we went from operational
down to up or vice-versa or whether we went on->off->on i.e looking at
those two flags alone is ambigous.
Note, you want to report both transitions and you also want to make
sure its not just simple noise.

> > -Also IFF_RUNNING seems to have inconsistent semantics in a lot of
> > drivers. It should really stand for "operational status" whereas
> > IFF_UP should stand for "admin status" -- anyone wanna shed historical
> > wisdom here?
>
> Ultimately, that's what I want IFF_RUNNING to become in linux.
>
> There are 14 files left below linux/drivers that still play with
> IFF_RUNNING instead of using the netif_carrier_o* functions. I'm willing
> to supply patches for them as they are broken anyway, but I won't be
> able to do more than a test if the modifications compile.
>

If we were to follow the BSD or CISCOish view of the world, then
removing a cable or death of allocated virtual circuit/tunnel would
mean !IFF_RUNNING. If i was to draw a truth table it would look like:

NO_CARRIER | IFF_RUNNING | meaning
-----------|-------------|------------------------------------
!set       | !set        | There is carrier, but no cable
           |             | no sense for ethernet; but may be useful
           |             | for PPP (for example line protocol is not up)
-----------|-------------|----------------------------------------
!set       | set         | operational up
           |             |
           |             |
-----------|-------------|--------------------------------------
set        | !set        | operational down
           |             |
           |             |
-----------|-------------|--------------------------------------
set        | set         | carrier off, cable on; not sure what this
           |             | means (may make sense for host scope)
           |             |
-----------|-------------|--------------------------------------

You still need to maintain state to catch the on->off->on. For example, no
point in reporting such state transition via netlink -- but you could call
that an optimization.
TB above is a little different from what you have in your code
(essentially you also add to the noise by mucking with IFF_RUNNING) which
in itself is philosophical in nature:
- Is it correct to get rid of the driver having a say in IFF_RUNNING?
For example is it correct that the core code sets IFF_RUNNING on the
device? Think of either a bonding, VLANs or extreme case where PPP
circuit is up for one protocol but not for another.
We need to disambiguate operational and admin status very clearly. Eg
should ifconfing-ing up a device mean it is also IFF_RUNNING (this is what
ifconfig does i think)

cheers,
jamal

Re: Patch: Device operative state notification against 2.5.7

[Stefan Rompf]

Hi Jamal,

first thanks for your long reply. I've moved the last part of your mail
to the front to answer on it first.

> If we were to follow the BSD or CISCOish view of the world, then
> removing a cable or death of allocated virtual circuit/tunnel would
> mean !IFF_RUNNING.

You've caught me, a lot of my networking experience involved cisco stuff
;-) Anyway, I thought the semantic decision has already been made during
the development cycle from 2.2 to 2.4. In 2.4, IFF_RUNNING of the
in-kernel dev->flags it totally unused except for broken drivers. What
you see in fact as IFF_RUNNING when querying an interface from
userspace, is the LINK_STATE_NOCARRIER bit. Look at
net/core/dev.c::dev_ifsioc(), case SIOCGIFFLAGS or the rtnetlink
equivalent. Consequently, dev_change_flags() does not touch the
IFF_RUNNING flag (part of IFF_VOLATILE).

> NO_CARRIER | IFF_RUNNING | meaning
> -----------|-------------|------------------------------------
> !set       | !set        | There is carrier, but no cable
>            |             | no sense for ethernet; but may be useful
>            |             | for PPP (for example line protocol is not up)

If LPCP in a PPP interface is not open and the interface is not waiting
for dial on demand, I'd consider it admin down, !IFF_UP. May be the term
"carrier" in the flag and functions is simply misnamed and we should
have a three states instead. If the interface is admin up, it's
operational state can be:

-up
 The driver is sure that it can transmit and receive packets over the
line. Depending on the driver, that can be HDLC framing, HDLC
keepalives, ethernet carrier, generically said some kind of heartbeat.
This state does not include any protocols that might be used over the
line, just that layer 1 is up -> IFF_RUNNING
-down
 The driver cannot transmit packets. This may be the inverse of one of
the conditions above, but can also something like a stalled ethernet
tranceiver (adding a fourth internal state) -> !IFF_RUNNING
-unknown
 Some dial on demand interface that is currently not bound to a physical
or logical line, mostly your truth table entry quoted above. May be
these interfaces should simply have an IFF_UNBOUND flag that is set in
this mode

> Think of either a bonding, VLANs

For these, operational state should follow the underlying interface(s).

> or extreme case where PPP
> circuit is up for one protocol but not for another.

That's a completely different case. If negotiation for a specific
protocol fails, the interface is still admin & operational up, but
cannot have a valid address for the protocol in question. In this case,
it is the users' decision if it makes sense to keep the interface admin
up.

> > Is it possible to send (netlink) messages from a softirq?
> 
> Why not?

You are right, but my thread actually calls netdev_state_change(), and
the event devinet.c::inetdev_event() requires holding the RTNL
semaphore. As I want in kernel notification, so that f.e. the VLAN
driver can react on state changes of underlying devices, some kind of
process context is needed.

> In regards to walking the whole dev list:
> There is no point in polling when there is no work to be done (and
> therefore no need for you to walk all the devices which have not shown
> interest). A simple "registration" by devices raising or lowering carrier
> would be the best way to do it (via netif_carrier_on/ff). The first
> device on the list also wakes up the thread (or as you do right now
> always wakes it up).

I'll have a look if I can implement such a list. There are three reasons
why I haven't done it in the current version:

-netif_carrier_on()/_off() seems to be designed to be called from
anywhere, interrupts, timers, processes, so I didn't want to add too
much locking and memory allocation in this place. Also I have to admit
that I'm not yet long enough in kernel programming to know which kind of
memory I may safely allocate for list elements from an interrupt ;-)
-When the thread goes through this list of events, it has either to
check for every element if the device still exists (currently only
possible by traversing the device list so that the thread would actually
get _less_ efficient) or the event list has to be consulted on interface
removal
-A runaway driver must be kept from allocating *lots* of kernel memory
by calling netif_carrier_on()/_off() in a tight loop

And yes, I was lazy and didn't want to overoptimize stuff that lies
dormant nearly all the time ;-)

> Note, you want to report both transitions and you also want to make
> sure its not just simple noise.

A better solution should forward an op down immediatly and delay op up
by a defined time to check if no other op down arrives, but this would
have to be done before allocating list elements. Should be possible with
timers and extending the device structure to hold these elements... I'll
think about it.

Cheers, Stefan

Re: Patch: Device operative state notification against 2.5.7

[jamal]

Stefan,

On Mon, 1 Apr 2002, Stefan Rompf wrote:

> Hi Jamal,
>
> first thanks for your long reply. I've moved the last part of your mail
> to the front to answer on it first.
>
> > If we were to follow the BSD or CISCOish view of the world, then
> > removing a cable or death of allocated virtual circuit/tunnel would
> > mean !IFF_RUNNING.
>
> You've caught me, a lot of my networking experience involved cisco stuff
> ;-) Anyway, I thought the semantic decision has already been made during
> the development cycle from 2.2 to 2.4. In 2.4, IFF_RUNNING of the
> in-kernel dev->flags it totally unused except for broken drivers. What
> you see in fact as IFF_RUNNING when querying an interface from
> userspace, is the LINK_STATE_NOCARRIER bit. Look at
> net/core/dev.c::dev_ifsioc(), case SIOCGIFFLAGS or the rtnetlink
> equivalent. Consequently, dev_change_flags() does not touch the
> IFF_RUNNING flag (part of IFF_VOLATILE).

Ok, I didnt know of this decision. In that case get rid of any calls
directly modifying IFF_RUNNING in the drivers and have them make calls to
netcarrier_*. LINK_STATE_NOCARRIER is also good enough to provide
synchronization/serialization issue i mentioned earlier

>
> > NO_CARRIER | IFF_RUNNING | meaning
> > -----------|-------------|------------------------------------
> > !set       | !set        | There is carrier, but no cable
> >            |             | no sense for ethernet; but may be useful
> >            |             | for PPP (for example line protocol is not up)
>
> If LPCP in a PPP interface is not open and the interface is not waiting
> for dial on demand, I'd consider it admin down, !IFF_UP.

I agree.
Lets forget this now that i know there was a call to use
LINK_STATE_NOCARRIER to reflect IFF_RUNNING.

> May be the term
> "carrier" in the flag and functions is simply misnamed and we should
> have a three states instead. If the interface is admin up, it's
> operational state can be:
>
> -up
>  The driver is sure that it can transmit and receive packets over the
> line. Depending on the driver, that can be HDLC framing, HDLC
> keepalives, ethernet carrier, generically said some kind of heartbeat.
> This state does not include any protocols that might be used over the
> line, just that layer 1 is up -> IFF_RUNNING
> -down
>  The driver cannot transmit packets. This may be the inverse of one of
> the conditions above, but can also something like a stalled ethernet
> tranceiver (adding a fourth internal state) -> !IFF_RUNNING
> -unknown
>  Some dial on demand interface that is currently not bound to a physical
> or logical line, mostly your truth table entry quoted above. May be
> these interfaces should simply have an IFF_UNBOUND flag that is set in
> this mode

I think we should use SNMPs view of the world. RFC2863 still hasnt
been obsoleted, AFAIK.
Look at sections 3.1.12-14. There is a correlation between
admin/IFF_UP and oper/IFF_RUNNING.

> > Think of either a bonding, VLANs
>
> For these, operational state should follow the underlying interface(s).
>
> > or extreme case where PPP
> > circuit is up for one protocol but not for another.
>
> That's a completely different case. If negotiation for a specific
> protocol fails, the interface is still admin & operational up, but
> cannot have a valid address for the protocol in question. In this case,
> it is the users' decision if it makes sense to keep the interface admin
> up.
>

RFC 2863 covers these overlays well actually, look at it and then lets
review again.

> > > Is it possible to send (netlink) messages from a softirq?
> >
> > Why not?
>
> You are right, but my thread actually calls netdev_state_change(), and
> the event devinet.c::inetdev_event() requires holding the RTNL
> semaphore. As I want in kernel notification, so that f.e. the VLAN
> driver can react on state changes of underlying devices, some kind of
> process context is needed.

Not sure if this is true. But like i said no big deal right now ..

>
> > In regards to walking the whole dev list:
> > There is no point in polling when there is no work to be done (and
> > therefore no need for you to walk all the devices which have not shown
> > interest). A simple "registration" by devices raising or lowering carrier
> > would be the best way to do it (via netif_carrier_on/ff). The first
> > device on the list also wakes up the thread (or as you do right now
> > always wakes it up).
>
> I'll have a look if I can implement such a list. There are three reasons
> why I haven't done it in the current version:
>
> -netif_carrier_on()/_off() seems to be designed to be called from
> anywhere, interrupts, timers, processes, so I didn't want to add too
> much locking and memory allocation in this place. Also I have to admit
> that I'm not yet long enough in kernel programming to know which kind of
> memory I may safely allocate for list elements from an interrupt ;-)

Which is why it is a good thing. You can serialize..

> -When the thread goes through this list of events, it has either to
> check for every element if the device still exists (currently only
> possible by traversing the device list so that the thread would actually
> get _less_ efficient) or the event list has to be consulted on interface
> removal

Checking device still exists has to be done always.

> -A runaway driver must be kept from allocating *lots* of kernel memory
> by calling netif_carrier_on()/_off() in a tight loop
>
> And yes, I was lazy and didn't want to overoptimize stuff that lies
> dormant nearly all the time ;-)

You might have misunderstood me. Lets defer this to later, first step is
semantics
A first step could be to get rid of the timer you have. Only wake up the
thread when necessary.
You can then look at the NAPI code for an example of registration
and the concept of not going through the whole list.

cheers,
jamal

Re: Patch: Device operative state notification against 2.5.7

[David Brownell]

Two comments on this stuff:

(A) I skimmed the 3.1.13 descriptions in RFC 2863.
If RUNNING is to correspond to ifAdminStatus,
and NO_CARRIER is inverted up/down ifOperStatus,
this starts to make sense.  For example, it defines
the set/set state in Jamal's table as a fault condition.

That doesn't quite seem like a complete match though,
and not only because one can't map N-ary ifOperStatus
(up, down, unknown, testing, dormant, notPresent, and
maybe more) to single bits like NO_CARRIER.


(B) In Jamal's table I'm thinking about how IP-over-USB
stacks would work.  Those tend to be point-to-point
links with Ethernet framing (easier to bridge :).

The devices themselves are all hotplugged, so their
interface names won't exist unless there's hardware
(maybe it's in the process of being unplugged).  But
that means there are three meaningful modes:

    - Only "my" end connected ... nobody on the
      other end (treated as NO_CARRIER) so
      IFF_RUNNING can't _ever_ usefully be set

    - Both ends connected (!NO_CARRIER)
      so IFF_RUNNING could be set or not.

    - "Indeterminate" ... some cables might not be
      able to report whether someone's on the other
      end.  Driver would necessarily treat as if both
      ends were always connected.

So eventually some linkwatch patch could be solving
a problem there:  no network hotplug events appear
for "carrier on" and "carrier off", which are the only
events that really matter here (not "register device" 
and "unregister device") for ifup/ifdown/... calls.

- Dave


> NO_CARRIER | IFF_RUNNING | meaning
> -----------|-------------|------------------------------------
> !set       | !set        | There is carrier, but no cable
>            |             | no sense for ethernet; but may be useful
>            |             | for PPP (for example line protocol is not up)
> -----------|-------------|----------------------------------------
> !set       | set         | operational up
>            |             |
>            |             |
> -----------|-------------|--------------------------------------
> set        | !set        | operational down
>            |             |
>            |             |
> -----------|-------------|--------------------------------------
> set        | set         | carrier off, cable on; not sure what this
>            |             | means (may make sense for host scope)
>            |             |
> -----------|-------------|--------------------------------------

Re: Patch: Device operative state notification against 2.5.7

[jamal]

On Mon, 1 Apr 2002, David Brownell wrote:

> Two comments on this stuff:
>
> (A) I skimmed the 3.1.13 descriptions in RFC 2863.
> If RUNNING is to correspond to ifAdminStatus,
> and NO_CARRIER is inverted up/down ifOperStatus,
> this starts to make sense.  For example, it defines
> the set/set state in Jamal's table as a fault condition.

Actually ifOperStatus == NO_CARRIER/IFF_RUNNING set
NO_CARRIER is used to reflect IFF_RUNNING in serialized way.
ifAdminStatus == IFF_UP
Someone needs to enumarate a different truth table with IFF_UP etc
based on the RFC.

>
> That doesn't quite seem like a complete match though,
> and not only because one can't map N-ary ifOperStatus
> (up, down, unknown, testing, dormant, notPresent, and
> maybe more) to single bits like NO_CARRIER.
>

I think we need three bits now to represent IFF_RUNNING (as opposed to the
old 1 bit); we also need two bits for IFF_UP; note both are defined as
integeers. My only concern is how to map NO_CARRIER now that we have
this.


> (B) In Jamal's table I'm thinking about how IP-over-USB
> stacks would work.  Those tend to be point-to-point
> links with Ethernet framing (easier to bridge :).

That table is now inappropriate; we need a new one to map
IFF_UP and the two bits in IFF_RUNNING.

>
> The devices themselves are all hotplugged, so their
> interface names won't exist unless there's hardware
> (maybe it's in the process of being unplugged).  But
> that means there are three meaningful modes:
>
>     - Only "my" end connected ... nobody on the
>       other end (treated as NO_CARRIER) so
>       IFF_RUNNING can't _ever_ usefully be set
>

	!IFF_UP and IFF_RUNNING == dormant

but once USB negotiation starts IFF_RUNNING  should transition to
notPresent
BTW, i think that IFF_RUNNING == notPresent should also apply for
ethernet where theres MII autonegotiation failures (speed mismatch)

>     - Both ends connected (!NO_CARRIER)
>       so IFF_RUNNING could be set or not.
>

        !IFF_UP and IFF_RUNNING == up

>     - "Indeterminate" ... some cables might not be
>       able to report whether someone's on the other
>       end.  Driver would necessarily treat as if both
>       ends were always connected.

I guess one of those other oper states would represent things
Admin status could be up/down.

IFF_UP goes to up only if an IP address has been assigned to
the USB interface for example via DHCP etc
I am kind of brain dead right now, but i am sure we can
map hotplug devices easily

cheers,
jamal

Re: Patch: Device operative state notification against 2.5.7

[jamal]

Stefan,

On Thu, 4 Apr 2002, Stefan Rompf wrote:

> Hi Jamal,
>
> > I think we should use SNMPs view of the world. RFC2863 still hasnt
> > been obsoleted, AFAIK.
> > Look at sections 3.1.12-14. There is a correlation between
> > admin/IFF_UP and oper/IFF_RUNNING.
>
> Agreed. If we want to be that specific, I'd prefer to go even further
> than the SNMP specification:
>
> -IFOP_UP
>  Interface is operational up.
> -IFOP_TESTING
>  Interface is just being tested
> -IFOP_UNKNOWN
>  Driver is not capable of testing the interface status
> -IFOP_DORMANT
>  Interface is sleeping, lower layer status may be unknown.

If i understand you correctly, this would still be something along the
lines of dial on demand PPP. It stays "dormant" until it receives outgoing
packet which initiates connection establishment activity. The type of
activity may vary ...
Cant remember what diald does, but if i am not mistaken it actually kills
the device and creates a new one everytime it detects activity

> -IFOP_DOWN_NOTPRESENT
>  Interface does not exist. This can be f.e. an USB device that has just
> been unplugged

What do you plan to do with this? eg, the name and stats for example wont
change. What do the USB people think?

> -IFOP_DOWN_LOWERLAYER
>  Lower layer is down, f.e. ethernet interfaces below a VLAN
> -IFOP_DOWN_NOCARRIER
>  Interface link beat/framing is missing (current netif_carrier_o* maps
> here)

Good naming convention for the above two.

> -IFOP_DOWN_KEEPALIVE
>  Another form of keepalive is missing

Not clear on this one.

> -IFOP_DOWN_TX_TIMEOUT
>  Transmit timeout signalled by scheduler

I take it you want to use this for informational purposes
i.e someone has to solicit for this information
(Note, it is legal to have transmit path timeout but receive
path just fine)

> Multiple down reasons can be active for an interface.
>

this could be tricky mostly because some of the DOWN values are
actually different states (in the state machine)
eg IFOP_DOWN_NOTPRESENT --> IFOP_DOWN_NOCARRIER when the USB hardware
is inserted
Also the only time IFOP_DOWN_LOWERLAYER makes sense is when the top
stacked device is actually admin up (but none of the lower devices are
operationally UP)

Maybe you mean there should be something generally refered to as IFOP_DOWN
and the reason would be one of IFOP_DOWN_* ?

> These states can be represented by atomic data types in the netdevice
> structure so that they can safely be set and queried from everywhere
> within the kernel. If IFOP_UP, IFOP_UNKNOWN or IFOP_DORMANT is set,
> IFF_RUNNING will be visible. If course it must be possible to get the
> detailed states from user mode and may be write to IFOP_DORMANT.
>

The only useful state transitions are to/from IFOP_DOWN_NOCARRIER
in my opinion. What do you think?
In which case you only set IFF_RUNNING when you transition from
IFOP_DOWN_NOCARRIER -> IFOP_UP and you clear it when
IFOP_UP -> IFOP_DOWN_NOCARRIER
Things like IFOP_DORMANT can probably be set only by the driver
and still retrieved via netlink or even ioctl when they are solicited
for. By default you only announce via netlink transitions to/from
IFOP_DOWN_NOCARRIER.

> Userspace notification via netlink should remain optional. Does it make
> sense to queue up events or is it enough to report the last state after
> a bunch of changes has happened? For me, the second would be ok,
> especially as all drivers that can do link beat detection now utilise a
> timer for this purpose and will lose too rapid changes anyway.

I think the transitions to/from IFOP_DOWN_NOCARRIER would be useful
for in kernel FIB or ARP updates as well; so maybe making these two
optional would not be a wise decision. And if you send within the kernel
then by default user space also gets it.

To avoid oscillations you only send the latest request _if it is still
valid_, otherwise dont send any. Your code structuring should be built
for this.
Also (this might be tricky to do), once you start flapping
like that, it would also make sense to ratelimit (in exponential manner)
the amount of netlink messages you send to announce carrier on/off.

One other thing you didnt talk about is the relation ship with IFF_UP;
essentially, !IFF_UP would also mean !IFF_RUNNING and generally
one of the DOWN states (NOCARRIER?).
when someone ifconfigs up, that should also invoke the device operational
discovery activity (eg via device open()), so no need to worry there.

Apologies for making life difficult Stefan; we need to get it right and we
have that opportunity ;->

cheers,
jamal

Re: Patch: Device operative state notification against 2.5.7

[jamal]

On Sun, 7 Apr 2002, jamal wrote:

> The only useful state transitions are to/from IFOP_DOWN_NOCARRIER
> in my opinion. What do you think?

I meant this view from a "listener" perspective; example from an
SNMP NMS perspective or even a dynamic route daemon, this is the only
really interesting state transition. So IMO, it only makes sense to send
netlink messages for these.

cheers,
jamal

Re: Patch: Device operative state notification against 2.5.7

[Michael Richardson]

-----BEGIN PGP SIGNED MESSAGE-----


>>>>> "jamal" == jamal  <hadi@cyberus.ca> writes:
    >> -IFOP_DOWN_LOWERLAYER
    >> Lower layer is down, f.e. ethernet interfaces below a VLAN
    >> -IFOP_DOWN_NOCARRIER
    >> Interface link beat/framing is missing (current netif_carrier_o* maps
    >> here)

  Are there docs on these interfaces somewhere?

    jamal> Good naming convention for the above two.

    >> -IFOP_DOWN_KEEPALIVE
    >> Another form of keepalive is missing

    jamal> Not clear on this one.

  PPP, for instance, does keepalives. 
  An IPsec tunnel might also use such a mechanism (we have plans).

    jamal> Also the only time IFOP_DOWN_LOWERLAYER makes sense is when the top
    jamal> stacked device is actually admin up (but none of the lower devices are
    jamal> operationally UP)

  There are very good operational reasons why this is often done...
  It would be nice to see this kind of stuff in diagnostics. 

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another NetBSD/notebook using, kernel hacking, security guy");  [

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3ia
Charset: latin1
Comment: Finger me for keys

iQCVAwUBPLDD9YqHRg3pndX9AQGvhwP9HYlhJ1BJ5uJUDkRjcbLuaYLkwFrNWX+8
X4hvPb1mOgwQOPs3Fwz40Ng7FEHk4GECwSbRQlccEoSbPU1BLRmUESywN8xmAMyJ
uGVqRt9UOzNzTtuvxOdpxk4LkNxZt06Flbs3DAvwv7G4d4W5Ciovu7I/JEjcnbCI
qAcy0HuyDqI=
=EQ18
-----END PGP SIGNATURE-----

Re: Patch: Device operative state notification against 2.5.7

[jamal]

On Sun, 7 Apr 2002, Michael Richardson wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
>
>
> >>>>> "jamal" == jamal  <hadi@cyberus.ca> writes:
>     >> -IFOP_DOWN_LOWERLAYER
>     >> Lower layer is down, f.e. ethernet interfaces below a VLAN
>     >> -IFOP_DOWN_NOCARRIER
>     >> Interface link beat/framing is missing (current netif_carrier_o* maps
>     >> here)
>
>   Are there docs on these interfaces somewhere?
>

I guess this discussion will result in some documentation...

>     jamal> Good naming convention for the above two.
>
>     >> -IFOP_DOWN_KEEPALIVE
>     >> Another form of keepalive is missing
>
>     jamal> Not clear on this one.
>
>   PPP, for instance, does keepalives.
>   An IPsec tunnel might also use such a mechanism (we have plans).
>

Well, in that case the device is actually IFOP_UP. And the keepalives
could be looked at as "carrier/link level diagnostics". Should the
heartbeats disappear, it would be fair to transition the device to
IFOP_DOWN_NOCARRIER. Should the device underneath have its carrier
dissapear (eg a cable removed on an ethernet which ipsec0 uses), then
the transition is very quickly to IFOP_DOWN_LOWERLAYER.... I guess the
device(ipsec0) may also transition to IFOP_DOWN_NOCARRIER since its
carrier path is doewn etc.

>     jamal> Also the only time IFOP_DOWN_LOWERLAYER makes sense is when the top
>     jamal> stacked device is actually admin up (but none of the lower devices are
>     jamal> operationally UP)
>
>   There are very good operational reasons why this is often done...
>   It would be nice to see this kind of stuff in diagnostics.

I think they should be accessible; whether you send netlink
advertisements everytime there is some state transition is the question.
In my opinion the only interesting things are the a transition from a case
where a carrier (ethernet cable) is connected to where it is disconnected.
The other direction is also important. For example, i think these are the
only two route daemons need to know about.

cheers,
jamal

Re: Patch: Device operative state notification against 2.5.7

[David Brownell]

> > The only useful state transitions are to/from IFOP_DOWN_NOCARRIER
> > in my opinion. What do you think?
> 
> I meant this view from a "listener" perspective; example from an
> SNMP NMS perspective or even a dynamic route daemon, this is the only
> really interesting state transition. So IMO, it only makes sense to send
> netlink messages for these.

But ... how about "carrier on"?  What else would be able to trigger
software to bring the interface up (so it could be routed or bridged)
if there were no notification for "carrier on"?  Not expecting some
administrator to be initating it (or eyeballing some status display);
one can ignore alerts, but one can't create them from nothing.

This is where my "asymmetry in design" alert triggers.  It's usually
been a good early warning system for trouble ... :)

- Dave

Re: Patch: Device operative state notification against 2.5.7

[Stefan Rompf]

Hi Jamal,

> [discussion whether IFOP_DOWN_LOWERLAYER should send netlink messages]
> 
> I think they should be accessible; whether you send netlink
> advertisements everytime there is some state transition is the question.

they should. For a routing daemon it is interesting to be notified when
interfaces becomes capable or incapable of handling packets, independant
of the reason. As long as we abstract virtual or subinterfaces as normal
interfaces to the upper layers (which is IMHO the correct decision),
IFOP_DOWN_LOWERLAYER must trigger a notification.

Stefan

Re: Patch: Device operative state notification against 2.5.7

[jamal]

On Tue, 9 Apr 2002, Stefan Rompf wrote:

> Hi Jamal,
>
> > [discussion whether IFOP_DOWN_LOWERLAYER should send netlink messages]
> >
> > I think they should be accessible; whether you send netlink
> > advertisements everytime there is some state transition is the question.
>
> they should. For a routing daemon it is interesting to be notified when
> interfaces becomes capable or incapable of handling packets, independant
> of the reason. As long as we abstract virtual or subinterfaces as normal
> interfaces to the upper layers (which is IMHO the correct decision),
> IFOP_DOWN_LOWERLAYER must trigger a notification.

Well ... ok. Note, the user space program could maintain state
and observe when all the interfaces are down to reach the same conclusion.

Ok, Stefan, i think you should be ready to roll some code now ;->

cheers,
jamal

Re: Patch: Device operative state notification against 2.5.7

[Stefan Rompf]

Hi Jamal,

> Ok, Stefan, i think you should be ready to roll some code now ;->

indeed. It will take a couple of days as I'm not motivated to sit
another four hours before the computer every evening ;-)

Cheers, Stefan