dropwatch for network drop monitor

dropwatch for network drop monitor

[Oliver Hartkopp]

Hi Neal,

i tried the dropwatch app from your git.

Compiling the source produces:

hartko@vwagwolkf320:~/dropwatch/src$ make
gcc -c -g main.c
gcc -c -g lookup.c
gcc -c -g lookup_bfd.c
gcc -c -g lookup_kas.c
lookup_kas.c: In function ‘lookup_kas_cache’:
lookup_kas.c:56: warning: cast from pointer to integer of different size
lookup_kas.c: In function ‘lookup_kas_proc’:
lookup_kas.c:100: warning: cast from pointer to integer of different size
lookup_kas.c:103: warning: cast from pointer to integer of different size
gcc -g -o dropwatch main.o lookup.o lookup_bfd.o lookup_kas.o -lbfd -lreadline -lnl 

You cast a 'pc' variable to __u64 - is your code depending on 64 bit ??

Starting the stuff brings:

vwagwolkf320:/home/hartko/dropwatch/src# ./dropwatch 
Initalizing kallsyms db
dropwatch> start
Enabling monitoring...
Waiting for activation ack....
Waiting for activation ack....
Waiting for activation ack....
Waiting for activation ack....
Waiting for activation ack....
Waiting for activation ack....
Failed activation request, error: Operation now in progress
Shutting down ...
vwagwolkf320:/home/hartko/dropwatch/src# 

And in my kernel log i get at the same time:
[ 7070.568482] netlink: 60 bytes leftover after parsing attributes.

Any ideas?

Oliver

Re: dropwatch for network drop monitor

[Neil Horman]

On Fri, Apr 17, 2009 at 02:57:24PM +0200, Oliver Hartkopp wrote:
> Hi Neal,
> 
> i tried the dropwatch app from your git.
> 
> Compiling the source produces:
> 
> hartko@vwagwolkf320:~/dropwatch/src$ make
> gcc -c -g main.c
> gcc -c -g lookup.c
> gcc -c -g lookup_bfd.c
> gcc -c -g lookup_kas.c
> lookup_kas.c: In function ‘lookup_kas_cache’:
> lookup_kas.c:56: warning: cast from pointer to integer of different size
> lookup_kas.c: In function ‘lookup_kas_proc’:
> lookup_kas.c:100: warning: cast from pointer to integer of different size
> lookup_kas.c:103: warning: cast from pointer to integer of different size
> gcc -g -o dropwatch main.o lookup.o lookup_bfd.o lookup_kas.o -lbfd -lreadline -lnl 
> 
Strange, I don't get this, are you building on 32 bit or 64 bit?  Although,
that shouldn't cause you any problems at the moment, since that code is still
not actually used.

> You cast a 'pc' variable to __u64 - is your code depending on 64 bit ??
> 
Not depending on, but I wanted to provide room in that data field to hold a void
pointer, so I cas't it to a __u64

> Starting the stuff brings:
> 
> vwagwolkf320:/home/hartko/dropwatch/src# ./dropwatch 
> Initalizing kallsyms db
> dropwatch> start
> Enabling monitoring...
> Waiting for activation ack....
> Waiting for activation ack....
> Waiting for activation ack....
> Waiting for activation ack....
> Waiting for activation ack....
> Waiting for activation ack....
> Failed activation request, error: Operation now in progress
> Shutting down ...
> vwagwolkf320:/home/hartko/dropwatch/src# 
> 
> And in my kernel log i get at the same time:
> [ 7070.568482] netlink: 60 bytes leftover after parsing attributes.
> 
> Any ideas?
> 
> Oliver
> 
> 
Not off the top of my head, I've run this on 64 bit and 32 bit and not seen
these errors.  I'll bring it back up on my 32 bit system and see what comes of
it.  Do me a favor and open a bug at the trac page for this.  Thanks!
Neil

Re: dropwatch for network drop monitor

[Oliver Hartkopp]

Hi Neil,

>> Initalizing kallsyms db
>> dropwatch> start
>> Enabling monitoring...
>> Waiting for activation ack....
>> Waiting for activation ack....
>> Waiting for activation ack....
>> Waiting for activation ack....
>> Waiting for activation ack....
>> Waiting for activation ack....
>> Failed activation request, error: Operation now in progress
>> Shutting down ...
>> vwagwolkf320:/home/hartko/dropwatch/src# 

Indeed i only get this the second time.

The first time i get

hartko@vwagwolkf320:~/dropwatch/src$ ./dropwatch
Initalizing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring

But Ctrl-C does not work. I only get 100% CPU load on the CPU executing dropwatch.

After killing the process, i get the above originally described behaviour.


>>
>> And in my kernel log i get at the same time:
>> [ 7070.568482] netlink: 60 bytes leftover after parsing attributes.

And this comes every time, i invoke dropwatch.

I'm running a debian system here on a Dell 830 Laptop with Core2Duo and the
latest linux-2.6 git

Regards,
Oliver

ps. Do i need to be root to run dropwatch?

Re: dropwatch for network drop monitor

[Neil Horman]

On Fri, Apr 17, 2009 at 04:16:10PM +0200, Oliver Hartkopp wrote:
> Hi Neil,
> 
> >> Initalizing kallsyms db
> >> dropwatch> start
> >> Enabling monitoring...
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Failed activation request, error: Operation now in progress
> >> Shutting down ...
> >> vwagwolkf320:/home/hartko/dropwatch/src# 
> 
> Indeed i only get this the second time.
> 
> The first time i get
> 
> hartko@vwagwolkf320:~/dropwatch/src$ ./dropwatch
> Initalizing kallsyms db
> dropwatch> start
> Enabling monitoring...
> Kernel monitoring activated.
> Issue Ctrl-C to stop monitoring
> 
> But Ctrl-C does not work. I only get 100% CPU load on the CPU executing dropwatch.
> 
> After killing the process, i get the above originally described behaviour.
> 
Thats interesting, can't say I've ever seen that behavior before.  I'm building
the latest net-next kernel on my s32 bit system right now, I'll let you know
what I come up with.

> 
> >>
> >> And in my kernel log i get at the same time:
> >> [ 7070.568482] netlink: 60 bytes leftover after parsing attributes.
> 
> And this comes every time, i invoke dropwatch.
> 
I'll watch for this.

> I'm running a debian system here on a Dell 830 Laptop with Core2Duo and the
> latest linux-2.6 git
> 
Thank you

> Regards,
> Oliver
> 
> ps. Do i need to be root to run dropwatch?
> 
IIRC, at the moment you do yes, since NETLINK_GENERIC family lookups are
restricted to root access only.  I hope to change that one day

Neil

> 

Re: dropwatch for network drop monitor

[Neil Horman]

On Fri, Apr 17, 2009 at 04:16:10PM +0200, Oliver Hartkopp wrote:
> Hi Neil,
> 
> >> Initalizing kallsyms db
> >> dropwatch> start
> >> Enabling monitoring...
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Waiting for activation ack....
> >> Failed activation request, error: Operation now in progress
> >> Shutting down ...
> >> vwagwolkf320:/home/hartko/dropwatch/src# 
> 
> Indeed i only get this the second time.
> 
> The first time i get
> 
> hartko@vwagwolkf320:~/dropwatch/src$ ./dropwatch
> Initalizing kallsyms db
> dropwatch> start
> Enabling monitoring...
> Kernel monitoring activated.
> Issue Ctrl-C to stop monitoring
> 
> But Ctrl-C does not work. I only get 100% CPU load on the CPU executing dropwatch.
> 
> After killing the process, i get the above originally described behaviour.
> 
> 
> >>
> >> And in my kernel log i get at the same time:
> >> [ 7070.568482] netlink: 60 bytes leftover after parsing attributes.
> 
> And this comes every time, i invoke dropwatch.
> 
> I'm running a debian system here on a Dell 830 Laptop with Core2Duo and the
> latest linux-2.6 git
> 
> Regards,
> Oliver
> 
> ps. Do i need to be root to run dropwatch?
> 
> 

Quick update for you, I've reproduced the problem.  The inability to restart
after the first few errors is actually by design.  Since you had to kill the
user space process, it left the kernel bits in a state where they were still
monitoring, so a second start command (when you restarted the user space daemon)
caused an error, and the program shutdown (which I probably shouldn't do,
returning instead to the command console).  You can work around this by issuing
a stop command, which successfully stops the kernel bits and lets you restart
them.

This also tells me that the kernel netlink warning is not fatal (as our netlink
message commands are getting processed properly).  I still need to figure out
why thats happening though.

As for the ctrl-c issue, I've got no idea whats going on there, its like the
recv operation is restarting after we catch the SIGINT.  I think I may just
switch that to a poll operation on the netlink socket and stdin input.  That
should fix that up quick.  Expect a git commit on that sometime this weekend,
and some subseqent netdev posts to fix the kernel bits. 

Thanks!
Neil

Re: dropwatch for network drop monitor

[Oliver Hartkopp]

Neil Horman wrote:

> 
> Quick update for you, I've reproduced the problem.  The inability to restart
> after the first few errors is actually by design.  Since you had to kill the
> user space process, it left the kernel bits in a state where they were still
> monitoring, so a second start command (when you restarted the user space daemon)
> caused an error, and the program shutdown (which I probably shouldn't do,
> returning instead to the command console).  You can work around this by issuing
> a stop command, which successfully stops the kernel bits and lets you restart
> them.

Hard terminations are mostly critical to handle. I'll try the workaround.

> 
> This also tells me that the kernel netlink warning is not fatal (as our netlink
> message commands are getting processed properly).  I still need to figure out
> why thats happening though.
> 

Ok.

> As for the ctrl-c issue, I've got no idea whats going on there, its like the
> recv operation is restarting after we catch the SIGINT.  I think I may just
> switch that to a poll operation on the netlink socket and stdin input.  That
> should fix that up quick.  Expect a git commit on that sometime this weekend,
> and some subseqent netdev posts to fix the kernel bits. 

You can also send me an off-list mail when you have something to test this
weekend.

I can produce a lot of drops on my machine ;-)

Regards,
Oliver