Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Stephen Hemminger]

Begin forwarded message:

Date: Wed, 20 Nov 2013 05:57:59 -0800
From: "bugzilla-daemon@bugzilla.kernel.org" <bugzilla-daemon@bugzilla.kernel.org>
To: "stephen@networkplumber.org" <stephen@networkplumber.org>
Subject: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds


https://bugzilla.kernel.org/show_bug.cgi?id=65261

            Bug ID: 65261
           Summary: Packet loss or excessive packet delay repeatedly for
                    some seconds
           Product: Networking
           Version: 2.5
    Kernel Version: 3.10.19
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: IPV4
          Assignee: shemminger@linux-foundation.org
          Reporter: arno@wagner.name
        Regression: No

Created attachment 115261
  --> https://bugzilla.kernel.org/attachment.cgi?id=115261&action=edit
Kernel config 3.10.19

I recently upgraded my development-server/firewall/NAT-box from 3.10.17 to
3.10.19. Since then I noticed increased DNS lookup failures on a connected
Windows box and occasional slow updates on putty-SSH logins when scrolling in
an editor (joe, takes something like an estimated 100-300ms for screen
updates). These update delays are repeatable for something like 10-20 seconds
or more, e.g. inserting a line and then deleting again, then vanish. The delays
make remote editing hard to do when they happen. The DNS lookup failures are
really annoying. 

I have not found a way to reliably trigger the problem.

Going back to 3.10.17 fixed the issue as far as I can tell. (Several hours
editing source code without it showing up.) The only change between the two
configurations was that I added the scsi CDROM driver (which should not be able
to cause this?).

The network connection has an iptables "all pass" on the server side for the
affected connection. The client side is a Win7 machine. The network link is GbE
with 2 Switches in there. No packet loss on ping/ping -f. Network hardware is 
Intel 82574L Gigabit card on both sides.

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Hannes Frederic Sowa]

On Wed, Nov 20, 2013 at 12:23:16PM -0800, Stephen Hemminger wrote:
> 
> 
> Begin forwarded message:
> 
> Date: Wed, 20 Nov 2013 05:57:59 -0800
> From: "bugzilla-daemon@bugzilla.kernel.org" <bugzilla-daemon@bugzilla.kernel.org>
> To: "stephen@networkplumber.org" <stephen@networkplumber.org>
> Subject: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds
> 
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=65261
> 
>             Bug ID: 65261
>            Summary: Packet loss or excessive packet delay repeatedly for
>                     some seconds
>            Product: Networking
>            Version: 2.5
>     Kernel Version: 3.10.19
>           Hardware: x86-64
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: high
>           Priority: P1
>          Component: IPV4
>           Assignee: shemminger@linux-foundation.org
>           Reporter: arno@wagner.name
>         Regression: No
> 
> Created attachment 115261
>   --> https://bugzilla.kernel.org/attachment.cgi?id=115261&action=edit
> Kernel config 3.10.19
> 
> I recently upgraded my development-server/firewall/NAT-box from 3.10.17 to
> 3.10.19. Since then I noticed increased DNS lookup failures on a connected
> Windows box and occasional slow updates on putty-SSH logins when scrolling in
> an editor (joe, takes something like an estimated 100-300ms for screen
> updates). These update delays are repeatable for something like 10-20 seconds
> or more, e.g. inserting a line and then deleting again, then vanish. The delays
> make remote editing hard to do when they happen. The DNS lookup failures are
> really annoying. 
> 
> I have not found a way to reliably trigger the problem.
> 
> Going back to 3.10.17 fixed the issue as far as I can tell. (Several hours
> editing source code without it showing up.) The only change between the two
> configurations was that I added the scsi CDROM driver (which should not be able
> to cause this?).
> 
> The network connection has an iptables "all pass" on the server side for the
> affected connection. The client side is a Win7 machine. The network link is GbE
> with 2 Switches in there. No packet loss on ping/ping -f. Network hardware is 
> Intel 82574L Gigabit card on both sides.

Could you try dropwatch while the packet drops are happening?
Often it is helpful to check ip monitor all if events happen during bursty
losses.

Greetings,

  Hannes

Re: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Arno Wagner]

On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> On Wed, Nov 20, 2013 at 12:23:16PM -0800, Stephen Hemminger wrote:
> > 
> > 
> > Begin forwarded message:
> > 
> > Date: Wed, 20 Nov 2013 05:57:59 -0800
> > From: "bugzilla-daemon@bugzilla.kernel.org" <bugzilla-daemon@bugzilla.kernel.org>
> > To: "stephen@networkplumber.org" <stephen@networkplumber.org>
> > Subject: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds
> > 
> > 
> > https://bugzilla.kernel.org/show_bug.cgi?id=65261
> > 
> >             Bug ID: 65261
> >            Summary: Packet loss or excessive packet delay repeatedly for
> >                     some seconds
> >            Product: Networking
> >            Version: 2.5
> >     Kernel Version: 3.10.19
> >           Hardware: x86-64
> >                 OS: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: high
> >           Priority: P1
> >          Component: IPV4
> >           Assignee: shemminger@linux-foundation.org
> >           Reporter: arno@wagner.name
> >         Regression: No
> > 
> > Created attachment 115261
> >   --> https://bugzilla.kernel.org/attachment.cgi?id=115261&action=edit
> > Kernel config 3.10.19
> > 
> > I recently upgraded my development-server/firewall/NAT-box from 3.10.17 to
> > 3.10.19. Since then I noticed increased DNS lookup failures on a connected
> > Windows box and occasional slow updates on putty-SSH logins when scrolling in
> > an editor (joe, takes something like an estimated 100-300ms for screen
> > updates). These update delays are repeatable for something like 10-20 seconds
> > or more, e.g. inserting a line and then deleting again, then vanish. The delays
> > make remote editing hard to do when they happen. The DNS lookup failures are
> > really annoying. 
> > 
> > I have not found a way to reliably trigger the problem.
> > 
> > Going back to 3.10.17 fixed the issue as far as I can tell. (Several hours
> > editing source code without it showing up.) The only change between the two
> > configurations was that I added the scsi CDROM driver (which should not be able
> > to cause this?).
> > 
> > The network connection has an iptables "all pass" on the server side for the
> > affected connection. The client side is a Win7 machine. The network link is GbE
> > with 2 Switches in there. No packet loss on ping/ping -f. Network hardware is 
> > Intel 82574L Gigabit card on both sides.
> 
> Could you try dropwatch while the packet drops are happening?
> Often it is helpful to check ip monitor all if events happen during bursty
> losses.

Sorry, but dropwatch is not in Debian and compiling it from sources
seem to require something called "rpmbuild". Hence I cannot use
it or at least would have to write my own Makefile for it. 

Arno

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
There are two ways of constructing a software design: One way is to make it
so simple that there are obviously no deficiencies, and the other way is to
make it so complicated that there are no obvious deficiencies. The first
method is far more difficult.  --Tony Hoare

Re: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
[...]
> > Could you try dropwatch while the packet drops are happening?
> > Often it is helpful to check ip monitor all if events happen during bursty
> > losses.
> 
> Sorry, but dropwatch is not in Debian and compiling it from sources
> seem to require something called "rpmbuild". Hence I cannot use
> it or at least would have to write my own Makefile for it. 

You could use 'perf script net_dropmonitor' instead.

Ben.

-- 
Ben Hutchings
The generation of random numbers is too important to be left to chance.
                                                            - Robert Coveyou

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Arno Wagner]

On Sat, Nov 23, 2013 at 17:45:58 CET, Ben Hutchings wrote:
> On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> > On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> [...]
> > > Could you try dropwatch while the packet drops are happening?
> > > Often it is helpful to check ip monitor all if events happen during bursty
> > > losses.
> > 
> > Sorry, but dropwatch is not in Debian and compiling it from sources
> > seem to require something called "rpmbuild". Hence I cannot use
> > it or at least would have to write my own Makefile for it. 
> 
> You could use 'perf script net_dropmonitor' instead.

That gets me "invalid or unsupported event: 'skb:kfree_skb'"
with 3.10.17, so I guess it will not work with 3.10.19 either.

Arno
-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
There are two ways of constructing a software design: One way is to make it
so simple that there are obviously no deficiencies, and the other way is to
make it so complicated that there are no obvious deficiencies. The first
method is far more difficult.  --Tony Hoare

Re: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1141 bytes --]

On Sat, 2013-11-23 at 21:20 +0100, Arno Wagner wrote:
> On Sat, Nov 23, 2013 at 17:45:58 CET, Ben Hutchings wrote:
> > On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> > > On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> > [...]
> > > > Could you try dropwatch while the packet drops are happening?
> > > > Often it is helpful to check ip monitor all if events happen during bursty
> > > > losses.
> > > 
> > > Sorry, but dropwatch is not in Debian and compiling it from sources
> > > seem to require something called "rpmbuild". Hence I cannot use
> > > it or at least would have to write my own Makefile for it. 
> > 
> > You could use 'perf script net_dropmonitor' instead.
> 
> That gets me "invalid or unsupported event: 'skb:kfree_skb'"
> with 3.10.17, so I guess it will not work with 3.10.19 either.

That is probably an obscure way of saying 'permission denied'; I see the
same if I don't run it as root.

Ben.

-- 
Ben Hutchings
The generation of random numbers is too important to be left to chance.
                                                            - Robert Coveyou

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Hannes Frederic Sowa]

On Sat, Nov 23, 2013 at 09:20:47PM +0100, Arno Wagner wrote:
> On Sat, Nov 23, 2013 at 17:45:58 CET, Ben Hutchings wrote:
> > On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> > > On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> > [...]
> > > > Could you try dropwatch while the packet drops are happening?
> > > > Often it is helpful to check ip monitor all if events happen during bursty
> > > > losses.
> > > 
> > > Sorry, but dropwatch is not in Debian and compiling it from sources
> > > seem to require something called "rpmbuild". Hence I cannot use
> > > it or at least would have to write my own Makefile for it. 
> > 
> > You could use 'perf script net_dropmonitor' instead.
> 
> That gets me "invalid or unsupported event: 'skb:kfree_skb'"
> with 3.10.17, so I guess it will not work with 3.10.19 either.

You need CONFIG_DROP_MONITOR enabled on your kernel build.

Re: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Arno Wagner]

On Sat, Nov 23, 2013 at 21:39:33 CET, Hannes Frederic Sowa wrote:
> On Sat, Nov 23, 2013 at 09:20:47PM +0100, Arno Wagner wrote:
> > On Sat, Nov 23, 2013 at 17:45:58 CET, Ben Hutchings wrote:
> > > On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> > > > On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> > > [...]
> > > > > Could you try dropwatch while the packet drops are happening?
> > > > > Often it is helpful to check ip monitor all if events happen during bursty
> > > > > losses.
> > > > 
> > > > Sorry, but dropwatch is not in Debian and compiling it from sources
> > > > seem to require something called "rpmbuild". Hence I cannot use
> > > > it or at least would have to write my own Makefile for it. 
> > > 
> > > You could use 'perf script net_dropmonitor' instead.
> > 
> > That gets me "invalid or unsupported event: 'skb:kfree_skb'"
> > with 3.10.17, so I guess it will not work with 3.10.19 either.
> 
> You need CONFIG_DROP_MONITOR enabled on your kernel build.

Interestingly, that option is not even there in .config. 
Must be someting that "make oldconfig" messed up along 
the way, as making a new .config with all default values 
does have it commented out as "not set".

After a bit of poking around, I found it and its pre-requisites 
net/Kconfig, where it says it depends on INET && TRACEPOINTS.
After a bit more poking I finally found TRACEPOINTS on init/Kconfig,
depending on nothing. But it does not get displayed as selectable
in "make menuconfig".  As far as I can tell, it is not constrained
and should show up? 

So I really have no clue how to turn this on...

Arno




-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
There are two ways of constructing a software design: One way is to make it
so simple that there are obviously no deficiencies, and the other way is to
make it so complicated that there are no obvious deficiencies. The first
method is far more difficult.  --Tony Hoare

Re: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Hannes Frederic Sowa]

On Sat, Nov 23, 2013 at 11:00:00PM +0100, Arno Wagner wrote:
> On Sat, Nov 23, 2013 at 21:39:33 CET, Hannes Frederic Sowa wrote:
> > On Sat, Nov 23, 2013 at 09:20:47PM +0100, Arno Wagner wrote:
> > > On Sat, Nov 23, 2013 at 17:45:58 CET, Ben Hutchings wrote:
> > > > On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> > > > > On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> > > > [...]
> > > > > > Could you try dropwatch while the packet drops are happening?
> > > > > > Often it is helpful to check ip monitor all if events happen during bursty
> > > > > > losses.
> > > > > 
> > > > > Sorry, but dropwatch is not in Debian and compiling it from sources
> > > > > seem to require something called "rpmbuild". Hence I cannot use
> > > > > it or at least would have to write my own Makefile for it. 
> > > > 
> > > > You could use 'perf script net_dropmonitor' instead.
> > > 
> > > That gets me "invalid or unsupported event: 'skb:kfree_skb'"
> > > with 3.10.17, so I guess it will not work with 3.10.19 either.
> > 
> > You need CONFIG_DROP_MONITOR enabled on your kernel build.
> 
> Interestingly, that option is not even there in .config. 
> Must be someting that "make oldconfig" messed up along 
> the way, as making a new .config with all default values 
> does have it commented out as "not set".
> 
> After a bit of poking around, I found it and its pre-requisites 
> net/Kconfig, where it says it depends on INET && TRACEPOINTS.
> After a bit more poking I finally found TRACEPOINTS on init/Kconfig,
> depending on nothing. But it does not get displayed as selectable
> in "make menuconfig".  As far as I can tell, it is not constrained
> and should show up? 
> 
> So I really have no clue how to turn this on...

Check Kernel Hacking -> Tracers. It should show up Networking options ->
Network testing.

Greetings,

  Hannes

Re2: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Arno Wagner]

On Sat, Nov 23, 2013 at 21:39:33 CET, Hannes Frederic Sowa wrote:
> On Sat, Nov 23, 2013 at 09:20:47PM +0100, Arno Wagner wrote:
> > On Sat, Nov 23, 2013 at 17:45:58 CET, Ben Hutchings wrote:
> > > On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> > > > On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> > > [...]
> > > > > Could you try dropwatch while the packet drops are happening?
> > > > > Often it is helpful to check ip monitor all if events happen during bursty
> > > > > losses.
> > > > 
> > > > Sorry, but dropwatch is not in Debian and compiling it from sources
> > > > seem to require something called "rpmbuild". Hence I cannot use
> > > > it or at least would have to write my own Makefile for it. 
> > > 
> > > You could use 'perf script net_dropmonitor' instead.
> > 
> > That gets me "invalid or unsupported event: 'skb:kfree_skb'"
> > with 3.10.17, so I guess it will not work with 3.10.19 either.
> 
> You need CONFIG_DROP_MONITOR enabled on your kernel build.

Found it. At least if it actually is "CONFIG_NET_DROP_MONITOR".

Is there any less obscure way than greping the kernel tree and
tracing the options back to find out what meny entry turns on
what option?

Arno

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
There are two ways of constructing a software design: One way is to make it
so simple that there are obviously no deficiencies, and the other way is to
make it so complicated that there are no obvious deficiencies. The first
method is far more difficult.  --Tony Hoare

Re: Re2: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Hannes Frederic Sowa]

On Sat, Nov 23, 2013 at 11:11:51PM +0100, Arno Wagner wrote:
> On Sat, Nov 23, 2013 at 21:39:33 CET, Hannes Frederic Sowa wrote:
> > On Sat, Nov 23, 2013 at 09:20:47PM +0100, Arno Wagner wrote:
> > > On Sat, Nov 23, 2013 at 17:45:58 CET, Ben Hutchings wrote:
> > > > On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> > > > > On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> > > > [...]
> > > > > > Could you try dropwatch while the packet drops are happening?
> > > > > > Often it is helpful to check ip monitor all if events happen during bursty
> > > > > > losses.
> > > > > 
> > > > > Sorry, but dropwatch is not in Debian and compiling it from sources
> > > > > seem to require something called "rpmbuild". Hence I cannot use
> > > > > it or at least would have to write my own Makefile for it. 
> > > > 
> > > > You could use 'perf script net_dropmonitor' instead.
> > > 
> > > That gets me "invalid or unsupported event: 'skb:kfree_skb'"
> > > with 3.10.17, so I guess it will not work with 3.10.19 either.
> > 
> > You need CONFIG_DROP_MONITOR enabled on your kernel build.
> 
> Found it. At least if it actually is "CONFIG_NET_DROP_MONITOR".
> 
> Is there any less obscure way than greping the kernel tree and
> tracing the options back to find out what meny entry turns on
> what option?

I normally search in menuconfig (press '/').

Re: Re2: Fw: [Bug 65261] New: Packet loss or excessive packet delay repeatedly for some seconds

[Arno Wagner]

On Sat, Nov 23, 2013 at 23:18:14 CET, Hannes Frederic Sowa wrote:
> On Sat, Nov 23, 2013 at 11:11:51PM +0100, Arno Wagner wrote:
> > On Sat, Nov 23, 2013 at 21:39:33 CET, Hannes Frederic Sowa wrote:
> > > On Sat, Nov 23, 2013 at 09:20:47PM +0100, Arno Wagner wrote:
> > > > On Sat, Nov 23, 2013 at 17:45:58 CET, Ben Hutchings wrote:
> > > > > On Sat, 2013-11-23 at 13:11 +0100, Arno Wagner wrote:
> > > > > > On Sat, Nov 23, 2013 at 05:33:59 CET, Hannes Frederic Sowa wrote:
> > > > > [...]
> > > > > > > Could you try dropwatch while the packet drops are happening?
> > > > > > > Often it is helpful to check ip monitor all if events happen during bursty
> > > > > > > losses.
> > > > > > 
> > > > > > Sorry, but dropwatch is not in Debian and compiling it from sources
> > > > > > seem to require something called "rpmbuild". Hence I cannot use
> > > > > > it or at least would have to write my own Makefile for it. 
> > > > > 
> > > > > You could use 'perf script net_dropmonitor' instead.
> > > > 
> > > > That gets me "invalid or unsupported event: 'skb:kfree_skb'"
> > > > with 3.10.17, so I guess it will not work with 3.10.19 either.
> > > 
> > > You need CONFIG_DROP_MONITOR enabled on your kernel build.
> > 
> > Found it. At least if it actually is "CONFIG_NET_DROP_MONITOR".
> > 
> > Is there any less obscure way than greping the kernel tree and
> > tracing the options back to find out what meny entry turns on
> > what option?
> 
> I normally search in menuconfig (press '/').

Thanks, did not know about that one. Looks quite useful.

Anyways, I will run a test with that option and the tool.
in the next few days.

Arno

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
There are two ways of constructing a software design: One way is to make it
so simple that there are obviously no deficiencies, and the other way is to
make it so complicated that there are no obvious deficiencies. The first
method is far more difficult.  --Tony Hoare