panic 2.6.27-rc3-git2, qdisc_dequeue

netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* panic 2.6.27-rc3-git2, qdisc_dequeue_head
@ 2008-08-15 19:09 Denys Fedoryshchenko
  2008-08-15 20:16 ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-15 19:09 UTC (permalink / raw)
  To: netdev

Aug 15 21:59:54 192.168.20.66 [ 1123.800035] BUG: unable to handle kernel
Aug 15 21:59:54 192.168.20.66 NULL pointer dereference
Aug 15 21:59:54 192.168.20.66 at 00000004
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] IP:
Aug 15 21:59:54 192.168.20.66 [<c0272579>] qdisc_dequeue_head+0x29/0x3d
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] Oops: 0002 [#1]
Aug 15 21:59:54 192.168.20.66 SMP
Aug 15 21:59:54 192.168.20.66
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] Modules linked in:
Aug 15 21:59:54 192.168.20.66 netconsole
Aug 15 21:59:54 192.168.20.66 configfs
Aug 15 21:59:54 192.168.20.66 nf_nat_pptp
Aug 15 21:59:54 192.168.20.66 nf_conntrack_pptp
Aug 15 21:59:54 192.168.20.66 nf_conntrack_proto_gre
Aug 15 21:59:54 192.168.20.66 nf_nat_proto_gre
Aug 15 21:59:54 192.168.20.66 iTCO_wdt
Aug 15 21:59:54 192.168.20.66 sch_ingress
Aug 15 21:59:54 192.168.20.66 e752x_edac
Aug 15 21:59:54 192.168.20.66 edac_core
Aug 15 21:59:54 192.168.20.66 sch_prio
Aug 15 21:59:54 192.168.20.66 cls_u32
Aug 15 21:59:54 192.168.20.66 em_meta
Aug 15 21:59:54 192.168.20.66 cls_basic
Aug 15 21:59:54 192.168.20.66 xt_dscp
Aug 15 21:59:54 192.168.20.66 xt_DSCP
Aug 15 21:59:54 192.168.20.66 ipt_REJECT
Aug 15 21:59:54 192.168.20.66 ts_bm
Aug 15 21:59:54 192.168.20.66 xt_string
Aug 15 21:59:54 192.168.20.66 ipt_ttl
Aug 15 21:59:54 192.168.20.66 ifb
Aug 15 21:59:54 192.168.20.66 cls_fw
Aug 15 21:59:54 192.168.20.66 sch_tbf
Aug 15 21:59:54 192.168.20.66 sch_htb
Aug 15 21:59:54 192.168.20.66 act_ipt
Aug 15 21:59:54 192.168.20.66 act_mirred
Aug 15 21:59:54 192.168.20.66 xt_MARK
Aug 15 21:59:54 192.168.20.66 pppoe
Aug 15 21:59:54 192.168.20.66 pppox
Aug 15 21:59:54 192.168.20.66 ppp_generic
Aug 15 21:59:54 192.168.20.66 slhc
Aug 15 21:59:54 192.168.20.66 xt_TCPMSS
Aug 15 21:59:54 192.168.20.66 xt_mark
Aug 15 21:59:54 192.168.20.66 xt_tcpudp
Aug 15 21:59:54 192.168.20.66 iptable_mangle
Aug 15 21:59:54 192.168.20.66 iptable_nat
Aug 15 21:59:54 192.168.20.66 nf_nat
Aug 15 21:59:54 192.168.20.66 nf_conntrack_ipv4
Aug 15 21:59:54 192.168.20.66 nf_conntrack
Aug 15 21:59:54 192.168.20.66 rtc_cmos
Aug 15 21:59:54 192.168.20.66 rtc_core
Aug 15 21:59:54 192.168.20.66 rtc_lib
Aug 15 21:59:54 192.168.20.66 iptable_filter
Aug 15 21:59:54 192.168.20.66 ip_tables
Aug 15 21:59:54 192.168.20.66 x_tables
Aug 15 21:59:54 192.168.20.66 8021q
Aug 15 21:59:54 192.168.20.66 garp
Aug 15 21:59:54 192.168.20.66 stp
Aug 15 21:59:54 192.168.20.66 llc
Aug 15 21:59:54 192.168.20.66 tun
Aug 15 21:59:54 192.168.20.66 loop
Aug 15 21:59:54 192.168.20.66 tulip
Aug 15 21:59:54 192.168.20.66 r8169
Aug 15 21:59:54 192.168.20.66 sky2
Aug 15 21:59:54 192.168.20.66 via_velocity
Aug 15 21:59:54 192.168.20.66 via_rhine
Aug 15 21:59:54 192.168.20.66 sis900
Aug 15 21:59:54 192.168.20.66 ne2k_pci
Aug 15 21:59:54 192.168.20.66 8390
Aug 15 21:59:54 192.168.20.66 skge
Aug 15 21:59:54 192.168.20.66 tg3
Aug 15 21:59:54 192.168.20.66 libphy
Aug 15 21:59:54 192.168.20.66 8139too
Aug 15 21:59:54 192.168.20.66 e1000
Aug 15 21:59:54 192.168.20.66 e100
Aug 15 21:59:54 192.168.20.66 usb_storage
Aug 15 21:59:54 192.168.20.66 mtdblock
Aug 15 21:59:54 192.168.20.66 mtd_blkdevs
Aug 15 21:59:54 192.168.20.66 usbhid
Aug 15 21:59:54 192.168.20.66 uhci_hcd
Aug 15 21:59:54 192.168.20.66 ehci_hcd
Aug 15 21:59:54 192.168.20.66 ohci_hcd
Aug 15 21:59:54 192.168.20.66 usbcore
Aug 15 21:59:54 192.168.20.66
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] Pid: 0, comm: swapper Not 
tainted (2.6.27-rc3-git2-build-0030 #9)
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] EIP: 0060:[<c0272579>] EFLAGS: 
00010206 CPU: 0
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] EIP is at 
qdisc_dequeue_head+0x29/0x3d
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] EAX: f2312b2c EBX: f2312b00 ECX: 
f2653800 EDX: 00000000
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] ESI: f46b2680 EDI: f1c0f6a0 EBP: 
c085ef5c ESP: c085ef58
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  DS: 007b ES: 007b FS: 00d8 GS: 
0000 SS: 0068
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] Process swapper (pid: 0, 
ti=c085e000 task=c03813b4 task.ti=c03a7000)
Aug 15 21:59:54 192.168.20.66
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] Stack:
Aug 15 21:59:54 192.168.20.66 00000000
Aug 15 21:59:54 192.168.20.66 c085ef98
Aug 15 21:59:54 192.168.20.66 f8a10539
Aug 15 21:59:54 192.168.20.66 f1c0f600
Aug 15 21:59:54 192.168.20.66 00000000
Aug 15 21:59:54 192.168.20.66 00000000
Aug 15 21:59:54 192.168.20.66 f1c0f638
Aug 15 21:59:54 192.168.20.66 f46b2680
Aug 15 21:59:54 192.168.20.66
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]
Aug 15 21:59:54 192.168.20.66 f1c0f600
Aug 15 21:59:54 192.168.20.66 c085ef98
Aug 15 21:59:54 192.168.20.66 c02bc930
Aug 15 21:59:54 192.168.20.66 00000000
Aug 15 21:59:54 192.168.20.66 00000002
Aug 15 21:59:54 192.168.20.66 00000000
Aug 15 21:59:54 192.168.20.66 f46b2680
Aug 15 21:59:54 192.168.20.66 f1c0f600
Aug 15 21:59:54 192.168.20.66
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]
Aug 15 21:59:54 192.168.20.66 c085efbc
Aug 15 21:59:54 192.168.20.66 c026e48f
Aug 15 21:59:54 192.168.20.66 000c9204
Aug 15 21:59:54 192.168.20.66 00000000
Aug 15 21:59:54 192.168.20.66 f1c0f638
Aug 15 21:59:54 192.168.20.66 f27a5000
Aug 15 21:59:54 192.168.20.66 f1c0f600
Aug 15 21:59:54 192.168.20.66 f1c0f624
Aug 15 21:59:54 192.168.20.66
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] Call Trace:
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<f8a10539>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 tbf_dequeue+0x1c/0x198 [sch_tbf]
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c02bc930>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 _spin_lock+0x22/0x2a
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c026e48f>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 __
Aug 15 21:59:54 192.168.20.66 qdisc_run+0x41/0x18b
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c025fb0b>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 net_tx_action+0xaf/0xcc
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c0126203>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 __do_softirq+0x7b/0xf4
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c0126188>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 __do_softirq+0x0/0xf4
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c01060b3>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 do_softirq+0x65/0xb6
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c0125e28>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 irq_exit+0x44/0x79
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c010ff15>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 smp_apic_timer_interrupt+0x6f/0x7d
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c010438d>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 apic_timer_interrupt+0x2d/0x34
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c013007b>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 find_get_pid+0x2e/0x4d
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c0108d8a>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 mwait_idle+0x39/0x43
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c01029ee>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 cpu_idle+0xbf/0xe1
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  [<c02afc86>]
Aug 15 21:59:54 192.168.20.66 ?
Aug 15 21:59:54 192.168.20.66 rest_init+0x4e/0x50
Aug 15 21:59:54 192.168.20.66 [ 1123.800035]  =======================
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] Code:
Aug 15 21:59:54 192.168.20.66 90
Aug 15 21:59:54 192.168.20.66 90
Aug 15 21:59:54 192.168.20.66 55
Aug 15 21:59:54 192.168.20.66 89
Aug 15 21:59:54 192.168.20.66 e5
Aug 15 21:59:54 192.168.20.66 53
Aug 15 21:59:54 192.168.20.66 89
Aug 15 21:59:54 192.168.20.66 c3
Aug 15 21:59:54 192.168.20.66 8b
Aug 15 21:59:54 192.168.20.66 4b
Aug 15 21:59:54 192.168.20.66 2c
Aug 15 21:59:54 192.168.20.66 8d
Aug 15 21:59:54 192.168.20.66 40
Aug 15 21:59:54 192.168.20.66 2c
Aug 15 21:59:54 192.168.20.66 39
Aug 15 21:59:54 192.168.20.66 c1
Aug 15 21:59:54 192.168.20.66 74
Aug 15 21:59:54 192.168.20.66 26
Aug 15 21:59:54 192.168.20.66 85
Aug 15 21:59:54 192.168.20.66 c9
Aug 15 21:59:54 192.168.20.66 74
Aug 15 21:59:54 192.168.20.66 24
Aug 15 21:59:54 192.168.20.66 ff
Aug 15 21:59:54 192.168.20.66 4b
Aug 15 21:59:54 192.168.20.66 34
Aug 15 21:59:54 192.168.20.66 8b
Aug 15 21:59:54 192.168.20.66 41
Aug 15 21:59:54 192.168.20.66 04
Aug 15 21:59:54 192.168.20.66 8b
Aug 15 21:59:54 192.168.20.66 11
Aug 15 21:59:54 192.168.20.66 c7
Aug 15 21:59:54 192.168.20.66 41
Aug 15 21:59:54 192.168.20.66 04
Aug 15 21:59:54 192.168.20.66 00
Aug 15 21:59:54 192.168.20.66 00
Aug 15 21:59:54 192.168.20.66 00
Aug 15 21:59:54 192.168.20.66 00
Aug 15 21:59:54 192.168.20.66 c7
Aug 15 21:59:54 192.168.20.66 01
Aug 15 21:59:54 192.168.20.66 00
Aug 15 21:59:54 192.168.20.66 00
Aug 15 21:59:54 192.168.20.66 00
Aug 15 21:59:54 192.168.20.66 00
Aug 15 21:59:54 192.168.20.66
Aug 15 21:59:54 192.168.20.66 42
Aug 15 21:59:54 192.168.20.66 04
Aug 15 21:59:54 192.168.20.66 89
Aug 15 21:59:54 192.168.20.66 10
Aug 15 21:59:54 192.168.20.66 8b
Aug 15 21:59:54 192.168.20.66 41
Aug 15 21:59:54 192.168.20.66 20
Aug 15 21:59:54 192.168.20.66 29
Aug 15 21:59:54 192.168.20.66 43
Aug 15 21:59:54 192.168.20.66 74
Aug 15 21:59:54 192.168.20.66 eb
Aug 15 21:59:54 192.168.20.66 02
Aug 15 21:59:54 192.168.20.66 31
Aug 15 21:59:54 192.168.20.66 c9
Aug 15 21:59:54 192.168.20.66 89
Aug 15 21:59:54 192.168.20.66 c8
Aug 15 21:59:54 192.168.20.66 5b
Aug 15 21:59:54 192.168.20.66 5d
Aug 15 21:59:54 192.168.20.66 c3
Aug 15 21:59:54 192.168.20.66 8b
Aug 15 21:59:54 192.168.20.66
Aug 15 21:59:54 192.168.20.66 [ 1123.800035] EIP: [<c0272579>]
Aug 15 21:59:54 192.168.20.66 qdisc_dequeue_head+0x29/0x3d
Aug 15 21:59:54 192.168.20.66 SS:ESP 0068:c085ef58
Aug 15 21:59:54 192.168.20.66 [ 1123.804374] Kernel panic - not syncing: 
Fatal exception in interrupt
Aug 15 21:59:54 192.168.20.66 [ 1123.804377] Rebooting in 5 seconds..


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.


^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-15 19:09 panic 2.6.27-rc3-git2, qdisc_dequeue_head Denys Fedoryshchenko
@ 2008-08-15 20:16 ` Jarek Poplawski
  2008-08-15 20:36   ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-15 20:16 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

Denys Fedoryshchenko wrote, On 08/15/2008 09:09 PM:

> Aug 15 21:59:54 192.168.20.66 [ 1123.800035] BUG: unable to handle kernel
> Aug 15 21:59:54 192.168.20.66 NULL pointer dereference
> Aug 15 21:59:54 192.168.20.66 at 00000004
> Aug 15 21:59:54 192.168.20.66 [ 1123.800035] IP:
> Aug 15 21:59:54 192.168.20.66 [<c0272579>] qdisc_dequeue_head+0x29/0x3d


Could you add some details like:
- device(eth, ppp, ifb...), net driver(s) on that device,
- types of filters, actions on the path to this tbf queue,
- were there any config changes done around the time when this happened?

Thanks,
Jarek P.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-15 20:16 ` Jarek Poplawski
@ 2008-08-15 20:36   ` Denys Fedoryshchenko
  2008-08-15 20:54     ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-15 20:36 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

On Friday 15 August 2008, Jarek Poplawski wrote:
> Denys Fedoryshchenko wrote, On 08/15/2008 09:09 PM:
> > Aug 15 21:59:54 192.168.20.66 [ 1123.800035] BUG: unable to handle kernel
> > Aug 15 21:59:54 192.168.20.66 NULL pointer dereference
> > Aug 15 21:59:54 192.168.20.66 at 00000004
> > Aug 15 21:59:54 192.168.20.66 [ 1123.800035] IP:
> > Aug 15 21:59:54 192.168.20.66 [<c0272579>] qdisc_dequeue_head+0x29/0x3d
>
> Could you add some details like:
> - device(eth, ppp, ifb...), net driver(s) on that device,
> - types of filters, actions on the path to this tbf queue,
> - were there any config changes done around the time when this happened?
>
> Thanks,
> Jarek P.


TBF on ppp
Probably multiple ppp interfaces coming up. (but tbf applied on each 
separately)

Rules, qdiscs and filters looks like this:

    echo "qdisc del dev $2 root"
    echo "qdisc add dev $2 root handle 1: prio"
    echo "qdisc add dev $2 parent 1:1 handle 2: est 1sec 8sec tbf buffer 
${burst} latency 500ms rate ${rate}bit peakrate ${peakrate}bit mtu 1512"
#    echo "Bypass code"
# !!!!!!!!!!!!!!!!!!WRONG!!!!!!!!!!!!
    echo "qdisc add dev $2 parent 1:1 handle 3: tbf buffer ${burst}kb latency 
500ms rate 1024000"
    echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src 
194.146.153.24/32 flowid 3:1"
    echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src 
194.146.153.26/32 flowid 3:1"
    echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src 
194.146.153.6/32 flowid 3:1"
    echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src 
194.146.153.19/32 flowid 3:1"
    echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src 
2.2.2.102/32 flowid 3:1"
    echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src 
2.2.2.105/32 flowid 3:1"
    echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src 
2.2.2.106/32 flowid 3:1"
#TEMP BYPASS CODE
#    echo "Final rule"
    echo "filter add dev $2 parent 1:0 protocol ip prio 10 u32 match ip dst 
0.0.0.0/0 flowid 2:1"                                                          


^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-15 20:36   ` Denys Fedoryshchenko
@ 2008-08-15 20:54     ` Denys Fedoryshchenko
  2008-08-16  8:55       ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-15 20:54 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Btw, it can happen on deletion also. At that moment there was 600 sessions 
established(and non-stop logging in/off), and it is difficult to tell, what 
was happening at that moment...

On Friday 15 August 2008, Denys Fedoryshchenko wrote:
> On Friday 15 August 2008, Jarek Poplawski wrote:
> > Denys Fedoryshchenko wrote, On 08/15/2008 09:09 PM:
> > > Aug 15 21:59:54 192.168.20.66 [ 1123.800035] BUG: unable to handle
> > > kernel Aug 15 21:59:54 192.168.20.66 NULL pointer dereference
> > > Aug 15 21:59:54 192.168.20.66 at 00000004
> > > Aug 15 21:59:54 192.168.20.66 [ 1123.800035] IP:
> > > Aug 15 21:59:54 192.168.20.66 [<c0272579>] qdisc_dequeue_head+0x29/0x3d
> >
> > Could you add some details like:
> > - device(eth, ppp, ifb...), net driver(s) on that device,
> > - types of filters, actions on the path to this tbf queue,
> > - were there any config changes done around the time when this happened?
> >
> > Thanks,
> > Jarek P.
>
> TBF on ppp
> Probably multiple ppp interfaces coming up. (but tbf applied on each
> separately)
>
> Rules, qdiscs and filters looks like this:
>
>     echo "qdisc del dev $2 root"
>     echo "qdisc add dev $2 root handle 1: prio"
>     echo "qdisc add dev $2 parent 1:1 handle 2: est 1sec 8sec tbf buffer
> ${burst} latency 500ms rate ${rate}bit peakrate ${peakrate}bit mtu 1512"
> #    echo "Bypass code"
> # !!!!!!!!!!!!!!!!!!WRONG!!!!!!!!!!!!
>     echo "qdisc add dev $2 parent 1:1 handle 3: tbf buffer ${burst}kb
> latency 500ms rate 1024000"
>     echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src
> 194.146.153.24/32 flowid 3:1"
>     echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src
> 194.146.153.26/32 flowid 3:1"
>     echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src
> 194.146.153.6/32 flowid 3:1"
>     echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src
> 194.146.153.19/32 flowid 3:1"
>     echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src
> 2.2.2.102/32 flowid 3:1"
>     echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src
> 2.2.2.105/32 flowid 3:1"
>     echo "filter add dev $2 parent 1:0 protocol ip prio 5 u32 match ip src
> 2.2.2.106/32 flowid 3:1"
> #TEMP BYPASS CODE
> #    echo "Final rule"
>     echo "filter add dev $2 parent 1:0 protocol ip prio 10 u32 match ip dst
> 0.0.0.0/0 flowid 2:1"
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-15 20:54     ` Denys Fedoryshchenko
@ 2008-08-16  8:55       ` Jarek Poplawski
  2008-08-16  9:13         ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-16  8:55 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Fri, Aug 15, 2008 at 11:54:24PM +0300, Denys Fedoryshchenko wrote:
> Btw, it can happen on deletion also. At that moment there was 600 sessions 
> established(and non-stop logging in/off), and it is difficult to tell, what 
> was happening at that moment...

Alas I can't find the reason, so here is some debugging patch (not tested!).

Jarek P. 

---

 include/linux/skbuff.h  |   43 +++++++++++++++++++++++++++++++------------
 net/sched/sch_generic.c |    1 +
 2 files changed, 32 insertions(+), 12 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 358661c..c46b1bd 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -120,6 +120,7 @@ struct sk_buff_head {
 
 	__u32		qlen;
 	spinlock_t	lock;
+	spinlock_t	lock_debug;
 };
 
 struct sk_buff;
@@ -657,6 +658,7 @@ static inline __u32 skb_queue_len(const struct sk_buff_head *list_)
 static inline void skb_queue_head_init(struct sk_buff_head *list)
 {
 	spin_lock_init(&list->lock);
+	spin_lock_init(&list->lock_debug);
 	list->prev = list->next = (struct sk_buff *)list;
 	list->qlen = 0;
 }
@@ -679,10 +681,15 @@ static inline void __skb_insert(struct sk_buff *newsk,
 				struct sk_buff *prev, struct sk_buff *next,
 				struct sk_buff_head *list)
 {
-	newsk->next = next;
-	newsk->prev = prev;
-	next->prev  = prev->next = newsk;
-	list->qlen++;
+	if (spin_trylock(&list->lock_debug)) {
+		newsk->next = next;
+		newsk->prev = prev;
+		next->prev  = prev->next = newsk;
+		list->qlen++;
+		spin_unlock(&list->lock_debug);
+	} else {
+		WARN_ON(1);
+	}
 }
 
 /**
@@ -775,10 +782,16 @@ static inline void __skb_unlink(struct sk_buff *skb, struct sk_buff_head *list)
 extern struct sk_buff *skb_dequeue(struct sk_buff_head *list);
 static inline struct sk_buff *__skb_dequeue(struct sk_buff_head *list)
 {
-	struct sk_buff *skb = skb_peek(list);
-	if (skb)
-		__skb_unlink(skb, list);
-	return skb;
+	if (spin_trylock(&list->lock_debug)) {
+		struct sk_buff *skb = skb_peek(list);
+		if (skb)
+			__skb_unlink(skb, list);
+		spin_unlock(&list->lock_debug);
+		return skb;
+	} else {
+		WARN_ON(1);
+		return NULL;
+	}
 }
 
 /**
@@ -792,10 +805,16 @@ static inline struct sk_buff *__skb_dequeue(struct sk_buff_head *list)
 extern struct sk_buff *skb_dequeue_tail(struct sk_buff_head *list);
 static inline struct sk_buff *__skb_dequeue_tail(struct sk_buff_head *list)
 {
-	struct sk_buff *skb = skb_peek_tail(list);
-	if (skb)
-		__skb_unlink(skb, list);
-	return skb;
+	if (spin_trylock(&list->lock_debug)) {
+		struct sk_buff *skb = skb_peek_tail(list);
+		if (skb)
+			__skb_unlink(skb, list);
+		spin_unlock(&list->lock_debug);
+		return skb;
+	} else {
+		WARN_ON(1);
+		return NULL;
+	}
 }
 
 
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 4685746..bffe6cb 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -540,6 +540,7 @@ static void __qdisc_destroy(struct rcu_head *head)
 
 	kfree_skb(qdisc->gso_skb);
 
+	memset(qdisc, 0xf0 , sizeof(*qdisc));
 	kfree((char *) qdisc - qdisc->padded);
 }
 

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16  8:55       ` Jarek Poplawski
@ 2008-08-16  9:13         ` Jarek Poplawski
  2008-08-16 10:03           ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-16  9:13 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sat, Aug 16, 2008 at 10:55:37AM +0200, Jarek Poplawski wrote:
> On Fri, Aug 15, 2008 at 11:54:24PM +0300, Denys Fedoryshchenko wrote:
> > Btw, it can happen on deletion also. At that moment there was 600 sessions 
> > established(and non-stop logging in/off), and it is difficult to tell, what 
> > was happening at that moment...
> 
> Alas I can't find the reason, so here is some debugging patch (not tested!).

Hold on a minute! Forget the previous patch - here is a take #2.

Jarek P. 

---

 include/linux/skbuff.h  |   44 ++++++++++++++++++++++++++++++++------------
 net/sched/sch_generic.c |    1 +
 2 files changed, 33 insertions(+), 12 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 358661c..506142e 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -120,6 +120,7 @@ struct sk_buff_head {
 
 	__u32		qlen;
 	spinlock_t	lock;
+	spinlock_t	lock_debug;
 };
 
 struct sk_buff;
@@ -657,6 +658,7 @@ static inline __u32 skb_queue_len(const struct sk_buff_head *list_)
 static inline void skb_queue_head_init(struct sk_buff_head *list)
 {
 	spin_lock_init(&list->lock);
+	spin_lock_init(&list->lock_debug);
 	list->prev = list->next = (struct sk_buff *)list;
 	list->qlen = 0;
 }
@@ -679,10 +681,16 @@ static inline void __skb_insert(struct sk_buff *newsk,
 				struct sk_buff *prev, struct sk_buff *next,
 				struct sk_buff_head *list)
 {
-	newsk->next = next;
-	newsk->prev = prev;
-	next->prev  = prev->next = newsk;
-	list->qlen++;
+	if (spin_trylock(&list->lock_debug)) {
+		newsk->next = next;
+		newsk->prev = prev;
+		next->prev  = prev->next = newsk;
+		list->qlen++;
+		spin_unlock(&list->lock_debug);
+	} else {
+		kfree_skb(newsk);
+		WARN_ON(1);
+	}
 }
 
 /**
@@ -775,10 +783,16 @@ static inline void __skb_unlink(struct sk_buff *skb, struct sk_buff_head *list)
 extern struct sk_buff *skb_dequeue(struct sk_buff_head *list);
 static inline struct sk_buff *__skb_dequeue(struct sk_buff_head *list)
 {
-	struct sk_buff *skb = skb_peek(list);
-	if (skb)
-		__skb_unlink(skb, list);
-	return skb;
+	if (spin_trylock(&list->lock_debug)) {
+		struct sk_buff *skb = skb_peek(list);
+		if (skb)
+			__skb_unlink(skb, list);
+		spin_unlock(&list->lock_debug);
+		return skb;
+	} else {
+		WARN_ON(1);
+		return NULL;
+	}
 }
 
 /**
@@ -792,10 +806,16 @@ static inline struct sk_buff *__skb_dequeue(struct sk_buff_head *list)
 extern struct sk_buff *skb_dequeue_tail(struct sk_buff_head *list);
 static inline struct sk_buff *__skb_dequeue_tail(struct sk_buff_head *list)
 {
-	struct sk_buff *skb = skb_peek_tail(list);
-	if (skb)
-		__skb_unlink(skb, list);
-	return skb;
+	if (spin_trylock(&list->lock_debug)) {
+		struct sk_buff *skb = skb_peek_tail(list);
+		if (skb)
+			__skb_unlink(skb, list);
+		spin_unlock(&list->lock_debug);
+		return skb;
+	} else {
+		WARN_ON(1);
+		return NULL;
+	}
 }
 
 
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 4685746..bffe6cb 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -540,6 +540,7 @@ static void __qdisc_destroy(struct rcu_head *head)
 
 	kfree_skb(qdisc->gso_skb);
 
+	memset(qdisc, 0xf0 , sizeof(*qdisc));
 	kfree((char *) qdisc - qdisc->padded);
 }
 

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16  9:13         ` Jarek Poplawski
@ 2008-08-16 10:03           ` Denys Fedoryshchenko
  2008-08-16 10:05             ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-16 10:03 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

I tried patch on less loaded pppoe
Aug 16 13:02:00 194.146.153.146 pppd[1940]: peer from calling number 
00:1E:68:0A:B8:0B authorized
Aug 16 13:02:00 194.146.153.146 pppd[1940]: local  IP address 172.16.131.1
Aug 16 13:02:00 194.146.153.146 pppd[1940]: remote IP address 172.16.131.4
Aug 16 13:02:00 194.146.153.146 ppp: User toni18 on interface ppp0 logged in
Aug 16 13:02:00 194.146.153.146 ppp: ip 172.16.131.4 = toni18
Aug 16 13:02:00 194.146.153.146 ppp: Shaper TX/RX set 56000/128000 on if ppp0 
for toni18
Aug 16 13:02:00 194.146.153.146 pppshaper: Shaper DOWN for interface ppp0 will 
be set  bit/s with burstsize  and burstspeed 
Aug 16 13:02:00 194.146.153.146 pppshaper: Shaper RX for interface ppp0 will 
be set 128000 bit/s with burstsize 2048000 and burstspeed 640000
Aug 16 13:02:00 194.146.153.146 pppshaper: Shaper TX for interface ppp0 will 
be set 56000 bit/s with burstsize 2048000 and burstspeed 280000
Aug 16 13:02:01 194.146.153.146 [   81.476254] BUG: unable to handle kernel 
Aug 16 13:02:01 194.146.153.146 paging request
Aug 16 13:02:01 194.146.153.146 at 00002d30
Aug 16 13:02:01 194.146.153.146 [   81.476254] IP:
Aug 16 13:02:01 194.146.153.146 [<c01643bd>] kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 [   81.476254] *pde = 00000000 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.476254] Oops: 0000 [#1] 
Aug 16 13:02:01 194.146.153.146 SMP 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.476254] Modules linked in:
Aug 16 13:02:01 194.146.153.146 sch_prio
Aug 16 13:02:01 194.146.153.146 netconsole
Aug 16 13:02:01 194.146.153.146 configfs
Aug 16 13:02:01 194.146.153.146 cls_u32
Aug 16 13:02:01 194.146.153.146 em_meta
Aug 16 13:02:01 194.146.153.146 cls_basic
Aug 16 13:02:01 194.146.153.146 xt_dscp
Aug 16 13:02:01 194.146.153.146 xt_DSCP
Aug 16 13:02:01 194.146.153.146 ipt_REJECT
Aug 16 13:02:01 194.146.153.146 ts_bm
Aug 16 13:02:01 194.146.153.146 xt_string
Aug 16 13:02:01 194.146.153.146 ipt_ttl
Aug 16 13:02:01 194.146.153.146 ifb
Aug 16 13:02:01 194.146.153.146 cls_fw
Aug 16 13:02:01 194.146.153.146 sch_tbf
Aug 16 13:02:01 194.146.153.146 sch_htb
Aug 16 13:02:01 194.146.153.146 act_ipt
Aug 16 13:02:01 194.146.153.146 act_mirred
Aug 16 13:02:01 194.146.153.146 xt_MARK
Aug 16 13:02:01 194.146.153.146 pppoe
Aug 16 13:02:01 194.146.153.146 pppox
Aug 16 13:02:01 194.146.153.146 ppp_generic
Aug 16 13:02:01 194.146.153.146 slhc
Aug 16 13:02:01 194.146.153.146 ipt_MASQUERADE
Aug 16 13:02:01 194.146.153.146 xt_TCPMSS
Aug 16 13:02:01 194.146.153.146 xt_mark
Aug 16 13:02:01 194.146.153.146 xt_tcpudp
Aug 16 13:02:01 194.146.153.146 iptable_mangle
Aug 16 13:02:01 194.146.153.146 iptable_nat
Aug 16 13:02:01 194.146.153.146 nf_nat
Aug 16 13:02:01 194.146.153.146 nf_conntrack_ipv4
Aug 16 13:02:01 194.146.153.146 nf_conntrack
Aug 16 13:02:01 194.146.153.146 rtc_cmos
Aug 16 13:02:01 194.146.153.146 rtc_core
Aug 16 13:02:01 194.146.153.146 rtc_lib
Aug 16 13:02:01 194.146.153.146 iptable_filter
Aug 16 13:02:01 194.146.153.146 ip_tables
Aug 16 13:02:01 194.146.153.146 x_tables
Aug 16 13:02:01 194.146.153.146 8021q
Aug 16 13:02:01 194.146.153.146 garp
Aug 16 13:02:01 194.146.153.146 stp
Aug 16 13:02:01 194.146.153.146 llc
Aug 16 13:02:01 194.146.153.146 tun
Aug 16 13:02:01 194.146.153.146 loop
Aug 16 13:02:01 194.146.153.146 tulip
Aug 16 13:02:01 194.146.153.146 r8169
Aug 16 13:02:01 194.146.153.146 sky2
Aug 16 13:02:01 194.146.153.146 via_velocity
Aug 16 13:02:01 194.146.153.146 via_rhine
Aug 16 13:02:01 194.146.153.146 sis900
Aug 16 13:02:01 194.146.153.146 ne2k_pci
Aug 16 13:02:01 194.146.153.146 8390
Aug 16 13:02:01 194.146.153.146 skge
Aug 16 13:02:01 194.146.153.146 tg3
Aug 16 13:02:01 194.146.153.146 libphy
Aug 16 13:02:01 194.146.153.146 8139too
Aug 16 13:02:01 194.146.153.146 e1000
Aug 16 13:02:01 194.146.153.146 e100
Aug 16 13:02:01 194.146.153.146 mtdblock
Aug 16 13:02:01 194.146.153.146 mtd_blkdevs
Aug 16 13:02:01 194.146.153.146 aic79xx
Aug 16 13:02:01 194.146.153.146 mptsas
Aug 16 13:02:01 194.146.153.146 scsi_transport_sas
Aug 16 13:02:01 194.146.153.146 mptspi
Aug 16 13:02:01 194.146.153.146 mptscsih
Aug 16 13:02:01 194.146.153.146 scsi_transport_spi
Aug 16 13:02:01 194.146.153.146 mptbase
Aug 16 13:02:01 194.146.153.146 sata_via
Aug 16 13:02:01 194.146.153.146 sata_uli
Aug 16 13:02:01 194.146.153.146 sata_svw
Aug 16 13:02:01 194.146.153.146 sata_sis
Aug 16 13:02:01 194.146.153.146 sata_sil24
Aug 16 13:02:01 194.146.153.146 sata_sil
Aug 16 13:02:01 194.146.153.146 sata_mv
Aug 16 13:02:01 194.146.153.146 ahci
Aug 16 13:02:01 194.146.153.146 pata_via
Aug 16 13:02:01 194.146.153.146 pata_sis
Aug 16 13:02:01 194.146.153.146 pata_serverworks
Aug 16 13:02:01 194.146.153.146 pata_oldpiix
Aug 16 13:02:01 194.146.153.146 pata_mpiix
Aug 16 13:02:01 194.146.153.146 pata_marvell
Aug 16 13:02:01 194.146.153.146 pata_ali
Aug 16 13:02:01 194.146.153.146 ata_generic
Aug 16 13:02:01 194.146.153.146 ata_piix
Aug 16 13:02:01 194.146.153.146 libata
Aug 16 13:02:01 194.146.153.146 dock
Aug 16 13:02:01 194.146.153.146 usbhid
Aug 16 13:02:01 194.146.153.146 uhci_hcd
Aug 16 13:02:01 194.146.153.146 ehci_hcd
Aug 16 13:02:01 194.146.153.146 ohci_hcd
Aug 16 13:02:01 194.146.153.146 usbcore
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] 
Aug 16 13:02:01 194.146.153.146 [   81.480253] Pid: 1998, comm: modprobe Not 
tainted (2.6.27-rc3-git2-build-0031 #13)
Aug 16 13:02:01 194.146.153.146 [   81.480253] EIP: 0060:[<c01643bd>] EFLAGS: 
00010002 CPU: 1
Aug 16 13:02:01 194.146.153.146 [   81.480253] EIP is at kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 [   81.480253] EAX: 00000001 EBX: f72e4380 
ECX: 00000000 EDX: 00002c90
Aug 16 13:02:01 194.146.153.146 [   81.480253] ESI: c1e44744 EDI: 063d5210 
EBP: c0861fa4 ESP: c0861f8c
Aug 16 13:02:01 194.146.153.146 [   81.480253]  DS: 007b ES: 007b FS: 00d8 GS: 
0000 SS: 0068
Aug 16 13:02:01 194.146.153.146 [   81.480253] Process modprobe (pid: 1998, 
ti=c0861000 task=f6ce55a0 task.ti=f6d0b000)
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] Stack: 
Aug 16 13:02:01 194.146.153.146 c026e926 
Aug 16 13:02:01 194.146.153.146 00002c90 
Aug 16 13:02:01 194.146.153.146 00000286 
Aug 16 13:02:01 194.146.153.146 f72e4380 
Aug 16 13:02:01 194.146.153.146 063d5210 
Aug 16 13:02:01 194.146.153.146 f72e43bc 
Aug 16 13:02:01 194.146.153.146 c0861fb8 
Aug 16 13:02:01 194.146.153.146 c026e926 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253]        
Aug 16 13:02:01 194.146.153.146 c1d19518 
Aug 16 13:02:01 194.146.153.146 f7708688 
Aug 16 13:02:01 194.146.153.146 00000003 
Aug 16 13:02:01 194.146.153.146 c0861fcc 
Aug 16 13:02:01 194.146.153.146 c014b127 
Aug 16 13:02:01 194.146.153.146 c039daa0 
Aug 16 13:02:01 194.146.153.146 c0859554 
Aug 16 13:02:01 194.146.153.146 00000001 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253]        
Aug 16 13:02:01 194.146.153.146 c0861fd4 
Aug 16 13:02:01 194.146.153.146 c014b19e 
Aug 16 13:02:01 194.146.153.146 c0861ff8 
Aug 16 13:02:01 194.146.153.146 c0126203 
Aug 16 13:02:01 194.146.153.146 c085c700 
Aug 16 13:02:01 194.146.153.146 c085c700 
Aug 16 13:02:01 194.146.153.146 0000000a 
Aug 16 13:02:01 194.146.153.146 00000001 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] Call Trace:
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c014b127>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __rcu_process_callbacks+0x165/0x1c2
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c014b19e>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 rcu_process_callbacks+0x1a/0x33
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c0126203>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x7b/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c0126188>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x0/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c01060b3>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 do_softirq+0x65/0xb6
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c0125e28>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 irq_exit+0x44/0x79
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c010ff15>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c010438d>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 apic_timer_interrupt+0x2d/0x34
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c012007b>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 sys_unshare+0xbf/0x258
Aug 16 13:02:01 194.146.153.146 [   81.480253]  =======================
Aug 16 13:02:01 194.146.153.146 [   81.480253] Code: 
Aug 16 13:02:01 194.146.153.146 0b 
Aug 16 13:02:01 194.146.153.146 eb 
Aug 16 13:02:01 194.146.153.146 fe 
Aug 16 13:02:01 194.146.153.146 89 
Aug 16 13:02:01 194.146.153.146 f0 
Aug 16 13:02:01 194.146.153.146 e8 
Aug 16 13:02:01 194.146.153.146 0d 
Aug 16 13:02:01 194.146.153.146 f2 
Aug 16 13:02:01 194.146.153.146 fe 
Aug 16 13:02:01 194.146.153.146 ff 
Aug 16 13:02:01 194.146.153.146 eb 
Aug 16 13:02:01 194.146.153.146 78 
Aug 16 13:02:01 194.146.153.146 8b 
Aug 16 13:02:01 194.146.153.146 45 
Aug 16 13:02:01 194.146.153.146 04 
Aug 16 13:02:01 194.146.153.146 89 
Aug 16 13:02:01 194.146.153.146 45 
Aug 16 13:02:01 194.146.153.146 e8 
Aug 16 13:02:01 194.146.153.146 8b 
Aug 16 13:02:01 194.146.153.146 56 
Aug 16 13:02:01 194.146.153.146 0c 
Aug 16 13:02:01 194.146.153.146 89 
Aug 16 13:02:01 194.146.153.146 55 
Aug 16 13:02:01 194.146.153.146 ec 
Aug 16 13:02:01 194.146.153.146 9c 
Aug 16 13:02:01 194.146.153.146 8f 
Aug 16 13:02:01 194.146.153.146 45 
Aug 16 13:02:01 194.146.153.146 f0 
Aug 16 13:02:01 194.146.153.146 fa 
Aug 16 13:02:01 194.146.153.146 e8 
Aug 16 13:02:01 194.146.153.146 19 
Aug 16 13:02:01 194.146.153.146 84 
Aug 16 13:02:01 194.146.153.146 fd 
Aug 16 13:02:01 194.146.153.146 ff 
Aug 16 13:02:01 194.146.153.146 8b 
Aug 16 13:02:01 194.146.153.146 55 
Aug 16 13:02:01 194.146.153.146 ec 
Aug 16 13:02:01 194.146.153.146 64 
Aug 16 13:02:01 194.146.153.146 a1 
Aug 16 13:02:01 194.146.153.146 04 
Aug 16 13:02:01 194.146.153.146 50 
Aug 16 13:02:01 194.146.153.146 85 
Aug 16 13:02:01 194.146.153.146 c0 
Aug 16 13:02:01 194.146.153.146 unparseable log message: "<8b> "
Aug 16 13:02:01 194.146.153.146 9c 
Aug 16 13:02:01 194.146.153.146 82 
Aug 16 13:02:01 194.146.153.146 9c 
Aug 16 13:02:01 194.146.153.146 00 
Aug 16 13:02:01 194.146.153.146 00 
Aug 16 13:02:01 194.146.153.146 00 
Aug 16 13:02:01 194.146.153.146 89 
Aug 16 13:02:01 194.146.153.146 f8 
Aug 16 13:02:01 194.146.153.146 8b 
Aug 16 13:02:01 194.146.153.146 53 
Aug 16 13:02:01 194.146.153.146 10 
Aug 16 13:02:01 194.146.153.146 e8 
Aug 16 13:02:01 194.146.153.146 f3 
Aug 16 13:02:01 194.146.153.146 96 
Aug 16 13:02:01 194.146.153.146 fd 
Aug 16 13:02:01 194.146.153.146 ff 
Aug 16 13:02:01 194.146.153.146 3b 
Aug 16 13:02:01 194.146.153.146 73 
Aug 16 13:02:01 194.146.153.146 04 
Aug 16 13:02:01 194.146.153.146 75 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] EIP: [<c01643bd>] 
Aug 16 13:02:01 194.146.153.146 kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 SS:ESP 0068:c0861f8c
Aug 16 13:02:01 194.146.153.146 [   81.480253] Kernel panic - not syncing: 
Fatal exception in interrupt
Aug 16 13:02:01 194.146.153.146 [   81.480253] ------------[ cut 
here ]------------
Aug 16 13:02:01 194.146.153.146 [   81.480253] WARNING: at kernel/smp.c:328 
smp_call_function_mask+0x28/0x181()
Aug 16 13:02:01 194.146.153.146 [   81.480253] Modules linked in:
Aug 16 13:02:01 194.146.153.146 sch_prio
Aug 16 13:02:01 194.146.153.146 netconsole
Aug 16 13:02:01 194.146.153.146 configfs
Aug 16 13:02:01 194.146.153.146 cls_u32
Aug 16 13:02:01 194.146.153.146 em_meta
Aug 16 13:02:01 194.146.153.146 cls_basic
Aug 16 13:02:01 194.146.153.146 xt_dscp
Aug 16 13:02:01 194.146.153.146 xt_DSCP
Aug 16 13:02:01 194.146.153.146 ipt_REJECT
Aug 16 13:02:01 194.146.153.146 ts_bm
Aug 16 13:02:01 194.146.153.146 xt_string
Aug 16 13:02:01 194.146.153.146 ipt_ttl
Aug 16 13:02:01 194.146.153.146 ifb
Aug 16 13:02:01 194.146.153.146 cls_fw
Aug 16 13:02:01 194.146.153.146 sch_tbf
Aug 16 13:02:01 194.146.153.146 sch_htb
Aug 16 13:02:01 194.146.153.146 act_ipt
Aug 16 13:02:01 194.146.153.146 act_mirred
Aug 16 13:02:01 194.146.153.146 xt_MARK
Aug 16 13:02:01 194.146.153.146 pppoe
Aug 16 13:02:01 194.146.153.146 pppox
Aug 16 13:02:01 194.146.153.146 ppp_generic
Aug 16 13:02:01 194.146.153.146 slhc
Aug 16 13:02:01 194.146.153.146 ipt_MASQUERADE
Aug 16 13:02:01 194.146.153.146 xt_TCPMSS
Aug 16 13:02:01 194.146.153.146 xt_mark
Aug 16 13:02:01 194.146.153.146 xt_tcpudp
Aug 16 13:02:01 194.146.153.146 iptable_mangle
Aug 16 13:02:01 194.146.153.146 iptable_nat
Aug 16 13:02:01 194.146.153.146 nf_nat
Aug 16 13:02:01 194.146.153.146 nf_conntrack_ipv4
Aug 16 13:02:01 194.146.153.146 nf_conntrack
Aug 16 13:02:01 194.146.153.146 rtc_cmos
Aug 16 13:02:01 194.146.153.146 rtc_core
Aug 16 13:02:01 194.146.153.146 rtc_lib
Aug 16 13:02:01 194.146.153.146 iptable_filter
Aug 16 13:02:01 194.146.153.146 ip_tables
Aug 16 13:02:01 194.146.153.146 x_tables
Aug 16 13:02:01 194.146.153.146 8021q
Aug 16 13:02:01 194.146.153.146 garp
Aug 16 13:02:01 194.146.153.146 stp
Aug 16 13:02:01 194.146.153.146 llc
Aug 16 13:02:01 194.146.153.146 tun
Aug 16 13:02:01 194.146.153.146 loop
Aug 16 13:02:01 194.146.153.146 tulip
Aug 16 13:02:01 194.146.153.146 r8169
Aug 16 13:02:01 194.146.153.146 sky2
Aug 16 13:02:01 194.146.153.146 via_velocity
Aug 16 13:02:01 194.146.153.146 via_rhine
Aug 16 13:02:01 194.146.153.146 sis900
Aug 16 13:02:01 194.146.153.146 ne2k_pci
Aug 16 13:02:01 194.146.153.146 8390
Aug 16 13:02:01 194.146.153.146 skge
Aug 16 13:02:01 194.146.153.146 tg3
Aug 16 13:02:01 194.146.153.146 libphy
Aug 16 13:02:01 194.146.153.146 8139too
Aug 16 13:02:01 194.146.153.146 e1000
Aug 16 13:02:01 194.146.153.146 e100
Aug 16 13:02:01 194.146.153.146 mtdblock
Aug 16 13:02:01 194.146.153.146 mtd_blkdevs
Aug 16 13:02:01 194.146.153.146 aic79xx
Aug 16 13:02:01 194.146.153.146 mptsas
Aug 16 13:02:01 194.146.153.146 scsi_transport_sas
Aug 16 13:02:01 194.146.153.146 mptspi
Aug 16 13:02:01 194.146.153.146 mptscsih
Aug 16 13:02:01 194.146.153.146 scsi_transport_spi
Aug 16 13:02:01 194.146.153.146 mptbase
Aug 16 13:02:01 194.146.153.146 sata_via
Aug 16 13:02:01 194.146.153.146 sata_uli
Aug 16 13:02:01 194.146.153.146 sata_svw
Aug 16 13:02:01 194.146.153.146 sata_sis
Aug 16 13:02:01 194.146.153.146 sata_sil24
Aug 16 13:02:01 194.146.153.146 sata_sil
Aug 16 13:02:01 194.146.153.146 sata_mv
Aug 16 13:02:01 194.146.153.146 ahci
Aug 16 13:02:01 194.146.153.146 pata_via
Aug 16 13:02:01 194.146.153.146 pata_sis
Aug 16 13:02:01 194.146.153.146 pata_serverworks
Aug 16 13:02:01 194.146.153.146 pata_oldpiix
Aug 16 13:02:01 194.146.153.146 pata_mpiix
Aug 16 13:02:01 194.146.153.146 pata_marvell
Aug 16 13:02:01 194.146.153.146 pata_ali
Aug 16 13:02:01 194.146.153.146 ata_generic
Aug 16 13:02:01 194.146.153.146 ata_piix
Aug 16 13:02:01 194.146.153.146 libata
Aug 16 13:02:01 194.146.153.146 dock
Aug 16 13:02:01 194.146.153.146 usbhid
Aug 16 13:02:01 194.146.153.146 uhci_hcd
Aug 16 13:02:01 194.146.153.146 ehci_hcd
Aug 16 13:02:01 194.146.153.146 ohci_hcd
Aug 16 13:02:01 194.146.153.146 usbcore
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] Pid: 1998, comm: modprobe 
Tainted: G      D   2.6.27-rc3-git2-build-0031 #13
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c02bb7e7>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 printk+0xf/0x18
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c0121be8>] 
Aug 16 13:02:01 194.146.153.146 warn_on_slowpath+0x41/0x65
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<f896b649>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 rtl8139_start_xmit+0xe6/0x10c [8139too]
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c026caf9>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 netpoll_send_skb+0xdc/0x15b
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c026cd49>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 netpoll_send_udp+0x1d1/0x1dc
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bdbe8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 _spin_unlock_irqrestore+0x2f/0x3c
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f8a98178>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 write_msg+0xb1/0xb9 [netconsole]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01430b4>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function_mask+0x28/0x181
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ef59>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 stop_this_cpu+0x0/0x3b
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f89a0225>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 velocity_init_registers+0x2c6/0x631 
[via_velocity]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f89a0020>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 velocity_init_registers+0xc1/0x631 
[via_velocity]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0160020>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 shmem_truncate_range+0x3f8/0x610
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014321f>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function+0x12/0x14
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010efaf>] 
Aug 16 13:02:01 194.146.153.146 native_smp_send_stop+0x1b/0x40
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bb740>] 
Aug 16 13:02:01 194.146.153.146 panic+0x48/0xe0
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0104945>] 
Aug 16 13:02:01 194.146.153.146 oops_end+0x73/0x87
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0105138>] 
Aug 16 13:02:01 194.146.153.146 die+0x5b/0x63
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0114512>] 
Aug 16 13:02:01 194.146.153.146 do_page_fault+0x532/0x624
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0104ad4>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 dump_trace+0xc5/0xe5
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ab8e>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 save_stack_address+0x0/0x2c
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013d566>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 mark_lock+0x495/0x7f8
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013e775>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __lock_acquire+0x59d/0x118a
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0113fe0>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 do_page_fault+0x0/0x624
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02be052>] 
Aug 16 13:02:01 194.146.153.146 error_code+0x72/0x78
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01643bd>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014b127>] 
Aug 16 13:02:01 194.146.153.146 __rcu_process_callbacks+0x165/0x1c2
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014b19e>] 
Aug 16 13:02:01 194.146.153.146 rcu_process_callbacks+0x1a/0x33
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0126203>] 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x7b/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0126188>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x0/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01060b3>] 
Aug 16 13:02:01 194.146.153.146 do_softirq+0x65/0xb6
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0125e28>] 
Aug 16 13:02:01 194.146.153.146 irq_exit+0x44/0x79
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ff15>] 
Aug 16 13:02:01 194.146.153.146 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010438d>] 
Aug 16 13:02:01 194.146.153.146 apic_timer_interrupt+0x2d/0x34
Aug 16 13:02:01 194.146.153.146 [   81.480254
Aug 16 13:02:01 194.146.153.146 ]  [<c012007b>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 sys_unshare+0xbf/0x258
Aug 16 13:02:01 194.146.153.146 [   81.480254]  =======================
Aug 16 13:02:01 194.146.153.146 [   81.480254] ---[ end trace 
bc171cacb91df283 ]---
Aug 16 13:02:01 194.146.153.146 [   81.480254] ------------[ cut 
here ]------------
Aug 16 13:02:01 194.146.153.146 [   81.480254] WARNING: at kernel/smp.c:217 
smp_call_function_single+0x2d/0x9d()
Aug 16 13:02:01 194.146.153.146 [   81.480254] Modules linked in:
Aug 16 13:02:01 194.146.153.146 sch_prio
Aug 16 13:02:01 194.146.153.146 netconsole
Aug 16 13:02:01 194.146.153.146 configfs
Aug 16 13:02:01 194.146.153.146 cls_u32
Aug 16 13:02:01 194.146.153.146 em_meta
Aug 16 13:02:01 194.146.153.146 cls_basic
Aug 16 13:02:01 194.146.153.146 xt_dscp
Aug 16 13:02:01 194.146.153.146 xt_DSCP
Aug 16 13:02:01 194.146.153.146 ipt_REJECT
Aug 16 13:02:01 194.146.153.146 ts_bm
Aug 16 13:02:01 194.146.153.146 xt_string
Aug 16 13:02:01 194.146.153.146 ipt_ttl
Aug 16 13:02:01 194.146.153.146 ifb
Aug 16 13:02:01 194.146.153.146 cls_fw
Aug 16 13:02:01 194.146.153.146 sch_tbf
Aug 16 13:02:01 194.146.153.146 sch_htb
Aug 16 13:02:01 194.146.153.146 act_ipt
Aug 16 13:02:01 194.146.153.146 act_mirred
Aug 16 13:02:01 194.146.153.146 xt_MARK
Aug 16 13:02:01 194.146.153.146 pppoe
Aug 16 13:02:01 194.146.153.146 pppox
Aug 16 13:02:01 194.146.153.146 ppp_generic
Aug 16 13:02:01 194.146.153.146 slhc
Aug 16 13:02:01 194.146.153.146 ipt_MASQUERADE
Aug 16 13:02:01 194.146.153.146 xt_TCPMSS
Aug 16 13:02:01 194.146.153.146 xt_mark
Aug 16 13:02:01 194.146.153.146 xt_tcpudp
Aug 16 13:02:01 194.146.153.146 iptable_mangle
Aug 16 13:02:01 194.146.153.146 iptable_nat
Aug 16 13:02:01 194.146.153.146 nf_nat
Aug 16 13:02:01 194.146.153.146 nf_conntrack_ipv4
Aug 16 13:02:01 194.146.153.146 nf_conntrack
Aug 16 13:02:01 194.146.153.146 rtc_cmos
Aug 16 13:02:01 194.146.153.146 rtc_core
Aug 16 13:02:01 194.146.153.146 rtc_lib
Aug 16 13:02:01 194.146.153.146 iptable_filter
Aug 16 13:02:01 194.146.153.146 ip_tables
Aug 16 13:02:01 194.146.153.146 x_tables
Aug 16 13:02:01 194.146.153.146 8021q
Aug 16 13:02:01 194.146.153.146 garp
Aug 16 13:02:01 194.146.153.146 stp
Aug 16 13:02:01 194.146.153.146 llc
Aug 16 13:02:01 194.146.153.146 tun
Aug 16 13:02:01 194.146.153.146 loop
Aug 16 13:02:01 194.146.153.146 tulip
Aug 16 13:02:01 194.146.153.146 r8169
Aug 16 13:02:01 194.146.153.146 sky2
Aug 16 13:02:01 194.146.153.146 via_velocity
Aug 16 13:02:01 194.146.153.146 via_rhine
Aug 16 13:02:01 194.146.153.146 sis900
Aug 16 13:02:01 194.146.153.146 ne2k_pci
Aug 16 13:02:01 194.146.153.146 8390
Aug 16 13:02:01 194.146.153.146 skge
Aug 16 13:02:01 194.146.153.146 tg3
Aug 16 13:02:01 194.146.153.146 libphy
Aug 16 13:02:01 194.146.153.146 8139too
Aug 16 13:02:01 194.146.153.146 e1000
Aug 16 13:02:01 194.146.153.146 e100
Aug 16 13:02:01 194.146.153.146 mtdblock
Aug 16 13:02:01 194.146.153.146 mtd_blkdevs
Aug 16 13:02:01 194.146.153.146 aic79xx
Aug 16 13:02:01 194.146.153.146 mptsas
Aug 16 13:02:01 194.146.153.146 scsi_transport_sas
Aug 16 13:02:01 194.146.153.146 mptspi
Aug 16 13:02:01 194.146.153.146 mptscsih
Aug 16 13:02:01 194.146.153.146 scsi_transport_spi
Aug 16 13:02:01 194.146.153.146 mptbase
Aug 16 13:02:01 194.146.153.146 sata_via
Aug 16 13:02:01 194.146.153.146 sata_uli
Aug 16 13:02:01 194.146.153.146 sata_svw
Aug 16 13:02:01 194.146.153.146 sata_sis
Aug 16 13:02:01 194.146.153.146 sata_sil24
Aug 16 13:02:01 194.146.153.146 sata_sil
Aug 16 13:02:01 194.146.153.146 sata_mv
Aug 16 13:02:01 194.146.153.146 ahci
Aug 16 13:02:01 194.146.153.146 pata_via
Aug 16 13:02:01 194.146.153.146 pata_sis
Aug 16 13:02:01 194.146.153.146 pata_serverworks
Aug 16 13:02:01 194.146.153.146 pata_oldpiix
Aug 16 13:02:01 194.146.153.146 pata_mpiix
Aug 16 13:02:01 194.146.153.146 pata_marvell
Aug 16 13:02:01 194.146.153.146 pata_ali
Aug 16 13:02:01 194.146.153.146 ata_generic
Aug 16 13:02:01 194.146.153.146 ata_piix
Aug 16 13:02:01 194.146.153.146 libata
Aug 16 13:02:01 194.146.153.146 dock
Aug 16 13:02:01 194.146.153.146 usbhid
Aug 16 13:02:01 194.146.153.146 uhci_hcd
Aug 16 13:02:01 194.146.153.146 ehci_hcd
Aug 16 13:02:01 194.146.153.146 ohci_hcd
Aug 16 13:02:01 194.146.153.146 usbcore
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480254] Pid: 1998, comm: modprobe 
Tainted: G      D W 2.6.27-rc3-git2-build-0031 #13
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bb7e7>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 printk+0xf/0x18
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0121be8>] 
Aug 16 13:02:01 194.146.153.146 warn_on_slowpath+0x41/0x65
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f896b649>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 rtl8139_start_xmit+0xe6/0x10c [8139too]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 netpoll_send_udp+0x1d1/0x1dc
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026caf9>] 
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 netpoll_send_skb+0xdc/0x15b
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026cd49>] 
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bdbe8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 _spin_unlock_irqrestore+0x2f/0x3c
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f8a98178>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 write_msg+0xb1/0xb9 [netconsole]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014301c>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function_single+0x2d/0x9d
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ef59>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 stop_this_cpu+0x0/0x3b
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0143103>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function_mask+0x77/0x181
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ef59>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 stop_this_cpu+0x0/0x3b
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f89a0225>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 velocity_init_registers+0x2c6/0x631 
[via_velocity]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f89a0020>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 velocity_init_registers+0xc1/0x631 
[via_velocity]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014321f>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function+0x12/0x14
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010efaf>] 
Aug 16 13:02:01 194.146.153.146 native_smp_send_stop+0x1b/0x40
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bb740>] 
Aug 16 13:02:01 194.146.153.146 panic+0x48/0xe0
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0104945>] 
Aug 16 13:02:01 194.146.153.146 oops_end+0x73/0x87
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0105138>] 
Aug 16 13:02:01 194.146.153.146 die+0x5b/0x63
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0114512>] 
Aug 16 13:02:01 194.146.153.146 do_page_fault+0x532/0x624
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0104ad4>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 dump_trace+0xc5/0xe5
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ab8e>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 save_stack_address+0x0/0x2c
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013d566>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 mark_lock+0x495/0x7f8
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013e775>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __lock_acquire+0x59d/0x118a
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0113fe0>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 do_page_fault+0x0/0x624
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02be052>] 
Aug 16 13:02:01 194.146.153.146 error_code+0x72/0x78
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01643bd>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014b127>] 
Aug 16 13:02:01 194.146.153.146 __rcu_process_callbacks+0x165/0x1c2
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014b19e>] 
Aug 16 13:02:01 194.146.153.146 rcu_process_callbacks+0x1a/0x33
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0126203>] 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x7b/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0126188>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x0/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01060b3>] 
Aug 16 13:02:01 194.146.153.146 do_softirq+0x65/0xb6
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0125e28>] 
Aug 16 13:02:01 194.146.153.146 irq_exit+0x44/0x79
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ff15>] 
Aug 16 13:02:01 194.146.153.146 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010438d>] 
Aug 16 13:02:01 194.146.153.146 apic_timer_interrupt+0x2d/0x34
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c012007b>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 sys_unshare+0xbf/0x258
Aug 16 13:02:01 194.146.153.146 [   81.480254]  =======================
Aug 16 13:02:01 194.146.153.146 [   81.480254] ---[ end trace 
bc171cacb91df283 ]---
Aug 16 13:02:01 194.146.153.146 [   81.480254] Rebooting in 5 seconds..

>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 10:03           ` Denys Fedoryshchenko
@ 2008-08-16 10:05             ` Denys Fedoryshchenko
  2008-08-16 12:27               ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-16 10:05 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Sorry, one more without wrapping

Aug 16 13:02:00 194.146.153.146 pppd[1940]: peer from calling number 00:1E:68:0A:B8:0B authorized
Aug 16 13:02:00 194.146.153.146 pppd[1940]: local  IP address 172.16.131.1
Aug 16 13:02:00 194.146.153.146 pppd[1940]: remote IP address 172.16.131.4
Aug 16 13:02:00 194.146.153.146 ppp: User toni18 on interface ppp0 logged in
Aug 16 13:02:00 194.146.153.146 ppp: ip 172.16.131.4 = toni18
Aug 16 13:02:00 194.146.153.146 ppp: Shaper TX/RX set 56000/128000 on if ppp0 for toni18
Aug 16 13:02:00 194.146.153.146 pppshaper: Shaper DOWN for interface ppp0 will be set  bit/s with burstsize  and burstspeed 
Aug 16 13:02:00 194.146.153.146 pppshaper: Shaper RX for interface ppp0 will be set 128000 bit/s with burstsize 2048000 and burstspeed 640000
Aug 16 13:02:00 194.146.153.146 pppshaper: Shaper TX for interface ppp0 will be set 56000 bit/s with burstsize 2048000 and burstspeed 280000
Aug 16 13:02:01 194.146.153.146 [   81.476254] BUG: unable to handle kernel 
Aug 16 13:02:01 194.146.153.146 paging request
Aug 16 13:02:01 194.146.153.146 at 00002d30
Aug 16 13:02:01 194.146.153.146 [   81.476254] IP:
Aug 16 13:02:01 194.146.153.146 [<c01643bd>] kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 [   81.476254] *pde = 00000000 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.476254] Oops: 0000 [#1] 
Aug 16 13:02:01 194.146.153.146 SMP 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.476254] Modules linked in:
Aug 16 13:02:01 194.146.153.146 sch_prio
Aug 16 13:02:01 194.146.153.146 netconsole
Aug 16 13:02:01 194.146.153.146 configfs
Aug 16 13:02:01 194.146.153.146 cls_u32
Aug 16 13:02:01 194.146.153.146 em_meta
Aug 16 13:02:01 194.146.153.146 cls_basic
Aug 16 13:02:01 194.146.153.146 xt_dscp
Aug 16 13:02:01 194.146.153.146 xt_DSCP
Aug 16 13:02:01 194.146.153.146 ipt_REJECT
Aug 16 13:02:01 194.146.153.146 ts_bm
Aug 16 13:02:01 194.146.153.146 xt_string
Aug 16 13:02:01 194.146.153.146 ipt_ttl
Aug 16 13:02:01 194.146.153.146 ifb
Aug 16 13:02:01 194.146.153.146 cls_fw
Aug 16 13:02:01 194.146.153.146 sch_tbf
Aug 16 13:02:01 194.146.153.146 sch_htb
Aug 16 13:02:01 194.146.153.146 act_ipt
Aug 16 13:02:01 194.146.153.146 act_mirred
Aug 16 13:02:01 194.146.153.146 xt_MARK
Aug 16 13:02:01 194.146.153.146 pppoe
Aug 16 13:02:01 194.146.153.146 pppox
Aug 16 13:02:01 194.146.153.146 ppp_generic
Aug 16 13:02:01 194.146.153.146 slhc
Aug 16 13:02:01 194.146.153.146 ipt_MASQUERADE
Aug 16 13:02:01 194.146.153.146 xt_TCPMSS
Aug 16 13:02:01 194.146.153.146 xt_mark
Aug 16 13:02:01 194.146.153.146 xt_tcpudp
Aug 16 13:02:01 194.146.153.146 iptable_mangle
Aug 16 13:02:01 194.146.153.146 iptable_nat
Aug 16 13:02:01 194.146.153.146 nf_nat
Aug 16 13:02:01 194.146.153.146 nf_conntrack_ipv4
Aug 16 13:02:01 194.146.153.146 nf_conntrack
Aug 16 13:02:01 194.146.153.146 rtc_cmos
Aug 16 13:02:01 194.146.153.146 rtc_core
Aug 16 13:02:01 194.146.153.146 rtc_lib
Aug 16 13:02:01 194.146.153.146 iptable_filter
Aug 16 13:02:01 194.146.153.146 ip_tables
Aug 16 13:02:01 194.146.153.146 x_tables
Aug 16 13:02:01 194.146.153.146 8021q
Aug 16 13:02:01 194.146.153.146 garp
Aug 16 13:02:01 194.146.153.146 stp
Aug 16 13:02:01 194.146.153.146 llc
Aug 16 13:02:01 194.146.153.146 tun
Aug 16 13:02:01 194.146.153.146 loop
Aug 16 13:02:01 194.146.153.146 tulip
Aug 16 13:02:01 194.146.153.146 r8169
Aug 16 13:02:01 194.146.153.146 sky2
Aug 16 13:02:01 194.146.153.146 via_velocity
Aug 16 13:02:01 194.146.153.146 via_rhine
Aug 16 13:02:01 194.146.153.146 sis900
Aug 16 13:02:01 194.146.153.146 ne2k_pci
Aug 16 13:02:01 194.146.153.146 8390
Aug 16 13:02:01 194.146.153.146 skge
Aug 16 13:02:01 194.146.153.146 tg3
Aug 16 13:02:01 194.146.153.146 libphy
Aug 16 13:02:01 194.146.153.146 8139too
Aug 16 13:02:01 194.146.153.146 e1000
Aug 16 13:02:01 194.146.153.146 e100
Aug 16 13:02:01 194.146.153.146 mtdblock
Aug 16 13:02:01 194.146.153.146 mtd_blkdevs
Aug 16 13:02:01 194.146.153.146 aic79xx
Aug 16 13:02:01 194.146.153.146 mptsas
Aug 16 13:02:01 194.146.153.146 scsi_transport_sas
Aug 16 13:02:01 194.146.153.146 mptspi
Aug 16 13:02:01 194.146.153.146 mptscsih
Aug 16 13:02:01 194.146.153.146 scsi_transport_spi
Aug 16 13:02:01 194.146.153.146 mptbase
Aug 16 13:02:01 194.146.153.146 sata_via
Aug 16 13:02:01 194.146.153.146 sata_uli
Aug 16 13:02:01 194.146.153.146 sata_svw
Aug 16 13:02:01 194.146.153.146 sata_sis
Aug 16 13:02:01 194.146.153.146 sata_sil24
Aug 16 13:02:01 194.146.153.146 sata_sil
Aug 16 13:02:01 194.146.153.146 sata_mv
Aug 16 13:02:01 194.146.153.146 ahci
Aug 16 13:02:01 194.146.153.146 pata_via
Aug 16 13:02:01 194.146.153.146 pata_sis
Aug 16 13:02:01 194.146.153.146 pata_serverworks
Aug 16 13:02:01 194.146.153.146 pata_oldpiix
Aug 16 13:02:01 194.146.153.146 pata_mpiix
Aug 16 13:02:01 194.146.153.146 pata_marvell
Aug 16 13:02:01 194.146.153.146 pata_ali
Aug 16 13:02:01 194.146.153.146 ata_generic
Aug 16 13:02:01 194.146.153.146 ata_piix
Aug 16 13:02:01 194.146.153.146 libata
Aug 16 13:02:01 194.146.153.146 dock
Aug 16 13:02:01 194.146.153.146 usbhid
Aug 16 13:02:01 194.146.153.146 uhci_hcd
Aug 16 13:02:01 194.146.153.146 ehci_hcd
Aug 16 13:02:01 194.146.153.146 ohci_hcd
Aug 16 13:02:01 194.146.153.146 usbcore
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] 
Aug 16 13:02:01 194.146.153.146 [   81.480253] Pid: 1998, comm: modprobe Not tainted (2.6.27-rc3-git2-build-0031 #13)
Aug 16 13:02:01 194.146.153.146 [   81.480253] EIP: 0060:[<c01643bd>] EFLAGS: 00010002 CPU: 1
Aug 16 13:02:01 194.146.153.146 [   81.480253] EIP is at kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 [   81.480253] EAX: 00000001 EBX: f72e4380 ECX: 00000000 EDX: 00002c90
Aug 16 13:02:01 194.146.153.146 [   81.480253] ESI: c1e44744 EDI: 063d5210 EBP: c0861fa4 ESP: c0861f8c
Aug 16 13:02:01 194.146.153.146 [   81.480253]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Aug 16 13:02:01 194.146.153.146 [   81.480253] Process modprobe (pid: 1998, ti=c0861000 task=f6ce55a0 task.ti=f6d0b000)
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] Stack: 
Aug 16 13:02:01 194.146.153.146 c026e926 
Aug 16 13:02:01 194.146.153.146 00002c90 
Aug 16 13:02:01 194.146.153.146 00000286 
Aug 16 13:02:01 194.146.153.146 f72e4380 
Aug 16 13:02:01 194.146.153.146 063d5210 
Aug 16 13:02:01 194.146.153.146 f72e43bc 
Aug 16 13:02:01 194.146.153.146 c0861fb8 
Aug 16 13:02:01 194.146.153.146 c026e926 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253]        
Aug 16 13:02:01 194.146.153.146 c1d19518 
Aug 16 13:02:01 194.146.153.146 f7708688 
Aug 16 13:02:01 194.146.153.146 00000003 
Aug 16 13:02:01 194.146.153.146 c0861fcc 
Aug 16 13:02:01 194.146.153.146 c014b127 
Aug 16 13:02:01 194.146.153.146 c039daa0 
Aug 16 13:02:01 194.146.153.146 c0859554 
Aug 16 13:02:01 194.146.153.146 00000001 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253]        
Aug 16 13:02:01 194.146.153.146 c0861fd4 
Aug 16 13:02:01 194.146.153.146 c014b19e 
Aug 16 13:02:01 194.146.153.146 c0861ff8 
Aug 16 13:02:01 194.146.153.146 c0126203 
Aug 16 13:02:01 194.146.153.146 c085c700 
Aug 16 13:02:01 194.146.153.146 c085c700 
Aug 16 13:02:01 194.146.153.146 0000000a 
Aug 16 13:02:01 194.146.153.146 00000001 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] Call Trace:
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c014b127>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __rcu_process_callbacks+0x165/0x1c2
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c014b19e>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 rcu_process_callbacks+0x1a/0x33
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c0126203>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x7b/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c0126188>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x0/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c01060b3>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 do_softirq+0x65/0xb6
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c0125e28>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 irq_exit+0x44/0x79
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c010ff15>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c010438d>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 apic_timer_interrupt+0x2d/0x34
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c012007b>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 sys_unshare+0xbf/0x258
Aug 16 13:02:01 194.146.153.146 [   81.480253]  =======================
Aug 16 13:02:01 194.146.153.146 [   81.480253] Code: 
Aug 16 13:02:01 194.146.153.146 0b 
Aug 16 13:02:01 194.146.153.146 eb 
Aug 16 13:02:01 194.146.153.146 fe 
Aug 16 13:02:01 194.146.153.146 89 
Aug 16 13:02:01 194.146.153.146 f0 
Aug 16 13:02:01 194.146.153.146 e8 
Aug 16 13:02:01 194.146.153.146 0d 
Aug 16 13:02:01 194.146.153.146 f2 
Aug 16 13:02:01 194.146.153.146 fe 
Aug 16 13:02:01 194.146.153.146 ff 
Aug 16 13:02:01 194.146.153.146 eb 
Aug 16 13:02:01 194.146.153.146 78 
Aug 16 13:02:01 194.146.153.146 8b 
Aug 16 13:02:01 194.146.153.146 45 
Aug 16 13:02:01 194.146.153.146 04 
Aug 16 13:02:01 194.146.153.146 89 
Aug 16 13:02:01 194.146.153.146 45 
Aug 16 13:02:01 194.146.153.146 e8 
Aug 16 13:02:01 194.146.153.146 8b 
Aug 16 13:02:01 194.146.153.146 56 
Aug 16 13:02:01 194.146.153.146 0c 
Aug 16 13:02:01 194.146.153.146 89 
Aug 16 13:02:01 194.146.153.146 55 
Aug 16 13:02:01 194.146.153.146 ec 
Aug 16 13:02:01 194.146.153.146 9c 
Aug 16 13:02:01 194.146.153.146 8f 
Aug 16 13:02:01 194.146.153.146 45 
Aug 16 13:02:01 194.146.153.146 f0 
Aug 16 13:02:01 194.146.153.146 fa 
Aug 16 13:02:01 194.146.153.146 e8 
Aug 16 13:02:01 194.146.153.146 19 
Aug 16 13:02:01 194.146.153.146 84 
Aug 16 13:02:01 194.146.153.146 fd 
Aug 16 13:02:01 194.146.153.146 ff 
Aug 16 13:02:01 194.146.153.146 8b 
Aug 16 13:02:01 194.146.153.146 55 
Aug 16 13:02:01 194.146.153.146 ec 
Aug 16 13:02:01 194.146.153.146 64 
Aug 16 13:02:01 194.146.153.146 a1 
Aug 16 13:02:01 194.146.153.146 04 
Aug 16 13:02:01 194.146.153.146 50 
Aug 16 13:02:01 194.146.153.146 85 
Aug 16 13:02:01 194.146.153.146 c0 
Aug 16 13:02:01 194.146.153.146 unparseable log message: "<8b> "
Aug 16 13:02:01 194.146.153.146 9c 
Aug 16 13:02:01 194.146.153.146 82 
Aug 16 13:02:01 194.146.153.146 9c 
Aug 16 13:02:01 194.146.153.146 00 
Aug 16 13:02:01 194.146.153.146 00 
Aug 16 13:02:01 194.146.153.146 00 
Aug 16 13:02:01 194.146.153.146 89 
Aug 16 13:02:01 194.146.153.146 f8 
Aug 16 13:02:01 194.146.153.146 8b 
Aug 16 13:02:01 194.146.153.146 53 
Aug 16 13:02:01 194.146.153.146 10 
Aug 16 13:02:01 194.146.153.146 e8 
Aug 16 13:02:01 194.146.153.146 f3 
Aug 16 13:02:01 194.146.153.146 96 
Aug 16 13:02:01 194.146.153.146 fd 
Aug 16 13:02:01 194.146.153.146 ff 
Aug 16 13:02:01 194.146.153.146 3b 
Aug 16 13:02:01 194.146.153.146 73 
Aug 16 13:02:01 194.146.153.146 04 
Aug 16 13:02:01 194.146.153.146 75 
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] EIP: [<c01643bd>] 
Aug 16 13:02:01 194.146.153.146 kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 SS:ESP 0068:c0861f8c
Aug 16 13:02:01 194.146.153.146 [   81.480253] Kernel panic - not syncing: Fatal exception in interrupt
Aug 16 13:02:01 194.146.153.146 [   81.480253] ------------[ cut here ]------------
Aug 16 13:02:01 194.146.153.146 [   81.480253] WARNING: at kernel/smp.c:328 smp_call_function_mask+0x28/0x181()
Aug 16 13:02:01 194.146.153.146 [   81.480253] Modules linked in:
Aug 16 13:02:01 194.146.153.146 sch_prio
Aug 16 13:02:01 194.146.153.146 netconsole
Aug 16 13:02:01 194.146.153.146 configfs
Aug 16 13:02:01 194.146.153.146 cls_u32
Aug 16 13:02:01 194.146.153.146 em_meta
Aug 16 13:02:01 194.146.153.146 cls_basic
Aug 16 13:02:01 194.146.153.146 xt_dscp
Aug 16 13:02:01 194.146.153.146 xt_DSCP
Aug 16 13:02:01 194.146.153.146 ipt_REJECT
Aug 16 13:02:01 194.146.153.146 ts_bm
Aug 16 13:02:01 194.146.153.146 xt_string
Aug 16 13:02:01 194.146.153.146 ipt_ttl
Aug 16 13:02:01 194.146.153.146 ifb
Aug 16 13:02:01 194.146.153.146 cls_fw
Aug 16 13:02:01 194.146.153.146 sch_tbf
Aug 16 13:02:01 194.146.153.146 sch_htb
Aug 16 13:02:01 194.146.153.146 act_ipt
Aug 16 13:02:01 194.146.153.146 act_mirred
Aug 16 13:02:01 194.146.153.146 xt_MARK
Aug 16 13:02:01 194.146.153.146 pppoe
Aug 16 13:02:01 194.146.153.146 pppox
Aug 16 13:02:01 194.146.153.146 ppp_generic
Aug 16 13:02:01 194.146.153.146 slhc
Aug 16 13:02:01 194.146.153.146 ipt_MASQUERADE
Aug 16 13:02:01 194.146.153.146 xt_TCPMSS
Aug 16 13:02:01 194.146.153.146 xt_mark
Aug 16 13:02:01 194.146.153.146 xt_tcpudp
Aug 16 13:02:01 194.146.153.146 iptable_mangle
Aug 16 13:02:01 194.146.153.146 iptable_nat
Aug 16 13:02:01 194.146.153.146 nf_nat
Aug 16 13:02:01 194.146.153.146 nf_conntrack_ipv4
Aug 16 13:02:01 194.146.153.146 nf_conntrack
Aug 16 13:02:01 194.146.153.146 rtc_cmos
Aug 16 13:02:01 194.146.153.146 rtc_core
Aug 16 13:02:01 194.146.153.146 rtc_lib
Aug 16 13:02:01 194.146.153.146 iptable_filter
Aug 16 13:02:01 194.146.153.146 ip_tables
Aug 16 13:02:01 194.146.153.146 x_tables
Aug 16 13:02:01 194.146.153.146 8021q
Aug 16 13:02:01 194.146.153.146 garp
Aug 16 13:02:01 194.146.153.146 stp
Aug 16 13:02:01 194.146.153.146 llc
Aug 16 13:02:01 194.146.153.146 tun
Aug 16 13:02:01 194.146.153.146 loop
Aug 16 13:02:01 194.146.153.146 tulip
Aug 16 13:02:01 194.146.153.146 r8169
Aug 16 13:02:01 194.146.153.146 sky2
Aug 16 13:02:01 194.146.153.146 via_velocity
Aug 16 13:02:01 194.146.153.146 via_rhine
Aug 16 13:02:01 194.146.153.146 sis900
Aug 16 13:02:01 194.146.153.146 ne2k_pci
Aug 16 13:02:01 194.146.153.146 8390
Aug 16 13:02:01 194.146.153.146 skge
Aug 16 13:02:01 194.146.153.146 tg3
Aug 16 13:02:01 194.146.153.146 libphy
Aug 16 13:02:01 194.146.153.146 8139too
Aug 16 13:02:01 194.146.153.146 e1000
Aug 16 13:02:01 194.146.153.146 e100
Aug 16 13:02:01 194.146.153.146 mtdblock
Aug 16 13:02:01 194.146.153.146 mtd_blkdevs
Aug 16 13:02:01 194.146.153.146 aic79xx
Aug 16 13:02:01 194.146.153.146 mptsas
Aug 16 13:02:01 194.146.153.146 scsi_transport_sas
Aug 16 13:02:01 194.146.153.146 mptspi
Aug 16 13:02:01 194.146.153.146 mptscsih
Aug 16 13:02:01 194.146.153.146 scsi_transport_spi
Aug 16 13:02:01 194.146.153.146 mptbase
Aug 16 13:02:01 194.146.153.146 sata_via
Aug 16 13:02:01 194.146.153.146 sata_uli
Aug 16 13:02:01 194.146.153.146 sata_svw
Aug 16 13:02:01 194.146.153.146 sata_sis
Aug 16 13:02:01 194.146.153.146 sata_sil24
Aug 16 13:02:01 194.146.153.146 sata_sil
Aug 16 13:02:01 194.146.153.146 sata_mv
Aug 16 13:02:01 194.146.153.146 ahci
Aug 16 13:02:01 194.146.153.146 pata_via
Aug 16 13:02:01 194.146.153.146 pata_sis
Aug 16 13:02:01 194.146.153.146 pata_serverworks
Aug 16 13:02:01 194.146.153.146 pata_oldpiix
Aug 16 13:02:01 194.146.153.146 pata_mpiix
Aug 16 13:02:01 194.146.153.146 pata_marvell
Aug 16 13:02:01 194.146.153.146 pata_ali
Aug 16 13:02:01 194.146.153.146 ata_generic
Aug 16 13:02:01 194.146.153.146 ata_piix
Aug 16 13:02:01 194.146.153.146 libata
Aug 16 13:02:01 194.146.153.146 dock
Aug 16 13:02:01 194.146.153.146 usbhid
Aug 16 13:02:01 194.146.153.146 uhci_hcd
Aug 16 13:02:01 194.146.153.146 ehci_hcd
Aug 16 13:02:01 194.146.153.146 ohci_hcd
Aug 16 13:02:01 194.146.153.146 usbcore
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480253] Pid: 1998, comm: modprobe Tainted: G      D   2.6.27-rc3-git2-build-0031 #13
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c02bb7e7>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 printk+0xf/0x18
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c0121be8>] 
Aug 16 13:02:01 194.146.153.146 warn_on_slowpath+0x41/0x65
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<f896b649>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 rtl8139_start_xmit+0xe6/0x10c [8139too]
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c026caf9>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 netpoll_send_skb+0xdc/0x15b
Aug 16 13:02:01 194.146.153.146 [   81.480253]  [<c026cd49>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 netpoll_send_udp+0x1d1/0x1dc
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bdbe8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 _spin_unlock_irqrestore+0x2f/0x3c
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f8a98178>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 write_msg+0xb1/0xb9 [netconsole]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01430b4>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function_mask+0x28/0x181
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ef59>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 stop_this_cpu+0x0/0x3b
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f89a0225>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 velocity_init_registers+0x2c6/0x631 [via_velocity]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f89a0020>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 velocity_init_registers+0xc1/0x631 [via_velocity]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0160020>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 shmem_truncate_range+0x3f8/0x610
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014321f>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function+0x12/0x14
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010efaf>] 
Aug 16 13:02:01 194.146.153.146 native_smp_send_stop+0x1b/0x40
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bb740>] 
Aug 16 13:02:01 194.146.153.146 panic+0x48/0xe0
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0104945>] 
Aug 16 13:02:01 194.146.153.146 oops_end+0x73/0x87
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0105138>] 
Aug 16 13:02:01 194.146.153.146 die+0x5b/0x63
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0114512>] 
Aug 16 13:02:01 194.146.153.146 do_page_fault+0x532/0x624
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0104ad4>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 dump_trace+0xc5/0xe5
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ab8e>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 save_stack_address+0x0/0x2c
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013d566>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 mark_lock+0x495/0x7f8
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013e775>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __lock_acquire+0x59d/0x118a
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0113fe0>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 do_page_fault+0x0/0x624
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02be052>] 
Aug 16 13:02:01 194.146.153.146 error_code+0x72/0x78
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01643bd>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014b127>] 
Aug 16 13:02:01 194.146.153.146 __rcu_process_callbacks+0x165/0x1c2
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014b19e>] 
Aug 16 13:02:01 194.146.153.146 rcu_process_callbacks+0x1a/0x33
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0126203>] 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x7b/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0126188>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x0/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01060b3>] 
Aug 16 13:02:01 194.146.153.146 do_softirq+0x65/0xb6
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0125e28>] 
Aug 16 13:02:01 194.146.153.146 irq_exit+0x44/0x79
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ff15>] 
Aug 16 13:02:01 194.146.153.146 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010438d>] 
Aug 16 13:02:01 194.146.153.146 apic_timer_interrupt+0x2d/0x34
Aug 16 13:02:01 194.146.153.146 [   81.480254
Aug 16 13:02:01 194.146.153.146 ]  [<c012007b>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 sys_unshare+0xbf/0x258
Aug 16 13:02:01 194.146.153.146 [   81.480254]  =======================
Aug 16 13:02:01 194.146.153.146 [   81.480254] ---[ end trace bc171cacb91df283 ]---
Aug 16 13:02:01 194.146.153.146 [   81.480254] ------------[ cut here ]------------
Aug 16 13:02:01 194.146.153.146 [   81.480254] WARNING: at kernel/smp.c:217 smp_call_function_single+0x2d/0x9d()
Aug 16 13:02:01 194.146.153.146 [   81.480254] Modules linked in:
Aug 16 13:02:01 194.146.153.146 sch_prio
Aug 16 13:02:01 194.146.153.146 netconsole
Aug 16 13:02:01 194.146.153.146 configfs
Aug 16 13:02:01 194.146.153.146 cls_u32
Aug 16 13:02:01 194.146.153.146 em_meta
Aug 16 13:02:01 194.146.153.146 cls_basic
Aug 16 13:02:01 194.146.153.146 xt_dscp
Aug 16 13:02:01 194.146.153.146 xt_DSCP
Aug 16 13:02:01 194.146.153.146 ipt_REJECT
Aug 16 13:02:01 194.146.153.146 ts_bm
Aug 16 13:02:01 194.146.153.146 xt_string
Aug 16 13:02:01 194.146.153.146 ipt_ttl
Aug 16 13:02:01 194.146.153.146 ifb
Aug 16 13:02:01 194.146.153.146 cls_fw
Aug 16 13:02:01 194.146.153.146 sch_tbf
Aug 16 13:02:01 194.146.153.146 sch_htb
Aug 16 13:02:01 194.146.153.146 act_ipt
Aug 16 13:02:01 194.146.153.146 act_mirred
Aug 16 13:02:01 194.146.153.146 xt_MARK
Aug 16 13:02:01 194.146.153.146 pppoe
Aug 16 13:02:01 194.146.153.146 pppox
Aug 16 13:02:01 194.146.153.146 ppp_generic
Aug 16 13:02:01 194.146.153.146 slhc
Aug 16 13:02:01 194.146.153.146 ipt_MASQUERADE
Aug 16 13:02:01 194.146.153.146 xt_TCPMSS
Aug 16 13:02:01 194.146.153.146 xt_mark
Aug 16 13:02:01 194.146.153.146 xt_tcpudp
Aug 16 13:02:01 194.146.153.146 iptable_mangle
Aug 16 13:02:01 194.146.153.146 iptable_nat
Aug 16 13:02:01 194.146.153.146 nf_nat
Aug 16 13:02:01 194.146.153.146 nf_conntrack_ipv4
Aug 16 13:02:01 194.146.153.146 nf_conntrack
Aug 16 13:02:01 194.146.153.146 rtc_cmos
Aug 16 13:02:01 194.146.153.146 rtc_core
Aug 16 13:02:01 194.146.153.146 rtc_lib
Aug 16 13:02:01 194.146.153.146 iptable_filter
Aug 16 13:02:01 194.146.153.146 ip_tables
Aug 16 13:02:01 194.146.153.146 x_tables
Aug 16 13:02:01 194.146.153.146 8021q
Aug 16 13:02:01 194.146.153.146 garp
Aug 16 13:02:01 194.146.153.146 stp
Aug 16 13:02:01 194.146.153.146 llc
Aug 16 13:02:01 194.146.153.146 tun
Aug 16 13:02:01 194.146.153.146 loop
Aug 16 13:02:01 194.146.153.146 tulip
Aug 16 13:02:01 194.146.153.146 r8169
Aug 16 13:02:01 194.146.153.146 sky2
Aug 16 13:02:01 194.146.153.146 via_velocity
Aug 16 13:02:01 194.146.153.146 via_rhine
Aug 16 13:02:01 194.146.153.146 sis900
Aug 16 13:02:01 194.146.153.146 ne2k_pci
Aug 16 13:02:01 194.146.153.146 8390
Aug 16 13:02:01 194.146.153.146 skge
Aug 16 13:02:01 194.146.153.146 tg3
Aug 16 13:02:01 194.146.153.146 libphy
Aug 16 13:02:01 194.146.153.146 8139too
Aug 16 13:02:01 194.146.153.146 e1000
Aug 16 13:02:01 194.146.153.146 e100
Aug 16 13:02:01 194.146.153.146 mtdblock
Aug 16 13:02:01 194.146.153.146 mtd_blkdevs
Aug 16 13:02:01 194.146.153.146 aic79xx
Aug 16 13:02:01 194.146.153.146 mptsas
Aug 16 13:02:01 194.146.153.146 scsi_transport_sas
Aug 16 13:02:01 194.146.153.146 mptspi
Aug 16 13:02:01 194.146.153.146 mptscsih
Aug 16 13:02:01 194.146.153.146 scsi_transport_spi
Aug 16 13:02:01 194.146.153.146 mptbase
Aug 16 13:02:01 194.146.153.146 sata_via
Aug 16 13:02:01 194.146.153.146 sata_uli
Aug 16 13:02:01 194.146.153.146 sata_svw
Aug 16 13:02:01 194.146.153.146 sata_sis
Aug 16 13:02:01 194.146.153.146 sata_sil24
Aug 16 13:02:01 194.146.153.146 sata_sil
Aug 16 13:02:01 194.146.153.146 sata_mv
Aug 16 13:02:01 194.146.153.146 ahci
Aug 16 13:02:01 194.146.153.146 pata_via
Aug 16 13:02:01 194.146.153.146 pata_sis
Aug 16 13:02:01 194.146.153.146 pata_serverworks
Aug 16 13:02:01 194.146.153.146 pata_oldpiix
Aug 16 13:02:01 194.146.153.146 pata_mpiix
Aug 16 13:02:01 194.146.153.146 pata_marvell
Aug 16 13:02:01 194.146.153.146 pata_ali
Aug 16 13:02:01 194.146.153.146 ata_generic
Aug 16 13:02:01 194.146.153.146 ata_piix
Aug 16 13:02:01 194.146.153.146 libata
Aug 16 13:02:01 194.146.153.146 dock
Aug 16 13:02:01 194.146.153.146 usbhid
Aug 16 13:02:01 194.146.153.146 uhci_hcd
Aug 16 13:02:01 194.146.153.146 ehci_hcd
Aug 16 13:02:01 194.146.153.146 ohci_hcd
Aug 16 13:02:01 194.146.153.146 usbcore
Aug 16 13:02:01 194.146.153.146 
Aug 16 13:02:01 194.146.153.146 [   81.480254] Pid: 1998, comm: modprobe Tainted: G      D W 2.6.27-rc3-git2-build-0031 #13
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bb7e7>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 printk+0xf/0x18
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0121be8>] 
Aug 16 13:02:01 194.146.153.146 warn_on_slowpath+0x41/0x65
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f896b649>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 rtl8139_start_xmit+0xe6/0x10c [8139too]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 netpoll_send_udp+0x1d1/0x1dc
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026caf9>] 
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013c7d8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 trace_hardirqs_off+0xb/0xd
Aug 16 13:02:01 194.146.153.146 netpoll_send_skb+0xdc/0x15b
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026cd49>] 
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bdbe8>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 _spin_unlock_irqrestore+0x2f/0x3c
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f8a98178>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 write_msg+0xb1/0xb9 [netconsole]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014301c>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function_single+0x2d/0x9d
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ef59>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 stop_this_cpu+0x0/0x3b
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0143103>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function_mask+0x77/0x181
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ef59>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 stop_this_cpu+0x0/0x3b
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f89a0225>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 velocity_init_registers+0x2c6/0x631 [via_velocity]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<f89a0020>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 velocity_init_registers+0xc1/0x631 [via_velocity]
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014321f>] 
Aug 16 13:02:01 194.146.153.146 smp_call_function+0x12/0x14
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010efaf>] 
Aug 16 13:02:01 194.146.153.146 native_smp_send_stop+0x1b/0x40
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02bb740>] 
Aug 16 13:02:01 194.146.153.146 panic+0x48/0xe0
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0104945>] 
Aug 16 13:02:01 194.146.153.146 oops_end+0x73/0x87
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0105138>] 
Aug 16 13:02:01 194.146.153.146 die+0x5b/0x63
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0114512>] 
Aug 16 13:02:01 194.146.153.146 do_page_fault+0x532/0x624
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0104ad4>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 dump_trace+0xc5/0xe5
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ab8e>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 save_stack_address+0x0/0x2c
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013d566>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 mark_lock+0x495/0x7f8
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c013e775>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __lock_acquire+0x59d/0x118a
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0113fe0>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 do_page_fault+0x0/0x624
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c02be052>] 
Aug 16 13:02:01 194.146.153.146 error_code+0x72/0x78
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01643bd>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 kfree+0x67/0xc8
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c026e926>] 
Aug 16 13:02:01 194.146.153.146 __qdisc_destroy+0x70/0x75
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014b127>] 
Aug 16 13:02:01 194.146.153.146 __rcu_process_callbacks+0x165/0x1c2
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c014b19e>] 
Aug 16 13:02:01 194.146.153.146 rcu_process_callbacks+0x1a/0x33
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0126203>] 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x7b/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0126188>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 __do_softirq+0x0/0xf4
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c01060b3>] 
Aug 16 13:02:01 194.146.153.146 do_softirq+0x65/0xb6
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c0125e28>] 
Aug 16 13:02:01 194.146.153.146 irq_exit+0x44/0x79
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010ff15>] 
Aug 16 13:02:01 194.146.153.146 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c010438d>] 
Aug 16 13:02:01 194.146.153.146 apic_timer_interrupt+0x2d/0x34
Aug 16 13:02:01 194.146.153.146 [   81.480254]  [<c012007b>] 
Aug 16 13:02:01 194.146.153.146 ? 
Aug 16 13:02:01 194.146.153.146 sys_unshare+0xbf/0x258
Aug 16 13:02:01 194.146.153.146 [   81.480254]  =======================
Aug 16 13:02:01 194.146.153.146 [   81.480254] ---[ end trace bc171cacb91df283 ]---
Aug 16 13:02:01 194.146.153.146 [   81.480254] Rebooting in 5 seconds..

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 12:27               ` Jarek Poplawski
@ 2008-08-16 12:24                 ` Denys Fedoryshchenko
  2008-08-16 12:34                   ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-16 12:24 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Do you want me to reverse previous one?

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 10:05             ` Denys Fedoryshchenko
@ 2008-08-16 12:27               ` Jarek Poplawski
  2008-08-16 12:24                 ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-16 12:27 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sat, Aug 16, 2008 at 01:05:46PM +0300, Denys Fedoryshchenko wrote:
> Sorry, one more without wrapping

So this patch proved without wrapping that I'm a moron!

Here is take 3, I hope more useful.

Sorry,
Jarek P.

---

 include/linux/skbuff.h  |   44 ++++++++++++++++++++++++++++++++------------
 net/sched/sch_generic.c |    5 ++++-
 2 files changed, 36 insertions(+), 13 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 358661c..506142e 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -120,6 +120,7 @@ struct sk_buff_head {
 
 	__u32		qlen;
 	spinlock_t	lock;
+	spinlock_t	lock_debug;
 };
 
 struct sk_buff;
@@ -657,6 +658,7 @@ static inline __u32 skb_queue_len(const struct sk_buff_head *list_)
 static inline void skb_queue_head_init(struct sk_buff_head *list)
 {
 	spin_lock_init(&list->lock);
+	spin_lock_init(&list->lock_debug);
 	list->prev = list->next = (struct sk_buff *)list;
 	list->qlen = 0;
 }
@@ -679,10 +681,16 @@ static inline void __skb_insert(struct sk_buff *newsk,
 				struct sk_buff *prev, struct sk_buff *next,
 				struct sk_buff_head *list)
 {
-	newsk->next = next;
-	newsk->prev = prev;
-	next->prev  = prev->next = newsk;
-	list->qlen++;
+	if (spin_trylock(&list->lock_debug)) {
+		newsk->next = next;
+		newsk->prev = prev;
+		next->prev  = prev->next = newsk;
+		list->qlen++;
+		spin_unlock(&list->lock_debug);
+	} else {
+		kfree_skb(newsk);
+		WARN_ON(1);
+	}
 }
 
 /**
@@ -775,10 +783,16 @@ static inline void __skb_unlink(struct sk_buff *skb, struct sk_buff_head *list)
 extern struct sk_buff *skb_dequeue(struct sk_buff_head *list);
 static inline struct sk_buff *__skb_dequeue(struct sk_buff_head *list)
 {
-	struct sk_buff *skb = skb_peek(list);
-	if (skb)
-		__skb_unlink(skb, list);
-	return skb;
+	if (spin_trylock(&list->lock_debug)) {
+		struct sk_buff *skb = skb_peek(list);
+		if (skb)
+			__skb_unlink(skb, list);
+		spin_unlock(&list->lock_debug);
+		return skb;
+	} else {
+		WARN_ON(1);
+		return NULL;
+	}
 }
 
 /**
@@ -792,10 +806,16 @@ static inline struct sk_buff *__skb_dequeue(struct sk_buff_head *list)
 extern struct sk_buff *skb_dequeue_tail(struct sk_buff_head *list);
 static inline struct sk_buff *__skb_dequeue_tail(struct sk_buff_head *list)
 {
-	struct sk_buff *skb = skb_peek_tail(list);
-	if (skb)
-		__skb_unlink(skb, list);
-	return skb;
+	if (spin_trylock(&list->lock_debug)) {
+		struct sk_buff *skb = skb_peek_tail(list);
+		if (skb)
+			__skb_unlink(skb, list);
+		spin_unlock(&list->lock_debug);
+		return skb;
+	} else {
+		WARN_ON(1);
+		return NULL;
+	}
 }
 
 
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 4685746..065a8b0 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -525,6 +525,7 @@ static void __qdisc_destroy(struct rcu_head *head)
 {
 	struct Qdisc *qdisc = container_of(head, struct Qdisc, q_rcu);
 	const struct Qdisc_ops  *ops = qdisc->ops;
+	int p;
 
 #ifdef CONFIG_NET_SCHED
 	qdisc_put_stab(qdisc->stab);
@@ -540,7 +541,9 @@ static void __qdisc_destroy(struct rcu_head *head)
 
 	kfree_skb(qdisc->gso_skb);
 
-	kfree((char *) qdisc - qdisc->padded);
+	p = qdisc->padded;
+	memset(qdisc, 0xf0 , sizeof(*qdisc));
+	kfree((char *) qdisc - p);
 }
 
 /* Under qdisc_lock(qdisc) and BH! */

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 12:24                 ` Denys Fedoryshchenko
@ 2008-08-16 12:34                   ` Jarek Poplawski
  2008-08-16 13:13                     ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-16 12:34 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sat, Aug 16, 2008 at 03:24:30PM +0300, Denys Fedoryshchenko wrote:
> Do you want me to reverse previous one?

Yes, please. I forgot to mention this.

Jarek P.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 12:34                   ` Jarek Poplawski
@ 2008-08-16 13:13                     ` Denys Fedoryshchenko
  2008-08-16 14:27                       ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-16 13:13 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

On Saturday 16 August 2008, Jarek Poplawski wrote:
> On Sat, Aug 16, 2008 at 03:24:30PM +0300, Denys Fedoryshchenko wrote:
> > Do you want me to reverse previous one?
>
> Yes, please. I forgot to mention this.
>
> Jarek P.

Here is what i got:
Aug 16 16:07:33 192.168.20.66 kernel: [   61.930066] ------------[ cut here ]------------
Aug 16 16:07:33 192.168.20.66 kernel: [   61.930066] WARNING: at include/linux/skbuff.h:692 qdisc_requeue+0x56/0x6e()
Aug 16 16:07:33 192.168.20.66 kernel: [   61.930066] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:07:33 192.168.20.66 kernel: [   61.934080] Pid: 6317, comm: modprobe Not tainted 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:07:33 192.168.20.66 kernel: [   61.934080]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:07:33 192.168.20.66 kernel: [   61.934080]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:07:33 192.168.20.66 kernel: [   61.934080]  [<c02bdc12>] ? _spin_unlock+0x1d/0x20
Aug 16 16:07:33 192.168.20.66 kernel: [   61.935017]  [<c013d91c>] ? mark_held_locks+0x53/0x6a
Aug 16 16:07:33 192.168.20.66 kernel: [   61.935069]  [<c013d91c>] ? mark_held_locks+0x53/0x6a
Aug 16 16:07:33 192.168.20.66 kernel: [   61.935069]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:07:33 192.168.20.66 kernel: [   61.935069]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:07:33 192.168.20.66 kernel: [   61.935069]  [<c013da7d>] ? trace_hardirqs_on_caller+0xcb/0x102
Aug 16 16:07:33 192.168.20.66 kernel: [   61.935069]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:07:33 192.168.20.66 kernel: [   61.936074]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:07:33 192.168.20.66 kernel: [   61.936074]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:07:33 192.168.20.66 kernel: [   61.936074]  [<c0272dbd>] qdisc_requeue+0x56/0x6e
Aug 16 16:07:33 192.168.20.66 kernel: [   61.936074]  [<f8a10689>] tbf_dequeue+0x166/0x19b [sch_tbf]
Aug 16 16:07:33 192.168.20.66 kernel: [   61.936074]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:07:33 192.168.20.66 kernel: [   61.937008]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:07:33 192.168.20.66 kernel: [   61.937066]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:07:33 192.168.20.66 kernel: [   61.937066]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:07:33 192.168.20.66 kernel: [   61.937066]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:07:33 192.168.20.66 kernel: [   61.937066]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:07:33 192.168.20.66 kernel: [   61.937066]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:07:33 192.168.20.66 kernel: [   61.937066]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:07:33 192.168.20.66 kernel: [   61.938072]  [<c01400d8>] ? lockdep_stats_show+0x2e0/0x357
Aug 16 16:07:33 192.168.20.66 kernel: [   61.938123]  [<c01dd719>] ? __copy_to_user_ll+0x50/0xd7
Aug 16 16:07:33 192.168.20.66 kernel: [   61.938123]  [<c014c03d>] file_read_actor+0x65/0xbf
Aug 16 16:07:33 192.168.20.66 kernel: [   61.938123]  [<c014df8e>] generic_file_aio_read+0x2d1/0x4fe
Aug 16 16:07:33 192.168.20.66 kernel: [   61.938123]  [<c0166f71>] do_sync_read+0xab/0xe9
Aug 16 16:07:33 192.168.20.66 kernel: [   61.938123]  [<c0132301>] ? autoremove_wake_function+0x0/0x33
Aug 16 16:07:33 192.168.20.66 kernel: [   61.939066]  [<c013da93>] ? trace_hardirqs_on_caller+0xe1/0x102
Aug 16 16:07:33 192.168.20.66 kernel: [   61.939066]  [<c02bcd89>] ? mutex_unlock+0x8/0xa
Aug 16 16:07:33 192.168.20.66 kernel: [   61.939066]  [<c0166ec6>] ? do_sync_read+0x0/0xe9
Aug 16 16:07:33 192.168.20.66 kernel: [   61.939066]  [<c01677c6>] vfs_read+0x83/0xed
Aug 16 16:07:33 192.168.20.66 kernel: [   61.939066]  [<c01678c9>] sys_read+0x3b/0x60
Aug 16 16:07:33 192.168.20.66 kernel: [   61.940071]  [<c0103861>] sysenter_do_call+0x12/0x35
Aug 16 16:07:33 192.168.20.66 kernel: [   61.940072]  [<c012007b>] ? sys_unshare+0xbf/0x258
Aug 16 16:07:33 192.168.20.66 kernel: [   61.940072]  =======================
Aug 16 16:07:33 192.168.20.66 kernel: [   61.940072] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:08:42 192.168.20.66 kernel: [  130.183597] ------------[ cut here ]------------
Aug 16 16:08:42 192.168.20.66 kernel: [  130.183597] WARNING: at include/linux/skbuff.h:692 bfifo_enqueue+0x66/0xb9()
Aug 16 16:08:42 192.168.20.66 kernel: [  130.183597] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c026a00f>] ? inet_proto_csum_replace4+0x33/0xab
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c025c3be>] ? skb_release_data+0x8e/0x92
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0272c61>] bfifo_enqueue+0x66/0xb9
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<f8a1071a>] tbf_enqueue+0x5c/0x92 [sch_tbf]
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<f8a5275a>] prio_enqueue+0x44/0x82 [sch_prio]
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0261b1c>] dev_queue_xmit+0x339/0x481
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c02804fa>] ip_finish_output+0x1df/0x209
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c02805a2>] ip_output+0x7e/0x83
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c027d1e7>] ip_forward_finish+0x2f/0x32
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c027d3f1>] ip_forward+0x207/0x256
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c027c1ce>] ip_rcv_finish+0x2a6/0x2bc
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c027c3d1>] ip_rcv+0x1ed/0x217
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0261169>] netif_receive_skb+0x3c3/0x3f6
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c026df14>] ? eth_type_trans+0x28/0xcc
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<f890f2f1>] e1000_clean_rx_irq+0x2b2/0x362 [e1000]
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<f89110ac>] e1000_clean+0x2d0/0x474 [e1000]
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c02bdcc0>] ? _spin_lock+0x22/0x2a
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c025f9de>] net_rx_action+0xbb/0x1e5
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c014a35d>] ? handle_fasteoi_irq+0x0/0xb6
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0106038>] do_IRQ+0xae/0xc4
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0104288>] common_interrupt+0x28/0x30
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  [<c02b1016>] rest_init+0x4e/0x50
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006]  =======================
Aug 16 16:08:42 192.168.20.66 kernel: [  130.185006] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:08:58 192.168.20.66 kernel: [  146.592004] ------------[ cut here ]------------
Aug 16 16:08:58 192.168.20.66 kernel: [  146.592004] WARNING: at include/linux/skbuff.h:793 qdisc_dequeue_head+0x5e/0x72()
Aug 16 16:08:58 192.168.20.66 kernel: [  146.592004] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c0250032>] ? hid_parse_report+0x1f/0x23d
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<f890ef35>] ? e1000_alloc_rx_buffers+0x16d/0x277 [e1000]
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<f890f379>] ? e1000_clean_rx_irq+0x33a/0x362 [e1000]
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c0272eb2>] qdisc_dequeue_head+0x5e/0x72
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<f8a1053f>] tbf_dequeue+0x1c/0x19b [sch_tbf]
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  [<c02b9b45>] start_secondary+0x184/0x189
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004]  =======================
Aug 16 16:08:58 192.168.20.66 kernel: [  146.594004] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:09:03 192.168.20.66 kernel: [  151.431066] ------------[ cut here ]------------
Aug 16 16:09:03 192.168.20.66 kernel: [  151.431066] WARNING: at include/linux/skbuff.h:692 bfifo_enqueue+0x66/0xb9()
Aug 16 16:09:03 192.168.20.66 kernel: [  151.431066] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:09:03 192.168.20.66 kernel: [  151.435005] Pid: 15973, comm: modprobe Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:09:03 192.168.20.66 kernel: [  151.435005]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:09:03 192.168.20.66 kernel: [  151.435005]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:09:03 192.168.20.66 kernel: [  151.435005]  [<c026a00f>] ? inet_proto_csum_replace4+0x33/0xab
Aug 16 16:09:03 192.168.20.66 kernel: [  151.436010]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:09:03 192.168.20.66 kernel: [  151.436040]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:09:03 192.168.20.66 kernel: [  151.436040]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:09:03 192.168.20.66 kernel: [  151.436040]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:09:03 192.168.20.66 kernel: [  151.436040]  [<c0272c61>] bfifo_enqueue+0x66/0xb9
Aug 16 16:09:03 192.168.20.66 kernel: [  151.436040]  [<f8a1071a>] tbf_enqueue+0x5c/0x92 [sch_tbf]
Aug 16 16:09:03 192.168.20.66 kernel: [  151.437013]  [<f8a5275a>] prio_enqueue+0x44/0x82 [sch_prio]
Aug 16 16:09:03 192.168.20.66 kernel: [  151.437013]  [<c0261b1c>] dev_queue_xmit+0x339/0x481
Aug 16 16:09:03 192.168.20.66 kernel: [  151.437013]  [<c02804fa>] ip_finish_output+0x1df/0x209
Aug 16 16:09:03 192.168.20.66 kernel: [  151.437013]  [<c02805a2>] ip_output+0x7e/0x83
Aug 16 16:09:03 192.168.20.66 kernel: [  151.437013]  [<c027d1e7>] ip_forward_finish+0x2f/0x32
Aug 16 16:09:03 192.168.20.66 kernel: [  151.437013]  [<c027d3f1>] ip_forward+0x207/0x256
Aug 16 16:09:03 192.168.20.66 kernel: [  151.438017]  [<c027c1ce>] ip_rcv_finish+0x2a6/0x2bc
Aug 16 16:09:03 192.168.20.66 kernel: [  151.438026]  [<c027c3d1>] ip_rcv+0x1ed/0x217
Aug 16 16:09:03 192.168.20.66 kernel: [  151.438026]  [<c0261169>] netif_receive_skb+0x3c3/0x3f6
Aug 16 16:09:03 192.168.20.66 kernel: [  151.438026]  [<c026df14>] ? eth_type_trans+0x28/0xcc
Aug 16 16:09:03 192.168.20.66 kernel: [  151.438026]  [<f890f2f1>] e1000_clean_rx_irq+0x2b2/0x362 [e1000]
Aug 16 16:09:03 192.168.20.66 kernel: [  151.438026]  [<f89110ac>] e1000_clean+0x2d0/0x474 [e1000]
Aug 16 16:09:03 192.168.20.66 kernel: [  151.439023]  [<c02bdc12>] ? _spin_unlock+0x1d/0x20
Aug 16 16:09:03 192.168.20.66 kernel: [  151.439023]  [<c02bdcc0>] ? _spin_lock+0x22/0x2a
Aug 16 16:09:03 192.168.20.66 kernel: [  151.439023]  [<c025f9de>] net_rx_action+0xbb/0x1e5
Aug 16 16:09:03 192.168.20.66 kernel: [  151.439023]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:09:03 192.168.20.66 kernel: [  151.439023]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:09:03 192.168.20.66 kernel: [  151.439023]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:09:03 192.168.20.66 kernel: [  151.440028]  [<c014a35d>] ? handle_fasteoi_irq+0x0/0xb6
Aug 16 16:09:03 192.168.20.66 kernel: [  151.440053]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:09:03 192.168.20.66 kernel: [  151.440053]  [<c0106038>] do_IRQ+0xae/0xc4
Aug 16 16:09:03 192.168.20.66 kernel: [  151.440053]  [<c0104288>] common_interrupt+0x28/0x30
Aug 16 16:09:03 192.168.20.66 kernel: [  151.440053]  [<c01400d8>] ? lockdep_stats_show+0x2e0/0x357
Aug 16 16:09:03 192.168.20.66 kernel: [  151.440053]  [<c01dd728>] ? __copy_to_user_ll+0x5f/0xd7
Aug 16 16:09:03 192.168.20.66 kernel: [  151.441058]  [<c014c03d>] file_read_actor+0x65/0xbf
Aug 16 16:09:03 192.168.20.66 kernel: [  151.441080]  [<c014df8e>] generic_file_aio_read+0x2d1/0x4fe
Aug 16 16:09:03 192.168.20.66 kernel: [  151.441080]  [<c0166f71>] do_sync_read+0xab/0xe9
Aug 16 16:09:03 192.168.20.66 kernel: [  151.441080]  [<c0132301>] ? autoremove_wake_function+0x0/0x33
Aug 16 16:09:03 192.168.20.66 kernel: [  151.441080]  [<c02bcd89>] ? mutex_unlock+0x8/0xa
Aug 16 16:09:03 192.168.20.66 kernel: [  151.441080]  [<c0166ec6>] ? do_sync_read+0x0/0xe9
Aug 16 16:09:03 192.168.20.66 kernel: [  151.442013]  [<c01677c6>] vfs_read+0x83/0xed
Aug 16 16:09:03 192.168.20.66 kernel: [  151.442013]  [<c01678c9>] sys_read+0x3b/0x60
Aug 16 16:09:03 192.168.20.66 kernel: [  151.442013]  [<c0103861>] sysenter_do_call+0x12/0x35
Aug 16 16:09:03 192.168.20.66 kernel: [  151.442013]  [<c012007b>] ? sys_unshare+0xbf/0x258
Aug 16 16:09:03 192.168.20.66 kernel: [  151.442013]  =======================
Aug 16 16:09:03 192.168.20.66 kernel: [  151.442013] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:09:33 192.168.20.66 kernel_read+0x34/0x46
Aug 16 16:09:33 192.168.20.66 kernel: [  181.262068] ------------[ cut here ]------------
Aug 16 16:09:33 192.168.20.66 kernel: [  181.262068] WARNING: at include/linux/skbuff.h:692 qdisc_requeue+0x56/0x6e()
Aug 16 16:09:33 192.168.20.66 kernel: [  181.263013] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:09:33 192.168.20.66 kernel: [  181.266061] Pid: 18331, comm: sh Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:09:33 192.168.20.66 kernel: [  181.267067]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:09:33 192.168.20.66 kernel: [  181.267091]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:09:33 192.168.20.66 kernel: [  181.267091]  [<c0261c31>] ? dev_queue_xmit+0x44e/0x481
Aug 16 16:09:33 192.168.20.66 kernel: [  181.267091]  [<c026e00a>] ? eth_header+0x0/0x8e
Aug 16 16:09:33 192.168.20.66 kernel: [  181.267091]  [<c026e00a>] ? eth_header+0x0/0x8e
Aug 16 16:09:33 192.168.20.66 kernel: [  181.267096]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:09:33 192.168.20.66 kernel: [  181.268034]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:09:33 192.168.20.66 kernel: [  181.268034]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:09:33 192.168.20.66 kernel: [  181.268034]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:09:33 192.168.20.66 kernel: [  181.268034]  [<c0272dbd>] qdisc_requeue+0x56/0x6e
Aug 16 16:09:33 192.168.20.66 kernel: [  181.268034]  [<f8a10689>] tbf_dequeue+0x166/0x19b [sch_tbf]
Aug 16 16:09:33 192.168.20.66 kernel: [  181.269021]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:09:33 192.168.20.66 kernel: [  181.269021]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:09:33 192.168.20.66 kernel: [  181.269021]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:09:33 192.168.20.66 kernel: [  181.269021]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:09:33 192.168.20.66 kernel: [  181.269021]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:09:33 192.168.20.66 kernel: [  181.269021]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:09:33 192.168.20.66 kernel: [  181.270027]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:09:33 192.168.20.66 kernel: [  181.270101]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:09:33 192.168.20.66 kernel: [  181.270101]  [<c0157078>] ? page_address+0x87/0xa9
Aug 16 16:09:33 192.168.20.66 kernel: [  181.270101]  [<c02bdbef>] ? _spin_unlock_irqrestore+0x36/0x3c
Aug 16 16:09:33 192.168.20.66 kernel: [  181.270101]  [<c02bdbf1>] ? _spin_unlock_irqrestore+0x38/0x3c
Aug 16 16:09:33 192.168.20.66 kernel: [  181.270107]  [<c0157078>] page_address+0x87/0xa9
Aug 16 16:09:33 192.168.20.66 kernel: [  181.271013]  [<c016cfca>] ? page_put_link+0x0/0x1b
Aug 16 16:09:33 192.168.20.66 kernel: [  181.271013]  [<c01570b1>] kunmap_high+0x17/0x8e
Aug 16 16:09:33 192.168.20.66 kernel: [  181.271013]  [<c0116630>] kunmap+0x4a/0x4c
Aug 16 16:09:33 192.168.20.66 kernel: [  181.271013]  [<c016cfdb>] page_put_link+0x11/0x1b
Aug 16 16:09:33 192.168.20.66 kernel: [  181.271013]  [<c016ee42>] __link_path_walk+0xa0a/0xb46
Aug 16 16:09:33 192.168.20.66 kernel: [  181.271013]  [<c016f0fd>] path_walk+0x4c/0x9b
Aug 16 16:09:33 192.168.20.66 kernel: [  181.272036]  [<c016f2c2>] do_path_lookup+0x129/0x143
Aug 16 16:09:33 192.168.20.66 kernel: [  181.272036]  [<c016f335>] __path_lookup_intent_open+0x45/0x76
Aug 16 16:09:33 192.168.20.66 kernel: [  181.272036]  [<c016f376>] path_lookup_open+0x10/0x12
Aug 16 16:09:33 192.168.20.66 kernel: [  181.272036]  [<c016b6aa>] open_exec+0x1f/0xa2
Aug 16 16:09:33 192.168.20.66 kernel: [  181.272036]  [<c016b5ae>] ? kernel_read+0x34/0x46
Aug 16 16:09:33 192.168.20.66 kernel: [  181.274041]  [<c018f48a>] load_elf_binary+0x1fe/0x1118
Aug 16 16:09:33 192.168.20.66 kernel: [  181.274165]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:09:33 192.168.20.66 kernel: [  181.274165]  [<c016ab15>] ? search_binary_handler+0x3a/0x1e9
Aug 16 16:09:33 192.168.20.66 kernel: [  181.274165]  [<c018f28c>] ? load_elf_binary+0x0/0x1118
Aug 16 16:09:33 192.168.20.66 kernel: [  181.274165]  [<c016ab5c>] search_binary_handler+0x81/0x1e9
Aug 16 16:09:33 192.168.20.66 kernel: [  181.275064]  [<c016bc35>] do_execve+0x170/0x1e2
Aug 16 16:09:33 192.168.20.66 kernel: [  181.275064]  [<c010236a>] sys_execve+0x29/0x55
Aug 16 16:09:33 192.168.20.66 kernel: [  181.275064]  [<c0103861>] sysenter_do_call+0x12/0x35
Aug 16 16:09:33 192.168.20.66 kernel: [  181.275064]  =======================
Aug 16 16:09:33 192.168.20.66 kernel: [  181.275064] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:10:02 192.168.20.66 kernel: [  210.034004] ------------[ cut here ]------------
Aug 16 16:10:02 192.168.20.66 kernel: [  210.034004] WARNING: at include/linux/skbuff.h:692 bfifo_enqueue+0x66/0xb9()
Aug 16 16:10:02 192.168.20.66 kernel: [  210.034004] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:10:02 192.168.20.66 kernel: [  210.045926] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:10:02 192.168.20.66 kernel: [  210.046915]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:10:02 192.168.20.66 kernel: [  210.046915]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:10:02 192.168.20.66 kernel: [  210.046915]  [<c02bdc12>] ? _spin_unlock+0x1d/0x20
Aug 16 16:10:02 192.168.20.66 kernel: [  210.046915]  [<c016275b>] ? add_partial+0x44/0x49
Aug 16 16:10:02 192.168.20.66 kernel: [  210.046915]  [<c0163309>] ? __slab_free+0x61/0x283
Aug 16 16:10:02 192.168.20.66 kernel: [  210.046915]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0272c61>] bfifo_enqueue+0x66/0xb9
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<f8a1071a>] tbf_enqueue+0x5c/0x92 [sch_tbf]
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<f8a5275a>] prio_enqueue+0x44/0x82 [sch_prio]
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0261b1c>] dev_queue_xmit+0x339/0x481
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c02804fa>] ip_finish_output+0x1df/0x209
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c02805a2>] ip_output+0x7e/0x83
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c027d1e7>] ip_forward_finish+0x2f/0x32
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c027d3f1>] ip_forward+0x207/0x256
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c027c1ce>] ip_rcv_finish+0x2a6/0x2bc
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c027c3d1>] ip_rcv+0x1ed/0x217
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0261169>] netif_receive_skb+0x3c3/0x3f6
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c026df14>] ? eth_type_trans+0x28/0xcc
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<f890f2f1>] e1000_clean_rx_irq+0x2b2/0x362 [e1000]
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<f89110ac>] e1000_clean+0x2d0/0x474 [e1000]
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c02bdcc0>] ? _spin_lock+0x22/0x2a
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c025f9de>] net_rx_action+0xbb/0x1e5
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c014a35d>] ? handle_fasteoi_irq+0x0/0xb6
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0106038>] do_IRQ+0xae/0xc4
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0104288>] common_interrupt+0x28/0x30
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  [<c02b1016>] rest_init+0x4e/0x50
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101]  =======================
Aug 16 16:10:02 192.168.20.66 kernel: [  210.047101] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:10:13 192.168.20.66 kernel: [  221.739003] ------------[ cut here ]------------
Aug 16 16:10:13 192.168.20.66 kernel: [  221.739003] WARNING: at include/linux/skbuff.h:692 qdisc_requeue+0x56/0x6e()
Aug 16 16:10:13 192.168.20.66 kernel: [  221.739003] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:10:13 192.168.20.66 kernel: [  221.769582] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:10:13 192.168.20.66 kernel: [  221.770028]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:10:13 192.168.20.66 kernel: [  221.770028]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:10:13 192.168.20.66 kernel: [  221.770028]  [<c02bdc12>] ? _spin_unlock+0x1d/0x20
Aug 16 16:10:13 192.168.20.66 kernel: [  221.770028]  [<c016275b>] ? add_partial+0x44/0x49
Aug 16 16:10:13 192.168.20.66 kernel: [  221.770028]  [<c0163309>] ? __slab_free+0x61/0x283
Aug 16 16:10:13 192.168.20.66 kernel: [  221.770028]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:10:13 192.168.20.66 kernel: [  221.770028]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c0272dbd>] qdisc_requeue+0x56/0x6e
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<f8a10689>] tbf_dequeue+0x166/0x19b [sch_tbf]
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  [<c02b1016>] rest_init+0x4e/0x50
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028]  =======================
Aug 16 16:10:13 192.168.20.66 kernel: [  221.771028] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002] ------------[ cut here ]------------
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002] WARNING: at include/linux/skbuff.h:692 bfifo_enqueue+0x66/0xb9()
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c02bdc12>] ? _spin_unlock+0x1d/0x20
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c016275b>] ? add_partial+0x44/0x49
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0163309>] ? __slab_free+0x61/0x283
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0272c61>] bfifo_enqueue+0x66/0xb9
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<f8a1071a>] tbf_enqueue+0x5c/0x92 [sch_tbf]
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<f8a5275a>] prio_enqueue+0x44/0x82 [sch_prio]
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0261b1c>] dev_queue_xmit+0x339/0x481
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c02804fa>] ip_finish_output+0x1df/0x209
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c02805a2>] ip_output+0x7e/0x83
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c027d1e7>] ip_forward_finish+0x2f/0x32
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c027d3f1>] ip_forward+0x207/0x256
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c027c1ce>] ip_rcv_finish+0x2a6/0x2bc
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c027c3d1>] ip_rcv+0x1ed/0x217
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0261169>] netif_receive_skb+0x3c3/0x3f6
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c026df14>] ? eth_type_trans+0x28/0xcc
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<f890f2f1>] e1000_clean_rx_irq+0x2b2/0x362 [e1000]
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<f89110ac>] e1000_clean+0x2d0/0x474 [e1000]
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c02bdcc0>] ? _spin_lock+0x22/0x2a
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c025f9de>] net_rx_action+0xbb/0x1e5
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c014a35d>] ? handle_fasteoi_irq+0x0/0xb6
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0106038>] do_IRQ+0xae/0xc4
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0104288>] common_interrupt+0x28/0x30
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  [<c02b1016>] rest_init+0x4e/0x50
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002]  =======================
Aug 16 16:10:26 192.168.20.66 kernel: [  234.546002] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:10:31 192.168.20.66 kernel: [  239.450003] ------------[ cut here ]------------
Aug 16 16:10:31 192.168.20.66 kernel: [  239.450003] WARNING: at include/linux/skbuff.h:793 qdisc_dequeue_head+0x5e/0x72()
Aug 16 16:10:31 192.168.20.66 kernel: [  239.450003] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<f89f0032>] ? tcp_packet+0x729/0xa57 [nf_conntrack]
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<c02bdb90>] ? _spin_unlock_bh+0x25/0x28
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<c0272eb2>] qdisc_dequeue_head+0x5e/0x72
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<f8a1053f>] tbf_dequeue+0x1c/0x19b [sch_tbf]
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<c02bdc12>] ? _spin_unlock+0x1d/0x20
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<f8a5201b>] prio_dequeue+0x1b/0x30 [sch_prio]
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<c0261b3c>] dev_queue_xmit+0x359/0x481
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451002]  [<c02804fa>] ip_finish_output+0x1df/0x209
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451003]  [<c02805a2>] ip_output+0x7e/0x83
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451003]  [<c027d1e7>] ip_forward_finish+0x2f/0x32
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451003]  [<c027d3f1>] ip_forward+0x207/0x256
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451003]  [<c027c1ce>] ip_rcv_finish+0x2a6/0x2bc
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451003]  [<c027c3d1>] ip_rcv+0x1ed/0x217
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451003]  [<c0261169>] netif_receive_skb+0x3c3/0x3f6
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451003]  [<c026df14>] ? eth_type_trans+0x28/0xcc
Aug 16 16:10:31 192.168.20.66 kernel: [  239.451003]  [<f890f2f1>] e1000_clean_rx_irq+0x2b2/0x362 [e1000]
Aug 16 16:10:31 192.168.20.66 kernel: [  239.463208]  [<f89110ac>] e1000_clean+0x2d0/0x474 [e1000]
Aug 16 16:10:31 192.168.20.66 kernel: [  239.463384]  [<c02bdcc0>] ? _spin_lock+0x22/0x2a
Aug 16 16:10:31 192.168.20.66 kernel: [  239.463747]  [<c025f9de>] net_rx_action+0xbb/0x1e5
Aug 16 16:10:31 192.168.20.66 kernel: [  239.464197]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:10:31 192.168.20.66 kernel: [  239.464197]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:10:31 192.168.20.66 kernel: [  239.464197]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465188]  [<c014a35d>] ? handle_fasteoi_irq+0x0/0xb6
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465188]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465188]  [<c0106038>] do_IRQ+0xae/0xc4
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465188]  [<c0104288>] common_interrupt+0x28/0x30
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465188]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465188]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465197]  [<c02b1016>] rest_init+0x4e/0x50
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465197]  =======================
Aug 16 16:10:31 192.168.20.66 kernel: [  239.465197] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:11:47 192.168.20.66 kernel: [  315.228593] ------------[ cut here ]------------
Aug 16 16:11:47 192.168.20.66 kernel: [  315.228593] WARNING: at include/linux/skbuff.h:793 qdisc_dequeue_head+0x5e/0x72()
Aug 16 16:11:47 192.168.20.66 kernel: [  315.228593] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c0266e18>] ? neigh_resolve_output+0x201/0x24b
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c0272eb2>] qdisc_dequeue_head+0x5e/0x72
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<f8a1053f>] tbf_dequeue+0x1c/0x19b [sch_tbf]
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  [<c02b1016>] rest_init+0x4e/0x50
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003]  =======================
Aug 16 16:11:47 192.168.20.66 kernel: [  315.229003] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006] ------------[ cut here ]------------
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006] WARNING: at include/linux/skbuff.h:793 qdisc_dequeue_head+0x5e/0x72()
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c0250032>] ? hid_parse_report+0x1f/0x23d
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c0272eb2>] qdisc_dequeue_head+0x5e/0x72
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<f8a1053f>] tbf_dequeue+0x1c/0x19b [sch_tbf]
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  [<c02b9b45>] start_secondary+0x184/0x189
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006]  =======================
Aug 16 16:11:50 192.168.20.66 kernel: [  318.892006] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:12:49 192.168.20.66 kernel: [  377.171065] ------------[ cut here ]------------
Aug 16 16:12:49 192.168.20.66 kernel: [  377.171065] WARNING: at include/linux/skbuff.h:692 qdisc_requeue+0x56/0x6e()
Aug 16 16:12:49 192.168.20.66 kernel: [  377.171065] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c0261c31>] ? dev_queue_xmit+0x44e/0x481
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c026e00a>] ? eth_header+0x0/0x8e
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c026e00a>] ? eth_header+0x0/0x8e
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c0272dbd>] qdisc_requeue+0x56/0x6e
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<f8a10689>] tbf_dequeue+0x166/0x19b [sch_tbf]
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  [<c02b9b45>] start_secondary+0x184/0x189
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010]  =======================
Aug 16 16:12:49 192.168.20.66 kernel: [  377.178010] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:12:52 192.168.20.66 kernel: [  380.360065] ------------[ cut here ]------------
Aug 16 16:12:52 192.168.20.66 kernel: [  380.360065] WARNING: at include/linux/skbuff.h:692 qdisc_requeue+0x56/0x6e()
Aug 16 16:12:52 192.168.20.66 kernel: [  380.360065] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:12:52 192.168.20.66 kernel:  ppp_generic slhc
Aug 16 16:12:52 192.168.20.66 kernel:  xt_TCPMSS
Aug 16 16:12:52 192.168.20.66 kernel:  xt_mark
Aug 16 16:12:52 192.168.20.66 kernel:  xt_tcpudp
Aug 16 16:12:52 192.168.20.66 kernel:  iptable_mangle
Aug 16 16:12:52 192.168.20.66 kernel:  iptable_nat
Aug 16 16:12:52 192.168.20.66 kernel:  nf_nat
Aug 16 16:12:52 192.168.20.66 kernel:  nf_conntrack_ipv4
Aug 16 16:12:52 192.168.20.66 kernel:  nf_conntrack
Aug 16 16:12:52 192.168.20.66 kernel:  rtc_cmos
Aug 16 16:12:52 192.168.20.66 kernel:  rtc_core
Aug 16 16:12:52 192.168.20.66 kernel:  rtc_lib iptable_filter
Aug 16 16:12:52 192.168.20.66 kernel:  ip_tables x_tables
Aug 16 16:12:52 192.168.20.66 kernel:  8021q
Aug 16 16:12:52 192.168.20.66 kernel:  garp stp
Aug 16 16:12:52 192.168.20.66 kernel:  llc tun
Aug 16 16:12:52 192.168.20.66 kernel:  loop tulip
Aug 16 16:12:52 192.168.20.66 kernel:  r8169 sky2
Aug 16 16:12:52 192.168.20.66 kernel:  via_velocity via_rhine sis900
Aug 16 16:12:52 192.168.20.66 kernel:  ne2k_pci
Aug 16 16:12:52 192.168.20.66 kernel:  8390
Aug 16 16:12:52 192.168.20.66 kernel:  skge tg3
Aug 16 16:12:52 192.168.20.66 kernel:  libphy 8139too
Aug 16 16:12:52 192.168.20.66 kernel:  e1000 e100
Aug 16 16:12:52 192.168.20.66 kernel:  usb_storage mtdblock mtd_blkdevs
Aug 16 16:12:52 192.168.20.66 kernel:  usbhid
Aug 16 16:12:52 192.168.20.66 kernel:  uhci_hcd
Aug 16 16:12:52 192.168.20.66 kernel:  ehci_hcd ohci_hcd
Aug 16 16:12:52 192.168.20.66 kernel:  usbcore
Aug 16 16:12:52 192.168.20.66 kernel: [  380.364109] Pid: 25174, comm: modprobe Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:12:52 192.168.20.66 kernel: [  380.364109]  [<c02bb7eb>]
Aug 16 16:12:52 192.168.20.66 kernel: ?
Aug 16 16:12:52 192.168.20.66 kernel: printk+0xf/0x14
Aug 16 16:12:52 192.168.20.66 kernel: [  380.364109]  [<c0121be8>]
Aug 16 16:12:52 192.168.20.66 kernel: warn_on_slowpath+0x41/0x65
Aug 16 16:12:52 192.168.20.66 kernel: [  380.365116]  [<c02bdc12>]
Aug 16 16:12:52 192.168.20.66 kernel: ?
Aug 16 16:12:52 192.168.20.66 kernel: _spin_unlock+0x1d/0x20
Aug 16 16:12:52 192.168.20.66 kernel: [  380.365153]  [<c016275b>]
Aug 16 16:12:52 192.168.20.66 kernel: ?
Aug 16 16:12:52 192.168.20.66 kernel: add_partial+0x44/0x49
Aug 16 16:12:52 192.168.20.66 kernel: [  380.365153]  [<c0163309>]
Aug 16 16:12:52 192.168.20.66 kernel: ?
Aug 16 16:12:52 192.168.20.66 kernel: __slab_free+0x61/0x283
Aug 16 16:12:52 192.168.20.66 kernel: [  380.365153]  [<c013dabf>]
Aug 16 16:12:52 192.168.20.66 kernel: ?
Aug 16 16:12:52 192.168.20.66 kernel: trace_hardirqs_on+0xb/0xd
Aug 16 16:12:52 192.168.20.66 kernel: [  380.365153]  [<c013dabf>]
Aug 16 16:12:52 192.168.20.66 kernel: ?
Aug 16 16:12:52 192.168.20.66 kernel: trace_hardirqs_on+0xb/0xd
Aug 16 16:12:52 192.168.20.66 kernel: [  380.366079]  [<c025c000>]
Aug 16 16:12:52 192.168.20.66 kernel: ?
Aug 16 16:12:52 192.168.20.66 kernel: __kfree_skb+0x63/0x66
Aug 16 16:12:52 192.168.20.66 kernel: [  380.366079]  [<c025c000>]
Aug 16 16:12:52 192.168.20.66 kernel: ?
Aug 16 16:12:52 192.168.20.66 kernel: __kfree_skb+0x63/0x66
Aug 16 16:12:52 192.168.20.66 kernel: [  380.366079]  [<c0272dbd>]
Aug 16 16:12:52 192.168.20.66 kernel: qdisc_requeue+0x56/0x6e
Aug 16 16:12:52 192.168.20.66 kernel: [  380.366079]  [<f8a10689>]
Aug 16 16:12:52 192.168.20.66 kernel: tbf_dequeue+0x166/0x19b [sch_tbf]
Aug 16 16:12:52 192.168.20.66 kernel: [  380.366079]  [<c026eace>]
Aug 16 16:12:52 192.168.20.66 kernel: __qdisc_run+0x41/0x18b
Aug 16 16:12:52 192.168.20.66 kernel: [  380.367079]  [<c025fd85>]
Aug 16 16:12:52 192.168.20.66 kernel: net_tx_action+0xaf/0xcc
Aug 16 16:12:52 192.168.20.66 kernel: [  380.367086]  [<c0126203>]
Aug 16 16:12:52 192.168.20.66 kernel: __do_softirq+0x7b/0xf4
Aug 16 16:12:52 192.168.20.66 kernel: [  380.367086]  [<c0126188>]
Aug 16 16:12:52 192.168.20.66 kernel: ? __do_softirq+0x0/0xf4
Aug 16 16:12:52 192.168.20.66 kernel: [  380.367086]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:12:52 192.168.20.66 kernel: [  380.367086]  [<c0125e28>]
Aug 16 16:12:52 192.168.20.66 kernel: irq_exit+0x44/0x79
Aug 16 16:12:52 192.168.20.66 kernel: [  380.368083]  [<c010ff15>]
Aug 16 16:12:52 192.168.20.66 kernel: smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:12:52 192.168.20.66 kernel: [  380.368083]  [<c010438d>]
Aug 16 16:12:52 192.168.20.66 kernel: apic_timer_interrupt+0x2d/0x34
Aug 16 16:12:52 192.168.20.66 kernel: [  380.368083]  [<c012007b>]
Aug 16 16:12:52 192.168.20.66 kernel: ? sys_unshare+0xbf/0x258
Aug 16 16:12:52 192.168.20.66 kernel: [  380.368083]  =======================
Aug 16 16:12:52 192.168.20.66 kernel: [  380.368083] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:13:11 192.168.20.66 kernel: [  399.643004] ------------[ cut here ]------------
Aug 16 16:13:11 192.168.20.66 kernel: [  399.643004] WARNING: at include/linux/skbuff.h:692 bfifo_enqueue+0x66/0xb9()
Aug 16 16:13:11 192.168.20.66 kernel: [  399.643004] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:13:11 192.168.20.66 kernel: [  399.643004] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:13:11 192.168.20.66 kernel: [  399.643004]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:13:11 192.168.20.66 kernel: [  399.643004]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:13:11 192.168.20.66 kernel: [  399.643004]  [<c026a00f>] ? inet_proto_csum_replace4+0x33/0xab
Aug 16 16:13:11 192.168.20.66 kernel: [  399.656071]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:13:11 192.168.20.66 kernel: [  399.656266]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:13:11 192.168.20.66 kernel: [  399.656649]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:13:11 192.168.20.66 kernel: [  399.657066]  [<c025c000>] ? __kfree_skb+0x63/0x66
Aug 16 16:13:11 192.168.20.66 kernel: [  399.657066]  [<c0272c61>] bfifo_enqueue+0x66/0xb9
Aug 16 16:13:11 192.168.20.66 kernel: [  399.657066]  [<f8a1071a>] tbf_enqueue+0x5c/0x92 [sch_tbf]
Aug 16 16:13:11 192.168.20.66 kernel: [  399.658066]  [<f8a5275a>] prio_enqueue+0x44/0x82 [sch_prio]
Aug 16 16:13:11 192.168.20.66 kernel: [  399.658066]  [<c0261b1c>] dev_queue_xmit+0x339/0x481
Aug 16 16:13:11 192.168.20.66 kernel: [  399.658066]  [<c02804fa>] ip_finish_output+0x1df/0x209
Aug 16 16:13:11 192.168.20.66 kernel: [  399.658066]  [<c02805a2>] ip_output+0x7e/0x83
Aug 16 16:13:11 192.168.20.66 kernel: [  399.658066]  [<c027d1e7>] ip_forward_finish+0x2f/0x32
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c027d3f1>] ip_forward+0x207/0x256
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c027c1ce>] ip_rcv_finish+0x2a6/0x2bc
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c027c3d1>] ip_rcv+0x1ed/0x217
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c0261169>] netif_receive_skb+0x3c3/0x3f6
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c026df14>] ? eth_type_trans+0x28/0xcc
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<f890f2f1>] e1000_clean_rx_irq+0x2b2/0x362 [e1000]
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<f89110ac>] e1000_clean+0x2d0/0x474 [e1000]
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c02bdcc0>] ? _spin_lock+0x22/0x2a
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c025f9de>] net_rx_action+0xbb/0x1e5
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c014a35d>] ? handle_fasteoi_irq+0x0/0xb6
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c0106038>] do_IRQ+0xae/0xc4
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c0104288>] common_interrupt+0x28/0x30
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  [<c02b9b45>] start_secondary+0x184/0x189
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065]  =======================
Aug 16 16:13:11 192.168.20.66 kernel: [  399.659065] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:13:42 192.168.20.66 kernel: [  430.623004] ------------[ cut here ]------------
Aug 16 16:13:42 192.168.20.66 kernel: [  430.623004] WARNING: at include/linux/skbuff.h:793 qdisc_dequeue_head+0x5e/0x72()
Aug 16 16:13:42 192.168.20.66 kernel: [  430.623004] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c0250032>] ? hid_parse_report+0x1f/0x23d
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<f890ef35>] ? e1000_alloc_rx_buffers+0x16d/0x277 [e1000]
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<f890f379>] ? e1000_clean_rx_irq+0x33a/0x362 [e1000]
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c0272eb2>] qdisc_dequeue_head+0x5e/0x72
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<f8a1053f>] tbf_dequeue+0x1c/0x19b [sch_tbf]
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  [<c02b1016>] rest_init+0x4e/0x50
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003]  =======================
Aug 16 16:13:42 192.168.20.66 kernel: [  430.625003] ---[ end trace 7e8f04d781fe52f3 ]---
Aug 16 16:14:02 192.168.20.66 kernel: [  450.101006] ------------[ cut here ]------------
Aug 16 16:14:02 192.168.20.66 kernel: [  450.101006] WARNING: at include/linux/skbuff.h:793 qdisc_dequeue_head+0x5e/0x72()
Aug 16 16:14:02 192.168.20.66 kernel: [  450.101006] Modules linked in: sch_ingress sch_prio netconsole configfs nf_nat_pptp nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_proto_gre iTCO_wdt 
e752x_edac edac_core cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008] Pid: 0, comm: swapper Tainted: G        W 2.6.27-rc3-git2-build-0031 #15
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c02bb7eb>] ? printk+0xf/0x14
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c0121be8>] warn_on_slowpath+0x41/0x65
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c0261c31>] ? dev_queue_xmit+0x44e/0x481
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c026e00a>] ? eth_header+0x0/0x8e
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c026e00a>] ? eth_header+0x0/0x8e
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<f8a056e2>] ? ppp_push+0x7f/0x407 [ppp_generic]
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c013dabf>] ? trace_hardirqs_on+0xb/0xd
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c012604d>] ? local_bh_enable_ip+0xa6/0xaa
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c02bdb90>] ? _spin_unlock_bh+0x25/0x28
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<f8a062b2>] ? ppp_xmit_process+0x459/0x461 [ppp_generic]
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c0272eb2>] qdisc_dequeue_head+0x5e/0x72
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<f8a1053f>] tbf_dequeue+0x1c/0x19b [sch_tbf]
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c02bdcc0>] ? _spin_lock+0x22/0x2a
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c026eace>] __qdisc_run+0x41/0x18b
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c025fd85>] net_tx_action+0xaf/0xcc
Aug 16 16:14:02 192.168.20.66 kernel: [  450.119008]  [<c0126203>] __do_softirq+0x7b/0xf4
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  [<c0126188>] ? __do_softirq+0x0/0xf4
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  [<c01060b3>] do_softirq+0x65/0xb6
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  [<c0125e28>] irq_exit+0x44/0x79
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  [<c010ff15>] smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  [<c010438d>] apic_timer_interrupt+0x2d/0x34
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  [<c0108d8a>] ? mwait_idle+0x39/0x43
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  [<c01029ee>] cpu_idle+0xbf/0xe1
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  [<c02b9b45>] start_secondary+0x184/0x189
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002]  =======================
Aug 16 16:14:02 192.168.20.66 kernel: [  450.125002] ---[ end trace 7e8f04d781fe52f3 ]---


^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 13:13                     ` Denys Fedoryshchenko
@ 2008-08-16 14:27                       ` Jarek Poplawski
  2008-08-16 15:28                         ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-16 14:27 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sat, Aug 16, 2008 at 04:13:59PM +0300, Denys Fedoryshchenko wrote:
...
> Here is what i got:
> Aug 16 16:07:33 192.168.20.66 kernel: [   61.930066] ------------[ cut here ]------------
> Aug 16 16:07:33 192.168.20.66 kernel: [   61.930066] WARNING: at include/linux/skbuff.h:692 qdisc_requeue+0x56/0x6e()

It's very helpful, thanks.

Here is another patch - please apply without removing the previous one yet.

Jarek P.

---

 include/net/sch_generic.h |    5 +++++
 net/sched/sch_api.c       |    2 +-
 2 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index a7abfda..4f28fa7 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -193,6 +193,11 @@ static inline struct Qdisc *qdisc_root(struct Qdisc *qdisc)
 	return qdisc->dev_queue->qdisc;
 }
 
+static inline struct Qdisc *qdisc_root_sleeping(struct Qdisc *qdisc)
+{
+	return qdisc->dev_queue->qdisc_sleeping;
+}
+
 /* The qdisc root lock is a mechanism by which to top level
  * of a qdisc tree can be locked from any qdisc node in the
  * forest.  This allows changing the configuration of some
diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
index c25465e..5c7ae56 100644
--- a/net/sched/sch_api.c
+++ b/net/sched/sch_api.c
@@ -435,7 +435,7 @@ void qdisc_watchdog_init(struct qdisc_watchdog *wd, struct Qdisc *qdisc)
 {
 	hrtimer_init(&wd->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
 	wd->timer.function = qdisc_watchdog;
-	wd->qdisc = qdisc;
+	wd->qdisc = qdisc_root_sleeping(qdisc);
 }
 EXPORT_SYMBOL(qdisc_watchdog_init);
 

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 14:27                       ` Jarek Poplawski
@ 2008-08-16 15:28                         ` Denys Fedoryshchenko
  2008-08-16 15:59                           ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-16 15:28 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Triggered panic by "batch" shaper "reapply".

Aug 16 18:26:29 192.168.20.66 [ 2920.823068] BUG: unable to handle kernel 
Aug 16 18:26:29 192.168.20.66 NULL pointer dereference
Aug 16 18:26:29 192.168.20.66 at 00000000
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] IP:
Aug 16 18:26:29 192.168.20.66 [<c013c083>] find_usage_forwards+0x7e/0x92
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] *pde = 00000000 
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Oops: 0000 [#1] 
Aug 16 18:26:29 192.168.20.66 SMP 
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Modules linked in:
Aug 16 18:26:29 192.168.20.66 sch_ingress
Aug 16 18:26:29 192.168.20.66 sch_prio
Aug 16 18:26:29 192.168.20.66 netconsole
Aug 16 18:26:29 192.168.20.66 configfs
Aug 16 18:26:29 192.168.20.66 nf_nat_pptp
Aug 16 18:26:29 192.168.20.66 nf_conntrack_pptp
Aug 16 18:26:29 192.168.20.66 nf_conntrack_proto_gre
Aug 16 18:26:29 192.168.20.66 nf_nat_proto_gre
Aug 16 18:26:29 192.168.20.66 iTCO_wdt
Aug 16 18:26:29 192.168.20.66 e752x_edac
Aug 16 18:26:29 192.168.20.66 edac_core
Aug 16 18:26:29 192.168.20.66 cls_u32
Aug 16 18:26:29 192.168.20.66 em_meta
Aug 16 18:26:29 192.168.20.66 cls_basic
Aug 16 18:26:29 192.168.20.66 xt_dscp
Aug 16 18:26:29 192.168.20.66 xt_DSCP
Aug 16 18:26:29 192.168.20.66 ipt_REJECT
Aug 16 18:26:29 192.168.20.66 ts_bm
Aug 16 18:26:29 192.168.20.66 xt_string
Aug 16 18:26:29 192.168.20.66 ipt_ttl
Aug 16 18:26:29 192.168.20.66 ifb
Aug 16 18:26:29 192.168.20.66 cls_fw
Aug 16 18:26:29 192.168.20.66 sch_tbf
Aug 16 18:26:29 192.168.20.66 sch_htb
Aug 16 18:26:29 192.168.20.66 act_ipt
Aug 16 18:26:29 192.168.20.66 act_mirred
Aug 16 18:26:29 192.168.20.66 xt_MARK
Aug 16 18:26:29 192.168.20.66 pppoe
Aug 16 18:26:29 192.168.20.66 pppox
Aug 16 18:26:29 192.168.20.66 ppp_generic
Aug 16 18:26:29 192.168.20.66 slhc
Aug 16 18:26:29 192.168.20.66 xt_TCPMSS
Aug 16 18:26:29 192.168.20.66 xt_mark
Aug 16 18:26:29 192.168.20.66 xt_tcpudp
Aug 16 18:26:29 192.168.20.66 iptable_mangle
Aug 16 18:26:29 192.168.20.66 iptable_nat
Aug 16 18:26:29 192.168.20.66 nf_nat
Aug 16 18:26:29 192.168.20.66 nf_conntrack_ipv4
Aug 16 18:26:29 192.168.20.66 nf_conntrack
Aug 16 18:26:29 192.168.20.66 rtc_cmos
Aug 16 18:26:29 192.168.20.66 rtc_core
Aug 16 18:26:29 192.168.20.66 rtc_lib
Aug 16 18:26:29 192.168.20.66 iptable_filter
Aug 16 18:26:29 192.168.20.66 ip_tables
Aug 16 18:26:29 192.168.20.66 x_tables
Aug 16 18:26:29 192.168.20.66 8021q
Aug 16 18:26:29 192.168.20.66 garp
Aug 16 18:26:29 192.168.20.66 stp
Aug 16 18:26:29 192.168.20.66 llc
Aug 16 18:26:29 192.168.20.66 tun
Aug 16 18:26:29 192.168.20.66 loop
Aug 16 18:26:29 192.168.20.66 tulip
Aug 16 18:26:29 192.168.20.66 r8169
Aug 16 18:26:29 192.168.20.66 sky2
Aug 16 18:26:29 192.168.20.66 via_velocity
Aug 16 18:26:29 192.168.20.66 via_rhine
Aug 16 18:26:29 192.168.20.66 sis900
Aug 16 18:26:29 192.168.20.66 ne2k_pci
Aug 16 18:26:29 192.168.20.66 8390
Aug 16 18:26:29 192.168.20.66 skge
Aug 16 18:26:29 192.168.20.66 tg3
Aug 16 18:26:29 192.168.20.66 libphy
Aug 16 18:26:29 192.168.20.66 8139too
Aug 16 18:26:29 192.168.20.66 e1000
Aug 16 18:26:29 192.168.20.66 e100
Aug 16 18:26:29 192.168.20.66 usb_storage
Aug 16 18:26:29 192.168.20.66 mtdblock
Aug 16 18:26:29 192.168.20.66 mtd_blkdevs
Aug 16 18:26:29 192.168.20.66 usbhid
Aug 16 18:26:29 192.168.20.66 uhci_hcd
Aug 16 18:26:29 192.168.20.66 ehci_hcd
Aug 16 18:26:29 192.168.20.66 ohci_hcd
Aug 16 18:26:29 192.168.20.66 usbcore
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Pid: 31408, comm: tc Not tainted (2.6.27-rc3-git2-build-0031 #17)
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] EIP: 0060:[<c013c083>] EFLAGS: 00010046 CPU: 1
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] EIP is at find_usage_forwards+0x7e/0x92
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] EAX: 00000008 EBX: 00000001 ECX: 00000003 EDX: 00000000
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] ESI: 00000000 EDI: c09ef1a0 EBP: c0861e98 ESP: c0861e8c
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Process tc (pid: 31408, ti=c0861000 task=f659bbf0 task.ti=f27a6000)
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Stack: 
Aug 16 18:26:29 192.168.20.66 f659bfa8 
Aug 16 18:26:29 192.168.20.66 f659bbf0 
Aug 16 18:26:29 192.168.20.66 f659bbf0 
Aug 16 18:26:29 192.168.20.66 c0861ea8 
Aug 16 18:26:29 192.168.20.66 c013d039 
Aug 16 18:26:29 192.168.20.66 f659bfa8 
Aug 16 18:26:29 192.168.20.66 00000001 
Aug 16 18:26:29 192.168.20.66 c0861ec4 
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]        
Aug 16 18:26:29 192.168.20.66 c013d477 
Aug 16 18:26:29 192.168.20.66 c03419ec 
Aug 16 18:26:29 192.168.20.66 00000002 
Aug 16 18:26:29 192.168.20.66 f659bfb8 
Aug 16 18:26:29 192.168.20.66 f659bc08 
Aug 16 18:26:29 192.168.20.66 f659bbf0 
Aug 16 18:26:29 192.168.20.66 c0861f38 
Aug 16 18:26:29 192.168.20.66 c013e6de 
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]        
Aug 16 18:26:29 192.168.20.66 00000000 
Aug 16 18:26:29 192.168.20.66 503b61a5 
Aug 16 18:26:29 192.168.20.66 f659bfc0 
Aug 16 18:26:29 192.168.20.66 62b1afea 
Aug 16 18:26:29 192.168.20.66 00000014 
Aug 16 18:26:29 192.168.20.66 00000002 
Aug 16 18:26:29 192.168.20.66 00000000 
Aug 16 18:26:29 192.168.20.66 00000000 
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Call Trace:
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013d039>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 check_usage_forwards+0x54/0x76
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013d477>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 mark_lock+0x3a6/0x7f8
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013e6de>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 __lock_acquire+0x506/0x118a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01dd5d0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_thunk+0xc/0x10
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013da7d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_caller+0xcb/0x102
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01dd5d0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_thunk+0xc/0x10
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c014a40d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 handle_fasteoi_irq+0xb0/0xb6
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013f3aa>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 lock_acquire+0x48/0x64
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x5
Aug 16 18:26:29 192.168.20.66 1/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bdcc1>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 _spin_lock+0x1b/0x2a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x51/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x51/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0129530>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 run_timer_softirq+0x11d/0x176
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025edc6>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x0/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0126203>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 __do_softirq+0x7b/0xf4
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0126188>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 __do_softirq+0x0/0xf4
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01060b3>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 do_softirq+0x65/0xb6
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0125e28>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 irq_exit+0x44/0x79
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010ff15>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010438d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 apic_timer_interrupt+0x2d/0x34
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  =======================
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Code: 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 eb 
Aug 16 18:26:29 192.168.20.66 35 
Aug 16 18:26:29 192.168.20.66 8b 
Aug 16 18:26:29 192.168.20.66 b3 
Aug 16 18:26:29 192.168.20.66 b0 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 8d 
Aug 16 18:26:29 192.168.20.66 bb 
Aug 16 18:26:29 192.168.20.66 b0 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 8d 
Aug 16 18:26:29 192.168.20.66 5a 
Aug 16 18:26:29 192.168.20.66 01 
Aug 16 18:26:29 192.168.20.66 eb 
Aug 16 18:26:29 192.168.20.66 15 
Aug 16 18:26:29 192.168.20.66 8b 
Aug 16 18:26:29 192.168.20.66 46 
Aug 16 18:26:29 192.168.20.66 08 
Aug 16 18:26:29 192.168.20.66 89 
Aug 16 18:26:29 192.168.20.66 da 
Aug 16 18:26:29 192.168.20.66 e8 
Aug 16 18:26:29 192.168.20.66 8d 
Aug 16 18:26:29 192.168.20.66 ff 
Aug 16 18:26:29 192.168.20.66 ff 
Aug 16 18:26:29 192.168.20.66 ff 
Aug 16 18:26:29 192.168.20.66 83 
Aug 16 18:26:29 192.168.20.66 f8 
Aug 16 18:26:29 192.168.20.66 02 
Aug 16 18:26:29 192.168.20.66 74 
Aug 16 18:26:29 192.168.20.66 15 
Aug 16 18:26:29 192.168.20.66 85 
Aug 16 18:26:29 192.168.20.66 c0 
Aug 16 18:26:29 192.168.20.66 74 
Aug 16 18:26:29 192.168.20.66 11 
Aug 16 18:26:29 192.168.20.66 8b 
Aug 16 18:26:29 192.168.20.66 36 
Aug 16 18:26:29 192.168.20.66 unparseable log message: "<8b> "
Aug 16 18:26:29 192.168.20.66 06 
Aug 16 18:26:29 192.168.20.66 0f 
Aug 16 18:26:29 192.168.20.66 18 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 90 
Aug 16 18:26:29 192.168.20.66 39 
Aug 16 18:26:29 192.168.20.66 fe 
Aug 16 18:26:29 192.168.20.66 75 
Aug 16 18:26:29 192.168.20.66 e1 
Aug 16 18:26:29 192.168.20.66 b8 
Aug 16 18:26:29 192.168.20.66 01 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 00 
Aug 16 18:26:29 192.168.20.66 5b 
Aug 16 18:26:29 192.168.20.66 5e 
Aug 16 18:26:29 192.168.20.66 5f 
Aug 16 18:26:29 192.168.20.66 5d 
Aug 16 18:26:29 192.168.20.66 c3 
Aug 16 18:26:29 192.168.20.66 55 
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] EIP: [<c013c083>] 
Aug 16 18:26:29 192.168.20.66 find_usage_forwards+0x7e/0x92
Aug 16 18:26:29 192.168.20.66 SS:ESP 0068:c0861e8c
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Kernel panic - not syncing: Fatal exception in interrupt
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] ------------[ cut here ]------------
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] WARNING: at kernel/smp.c:328 smp_call_function_mask+0x28/0x181()
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Modules linked in:
Aug 16 18:26:29 192.168.20.66 sch_ingress
Aug 16 18:26:29 192.168.20.66 sch_prio
Aug 16 18:26:29 192.168.20.66 netconsole
Aug 16 18:26:29 192.168.20.66 configfs
Aug 16 18:26:29 192.168.20.66 nf_nat_pptp
Aug 16 18:26:29 192.168.20.66 nf_conntrack_pptp
Aug 16 18:26:29 192.168.20.66 nf_conntrack_proto_gre
Aug 16 18:26:29 192.168.20.66 nf_nat_proto_gre
Aug 16 18:26:29 192.168.20.66 iTCO_wdt
Aug 16 18:26:29 192.168.20.66 e752x_edac
Aug 16 18:26:29 192.168.20.66 edac_core
Aug 16 18:26:29 192.168.20.66 cls_u32
Aug 16 18:26:29 192.168.20.66 em_meta
Aug 16 18:26:29 192.168.20.66 cls_basic
Aug 16 18:26:29 192.168.20.66 xt_dscp
Aug 16 18:26:29 192.168.20.66 xt_DSCP
Aug 16 18:26:29 192.168.20.66 ipt_REJECT
Aug 16 18:26:29 192.168.20.66 ts_bm
Aug 16 18:26:29 192.168.20.66 xt_string
Aug 16 18:26:29 192.168.20.66 ipt_ttl
Aug 16 18:26:29 192.168.20.66 ifb
Aug 16 18:26:29 192.168.20.66 cls_fw
Aug 16 18:26:29 192.168.20.66 sch_tbf
Aug 16 18:26:29 192.168.20.66 sch_htb
Aug 16 18:26:29 192.168.20.66 act_ipt
Aug 16 18:26:29 192.168.20.66 act_mirred
Aug 16 18:26:29 192.168.20.66 xt_MARK
Aug 16 18:26:29 192.168.20.66 pppoe
Aug 16 18:26:29 192.168.20.66 pppox
Aug 16 18:26:29 192.168.20.66 ppp_generic
Aug 16 18:26:29 192.168.20.66 slhc
Aug 16 18:26:29 192.168.20.66 xt_TCPMSS
Aug 16 18:26:29 192.168.20.66 xt_mark
Aug 16 18:26:29 192.168.20.66 xt_tcpudp
Aug 16 18:26:29 192.168.20.66 iptable_mangle
Aug 16 18:26:29 192.168.20.66 iptable_nat
Aug 16 18:26:29 192.168.20.66 nf_nat
Aug 16 18:26:29 192.168.20.66 nf_conntrack_ipv4
Aug 16 18:26:29 192.168.20.66 nf_conntrack
Aug 16 18:26:29 192.168.20.66 rtc_cmos
Aug 16 18:26:29 192.168.20.66 rtc_core
Aug 16 18:26:29 192.168.20.66 rtc_lib
Aug 16 18:26:29 192.168.20.66 iptable_filter
Aug 16 18:26:29 192.168.20.66 ip_tables
Aug 16 18:26:29 192.168.20.66 x_tables
Aug 16 18:26:29 192.168.20.66 8021q
Aug 16 18:26:29 192.168.20.66 garp
Aug 16 18:26:29 192.168.20.66 stp
Aug 16 18:26:29 192.168.20.66 llc
Aug 16 18:26:29 192.168.20.66 tun
Aug 16 18:26:29 192.168.20.66 loop
Aug 16 18:26:29 192.168.20.66 tulip
Aug 16 18:26:29 192.168.20.66 r8169
Aug 16 18:26:29 192.168.20.66 sky2
Aug 16 18:26:29 192.168.20.66 via_velocity
Aug 16 18:26:29 192.168.20.66 via_rhine
Aug 16 18:26:29 192.168.20.66 sis900
Aug 16 18:26:29 192.168.20.66 ne2k_pci
Aug 16 18:26:29 192.168.20.66 8390
Aug 16 18:26:29 192.168.20.66 skge
Aug 16 18:26:29 192.168.20.66 tg3
Aug 16 18:26:29 192.168.20.66 libphy
Aug 16 18:26:29 192.168.20.66 8139too
Aug 16 18:26:29 192.168.20.66 e1000
Aug 16 18:26:29 192.168.20.66 e100
Aug 16 18:26:29 192.168.20.66 usb_storage
Aug 16 18:26:29 192.168.20.66 mtdblock
Aug 16 18:26:29 192.168.20.66 mtd_blkdevs
Aug 16 18:26:29 192.168.20.66 usbhid
Aug 16 18:26:29 192.168.20.66 uhci_hcd
Aug 16 18:26:29 192.168.20.66 ehci_hcd
Aug 16 18:26:29 192.168.20.66 ohci_hcd
Aug 16 18:26:29 192.168.20.66 usbcore
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Pid: 31408, comm: tc Tainted: G      D   2.6.27-rc3-git2-build-0031 #17
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bb7f3>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 printk+0xf/0x14
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0121be8>] 
Aug 16 18:26:29 192.168.20.66 warn_on_slowpath+0x41/0x65
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013c7d8>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_off+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c026caf9>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 netpoll_send_skb+0xdc/0x15b
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c026cd49>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 netpoll_send_udp+0x1d1/0x1dc
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013c7d8>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_off+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bdbf0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 _spin_unlock_irqrestore+0x2f/0x3c
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<f8a56178>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 write_msg+0xb1/0xb9 [netconsole]
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013c7d8>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_off+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01430b4>] 
Aug 16 18:26:29 192.168.20.66 smp_call_function_mask+0x28/0x181
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010ef59>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 stop_this_cpu+0x0/0x3b
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01f1a97>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 vgacon_cursor+0x170/0x177
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c014321f>] 
Aug 16 18:26:29 192.168.20.66 smp_call_function+0x12/0x14
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010efaf>] 
Aug 16 18:26:29 192.168.20.66 native_smp_send_stop+0x1b/0x40
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bb74c>] 
Aug 16 18:26:29 192.168.20.66 panic+0x48/0xe0
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0104945>] 
Aug 16 18:26:29 192.168.20.66 oops_end+0x73/0x87
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0105138>] 
Aug 16 18:26:29 192.168.20.66 die+0x5b/0x63
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0114512>] 
Aug 16 18:26:29 192.168.20.66 do_page_fault+0x532/0x624
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013dabf>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013da7d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_caller+0xcb/0x102
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025c02c>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 kfree_skb+0x29/0x2b
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02afbe2>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 packet_rcv+0x313/0x31d
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013d91c>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 mark_held_locks+0x53/0x6a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013dabf>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013da7d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_caller+0xcb/0x102
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0261c31>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 dev_queue_xmit+0x44e/0x481
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013dabf>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0113fe0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 do_page_fault+0x0/0x624
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02be05a>] 
Aug 16 18:26:29 192.168.20.66 error_code+0x72/0x78
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013c083>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 find_usage_forwards+0x7e/0x92
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013d039>] 
Aug 16 18:26:29 192.168.20.66 check_usage_forwards+0x54/0x76
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013d477>] 
Aug 16 18:26:29 192.168.20.66 mark_lock+0x3a6/0x7f8
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013e6de>] 
Aug 16 18:26:29 192.168.20.66 __lock_acquire+0x506/0x118a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01dd5d0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_thunk+0xc/0x10
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013da7d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_caller+0xcb/0x102
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01dd5d0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_thunk+0xc/0x10
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c014a40d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 handle_fasteoi_irq+0xb0/0xb6
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013f3aa>] 
Aug 16 18:26:29 192.168.20.66 lock_acquire+0x48/0x64
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x51/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bdcc1>] 
Aug 16 18:26:29 192.168.20.66 _spin_lock+0x1b/0x2a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x51/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 est_timer+0x51/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0129530>] 
Aug 16 18:26:29 192.168.20.66 run_timer_softirq+0x11d/0x176
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025edc6>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x0/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0126203>] 
Aug 16 18:26:29 192.168.20.66 __do_softirq+0x7b/0xf4
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0126188>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 __do_softirq+0x0/0xf4
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01060b3>] 
Aug 16 18:26:29 192.168.20.66 do_softirq+0x65/0xb6
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0125e28>] 
Aug 16 18:26:29 192.168.20.66 irq_exit+0x44/0x79
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010ff15>] 
Aug 16 18:26:29 192.168.20.66 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010438d>] 
Aug 16 18:26:29 192.168.20.66 apic_timer_interrupt+0x2d/0x34
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  =======================
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] ---[ end trace 188e55215fc92b42 ]---
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] ------------[ cut here ]------------
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] WARNING: at kernel/smp.c:217 smp_call_function_single+0x2d/0x9d()
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Modules linked in:
Aug 16 18:26:29 192.168.20.66 sch_ingress
Aug 16 18:26:29 192.168.20.66 sch_prio
Aug 16 18:26:29 192.168.20.66 netconsole
Aug 16 18:26:29 192.168.20.66 configfs
Aug 16 18:26:29 192.168.20.66 nf_nat_pptp
Aug 16 18:26:29 192.168.20.66 nf_conntrack_pptp
Aug 16 18:26:29 192.168.20.66 nf_conntrack_proto_gre
Aug 16 18:26:29 192.168.20.66 nf_nat_proto_gre
Aug 16 18:26:29 192.168.20.66 iTCO_wdt
Aug 16 18:26:29 192.168.20.66 e752x_edac
Aug 16 18:26:29 192.168.20.66 edac_core
Aug 16 18:26:29 192.168.20.66 cls_u32
Aug 16 18:26:29 192.168.20.66 em_meta
Aug 16 18:26:29 192.168.20.66 cls_basic
Aug 16 18:26:29 192.168.20.66 xt_dscp
Aug 16 18:26:29 192.168.20.66 xt_DSCP
Aug 16 18:26:29 192.168.20.66 ipt_REJECT
Aug 16 18:26:29 192.168.20.66 ts_bm
Aug 16 18:26:29 192.168.20.66 xt_string
Aug 16 18:26:29 192.168.20.66 ipt_ttl
Aug 16 18:26:29 192.168.20.66 ifb
Aug 16 18:26:29 192.168.20.66 cls_fw
Aug 16 18:26:29 192.168.20.66 sch_tbf
Aug 16 18:26:29 192.168.20.66 sch_htb
Aug 16 18:26:29 192.168.20.66 act_ipt
Aug 16 18:26:29 192.168.20.66 act_mirred
Aug 16 18:26:29 192.168.20.66 xt_MARK
Aug 16 18:26:29 192.168.20.66 pppoe
Aug 16 18:26:29 192.168.20.66 pppox
Aug 16 18:26:29 192.168.20.66 ppp_generic
Aug 16 18:26:29 192.168.20.66 slhc
Aug 16 18:26:29 192.168.20.66 xt_TCPMSS
Aug 16 18:26:29 192.168.20.66 xt_mark
Aug 16 18:26:29 192.168.20.66 xt_tcpudp
Aug 16 18:26:29 192.168.20.66 iptable_mangle
Aug 16 18:26:29 192.168.20.66 iptable_nat
Aug 16 18:26:29 192.168.20.66 nf_nat
Aug 16 18:26:29 192.168.20.66 nf_conntrack_ipv4
Aug 16 18:26:29 192.168.20.66 nf_conntrack
Aug 16 18:26:29 192.168.20.66 rtc_cmos
Aug 16 18:26:29 192.168.20.66 rtc_core
Aug 16 18:26:29 192.168.20.66 rtc_lib
Aug 16 18:26:29 192.168.20.66 iptable_filter
Aug 16 18:26:29 192.168.20.66 ip_tables
Aug 16 18:26:29 192.168.20.66 x_tables
Aug 16 18:26:29 192.168.20.66 8021q
Aug 16 18:26:29 192.168.20.66 garp
Aug 16 18:26:29 192.168.20.66 stp
Aug 16 18:26:29 192.168.20.66 llc
Aug 16 18:26:29 192.168.20.66 tun
Aug 16 18:26:29 192.168.20.66 loop
Aug 16 18:26:29 192.168.20.66 tulip
Aug 16 18:26:29 192.168.20.66 r8169
Aug 16 18:26:29 192.168.20.66 sky2
Aug 16 18:26:29 192.168.20.66 via_velocity
Aug 16 18:26:29 192.168.20.66 via_rhine
Aug 16 18:26:29 192.168.20.66 sis900
Aug 16 18:26:29 192.168.20.66 ne2k_pci
Aug 16 18:26:29 192.168.20.66 8390
Aug 16 18:26:29 192.168.20.66 skge
Aug 16 18:26:29 192.168.20.66 tg3
Aug 16 18:26:29 192.168.20.66 libphy
Aug 16 18:26:29 192.168.20.66 8139too
Aug 16 18:26:29 192.168.20.66 e1000
Aug 16 18:26:29 192.168.20.66 e100
Aug 16 18:26:29 192.168.20.66 usb_storage
Aug 16 18:26:29 192.168.20.66 mtdblock
Aug 16 18:26:29 192.168.20.66 mtd_blkdevs
Aug 16 18:26:29 192.168.20.66 usbhid
Aug 16 18:26:29 192.168.20.66 uhci_hcd
Aug 16 18:26:29 192.168.20.66 ehci_hcd
Aug 16 18:26:29 192.168.20.66 ohci_hcd
Aug 16 18:26:29 192.168.20.66 usbcore
Aug 16 18:26:29 192.168.20.66 
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Pid: 31408, comm: tc Tainted: G      D W 2.6.27-rc3-git2-build-0031 #17
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bb7f3>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 printk+0xf/0x14
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0121be8>] 
Aug 16 18:26:29 192.168.20.66 warn_on_slowpath+0x41/0x65
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013c7d8>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_off+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c026caf9>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 netpoll_send_skb+0xdc/0x15b
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c026cd49>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 netpoll_send_udp+0x1d1/0x1dc
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013c7d8>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_off+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bdbf0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 _spin_unlock_irqrestore+0x2f/0x3c
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<f8a56178>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 write_msg+0xb1/0xb9 [netconsole]
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c014301c>] 
Aug 16 18:26:29 192.168.20.66 smp_call_function_single+0x2d/0x9d
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010ef59>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 stop_this_cpu+0x0/0x3b
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0143103>] 
Aug 16 18:26:29 192.168.20.66 smp_call_function_mask+0x77/0x181
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010ef59>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 stop_this_cpu+0x0/0x3b
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c014321f>] 
Aug 16 18:26:29 192.168.20.66 smp_call_function+0x12/0x14
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010efaf>] 
Aug 16 18:26:29 192.168.20.66 native_smp_send_stop+0x1b/0x40
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bb74c>] 
Aug 16 18:26:29 192.168.20.66 panic+0x48/0xe0
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0104945>] 
Aug 16 18:26:29 192.168.20.66 oops_end+0x73/0x87
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0105138>] 
Aug 16 18:26:29 192.168.20.66 die+0x5b/0x63
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0114512>] 
Aug 16 18:26:29 192.168.20.66 do_page_fault+0x532/0x624
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013dabf>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013da7d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_caller+0xcb/0x102
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025c02c>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 kfree_skb+0x29/0x2b
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02afbe2>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 packet_rcv+0x313/0x31d
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013d91c>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 mark_held_locks+0x53/0x6a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013dabf>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013da7d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_caller+0xcb/0x102
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0261c31>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 dev_queue_xmit+0x44e/0x481
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013dabf>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on+0xb/0xd
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0113fe0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 do_page_fault+0x0/0x624
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02be05a>] 
Aug 16 18:26:29 192.168.20.66 error_code+0x72/0x78
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013c083>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 find_usage_forwards+0x7e/0x92
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013d039>] 
Aug 16 18:26:29 192.168.20.66 check_usage_forwards+0x54/0x76
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013d477>] 
Aug 16 18:26:29 192.168.20.66 mark_lock+0x3a6/0x7f8
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013e6de>] 
Aug 16 18:26:29 192.168.20.66 __lock_acquire+0x506/0x118a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01dd5d0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_thunk+0xc/0x10
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013da7d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_caller+0xcb/0x102
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01dd5d0>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 trace_hardirqs_on_thunk+0xc/0x10
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c014a40d>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 handle_fasteoi_irq+0xb0/0xb6
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c013f3aa>] 
Aug 16 18:26:29 192.168.20.66 lock_acquire+0x48/0x64
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x51/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c02bdcc1>] 
Aug 16 18:26:29 192.168.20.66 _spin_lock+0x1b/0x2a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x51/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025ee17>] 
Aug 16 18:26:29 192.168.20.66 est_timer+0x51/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0129530>] 
Aug 16 18:26:29 192.168.20.66 run_timer_softirq+0x11d/0x176
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c025edc6>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 est_timer+0x0/0x14a
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0126203>] 
Aug 16 18:26:29 192.168.20.66 __do_softirq+0x7b/0xf4
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0126188>] 
Aug 16 18:26:29 192.168.20.66 ? 
Aug 16 18:26:29 192.168.20.66 __do_softirq+0x0/0xf4
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c01060b3>] 
Aug 16 18:26:29 192.168.20.66 do_softirq+0x65/0xb6
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c0125e28>] 
Aug 16 18:26:29 192.168.20.66 irq_exit+0x44/0x79
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010ff15>] 
Aug 16 18:26:29 192.168.20.66 smp_apic_timer_interrupt+0x6f/0x7d
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  [<c010438d>] 
Aug 16 18:26:29 192.168.20.66 apic_timer_interrupt+0x2d/0x34
Aug 16 18:26:29 192.168.20.66 [ 2920.823068]  =======================
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] ---[ end trace 188e55215fc92b42 ]---
Aug 16 18:26:29 192.168.20.66 [ 2920.823068] Rebooting in 5 seconds..


On Saturday 16 August 2008, Jarek Poplawski wrote:
> On Sat, Aug 16, 2008 at 04:13:59PM +0300, Denys Fedoryshchenko wrote:
> ...
>

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 15:28                         ` Denys Fedoryshchenko
@ 2008-08-16 15:59                           ` Jarek Poplawski
  2008-08-16 18:15                             ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-16 15:59 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sat, Aug 16, 2008 at 06:28:48PM +0300, Denys Fedoryshchenko wrote:
> Triggered panic by "batch" shaper "reapply".
> 
> Aug 16 18:26:29 192.168.20.66 [ 2920.823068] BUG: unable to handle kernel 
> Aug 16 18:26:29 192.168.20.66 NULL pointer dereference
> Aug 16 18:26:29 192.168.20.66 at 00000000
> Aug 16 18:26:29 192.168.20.66 [ 2920.823068] IP:
> Aug 16 18:26:29 192.168.20.66 [<c013c083>] find_usage_forwards+0x7e/0x92
...

This seems to be a different bug. Did you run this earlier (before the
last two patches)? I'd prefer to fix this oops from the beginning of
this thread first, so it would be better to test these patches without
changing conditions.

Jarek P.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 15:59                           ` Jarek Poplawski
@ 2008-08-16 18:15                             ` Denys Fedoryshchenko
  2008-08-16 18:52                               ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-16 18:15 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Yes, i did apply fix for another problem.

Other patches applied:
[PATCH] pkt_sched: Fix unlocking in tc_ctl_tfilter()
[PATCH] pkt_sched: Add lockdep annotation for qdisc locks

On Saturday 16 August 2008, Jarek Poplawski wrote:
> On Sat, Aug 16, 2008 at 06:28:48PM +0300, Denys Fedoryshchenko wrote:
> > Triggered panic by "batch" shaper "reapply".
> >
> > Aug 16 18:26:29 192.168.20.66 [ 2920.823068] BUG: unable to handle kernel
> > Aug 16 18:26:29 192.168.20.66 NULL pointer dereference
> > Aug 16 18:26:29 192.168.20.66 at 00000000
> > Aug 16 18:26:29 192.168.20.66 [ 2920.823068] IP:
> > Aug 16 18:26:29 192.168.20.66 [<c013c083>] find_usage_forwards+0x7e/0x92
>
> ...
>
> This seems to be a different bug. Did you run this earlier (before the
> last two patches)? I'd prefer to fix this oops from the beginning of
> this thread first, so it would be better to test these patches without
> changing conditions.
>
> Jarek P.
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 18:15                             ` Denys Fedoryshchenko
@ 2008-08-16 18:52                               ` Jarek Poplawski
  2008-08-16 19:22                                 ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-16 18:52 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sat, Aug 16, 2008 at 09:15:16PM +0300, Denys Fedoryshchenko wrote:
> Yes, i did apply fix for another problem.
> 
> Other patches applied:
> [PATCH] pkt_sched: Fix unlocking in tc_ctl_tfilter()
> [PATCH] pkt_sched: Add lockdep annotation for qdisc locks

These patches shouldn't change so much. I meant: was this last patch
(with previous one ie. take #3) tested the same as earlier take #3
alone? (There were warnings only, no oopses.)

BTW, here is a patch with another way of fixing this. I would be glad
if you could try this too. The previous patch should be reverted, so
we need only: "take #3", this one and these two you mentioned above.

Jarek P.

---

 include/linux/netdevice.h |    1 +
 include/net/sch_generic.h |    5 ++++-
 net/core/dev.c            |    1 +
 3 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 488c56e..7041c2c 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -445,6 +445,7 @@ struct netdev_queue {
 	struct net_device	*dev;
 	struct Qdisc		*qdisc;
 	unsigned long		state;
+	spinlock_t		qdisc_lock;
 	spinlock_t		_xmit_lock;
 	int			xmit_lock_owner;
 	struct Qdisc		*qdisc_sleeping;
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index a7abfda..f84e96c 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -185,7 +185,10 @@ static inline struct qdisc_skb_cb *qdisc_skb_cb(struct sk_buff *skb)
 
 static inline spinlock_t *qdisc_lock(struct Qdisc *qdisc)
 {
-	return &qdisc->q.lock;
+	if (unlikely(qdisc->flags & TCQ_F_BUILTIN))
+		return &qdisc->q.lock;
+
+	return &qdisc->dev_queue->qdisc_lock;
 }
 
 static inline struct Qdisc *qdisc_root(struct Qdisc *qdisc)
diff --git a/net/core/dev.c b/net/core/dev.c
index 600bb23..9ec20e0 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3859,6 +3859,7 @@ static void __netdev_init_queue_locks_one(struct net_device *dev,
 					  void *_unused)
 {
 	spin_lock_init(&dev_queue->_xmit_lock);
+	spin_lock_init(&dev_queue->qdisc_lock);
 	netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
 	dev_queue->xmit_lock_owner = -1;
 }

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 18:52                               ` Jarek Poplawski
@ 2008-08-16 19:22                                 ` Denys Fedoryshchenko
  2008-08-16 19:47                                   ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-16 19:22 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

> These patches shouldn't change so much. I meant: was this last patch
> (with previous one ie. take #3) tested the same as earlier take #3
> alone? (There were warnings only, no oopses.)
Yes, sure with take #3 also.

Warnings disappeared, but kernel paniced, and it is really looks like another 
bug. I will try also to enable more debug options at night (after 2-3 hours, 
when traffic is low), including intensive memory debugging, maybe i can catch 
some memory poisoning.

>
> BTW, here is a patch with another way of fixing this. I would be glad
> if you could try this too. The previous patch should be reverted, so
> we need only: "take #3", this one and these two you mentioned above.
>
> Jarek P.
>
> ---
>
>  include/linux/netdevice.h |    1 +
>  include/net/sch_generic.h |    5 ++++-
>  net/core/dev.c            |    1 +
>  3 files changed, 6 insertions(+), 1 deletions(-)
>
> diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> index 488c56e..7041c2c 100644
> --- a/include/linux/netdevice.h
> +++ b/include/linux/netdevice.h
> @@ -445,6 +445,7 @@ struct netdev_queue {
>  	struct net_device	*dev;
>  	struct Qdisc		*qdisc;
>  	unsigned long		state;
> +	spinlock_t		qdisc_lock;
>  	spinlock_t		_xmit_lock;
>  	int			xmit_lock_owner;
>  	struct Qdisc		*qdisc_sleeping;
> diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
> index a7abfda..f84e96c 100644
> --- a/include/net/sch_generic.h
> +++ b/include/net/sch_generic.h
> @@ -185,7 +185,10 @@ static inline struct qdisc_skb_cb *qdisc_skb_cb(struct
> sk_buff *skb)
>
>  static inline spinlock_t *qdisc_lock(struct Qdisc *qdisc)
>  {
> -	return &qdisc->q.lock;
> +	if (unlikely(qdisc->flags & TCQ_F_BUILTIN))
> +		return &qdisc->q.lock;
> +
> +	return &qdisc->dev_queue->qdisc_lock;
>  }
>
>  static inline struct Qdisc *qdisc_root(struct Qdisc *qdisc)
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 600bb23..9ec20e0 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -3859,6 +3859,7 @@ static void __netdev_init_queue_locks_one(struct
> net_device *dev, void *_unused)
>  {
>  	spin_lock_init(&dev_queue->_xmit_lock);
> +	spin_lock_init(&dev_queue->qdisc_lock);
>  	netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
>  	dev_queue->xmit_lock_owner = -1;
>  }
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 19:22                                 ` Denys Fedoryshchenko
@ 2008-08-16 19:47                                   ` Jarek Poplawski
  2008-08-17  9:02                                     ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-16 19:47 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sat, Aug 16, 2008 at 10:22:56PM +0300, Denys Fedoryshchenko wrote:
> > These patches shouldn't change so much. I meant: was this last patch
> > (with previous one ie. take #3) tested the same as earlier take #3
> > alone? (There were warnings only, no oopses.)
> Yes, sure with take #3 also.
> 
> Warnings disappeared, but kernel paniced, and it is really looks like another 
> bug. I will try also to enable more debug options at night (after 2-3 hours, 
> when traffic is low), including intensive memory debugging, maybe i can catch 
> some memory poisoning.

I only wanted to be sure if the reason(s) of this first oops was found
before looking for the next problem. They could be connected yet.
Anyway, I'm currently most interested in this patch below, so I could
know the better base for next fixes.

Jarek P. 

> 
> >
> > BTW, here is a patch with another way of fixing this. I would be glad
> > if you could try this too. The previous patch should be reverted, so
> > we need only: "take #3", this one and these two you mentioned above.
> >
> > Jarek P.
> >
> > ---
> >
> >  include/linux/netdevice.h |    1 +
> >  include/net/sch_generic.h |    5 ++++-
> >  net/core/dev.c            |    1 +
> >  3 files changed, 6 insertions(+), 1 deletions(-)
> >
> > diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> > index 488c56e..7041c2c 100644
> > --- a/include/linux/netdevice.h
> > +++ b/include/linux/netdevice.h
> > @@ -445,6 +445,7 @@ struct netdev_queue {
> >  	struct net_device	*dev;
> >  	struct Qdisc		*qdisc;
> >  	unsigned long		state;
> > +	spinlock_t		qdisc_lock;
> >  	spinlock_t		_xmit_lock;
> >  	int			xmit_lock_owner;
> >  	struct Qdisc		*qdisc_sleeping;
> > diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
> > index a7abfda..f84e96c 100644
> > --- a/include/net/sch_generic.h
> > +++ b/include/net/sch_generic.h
> > @@ -185,7 +185,10 @@ static inline struct qdisc_skb_cb *qdisc_skb_cb(struct
> > sk_buff *skb)
> >
> >  static inline spinlock_t *qdisc_lock(struct Qdisc *qdisc)
> >  {
> > -	return &qdisc->q.lock;
> > +	if (unlikely(qdisc->flags & TCQ_F_BUILTIN))
> > +		return &qdisc->q.lock;
> > +
> > +	return &qdisc->dev_queue->qdisc_lock;
> >  }
> >
> >  static inline struct Qdisc *qdisc_root(struct Qdisc *qdisc)
> > diff --git a/net/core/dev.c b/net/core/dev.c
> > index 600bb23..9ec20e0 100644
> > --- a/net/core/dev.c
> > +++ b/net/core/dev.c
> > @@ -3859,6 +3859,7 @@ static void __netdev_init_queue_locks_one(struct
> > net_device *dev, void *_unused)
> >  {
> >  	spin_lock_init(&dev_queue->_xmit_lock);
> > +	spin_lock_init(&dev_queue->qdisc_lock);
> >  	netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
> >  	dev_queue->xmit_lock_owner = -1;
> >  }
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-16 19:47                                   ` Jarek Poplawski
@ 2008-08-17  9:02                                     ` Denys Fedoryshchenko
  2008-08-17  9:34                                       ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17  9:02 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Is it was a fix? Because for now it is running fine from night (but i have 
enabled some minor debug options in kernel). No crash or warning yet.

I disable now debug as it was before and testing more.

On Saturday 16 August 2008, Jarek Poplawski wrote:
> > >  	spin_lock_init(&dev_queue->_xmit_lock);
> > > +	spin_lock_init(&dev_queue->qdisc_lock);
> > >  	netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
> > >  	dev_queue->xmit_lock_owner = -1;
> > >  }
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17  9:02                                     ` Denys Fedoryshchenko
@ 2008-08-17  9:34                                       ` Jarek Poplawski
  2008-08-17  9:35                                         ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17  9:34 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sun, Aug 17, 2008 at 12:02:13PM +0300, Denys Fedoryshchenko wrote:
> Is it was a fix? Because for now it is running fine from night (but i have 
> enabled some minor debug options in kernel). No crash or warning yet.
> 
> I disable now debug as it was before and testing more.
> 

Only the patch named "take #3" is debugging only, and the latter 2
patches were intended to fix things. As a matter of fact the first
fixing patch should be needed as well, so if it's possible try to add
it again without reverting anything. It can show some minor offset
while applying now. I resend it below for any case.

Thanks,
Jarek P.

---

 include/net/sch_generic.h |    5 +++++
 net/sched/sch_api.c       |    2 +-
 2 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index a7abfda..4f28fa7 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -193,6 +193,11 @@ static inline struct Qdisc *qdisc_root(struct Qdisc *qdisc)
 	return qdisc->dev_queue->qdisc;
 }
 
+static inline struct Qdisc *qdisc_root_sleeping(struct Qdisc *qdisc)
+{
+	return qdisc->dev_queue->qdisc_sleeping;
+}
+
 /* The qdisc root lock is a mechanism by which to top level
  * of a qdisc tree can be locked from any qdisc node in the
  * forest.  This allows changing the configuration of some
diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
index c25465e..5c7ae56 100644
--- a/net/sched/sch_api.c
+++ b/net/sched/sch_api.c
@@ -435,7 +435,7 @@ void qdisc_watchdog_init(struct qdisc_watchdog *wd, struct Qdisc *qdisc)
 {
 	hrtimer_init(&wd->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
 	wd->timer.function = qdisc_watchdog;
-	wd->qdisc = qdisc;
+	wd->qdisc = qdisc_root_sleeping(qdisc);
 }
 EXPORT_SYMBOL(qdisc_watchdog_init);
 

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17  9:34                                       ` Jarek Poplawski
@ 2008-08-17  9:35                                         ` Denys Fedoryshchenko
  2008-08-17  9:52                                           ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17  9:35 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

I have applied now 
01-fix-pkt_sched: Fix unloc~ in tc_ctl_tfilter()
02-lockdep.patch (it is adding in sch_api.c lockdep_set_class)
"take 3" patch
and latest patch you sent

Is it correct?

On Sunday 17 August 2008, Jarek Poplawski wrote:
> On Sun, Aug 17, 2008 at 12:02:13PM +0300, Denys Fedoryshchenko wrote:
> > Is it was a fix? Because for now it is running fine from night (but i
> > have enabled some minor debug options in kernel). No crash or warning
> > yet.
> >
> > I disable now debug as it was before and testing more.
>
> Only the patch named "take #3" is debugging only, and the latter 2
> patches were intended to fix things. As a matter of fact the first
> fixing patch should be needed as well, so if it's possible try to add
> it again without reverting anything. It can show some minor offset
> while applying now. I resend it below for any case.
>
> Thanks,
> Jarek P.
>
> ---
>
>  include/net/sch_generic.h |    5 +++++
>  net/sched/sch_api.c       |    2 +-
>  2 files changed, 6 insertions(+), 1 deletions(-)
>
> diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
> index a7abfda..4f28fa7 100644
> --- a/include/net/sch_generic.h
> +++ b/include/net/sch_generic.h
> @@ -193,6 +193,11 @@ static inline struct Qdisc *qdisc_root(struct Qdisc
> *qdisc) return qdisc->dev_queue->qdisc;
>  }
>
> +static inline struct Qdisc *qdisc_root_sleeping(struct Qdisc *qdisc)
> +{
> +	return qdisc->dev_queue->qdisc_sleeping;
> +}
> +
>  /* The qdisc root lock is a mechanism by which to top level
>   * of a qdisc tree can be locked from any qdisc node in the
>   * forest.  This allows changing the configuration of some
> diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
> index c25465e..5c7ae56 100644
> --- a/net/sched/sch_api.c
> +++ b/net/sched/sch_api.c
> @@ -435,7 +435,7 @@ void qdisc_watchdog_init(struct qdisc_watchdog *wd,
> struct Qdisc *qdisc) {
>  	hrtimer_init(&wd->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
>  	wd->timer.function = qdisc_watchdog;
> -	wd->qdisc = qdisc;
> +	wd->qdisc = qdisc_root_sleeping(qdisc);
>  }
>  EXPORT_SYMBOL(qdisc_watchdog_init);



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17  9:35                                         ` Denys Fedoryshchenko
@ 2008-08-17  9:52                                           ` Jarek Poplawski
  2008-08-17 10:01                                             ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17  9:52 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sun, Aug 17, 2008 at 12:35:46PM +0300, Denys Fedoryshchenko wrote:
> I have applied now 
> 01-fix-pkt_sched: Fix unloc~ in tc_ctl_tfilter()
> 02-lockdep.patch (it is adding in sch_api.c lockdep_set_class)
> "take 3" patch
+ fixing patch #1 (resend just before)
+ fixing patch #2 (resend below)

Jarek P.

--- (resend fixing patch #2)

 include/linux/netdevice.h |    1 +
 include/net/sch_generic.h |    5 ++++-
 net/core/dev.c            |    1 +
 3 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 488c56e..7041c2c 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -445,6 +445,7 @@ struct netdev_queue {
 	struct net_device	*dev;
 	struct Qdisc		*qdisc;
 	unsigned long		state;
+	spinlock_t		qdisc_lock;
 	spinlock_t		_xmit_lock;
 	int			xmit_lock_owner;
 	struct Qdisc		*qdisc_sleeping;
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index a7abfda..f84e96c 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -185,7 +185,10 @@ static inline struct qdisc_skb_cb *qdisc_skb_cb(struct sk_buff *skb)
 
 static inline spinlock_t *qdisc_lock(struct Qdisc *qdisc)
 {
-	return &qdisc->q.lock;
+	if (unlikely(qdisc->flags & TCQ_F_BUILTIN))
+		return &qdisc->q.lock;
+
+	return &qdisc->dev_queue->qdisc_lock;
 }
 
 static inline struct Qdisc *qdisc_root(struct Qdisc *qdisc)
diff --git a/net/core/dev.c b/net/core/dev.c
index 600bb23..9ec20e0 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3859,6 +3859,7 @@ static void __netdev_init_queue_locks_one(struct net_device *dev,
 					  void *_unused)
 {
 	spin_lock_init(&dev_queue->_xmit_lock);
+	spin_lock_init(&dev_queue->qdisc_lock);
 	netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
 	dev_queue->xmit_lock_owner = -1;
 }

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17  9:52                                           ` Jarek Poplawski
@ 2008-08-17 10:01                                             ` Denys Fedoryshchenko
  2008-08-17 10:31                                               ` Jarek Poplawski
  2008-08-17 10:55                                               ` Jarek Poplawski
  0 siblings, 2 replies; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17 10:01 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Ok, now things finally organized.
fixing patch #1 was missing.

Rebooting system to this kernel. But strange thing, without fixing patch #1 it 
didn't crash for whole night, and 1-3 hours now (with same options as 
before).

Let's test now complete set of patches. I will keep around 2 hours on most 
loaded pppoe NAS, then distribute to 2-3 servers more if it doesn't crash.

On Sunday 17 August 2008, Jarek Poplawski wrote:
> On Sun, Aug 17, 2008 at 12:35:46PM +0300, Denys Fedoryshchenko wrote:
> > I have applied now
> > 01-fix-pkt_sched: Fix unloc~ in tc_ctl_tfilter()
> > 02-lockdep.patch (it is adding in sch_api.c lockdep_set_class)
> > "take 3" patch
>
> + fixing patch #1 (resend just before)
> + fixing patch #2 (resend below)
>
> Jarek P.
>
> --- (resend fixing patch #2)
>
>  include/linux/netdevice.h |    1 +
>  include/net/sch_generic.h |    5 ++++-
>  net/core/dev.c            |    1 +
>  3 files changed, 6 insertions(+), 1 deletions(-)
>
> diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> index 488c56e..7041c2c 100644
> --- a/include/linux/netdevice.h
> +++ b/include/linux/netdevice.h
> @@ -445,6 +445,7 @@ struct netdev_queue {
>  	struct net_device	*dev;
>  	struct Qdisc		*qdisc;
>  	unsigned long		state;
> +	spinlock_t		qdisc_lock;
>  	spinlock_t		_xmit_lock;
>  	int			xmit_lock_owner;
>  	struct Qdisc		*qdisc_sleeping;
> diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
> index a7abfda..f84e96c 100644
> --- a/include/net/sch_generic.h
> +++ b/include/net/sch_generic.h
> @@ -185,7 +185,10 @@ static inline struct qdisc_skb_cb *qdisc_skb_cb(struct
> sk_buff *skb)
>
>  static inline spinlock_t *qdisc_lock(struct Qdisc *qdisc)
>  {
> -	return &qdisc->q.lock;
> +	if (unlikely(qdisc->flags & TCQ_F_BUILTIN))
> +		return &qdisc->q.lock;
> +
> +	return &qdisc->dev_queue->qdisc_lock;
>  }
>
>  static inline struct Qdisc *qdisc_root(struct Qdisc *qdisc)
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 600bb23..9ec20e0 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -3859,6 +3859,7 @@ static void __netdev_init_queue_locks_one(struct
> net_device *dev, void *_unused)
>  {
>  	spin_lock_init(&dev_queue->_xmit_lock);
> +	spin_lock_init(&dev_queue->qdisc_lock);
>  	netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
>  	dev_queue->xmit_lock_owner = -1;
>  }
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 10:01                                             ` Denys Fedoryshchenko
@ 2008-08-17 10:31                                               ` Jarek Poplawski
  2008-08-17 10:55                                               ` Jarek Poplawski
  1 sibling, 0 replies; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17 10:31 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sun, Aug 17, 2008 at 01:01:55PM +0300, Denys Fedoryshchenko wrote:
> Ok, now things finally organized.
> fixing patch #1 was missing.
> 
> Rebooting system to this kernel. But strange thing, without fixing patch #1 it 
> didn't crash for whole night, and 1-3 hours now (with same options as 
> before).

The fixing patch #1 was intended to fix only one issue shown in the
warnings: qdisc_watchdog scheduled leaf qdiscs for qdisc_run(), so
they were dequeued directly (skipping root qdiscs) and with wrong
locks.

The fixing patch #2 changes qdisc locking generally, so it probably
fixed other locking issues btw. like this last oops seen after the
patch #1. But still there is this (not critical) issue of skipping
root qdiscs while dequeuing after qdisc_watchdog, which has to be done
by the fixing patch #1.

Jarek P.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 10:01                                             ` Denys Fedoryshchenko
  2008-08-17 10:31                                               ` Jarek Poplawski
@ 2008-08-17 10:55                                               ` Jarek Poplawski
  2008-08-17 12:06                                                 ` Denys Fedoryshchenko
  1 sibling, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17 10:55 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sun, Aug 17, 2008 at 01:01:55PM +0300, Denys Fedoryshchenko wrote:
> Ok, now things finally organized.
> fixing patch #1 was missing.
> 
> Rebooting system to this kernel. But strange thing, without fixing patch #1 it 
> didn't crash for whole night, and 1-3 hours now (with same options as 
> before).
> 
> Let's test now complete set of patches. I will keep around 2 hours on most 
> loaded pppoe NAS, then distribute to 2-3 servers more if it doesn't crash.

BTW, after you complete this testing (no hurry) I would be glad if you
could try one more patch which I send earlier to the list. IMHO, it's
needed to fix some other locking problems. This patch could be applied
and tested as an addition to all currently tested patches (but let's
first be sure they really work).

Thanks,
Jarek P.

------------->

pkt_sched: Destroy qdiscs under rtnl_lock again.

We don't need to trigger __qdisc_destroy() as an RCU callback because
the use of qdisc isn't controlled by RCU alone: after querying RCU
with synchronize_rcu() in dev_deactivate() we additionaly wait in a
loop checking some flags. After the loop is done there could be no
outstanding use of the qdisc, so call_rcu() doesn't make any sense.

On the other hand, current calling Qdisc's ->destroy() from a softirq
context without locking (rtnl) can break various things like:
qdisc_put_rtab(), tcf_destroy_chain() (e.g. u32_destroy()), and
probably more.


Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>

---

 net/sched/sch_generic.c |    8 ++------
 1 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 4685746..e7379d2 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -518,12 +518,8 @@ void qdisc_reset(struct Qdisc *qdisc)
 }
 EXPORT_SYMBOL(qdisc_reset);
 
-/* this is the rcu callback function to clean up a qdisc when there
- * are no further references to it */
-
-static void __qdisc_destroy(struct rcu_head *head)
+static void __qdisc_destroy(struct Qdisc *qdisc)
 {
-	struct Qdisc *qdisc = container_of(head, struct Qdisc, q_rcu);
 	const struct Qdisc_ops  *ops = qdisc->ops;
 
 #ifdef CONFIG_NET_SCHED
@@ -554,7 +550,7 @@ void qdisc_destroy(struct Qdisc *qdisc)
 	if (qdisc->parent)
 		list_del(&qdisc->list);
 
-	call_rcu(&qdisc->q_rcu, __qdisc_destroy);
+	__qdisc_destroy(qdisc);
 }
 EXPORT_SYMBOL(qdisc_destroy);
 
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 10:55                                               ` Jarek Poplawski
@ 2008-08-17 12:06                                                 ` Denys Fedoryshchenko
  2008-08-17 12:18                                                   ` Denys Fedoryshchenko
  2008-08-17 12:38                                                   ` Jarek Poplawski
  0 siblings, 2 replies; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17 12:06 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Seems stable (without this patch yet) on normal shaper, and 
crashing on new version of shaper (it is doing very intensive 
classes creation/destroy).

I'm not sure if it is same bug, i will try to test on regular shaper
more things.

I will try to apply last patch also and test on another pppoe 
server.


Here is panic i got when i try to run alternative shaper.

Aug 17 15:01:54 192.168.20.66 [ 7222.089634] BUG: unable to handle kernel
Aug 17 15:01:54 192.168.20.66 paging request
Aug 17 15:01:54 192.168.20.66 at 50639a62
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] IP:
Aug 17 15:01:54 192.168.20.66 [<c025fd4e>] net_tx_action+0x78/0xd8
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Oops: 0000 [#1]
Aug 17 15:01:54 192.168.20.66 SMP
Aug 17 15:01:54 192.168.20.66
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Modules linked in:
Aug 17 15:01:54 192.168.20.66 cls_flow
Aug 17 15:01:54 192.168.20.66 act_police
Aug 17 15:01:54 192.168.20.66 sch_ingress
Aug 17 15:01:54 192.168.20.66 sch_prio
Aug 17 15:01:54 192.168.20.66 netconsole
Aug 17 15:01:54 192.168.20.66 configfs
Aug 17 15:01:54 192.168.20.66 nf_nat_pptp
Aug 17 15:01:54 192.168.20.66 nf_nat_proto_gre
Aug 17 15:01:54 192.168.20.66 nf_conntrack_pptp
Aug 17 15:01:54 192.168.20.66 nf_conntrack_proto_gre
Aug 17 15:01:54 192.168.20.66 iTCO_wdt
Aug 17 15:01:54 192.168.20.66 e752x_edac
Aug 17 15:01:54 192.168.20.66 edac_core
Aug 17 15:01:54 192.168.20.66 cls_u32
Aug 17 15:01:54 192.168.20.66 em_meta
Aug 17 15:01:54 192.168.20.66 cls_basic
Aug 17 15:01:54 192.168.20.66 xt_dscp
Aug 17 15:01:54 192.168.20.66 xt_DSCP
Aug 17 15:01:54 192.168.20.66 ipt_REJECT
Aug 17 15:01:54 192.168.20.66 ts_bm
Aug 17 15:01:54 192.168.20.66 xt_string
Aug 17 15:01:54 192.168.20.66 ipt_ttl
Aug 17 15:01:54 192.168.20.66 ifb
Aug 17 15:01:54 192.168.20.66 cls_fw
Aug 17 15:01:54 192.168.20.66 sch_tbf
Aug 17 15:01:54 192.168.20.66 sch_htb
Aug 17 15:01:54 192.168.20.66 act_ipt
Aug 17 15:01:54 192.168.20.66 act_mirred
Aug 17 15:01:54 192.168.20.66 xt_MARK
Aug 17 15:01:54 192.168.20.66 pppoe
Aug 17 15:01:54 192.168.20.66 pppox
Aug 17 15:01:54 192.168.20.66 ppp_generic
Aug 17 15:01:54 192.168.20.66 slhc
Aug 17 15:01:54 192.168.20.66 xt_TCPMSS
Aug 17 15:01:54 192.168.20.66 xt_mark
Aug 17 15:01:54 192.168.20.66 xt_tcpudp
Aug 17 15:01:54 192.168.20.66 iptable_mangle
Aug 17 15:01:54 192.168.20.66 iptable_nat
Aug 17 15:01:54 192.168.20.66 nf_nat
Aug 17 15:01:54 192.168.20.66 nf_conntrack_ipv4
Aug 17 15:01:54 192.168.20.66 nf_conntrack
Aug 17 15:01:54 192.168.20.66 rtc_cmos
Aug 17 15:01:54 192.168.20.66 rtc_core
Aug 17 15:01:54 192.168.20.66 rtc_lib
Aug 17 15:01:54 192.168.20.66 iptable_filter
Aug 17 15:01:54 192.168.20.66 ip_tables
Aug 17 15:01:54 192.168.20.66 x_tables
Aug 17 15:01:54 192.168.20.66 8021q
Aug 17 15:01:54 192.168.20.66 garp
Aug 17 15:01:54 192.168.20.66 stp
Aug 17 15:01:54 192.168.20.66 llc
Aug 17 15:01:54 192.168.20.66 tun
Aug 17 15:01:54 192.168.20.66 loop
Aug 17 15:01:54 192.168.20.66 tulip
Aug 17 15:01:54 192.168.20.66 r8169
Aug 17 15:01:54 192.168.20.66 sky2
Aug 17 15:01:54 192.168.20.66 via_velocity
Aug 17 15:01:54 192.168.20.66 via_rhine
Aug 17 15:01:54 192.168.20.66 sis900
Aug 17 15:01:54 192.168.20.66 ne2k_pci
Aug 17 15:01:54 192.168.20.66 8390
Aug 17 15:01:54 192.168.20.66 skge
Aug 17 15:01:54 192.168.20.66 tg3
Aug 17 15:01:54 192.168.20.66 libphy
Aug 17 15:01:54 192.168.20.66 8139too
Aug 17 15:01:54 192.168.20.66 e1000
Aug 17 15:01:54 192.168.20.66 e100
Aug 17 15:01:54 192.168.20.66 usb_storage
Aug 17 15:01:54 192.168.20.66 mtdblock
Aug 17 15:01:54 192.168.20.66 mtd_blkdevs
Aug 17 15:01:54 192.168.20.66 usbhid
Aug 17 15:01:54 192.168.20.66 uhci_hcd
Aug 17 15:01:54 192.168.20.66 ehci_hcd
Aug 17 15:01:54 192.168.20.66 ohci_hcd
Aug 17 15:01:54 192.168.20.66 usbcore
Aug 17 15:01:54 192.168.20.66
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Pid: 4, comm: ksoftirqd/0 Not tainted (2.6.27-rc3-git3-build-0031 #7)
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP: 0060:[<c025fd4e>] EFLAGS: 00010206 CPU: 0
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP is at net_tx_action+0x78/0xd8
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EAX: 01deb6a8 EBX: 506399ee ECX: 00000002 EDX: 00000001
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] ESI: f6429224 EDI: f6429238 EBP: c0861fd4 ESP: c0861fc4
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Process ksoftirqd/0 (pid: 4, ti=c0861000 task=f78399b0 task.ti=f7841000)
Aug 17 15:01:54 192.168.20.66
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Stack:
Aug 17 15:01:54 192.168.20.66 506399ee
Aug 17 15:01:54 192.168.20.66 c039ea88
Aug 17 15:01:54 192.168.20.66 c085a554
Aug 17 15:01:54 192.168.20.66 00000003
Aug 17 15:01:54 192.168.20.66 c0861ff8
Aug 17 15:01:54 192.168.20.66 c012626b
Aug 17 15:01:54 192.168.20.66 c085d700
Aug 17 15:01:54 192.168.20.66 c085d700
Aug 17 15:01:54 192.168.20.66
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
Aug 17 15:01:54 192.168.20.66 00000006
Aug 17 15:01:54 192.168.20.66 00000000
Aug 17 15:01:54 192.168.20.66 f7841fac
Aug 17 15:01:54 192.168.20.66 f7841000
Aug 17 15:01:54 192.168.20.66 c01261f0
Aug 17 15:01:54 192.168.20.66 f7841fbc
Aug 17 15:01:54 192.168.20.66 c01060b3
Aug 17 15:01:54 192.168.20.66
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Call Trace:
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c012626b>]
Aug 17 15:01:54 192.168.20.66 ?
Aug 17 15:01:54 192.168.20.66 __do_softirq+0x7b/0xf4
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01261f0>]
Aug 17 15:01:54 192.168.20.66 ?
Aug 17 15:01:54 192.168.20.66 __do_softirq+0x0/0xf4
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01060b3>]
Aug 17 15:01:54 192.168.20.66 ?
Aug 17 15:01:54 192.168.20.66 do_softirq+0x65/0xb6
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125cfc>]
Aug 17 15:01:54 192.168.20.66 ?
Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x59/0xc8
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125ca3>]
Aug 17 15:01:54 192.168.20.66 ?
Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x0/0xc8
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320e5>]
Aug 17 15:01:54 192.168.20.66 ?
Aug 17 15:01:54 192.168.20.66 k
Aug 17 15:01:54 192.168.20.66 thread+0x3b/0x61
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320aa>]
Aug 17 15:01:54 192.168.20.66 ?
Aug 17 15:01:54 192.168.20.66 kthread+0x0/0x61
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c010451b>]
Aug 17 15:01:54 192.168.20.66 ?
Aug 17 15:01:54 192.168.20.66 kernel_thread_helper+0x7/0x10
Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  =======================
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Code:
Aug 17 15:01:54 192.168.20.66 1e
Aug 17 15:01:54 192.168.20.66 ec
Aug 17 15:01:54 192.168.20.66 ff
Aug 17 15:01:54 192.168.20.66 89
Aug 17 15:01:54 192.168.20.66 d8
Aug 17 15:01:54 192.168.20.66 89
Aug 17 15:01:54 192.168.20.66 fb
Aug 17 15:01:54 192.168.20.66 e8
Aug 17 15:01:54 192.168.20.66 6e
Aug 17 15:01:54 192.168.20.66 c2
Aug 17 15:01:54 192.168.20.66 ff
Aug 17 15:01:54 192.168.20.66 ff
Aug 17 15:01:54 192.168.20.66 85
Aug 17 15:01:54 192.168.20.66 db
Aug 17 15:01:54 192.168.20.66 75
Aug 17 15:01:54 192.168.20.66 d9
Aug 17 15:01:54 192.168.20.66 83
Aug 17 15:01:54 192.168.20.66 3e
Aug 17 15:01:54 192.168.20.66 00
Aug 17 15:01:54 192.168.20.66 74
Aug 17 15:01:54 192.168.20.66 70
Aug 17 15:01:54 192.168.20.66 fa
Aug 17 15:01:54 192.168.20.66 e8
Aug 17 15:01:54 192.168.20.66 e7
Aug 17 15:01:54 192.168.20.66 ca
Aug 17 15:01:54 192.168.20.66 ed
Aug 17 15:01:54 192.168.20.66 ff
Aug 17 15:01:54 192.168.20.66 8b
Aug 17 15:01:54 192.168.20.66 1e
Aug 17 15:01:54 192.168.20.66 c7
Aug 17 15:01:54 192.168.20.66 06
Aug 17 15:01:54 192.168.20.66 00
Aug 17 15:01:54 192.168.20.66 00
Aug 17 15:01:54 192.168.20.66 00
Aug 17 15:01:54 192.168.20.66 00
Aug 17 15:01:54 192.168.20.66 e8
Aug 17 15:01:54 192.168.20.66 c1
Aug 17 15:01:54 192.168.20.66 dd
Aug 17 15:01:54 192.168.20.66 ed
Aug 17 15:01:54 192.168.20.66 ff
Aug 17 15:01:54 192.168.20.66 fb
Aug 17 15:01:54 192.168.20.66 eb
Aug 17 15:01:54 192.168.20.66 56
Aug 17 15:01:54 192.168.20.66 unparseable log message: "<8b> "
Aug 17 15:01:54 192.168.20.66 43
Aug 17 15:01:54 192.168.20.66 74
Aug 17 15:01:54 192.168.20.66 89
Aug 17 15:01:54 192.168.20.66 45
Aug 17 15:01:54 192.168.20.66 f0
Aug 17 15:01:54 192.168.20.66 8d
Aug 17 15:01:54 192.168.20.66 73
Aug 17 15:01:54 192.168.20.66 24
Aug 17 15:01:54 192.168.20.66 f0
Aug 17 15:01:54 192.168.20.66 80
Aug 17 15:01:54 192.168.20.66 63
Aug 17 15:01:54 192.168.20.66 24
Aug 17 15:01:54 192.168.20.66 fd
Aug 17 15:01:54 192.168.20.66 f6
Aug 17 15:01:54 192.168.20.66 43
Aug 17 15:01:54 192.168.20.66 08
Aug 17 15:01:54 192.168.20.66 01
Aug 17 15:01:54 192.168.20.66 8d
Aug 17 15:01:54 192.168.20.66 7b
Aug 17 15:01:54 192.168.20.66 38
Aug 17 15:01:54 192.168.20.66
Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP: [<c025fd4e>]
Aug 17 15:01:54 192.168.20.66 net_tx_action+0x78/0xd8
Aug 17 15:01:54 192.168.20.66 SS:ESP 0068:c0861fc4
Aug 17 15:01:54 192.168.20.66 [ 7222.094000] Kernel panic - not syncing: Fatal exception in interrupt
Aug 17 15:01:54 192.168.20.66 [ 7222.094626] Rebooting in 5 seconds..


On Sunday 17 August 2008, Jarek Poplawski wrote:
> On Sun, Aug 17, 2008 at 01:01:55PM +0300, Denys Fedoryshchenko wrote:
> > Ok, now things finally organized.
> > fixing patch #1 was missing.
> >
> > Rebooting system to this kernel. But strange thing, without fixing patch
> > #1 it didn't crash for whole night, and 1-3 hours now (with same options
> > as before).
> >
> > Let's test now complete set of patches. I will keep around 2 hours on
> > most loaded pppoe NAS, then distribute to 2-3 servers more if it doesn't
> > crash.
>
> BTW, after you complete this testing (no hurry) I would be glad if you
> could try one more patch which I send earlier to the list. IMHO, it's
> needed to fix some other locking problems. This patch could be applied
> and tested as an addition to all currently tested patches (but let's
> first be sure they really work).
>
> Thanks,
> Jarek P.
>
> ------------->
>
> pkt_sched: Destroy qdiscs under rtnl_lock again.
>
> We don't need to trigger __qdisc_destroy() as an RCU callback because
> the use of qdisc isn't controlled by RCU alone: after querying RCU
> with synchronize_rcu() in dev_deactivate() we additionaly wait in a
> loop checking some flags. After the loop is done there could be no
> outstanding use of the qdisc, so call_rcu() doesn't make any sense.
>
> On the other hand, current calling Qdisc's ->destroy() from a softirq
> context without locking (rtnl) can break various things like:
> qdisc_put_rtab(), tcf_destroy_chain() (e.g. u32_destroy()), and
> probably more.
>
>
> Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>
>
> ---
>
>  net/sched/sch_generic.c |    8 ++------
>  1 files changed, 2 insertions(+), 6 deletions(-)
>
> diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> index 4685746..e7379d2 100644
> --- a/net/sched/sch_generic.c
> +++ b/net/sched/sch_generic.c
> @@ -518,12 +518,8 @@ void qdisc_reset(struct Qdisc *qdisc)
>  }
>  EXPORT_SYMBOL(qdisc_reset);
>
> -/* this is the rcu callback function to clean up a qdisc when there
> - * are no further references to it */
> -
> -static void __qdisc_destroy(struct rcu_head *head)
> +static void __qdisc_destroy(struct Qdisc *qdisc)
>  {
> -	struct Qdisc *qdisc = container_of(head, struct Qdisc, q_rcu);
>  	const struct Qdisc_ops  *ops = qdisc->ops;
>
>  #ifdef CONFIG_NET_SCHED
> @@ -554,7 +550,7 @@ void qdisc_destroy(struct Qdisc *qdisc)
>  	if (qdisc->parent)
>  		list_del(&qdisc->list);
>
> -	call_rcu(&qdisc->q_rcu, __qdisc_destroy);
> +	__qdisc_destroy(qdisc);
>  }
>  EXPORT_SYMBOL(qdisc_destroy);
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 12:06                                                 ` Denys Fedoryshchenko
@ 2008-08-17 12:18                                                   ` Denys Fedoryshchenko
  2008-08-17 12:38                                                   ` Jarek Poplawski
  1 sibling, 0 replies; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17 12:18 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Patch applied. Got:

[   56.081296]
[   56.081296] =========================
[   56.081296] [ BUG: held lock freed! ]
[   56.081296] -------------------------
[   56.081296] tc/1973 is freeing memory f6ca2800-f6ca29ff, with a lock still held there!
[   56.081296]  (&list->lock#5){-...}, at: [<c026fc46>] notify_and_destroy+0x34/0x4a
[   56.081296] 2 locks held by tc/1973:
[   56.081296]  #0:  (rtnl_mutex){--..}, at: [<c026999e>] rtnl_lock+0xf/0x11
[   56.082296]  #1:  (&list->lock#5){-...}, at: [<c026fc46>] notify_and_destroy+0x34/0x4a
[   56.082296]
[   56.082296] stack backtrace:
[   56.082296] Pid: 1973, comm: tc Not tainted 2.6.27-rc3-git3-build-0031 #9
[   56.082296]  [<c02bb92b>] ? printk+0xf/0x14
[   56.082296]  [<c013dbe7>] debug_check_no_locks_freed+0xce/0xff
[   56.082296]  [<c01643da>] kfree+0x78/0xc8
[   56.082296]  [<c026e8f2>] ? qdisc_destroy+0xa7/0xab
[   56.082296]  [<c026e8f2>] qdisc_destroy+0xa7/0xab
[   56.082296]  [<c026fc4d>] notify_and_destroy+0x3b/0x4a
[   56.082296]  [<c026fd9a>] qdisc_graft+0x13e/0x1c9
[   56.082296]  [<c0270876>] tc_modify_qdisc+0x354/0x3ed
[   56.082296]  [<c013d974>] ? mark_held_locks+0x53/0x6a
[   56.082296]  [<c013db17>] ? trace_hardirqs_on+0xb/0xd
[   56.082296]  [<c013daeb>] ? trace_hardirqs_on_caller+0xe1/0x102
[   56.082296]  [<c0270522>] ? tc_modify_qdisc+0x0/0x3ed
[   56.082296]  [<c0269b5e>] rtnetlink_rcv_msg+0x198/0x1b2
[   56.082296]  [<c02699c6>] ? rtnetlink_rcv_msg+0x0/0x1b2
[   56.082296]  [<c0275b77>] netlink_rcv_skb+0x30/0x75
[   56.082296]  [<c02699be>] rtnetlink_rcv+0x1e/0x26
[   56.082296]  [<c027570e>] netlink_unicast+0x1a9/0x20a
[   56.082296]  [<c0275991>] netlink_sendmsg+0x222/0x22f
[   56.082296]  [<c0256fd1>] sock_sendmsg+0xca/0xe1
[   56.082296]  [<c0132359>] ? autoremove_wake_function+0x0/0x33
[   56.082296]  [<c014c6c7>] ? unlock_page+0x48/0x4b
[   56.082296]  [<c0157c64>] ? __do_fault+0x286/0x2be
[   56.082296]  [<c025d641>] ? verify_iovec+0x40/0x6f
[   56.082296]  [<c0257127>] sys_sendmsg+0x13f/0x192
[   56.082296]  [<c0135539>] ? up_read+0x16/0x29
[   56.082296]  [<c01142c5>] ? do_page_fault+0x2e5/0x624
[   56.082296]  [<c016fb47>] ? user_path_at+0x4a/0x67
[   56.082296]  [<c016fb47>] ? user_path_at+0x4a/0x67
[   56.082296]  [<c0257f57>] sys_socketcall+0x14b/0x180
[   56.082296]  [<c01dd5e0>] ? trace_hardirqs_on_thunk+0xc/0x10
[   56.082296]  [<c0103861>] sysenter_do_call+0x12/0x35
[   56.082296]  =======================

On Sunday 17 August 2008, Denys Fedoryshchenko wrote:
> Seems stable (without this patch yet) on normal shaper, and
> crashing on new version of shaper (it is doing very intensive
> classes creation/destroy).
>
> I'm not sure if it is same bug, i will try to test on regular shaper
> more things.
>
> I will try to apply last patch also and test on another pppoe
> server.
>
>
> Here is panic i got when i try to run alternative shaper.
>
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] BUG: unable to handle kernel
> Aug 17 15:01:54 192.168.20.66 paging request
> Aug 17 15:01:54 192.168.20.66 at 50639a62
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] IP:
> Aug 17 15:01:54 192.168.20.66 [<c025fd4e>] net_tx_action+0x78/0xd8
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Oops: 0000 [#1]
> Aug 17 15:01:54 192.168.20.66 SMP
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Modules linked in:
> Aug 17 15:01:54 192.168.20.66 cls_flow
> Aug 17 15:01:54 192.168.20.66 act_police
> Aug 17 15:01:54 192.168.20.66 sch_ingress
> Aug 17 15:01:54 192.168.20.66 sch_prio
> Aug 17 15:01:54 192.168.20.66 netconsole
> Aug 17 15:01:54 192.168.20.66 configfs
> Aug 17 15:01:54 192.168.20.66 nf_nat_pptp
> Aug 17 15:01:54 192.168.20.66 nf_nat_proto_gre
> Aug 17 15:01:54 192.168.20.66 nf_conntrack_pptp
> Aug 17 15:01:54 192.168.20.66 nf_conntrack_proto_gre
> Aug 17 15:01:54 192.168.20.66 iTCO_wdt
> Aug 17 15:01:54 192.168.20.66 e752x_edac
> Aug 17 15:01:54 192.168.20.66 edac_core
> Aug 17 15:01:54 192.168.20.66 cls_u32
> Aug 17 15:01:54 192.168.20.66 em_meta
> Aug 17 15:01:54 192.168.20.66 cls_basic
> Aug 17 15:01:54 192.168.20.66 xt_dscp
> Aug 17 15:01:54 192.168.20.66 xt_DSCP
> Aug 17 15:01:54 192.168.20.66 ipt_REJECT
> Aug 17 15:01:54 192.168.20.66 ts_bm
> Aug 17 15:01:54 192.168.20.66 xt_string
> Aug 17 15:01:54 192.168.20.66 ipt_ttl
> Aug 17 15:01:54 192.168.20.66 ifb
> Aug 17 15:01:54 192.168.20.66 cls_fw
> Aug 17 15:01:54 192.168.20.66 sch_tbf
> Aug 17 15:01:54 192.168.20.66 sch_htb
> Aug 17 15:01:54 192.168.20.66 act_ipt
> Aug 17 15:01:54 192.168.20.66 act_mirred
> Aug 17 15:01:54 192.168.20.66 xt_MARK
> Aug 17 15:01:54 192.168.20.66 pppoe
> Aug 17 15:01:54 192.168.20.66 pppox
> Aug 17 15:01:54 192.168.20.66 ppp_generic
> Aug 17 15:01:54 192.168.20.66 slhc
> Aug 17 15:01:54 192.168.20.66 xt_TCPMSS
> Aug 17 15:01:54 192.168.20.66 xt_mark
> Aug 17 15:01:54 192.168.20.66 xt_tcpudp
> Aug 17 15:01:54 192.168.20.66 iptable_mangle
> Aug 17 15:01:54 192.168.20.66 iptable_nat
> Aug 17 15:01:54 192.168.20.66 nf_nat
> Aug 17 15:01:54 192.168.20.66 nf_conntrack_ipv4
> Aug 17 15:01:54 192.168.20.66 nf_conntrack
> Aug 17 15:01:54 192.168.20.66 rtc_cmos
> Aug 17 15:01:54 192.168.20.66 rtc_core
> Aug 17 15:01:54 192.168.20.66 rtc_lib
> Aug 17 15:01:54 192.168.20.66 iptable_filter
> Aug 17 15:01:54 192.168.20.66 ip_tables
> Aug 17 15:01:54 192.168.20.66 x_tables
> Aug 17 15:01:54 192.168.20.66 8021q
> Aug 17 15:01:54 192.168.20.66 garp
> Aug 17 15:01:54 192.168.20.66 stp
> Aug 17 15:01:54 192.168.20.66 llc
> Aug 17 15:01:54 192.168.20.66 tun
> Aug 17 15:01:54 192.168.20.66 loop
> Aug 17 15:01:54 192.168.20.66 tulip
> Aug 17 15:01:54 192.168.20.66 r8169
> Aug 17 15:01:54 192.168.20.66 sky2
> Aug 17 15:01:54 192.168.20.66 via_velocity
> Aug 17 15:01:54 192.168.20.66 via_rhine
> Aug 17 15:01:54 192.168.20.66 sis900
> Aug 17 15:01:54 192.168.20.66 ne2k_pci
> Aug 17 15:01:54 192.168.20.66 8390
> Aug 17 15:01:54 192.168.20.66 skge
> Aug 17 15:01:54 192.168.20.66 tg3
> Aug 17 15:01:54 192.168.20.66 libphy
> Aug 17 15:01:54 192.168.20.66 8139too
> Aug 17 15:01:54 192.168.20.66 e1000
> Aug 17 15:01:54 192.168.20.66 e100
> Aug 17 15:01:54 192.168.20.66 usb_storage
> Aug 17 15:01:54 192.168.20.66 mtdblock
> Aug 17 15:01:54 192.168.20.66 mtd_blkdevs
> Aug 17 15:01:54 192.168.20.66 usbhid
> Aug 17 15:01:54 192.168.20.66 uhci_hcd
> Aug 17 15:01:54 192.168.20.66 ehci_hcd
> Aug 17 15:01:54 192.168.20.66 ohci_hcd
> Aug 17 15:01:54 192.168.20.66 usbcore
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Pid: 4, comm: ksoftirqd/0 Not
> tainted (2.6.27-rc3-git3-build-0031 #7) Aug 17 15:01:54 192.168.20.66 [
> 7222.089634] EIP: 0060:[<c025fd4e>] EFLAGS: 00010206 CPU: 0 Aug 17 15:01:54
> 192.168.20.66 [ 7222.089634] EIP is at net_tx_action+0x78/0xd8 Aug 17
> 15:01:54 192.168.20.66 [ 7222.089634] EAX: 01deb6a8 EBX: 506399ee ECX:
> 00000002 EDX: 00000001 Aug 17 15:01:54 192.168.20.66 [ 7222.089634] ESI:
> f6429224 EDI: f6429238 EBP: c0861fd4 ESP: c0861fc4 Aug 17 15:01:54
> 192.168.20.66 [ 7222.089634]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Process ksoftirqd/0 (pid: 4,
> ti=c0861000 task=f78399b0 task.ti=f7841000) Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Stack:
> Aug 17 15:01:54 192.168.20.66 506399ee
> Aug 17 15:01:54 192.168.20.66 c039ea88
> Aug 17 15:01:54 192.168.20.66 c085a554
> Aug 17 15:01:54 192.168.20.66 00000003
> Aug 17 15:01:54 192.168.20.66 c0861ff8
> Aug 17 15:01:54 192.168.20.66 c012626b
> Aug 17 15:01:54 192.168.20.66 c085d700
> Aug 17 15:01:54 192.168.20.66 c085d700
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> Aug 17 15:01:54 192.168.20.66 00000006
> Aug 17 15:01:54 192.168.20.66 00000000
> Aug 17 15:01:54 192.168.20.66 f7841fac
> Aug 17 15:01:54 192.168.20.66 f7841000
> Aug 17 15:01:54 192.168.20.66 c01261f0
> Aug 17 15:01:54 192.168.20.66 f7841fbc
> Aug 17 15:01:54 192.168.20.66 c01060b3
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Call Trace:
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c012626b>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 __do_softirq+0x7b/0xf4
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01261f0>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 __do_softirq+0x0/0xf4
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01060b3>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 do_softirq+0x65/0xb6
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125cfc>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x59/0xc8
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125ca3>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x0/0xc8
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320e5>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 k
> Aug 17 15:01:54 192.168.20.66 thread+0x3b/0x61
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320aa>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 kthread+0x0/0x61
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c010451b>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 kernel_thread_helper+0x7/0x10
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  =======================
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Code:
> Aug 17 15:01:54 192.168.20.66 1e
> Aug 17 15:01:54 192.168.20.66 ec
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 89
> Aug 17 15:01:54 192.168.20.66 d8
> Aug 17 15:01:54 192.168.20.66 89
> Aug 17 15:01:54 192.168.20.66 fb
> Aug 17 15:01:54 192.168.20.66 e8
> Aug 17 15:01:54 192.168.20.66 6e
> Aug 17 15:01:54 192.168.20.66 c2
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 85
> Aug 17 15:01:54 192.168.20.66 db
> Aug 17 15:01:54 192.168.20.66 75
> Aug 17 15:01:54 192.168.20.66 d9
> Aug 17 15:01:54 192.168.20.66 83
> Aug 17 15:01:54 192.168.20.66 3e
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 74
> Aug 17 15:01:54 192.168.20.66 70
> Aug 17 15:01:54 192.168.20.66 fa
> Aug 17 15:01:54 192.168.20.66 e8
> Aug 17 15:01:54 192.168.20.66 e7
> Aug 17 15:01:54 192.168.20.66 ca
> Aug 17 15:01:54 192.168.20.66 ed
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 8b
> Aug 17 15:01:54 192.168.20.66 1e
> Aug 17 15:01:54 192.168.20.66 c7
> Aug 17 15:01:54 192.168.20.66 06
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 e8
> Aug 17 15:01:54 192.168.20.66 c1
> Aug 17 15:01:54 192.168.20.66 dd
> Aug 17 15:01:54 192.168.20.66 ed
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 fb
> Aug 17 15:01:54 192.168.20.66 eb
> Aug 17 15:01:54 192.168.20.66 56
> Aug 17 15:01:54 192.168.20.66 unparseable log message: "<8b> "
> Aug 17 15:01:54 192.168.20.66 43
> Aug 17 15:01:54 192.168.20.66 74
> Aug 17 15:01:54 192.168.20.66 89
> Aug 17 15:01:54 192.168.20.66 45
> Aug 17 15:01:54 192.168.20.66 f0
> Aug 17 15:01:54 192.168.20.66 8d
> Aug 17 15:01:54 192.168.20.66 73
> Aug 17 15:01:54 192.168.20.66 24
> Aug 17 15:01:54 192.168.20.66 f0
> Aug 17 15:01:54 192.168.20.66 80
> Aug 17 15:01:54 192.168.20.66 63
> Aug 17 15:01:54 192.168.20.66 24
> Aug 17 15:01:54 192.168.20.66 fd
> Aug 17 15:01:54 192.168.20.66 f6
> Aug 17 15:01:54 192.168.20.66 43
> Aug 17 15:01:54 192.168.20.66 08
> Aug 17 15:01:54 192.168.20.66 01
> Aug 17 15:01:54 192.168.20.66 8d
> Aug 17 15:01:54 192.168.20.66 7b
> Aug 17 15:01:54 192.168.20.66 38
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP: [<c025fd4e>]
> Aug 17 15:01:54 192.168.20.66 net_tx_action+0x78/0xd8
> Aug 17 15:01:54 192.168.20.66 SS:ESP 0068:c0861fc4
> Aug 17 15:01:54 192.168.20.66 [ 7222.094000] Kernel panic - not syncing:
> Fatal exception in interrupt Aug 17 15:01:54 192.168.20.66 [ 7222.094626]
> Rebooting in 5 seconds..
>
> On Sunday 17 August 2008, Jarek Poplawski wrote:
> > On Sun, Aug 17, 2008 at 01:01:55PM +0300, Denys Fedoryshchenko wrote:
> > > Ok, now things finally organized.
> > > fixing patch #1 was missing.
> > >
> > > Rebooting system to this kernel. But strange thing, without fixing
> > > patch #1 it didn't crash for whole night, and 1-3 hours now (with same
> > > options as before).
> > >
> > > Let's test now complete set of patches. I will keep around 2 hours on
> > > most loaded pppoe NAS, then distribute to 2-3 servers more if it
> > > doesn't crash.
> >
> > BTW, after you complete this testing (no hurry) I would be glad if you
> > could try one more patch which I send earlier to the list. IMHO, it's
> > needed to fix some other locking problems. This patch could be applied
> > and tested as an addition to all currently tested patches (but let's
> > first be sure they really work).
> >
> > Thanks,
> > Jarek P.
> >
> > ------------->
> >
> > pkt_sched: Destroy qdiscs under rtnl_lock again.
> >
> > We don't need to trigger __qdisc_destroy() as an RCU callback because
> > the use of qdisc isn't controlled by RCU alone: after querying RCU
> > with synchronize_rcu() in dev_deactivate() we additionaly wait in a
> > loop checking some flags. After the loop is done there could be no
> > outstanding use of the qdisc, so call_rcu() doesn't make any sense.
> >
> > On the other hand, current calling Qdisc's ->destroy() from a softirq
> > context without locking (rtnl) can break various things like:
> > qdisc_put_rtab(), tcf_destroy_chain() (e.g. u32_destroy()), and
> > probably more.
> >
> >
> > Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>
> >
> > ---
> >
> >  net/sched/sch_generic.c |    8 ++------
> >  1 files changed, 2 insertions(+), 6 deletions(-)
> >
> > diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> > index 4685746..e7379d2 100644
> > --- a/net/sched/sch_generic.c
> > +++ b/net/sched/sch_generic.c
> > @@ -518,12 +518,8 @@ void qdisc_reset(struct Qdisc *qdisc)
> >  }
> >  EXPORT_SYMBOL(qdisc_reset);
> >
> > -/* this is the rcu callback function to clean up a qdisc when there
> > - * are no further references to it */
> > -
> > -static void __qdisc_destroy(struct rcu_head *head)
> > +static void __qdisc_destroy(struct Qdisc *qdisc)
> >  {
> > -	struct Qdisc *qdisc = container_of(head, struct Qdisc, q_rcu);
> >  	const struct Qdisc_ops  *ops = qdisc->ops;
> >
> >  #ifdef CONFIG_NET_SCHED
> > @@ -554,7 +550,7 @@ void qdisc_destroy(struct Qdisc *qdisc)
> >  	if (qdisc->parent)
> >  		list_del(&qdisc->list);
> >
> > -	call_rcu(&qdisc->q_rcu, __qdisc_destroy);
> > +	__qdisc_destroy(qdisc);
> >  }
> >  EXPORT_SYMBOL(qdisc_destroy);
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 12:06                                                 ` Denys Fedoryshchenko
  2008-08-17 12:18                                                   ` Denys Fedoryshchenko
@ 2008-08-17 12:38                                                   ` Jarek Poplawski
  2008-08-17 12:50                                                     ` Denys Fedoryshchenko
  1 sibling, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17 12:38 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sun, Aug 17, 2008 at 03:06:06PM +0300, Denys Fedoryshchenko wrote:
> Seems stable (without this patch yet) on normal shaper, and 
> crashing on new version of shaper (it is doing very intensive 
> classes creation/destroy).
> 
> I'm not sure if it is same bug, i will try to test on regular shaper
> more things.
> 
> I will try to apply last patch also and test on another pppoe 
> server.

Denys, I would like to be sure: do you mean here the last patch
attached below? So, the alternative shaper works with previous
patches, but not this one? If so, than no need to test this patch
anymore, only confirm this, please.

Jarek P.

> 
> 
> Here is panic i got when i try to run alternative shaper.
> 
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] BUG: unable to handle kernel
> Aug 17 15:01:54 192.168.20.66 paging request
> Aug 17 15:01:54 192.168.20.66 at 50639a62
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] IP:
> Aug 17 15:01:54 192.168.20.66 [<c025fd4e>] net_tx_action+0x78/0xd8
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Oops: 0000 [#1]
> Aug 17 15:01:54 192.168.20.66 SMP
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Modules linked in:
> Aug 17 15:01:54 192.168.20.66 cls_flow
> Aug 17 15:01:54 192.168.20.66 act_police
> Aug 17 15:01:54 192.168.20.66 sch_ingress
> Aug 17 15:01:54 192.168.20.66 sch_prio
> Aug 17 15:01:54 192.168.20.66 netconsole
> Aug 17 15:01:54 192.168.20.66 configfs
> Aug 17 15:01:54 192.168.20.66 nf_nat_pptp
> Aug 17 15:01:54 192.168.20.66 nf_nat_proto_gre
> Aug 17 15:01:54 192.168.20.66 nf_conntrack_pptp
> Aug 17 15:01:54 192.168.20.66 nf_conntrack_proto_gre
> Aug 17 15:01:54 192.168.20.66 iTCO_wdt
> Aug 17 15:01:54 192.168.20.66 e752x_edac
> Aug 17 15:01:54 192.168.20.66 edac_core
> Aug 17 15:01:54 192.168.20.66 cls_u32
> Aug 17 15:01:54 192.168.20.66 em_meta
> Aug 17 15:01:54 192.168.20.66 cls_basic
> Aug 17 15:01:54 192.168.20.66 xt_dscp
> Aug 17 15:01:54 192.168.20.66 xt_DSCP
> Aug 17 15:01:54 192.168.20.66 ipt_REJECT
> Aug 17 15:01:54 192.168.20.66 ts_bm
> Aug 17 15:01:54 192.168.20.66 xt_string
> Aug 17 15:01:54 192.168.20.66 ipt_ttl
> Aug 17 15:01:54 192.168.20.66 ifb
> Aug 17 15:01:54 192.168.20.66 cls_fw
> Aug 17 15:01:54 192.168.20.66 sch_tbf
> Aug 17 15:01:54 192.168.20.66 sch_htb
> Aug 17 15:01:54 192.168.20.66 act_ipt
> Aug 17 15:01:54 192.168.20.66 act_mirred
> Aug 17 15:01:54 192.168.20.66 xt_MARK
> Aug 17 15:01:54 192.168.20.66 pppoe
> Aug 17 15:01:54 192.168.20.66 pppox
> Aug 17 15:01:54 192.168.20.66 ppp_generic
> Aug 17 15:01:54 192.168.20.66 slhc
> Aug 17 15:01:54 192.168.20.66 xt_TCPMSS
> Aug 17 15:01:54 192.168.20.66 xt_mark
> Aug 17 15:01:54 192.168.20.66 xt_tcpudp
> Aug 17 15:01:54 192.168.20.66 iptable_mangle
> Aug 17 15:01:54 192.168.20.66 iptable_nat
> Aug 17 15:01:54 192.168.20.66 nf_nat
> Aug 17 15:01:54 192.168.20.66 nf_conntrack_ipv4
> Aug 17 15:01:54 192.168.20.66 nf_conntrack
> Aug 17 15:01:54 192.168.20.66 rtc_cmos
> Aug 17 15:01:54 192.168.20.66 rtc_core
> Aug 17 15:01:54 192.168.20.66 rtc_lib
> Aug 17 15:01:54 192.168.20.66 iptable_filter
> Aug 17 15:01:54 192.168.20.66 ip_tables
> Aug 17 15:01:54 192.168.20.66 x_tables
> Aug 17 15:01:54 192.168.20.66 8021q
> Aug 17 15:01:54 192.168.20.66 garp
> Aug 17 15:01:54 192.168.20.66 stp
> Aug 17 15:01:54 192.168.20.66 llc
> Aug 17 15:01:54 192.168.20.66 tun
> Aug 17 15:01:54 192.168.20.66 loop
> Aug 17 15:01:54 192.168.20.66 tulip
> Aug 17 15:01:54 192.168.20.66 r8169
> Aug 17 15:01:54 192.168.20.66 sky2
> Aug 17 15:01:54 192.168.20.66 via_velocity
> Aug 17 15:01:54 192.168.20.66 via_rhine
> Aug 17 15:01:54 192.168.20.66 sis900
> Aug 17 15:01:54 192.168.20.66 ne2k_pci
> Aug 17 15:01:54 192.168.20.66 8390
> Aug 17 15:01:54 192.168.20.66 skge
> Aug 17 15:01:54 192.168.20.66 tg3
> Aug 17 15:01:54 192.168.20.66 libphy
> Aug 17 15:01:54 192.168.20.66 8139too
> Aug 17 15:01:54 192.168.20.66 e1000
> Aug 17 15:01:54 192.168.20.66 e100
> Aug 17 15:01:54 192.168.20.66 usb_storage
> Aug 17 15:01:54 192.168.20.66 mtdblock
> Aug 17 15:01:54 192.168.20.66 mtd_blkdevs
> Aug 17 15:01:54 192.168.20.66 usbhid
> Aug 17 15:01:54 192.168.20.66 uhci_hcd
> Aug 17 15:01:54 192.168.20.66 ehci_hcd
> Aug 17 15:01:54 192.168.20.66 ohci_hcd
> Aug 17 15:01:54 192.168.20.66 usbcore
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Pid: 4, comm: ksoftirqd/0 Not tainted (2.6.27-rc3-git3-build-0031 #7)
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP: 0060:[<c025fd4e>] EFLAGS: 00010206 CPU: 0
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP is at net_tx_action+0x78/0xd8
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EAX: 01deb6a8 EBX: 506399ee ECX: 00000002 EDX: 00000001
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] ESI: f6429224 EDI: f6429238 EBP: c0861fd4 ESP: c0861fc4
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Process ksoftirqd/0 (pid: 4, ti=c0861000 task=f78399b0 task.ti=f7841000)
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Stack:
> Aug 17 15:01:54 192.168.20.66 506399ee
> Aug 17 15:01:54 192.168.20.66 c039ea88
> Aug 17 15:01:54 192.168.20.66 c085a554
> Aug 17 15:01:54 192.168.20.66 00000003
> Aug 17 15:01:54 192.168.20.66 c0861ff8
> Aug 17 15:01:54 192.168.20.66 c012626b
> Aug 17 15:01:54 192.168.20.66 c085d700
> Aug 17 15:01:54 192.168.20.66 c085d700
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> Aug 17 15:01:54 192.168.20.66 00000006
> Aug 17 15:01:54 192.168.20.66 00000000
> Aug 17 15:01:54 192.168.20.66 f7841fac
> Aug 17 15:01:54 192.168.20.66 f7841000
> Aug 17 15:01:54 192.168.20.66 c01261f0
> Aug 17 15:01:54 192.168.20.66 f7841fbc
> Aug 17 15:01:54 192.168.20.66 c01060b3
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Call Trace:
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c012626b>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 __do_softirq+0x7b/0xf4
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01261f0>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 __do_softirq+0x0/0xf4
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01060b3>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 do_softirq+0x65/0xb6
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125cfc>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x59/0xc8
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125ca3>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x0/0xc8
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320e5>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 k
> Aug 17 15:01:54 192.168.20.66 thread+0x3b/0x61
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320aa>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 kthread+0x0/0x61
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c010451b>]
> Aug 17 15:01:54 192.168.20.66 ?
> Aug 17 15:01:54 192.168.20.66 kernel_thread_helper+0x7/0x10
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  =======================
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Code:
> Aug 17 15:01:54 192.168.20.66 1e
> Aug 17 15:01:54 192.168.20.66 ec
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 89
> Aug 17 15:01:54 192.168.20.66 d8
> Aug 17 15:01:54 192.168.20.66 89
> Aug 17 15:01:54 192.168.20.66 fb
> Aug 17 15:01:54 192.168.20.66 e8
> Aug 17 15:01:54 192.168.20.66 6e
> Aug 17 15:01:54 192.168.20.66 c2
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 85
> Aug 17 15:01:54 192.168.20.66 db
> Aug 17 15:01:54 192.168.20.66 75
> Aug 17 15:01:54 192.168.20.66 d9
> Aug 17 15:01:54 192.168.20.66 83
> Aug 17 15:01:54 192.168.20.66 3e
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 74
> Aug 17 15:01:54 192.168.20.66 70
> Aug 17 15:01:54 192.168.20.66 fa
> Aug 17 15:01:54 192.168.20.66 e8
> Aug 17 15:01:54 192.168.20.66 e7
> Aug 17 15:01:54 192.168.20.66 ca
> Aug 17 15:01:54 192.168.20.66 ed
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 8b
> Aug 17 15:01:54 192.168.20.66 1e
> Aug 17 15:01:54 192.168.20.66 c7
> Aug 17 15:01:54 192.168.20.66 06
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 00
> Aug 17 15:01:54 192.168.20.66 e8
> Aug 17 15:01:54 192.168.20.66 c1
> Aug 17 15:01:54 192.168.20.66 dd
> Aug 17 15:01:54 192.168.20.66 ed
> Aug 17 15:01:54 192.168.20.66 ff
> Aug 17 15:01:54 192.168.20.66 fb
> Aug 17 15:01:54 192.168.20.66 eb
> Aug 17 15:01:54 192.168.20.66 56
> Aug 17 15:01:54 192.168.20.66 unparseable log message: "<8b> "
> Aug 17 15:01:54 192.168.20.66 43
> Aug 17 15:01:54 192.168.20.66 74
> Aug 17 15:01:54 192.168.20.66 89
> Aug 17 15:01:54 192.168.20.66 45
> Aug 17 15:01:54 192.168.20.66 f0
> Aug 17 15:01:54 192.168.20.66 8d
> Aug 17 15:01:54 192.168.20.66 73
> Aug 17 15:01:54 192.168.20.66 24
> Aug 17 15:01:54 192.168.20.66 f0
> Aug 17 15:01:54 192.168.20.66 80
> Aug 17 15:01:54 192.168.20.66 63
> Aug 17 15:01:54 192.168.20.66 24
> Aug 17 15:01:54 192.168.20.66 fd
> Aug 17 15:01:54 192.168.20.66 f6
> Aug 17 15:01:54 192.168.20.66 43
> Aug 17 15:01:54 192.168.20.66 08
> Aug 17 15:01:54 192.168.20.66 01
> Aug 17 15:01:54 192.168.20.66 8d
> Aug 17 15:01:54 192.168.20.66 7b
> Aug 17 15:01:54 192.168.20.66 38
> Aug 17 15:01:54 192.168.20.66
> Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP: [<c025fd4e>]
> Aug 17 15:01:54 192.168.20.66 net_tx_action+0x78/0xd8
> Aug 17 15:01:54 192.168.20.66 SS:ESP 0068:c0861fc4
> Aug 17 15:01:54 192.168.20.66 [ 7222.094000] Kernel panic - not syncing: Fatal exception in interrupt
> Aug 17 15:01:54 192.168.20.66 [ 7222.094626] Rebooting in 5 seconds..
> 
> 
> On Sunday 17 August 2008, Jarek Poplawski wrote:
> > On Sun, Aug 17, 2008 at 01:01:55PM +0300, Denys Fedoryshchenko wrote:
> > > Ok, now things finally organized.
> > > fixing patch #1 was missing.
> > >
> > > Rebooting system to this kernel. But strange thing, without fixing patch
> > > #1 it didn't crash for whole night, and 1-3 hours now (with same options
> > > as before).
> > >
> > > Let's test now complete set of patches. I will keep around 2 hours on
> > > most loaded pppoe NAS, then distribute to 2-3 servers more if it doesn't
> > > crash.
> >
> > BTW, after you complete this testing (no hurry) I would be glad if you
> > could try one more patch which I send earlier to the list. IMHO, it's
> > needed to fix some other locking problems. This patch could be applied
> > and tested as an addition to all currently tested patches (but let's
> > first be sure they really work).
> >
> > Thanks,
> > Jarek P.
> >
> > ------------->
> >
> > pkt_sched: Destroy qdiscs under rtnl_lock again.
> >
> > We don't need to trigger __qdisc_destroy() as an RCU callback because
> > the use of qdisc isn't controlled by RCU alone: after querying RCU
> > with synchronize_rcu() in dev_deactivate() we additionaly wait in a
> > loop checking some flags. After the loop is done there could be no
> > outstanding use of the qdisc, so call_rcu() doesn't make any sense.
> >
> > On the other hand, current calling Qdisc's ->destroy() from a softirq
> > context without locking (rtnl) can break various things like:
> > qdisc_put_rtab(), tcf_destroy_chain() (e.g. u32_destroy()), and
> > probably more.
> >
> >
> > Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>
> >
> > ---
> >
> >  net/sched/sch_generic.c |    8 ++------
> >  1 files changed, 2 insertions(+), 6 deletions(-)
> >
> > diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> > index 4685746..e7379d2 100644
> > --- a/net/sched/sch_generic.c
> > +++ b/net/sched/sch_generic.c
> > @@ -518,12 +518,8 @@ void qdisc_reset(struct Qdisc *qdisc)
> >  }
> >  EXPORT_SYMBOL(qdisc_reset);
> >
> > -/* this is the rcu callback function to clean up a qdisc when there
> > - * are no further references to it */
> > -
> > -static void __qdisc_destroy(struct rcu_head *head)
> > +static void __qdisc_destroy(struct Qdisc *qdisc)
> >  {
> > -	struct Qdisc *qdisc = container_of(head, struct Qdisc, q_rcu);
> >  	const struct Qdisc_ops  *ops = qdisc->ops;
> >
> >  #ifdef CONFIG_NET_SCHED
> > @@ -554,7 +550,7 @@ void qdisc_destroy(struct Qdisc *qdisc)
> >  	if (qdisc->parent)
> >  		list_del(&qdisc->list);
> >
> > -	call_rcu(&qdisc->q_rcu, __qdisc_destroy);
> > +	__qdisc_destroy(qdisc);
> >  }
> >  EXPORT_SYMBOL(qdisc_destroy);
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 12:38                                                   ` Jarek Poplawski
@ 2008-08-17 12:50                                                     ` Denys Fedoryshchenko
  2008-08-17 13:03                                                       ` Denys Fedoryshchenko
  2008-08-17 13:16                                                       ` Jarek Poplawski
  0 siblings, 2 replies; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17 12:50 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

On Sunday 17 August 2008, Jarek Poplawski wrote:

> Denys, I would like to be sure: do you mean here the last patch
> attached below? So, the alternative shaper works with previous
> patches, but not this one? If so, than no need to test this patch
> anymore, only confirm this, please.
No.
Alternative shaper didn't work without applying latest patch (just i had old 
kernel, and i decide just to try alternative shaper).

Latest warning with all patches applied. I will test alternative shaper with 
all patches applied now.



>
> Jarek P.
>
> > Here is panic i got when i try to run alternative shaper.
> >
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] BUG: unable to handle kernel
> > Aug 17 15:01:54 192.168.20.66 paging request
> > Aug 17 15:01:54 192.168.20.66 at 50639a62
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] IP:
> > Aug 17 15:01:54 192.168.20.66 [<c025fd4e>] net_tx_action+0x78/0xd8
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Oops: 0000 [#1]
> > Aug 17 15:01:54 192.168.20.66 SMP
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Modules linked in:
> > Aug 17 15:01:54 192.168.20.66 cls_flow
> > Aug 17 15:01:54 192.168.20.66 act_police
> > Aug 17 15:01:54 192.168.20.66 sch_ingress
> > Aug 17 15:01:54 192.168.20.66 sch_prio
> > Aug 17 15:01:54 192.168.20.66 netconsole
> > Aug 17 15:01:54 192.168.20.66 configfs
> > Aug 17 15:01:54 192.168.20.66 nf_nat_pptp
> > Aug 17 15:01:54 192.168.20.66 nf_nat_proto_gre
> > Aug 17 15:01:54 192.168.20.66 nf_conntrack_pptp
> > Aug 17 15:01:54 192.168.20.66 nf_conntrack_proto_gre
> > Aug 17 15:01:54 192.168.20.66 iTCO_wdt
> > Aug 17 15:01:54 192.168.20.66 e752x_edac
> > Aug 17 15:01:54 192.168.20.66 edac_core
> > Aug 17 15:01:54 192.168.20.66 cls_u32
> > Aug 17 15:01:54 192.168.20.66 em_meta
> > Aug 17 15:01:54 192.168.20.66 cls_basic
> > Aug 17 15:01:54 192.168.20.66 xt_dscp
> > Aug 17 15:01:54 192.168.20.66 xt_DSCP
> > Aug 17 15:01:54 192.168.20.66 ipt_REJECT
> > Aug 17 15:01:54 192.168.20.66 ts_bm
> > Aug 17 15:01:54 192.168.20.66 xt_string
> > Aug 17 15:01:54 192.168.20.66 ipt_ttl
> > Aug 17 15:01:54 192.168.20.66 ifb
> > Aug 17 15:01:54 192.168.20.66 cls_fw
> > Aug 17 15:01:54 192.168.20.66 sch_tbf
> > Aug 17 15:01:54 192.168.20.66 sch_htb
> > Aug 17 15:01:54 192.168.20.66 act_ipt
> > Aug 17 15:01:54 192.168.20.66 act_mirred
> > Aug 17 15:01:54 192.168.20.66 xt_MARK
> > Aug 17 15:01:54 192.168.20.66 pppoe
> > Aug 17 15:01:54 192.168.20.66 pppox
> > Aug 17 15:01:54 192.168.20.66 ppp_generic
> > Aug 17 15:01:54 192.168.20.66 slhc
> > Aug 17 15:01:54 192.168.20.66 xt_TCPMSS
> > Aug 17 15:01:54 192.168.20.66 xt_mark
> > Aug 17 15:01:54 192.168.20.66 xt_tcpudp
> > Aug 17 15:01:54 192.168.20.66 iptable_mangle
> > Aug 17 15:01:54 192.168.20.66 iptable_nat
> > Aug 17 15:01:54 192.168.20.66 nf_nat
> > Aug 17 15:01:54 192.168.20.66 nf_conntrack_ipv4
> > Aug 17 15:01:54 192.168.20.66 nf_conntrack
> > Aug 17 15:01:54 192.168.20.66 rtc_cmos
> > Aug 17 15:01:54 192.168.20.66 rtc_core
> > Aug 17 15:01:54 192.168.20.66 rtc_lib
> > Aug 17 15:01:54 192.168.20.66 iptable_filter
> > Aug 17 15:01:54 192.168.20.66 ip_tables
> > Aug 17 15:01:54 192.168.20.66 x_tables
> > Aug 17 15:01:54 192.168.20.66 8021q
> > Aug 17 15:01:54 192.168.20.66 garp
> > Aug 17 15:01:54 192.168.20.66 stp
> > Aug 17 15:01:54 192.168.20.66 llc
> > Aug 17 15:01:54 192.168.20.66 tun
> > Aug 17 15:01:54 192.168.20.66 loop
> > Aug 17 15:01:54 192.168.20.66 tulip
> > Aug 17 15:01:54 192.168.20.66 r8169
> > Aug 17 15:01:54 192.168.20.66 sky2
> > Aug 17 15:01:54 192.168.20.66 via_velocity
> > Aug 17 15:01:54 192.168.20.66 via_rhine
> > Aug 17 15:01:54 192.168.20.66 sis900
> > Aug 17 15:01:54 192.168.20.66 ne2k_pci
> > Aug 17 15:01:54 192.168.20.66 8390
> > Aug 17 15:01:54 192.168.20.66 skge
> > Aug 17 15:01:54 192.168.20.66 tg3
> > Aug 17 15:01:54 192.168.20.66 libphy
> > Aug 17 15:01:54 192.168.20.66 8139too
> > Aug 17 15:01:54 192.168.20.66 e1000
> > Aug 17 15:01:54 192.168.20.66 e100
> > Aug 17 15:01:54 192.168.20.66 usb_storage
> > Aug 17 15:01:54 192.168.20.66 mtdblock
> > Aug 17 15:01:54 192.168.20.66 mtd_blkdevs
> > Aug 17 15:01:54 192.168.20.66 usbhid
> > Aug 17 15:01:54 192.168.20.66 uhci_hcd
> > Aug 17 15:01:54 192.168.20.66 ehci_hcd
> > Aug 17 15:01:54 192.168.20.66 ohci_hcd
> > Aug 17 15:01:54 192.168.20.66 usbcore
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Pid: 4, comm: ksoftirqd/0
> > Not tainted (2.6.27-rc3-git3-build-0031 #7) Aug 17 15:01:54 192.168.20.66
> > [ 7222.089634] EIP: 0060:[<c025fd4e>] EFLAGS: 00010206 CPU: 0 Aug 17
> > 15:01:54 192.168.20.66 [ 7222.089634] EIP is at net_tx_action+0x78/0xd8
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EAX: 01deb6a8 EBX: 506399ee
> > ECX: 00000002 EDX: 00000001 Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> > ESI: f6429224 EDI: f6429238 EBP: c0861fd4 ESP: c0861fc4 Aug 17 15:01:54
> > 192.168.20.66 [ 7222.089634]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS:
> > 0068 Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Process ksoftirqd/0
> > (pid: 4, ti=c0861000 task=f78399b0 task.ti=f7841000) Aug 17 15:01:54
> > 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Stack:
> > Aug 17 15:01:54 192.168.20.66 506399ee
> > Aug 17 15:01:54 192.168.20.66 c039ea88
> > Aug 17 15:01:54 192.168.20.66 c085a554
> > Aug 17 15:01:54 192.168.20.66 00000003
> > Aug 17 15:01:54 192.168.20.66 c0861ff8
> > Aug 17 15:01:54 192.168.20.66 c012626b
> > Aug 17 15:01:54 192.168.20.66 c085d700
> > Aug 17 15:01:54 192.168.20.66 c085d700
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> > Aug 17 15:01:54 192.168.20.66 00000006
> > Aug 17 15:01:54 192.168.20.66 00000000
> > Aug 17 15:01:54 192.168.20.66 f7841fac
> > Aug 17 15:01:54 192.168.20.66 f7841000
> > Aug 17 15:01:54 192.168.20.66 c01261f0
> > Aug 17 15:01:54 192.168.20.66 f7841fbc
> > Aug 17 15:01:54 192.168.20.66 c01060b3
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Call Trace:
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c012626b>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 __do_softirq+0x7b/0xf4
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01261f0>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 __do_softirq+0x0/0xf4
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01060b3>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 do_softirq+0x65/0xb6
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125cfc>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x59/0xc8
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125ca3>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x0/0xc8
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320e5>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 k
> > Aug 17 15:01:54 192.168.20.66 thread+0x3b/0x61
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320aa>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 kthread+0x0/0x61
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c010451b>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 kernel_thread_helper+0x7/0x10
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  =======================
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Code:
> > Aug 17 15:01:54 192.168.20.66 1e
> > Aug 17 15:01:54 192.168.20.66 ec
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 89
> > Aug 17 15:01:54 192.168.20.66 d8
> > Aug 17 15:01:54 192.168.20.66 89
> > Aug 17 15:01:54 192.168.20.66 fb
> > Aug 17 15:01:54 192.168.20.66 e8
> > Aug 17 15:01:54 192.168.20.66 6e
> > Aug 17 15:01:54 192.168.20.66 c2
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 85
> > Aug 17 15:01:54 192.168.20.66 db
> > Aug 17 15:01:54 192.168.20.66 75
> > Aug 17 15:01:54 192.168.20.66 d9
> > Aug 17 15:01:54 192.168.20.66 83
> > Aug 17 15:01:54 192.168.20.66 3e
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 74
> > Aug 17 15:01:54 192.168.20.66 70
> > Aug 17 15:01:54 192.168.20.66 fa
> > Aug 17 15:01:54 192.168.20.66 e8
> > Aug 17 15:01:54 192.168.20.66 e7
> > Aug 17 15:01:54 192.168.20.66 ca
> > Aug 17 15:01:54 192.168.20.66 ed
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 8b
> > Aug 17 15:01:54 192.168.20.66 1e
> > Aug 17 15:01:54 192.168.20.66 c7
> > Aug 17 15:01:54 192.168.20.66 06
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 e8
> > Aug 17 15:01:54 192.168.20.66 c1
> > Aug 17 15:01:54 192.168.20.66 dd
> > Aug 17 15:01:54 192.168.20.66 ed
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 fb
> > Aug 17 15:01:54 192.168.20.66 eb
> > Aug 17 15:01:54 192.168.20.66 56
> > Aug 17 15:01:54 192.168.20.66 unparseable log message: "<8b> "
> > Aug 17 15:01:54 192.168.20.66 43
> > Aug 17 15:01:54 192.168.20.66 74
> > Aug 17 15:01:54 192.168.20.66 89
> > Aug 17 15:01:54 192.168.20.66 45
> > Aug 17 15:01:54 192.168.20.66 f0
> > Aug 17 15:01:54 192.168.20.66 8d
> > Aug 17 15:01:54 192.168.20.66 73
> > Aug 17 15:01:54 192.168.20.66 24
> > Aug 17 15:01:54 192.168.20.66 f0
> > Aug 17 15:01:54 192.168.20.66 80
> > Aug 17 15:01:54 192.168.20.66 63
> > Aug 17 15:01:54 192.168.20.66 24
> > Aug 17 15:01:54 192.168.20.66 fd
> > Aug 17 15:01:54 192.168.20.66 f6
> > Aug 17 15:01:54 192.168.20.66 43
> > Aug 17 15:01:54 192.168.20.66 08
> > Aug 17 15:01:54 192.168.20.66 01
> > Aug 17 15:01:54 192.168.20.66 8d
> > Aug 17 15:01:54 192.168.20.66 7b
> > Aug 17 15:01:54 192.168.20.66 38
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP: [<c025fd4e>]
> > Aug 17 15:01:54 192.168.20.66 net_tx_action+0x78/0xd8
> > Aug 17 15:01:54 192.168.20.66 SS:ESP 0068:c0861fc4
> > Aug 17 15:01:54 192.168.20.66 [ 7222.094000] Kernel panic - not syncing:
> > Fatal exception in interrupt Aug 17 15:01:54 192.168.20.66 [ 7222.094626]
> > Rebooting in 5 seconds..
> >
> > On Sunday 17 August 2008, Jarek Poplawski wrote:
> > > On Sun, Aug 17, 2008 at 01:01:55PM +0300, Denys Fedoryshchenko wrote:
> > > > Ok, now things finally organized.
> > > > fixing patch #1 was missing.
> > > >
> > > > Rebooting system to this kernel. But strange thing, without fixing
> > > > patch #1 it didn't crash for whole night, and 1-3 hours now (with
> > > > same options as before).
> > > >
> > > > Let's test now complete set of patches. I will keep around 2 hours on
> > > > most loaded pppoe NAS, then distribute to 2-3 servers more if it
> > > > doesn't crash.
> > >
> > > BTW, after you complete this testing (no hurry) I would be glad if you
> > > could try one more patch which I send earlier to the list. IMHO, it's
> > > needed to fix some other locking problems. This patch could be applied
> > > and tested as an addition to all currently tested patches (but let's
> > > first be sure they really work).
> > >
> > > Thanks,
> > > Jarek P.
> > >
> > > ------------->
> > >
> > > pkt_sched: Destroy qdiscs under rtnl_lock again.
> > >
> > > We don't need to trigger __qdisc_destroy() as an RCU callback because
> > > the use of qdisc isn't controlled by RCU alone: after querying RCU
> > > with synchronize_rcu() in dev_deactivate() we additionaly wait in a
> > > loop checking some flags. After the loop is done there could be no
> > > outstanding use of the qdisc, so call_rcu() doesn't make any sense.
> > >
> > > On the other hand, current calling Qdisc's ->destroy() from a softirq
> > > context without locking (rtnl) can break various things like:
> > > qdisc_put_rtab(), tcf_destroy_chain() (e.g. u32_destroy()), and
> > > probably more.
> > >
> > >
> > > Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>
> > >
> > > ---
> > >
> > >  net/sched/sch_generic.c |    8 ++------
> > >  1 files changed, 2 insertions(+), 6 deletions(-)
> > >
> > > diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> > > index 4685746..e7379d2 100644
> > > --- a/net/sched/sch_generic.c
> > > +++ b/net/sched/sch_generic.c
> > > @@ -518,12 +518,8 @@ void qdisc_reset(struct Qdisc *qdisc)
> > >  }
> > >  EXPORT_SYMBOL(qdisc_reset);
> > >
> > > -/* this is the rcu callback function to clean up a qdisc when there
> > > - * are no further references to it */
> > > -
> > > -static void __qdisc_destroy(struct rcu_head *head)
> > > +static void __qdisc_destroy(struct Qdisc *qdisc)
> > >  {
> > > -	struct Qdisc *qdisc = container_of(head, struct Qdisc, q_rcu);
> > >  	const struct Qdisc_ops  *ops = qdisc->ops;
> > >
> > >  #ifdef CONFIG_NET_SCHED
> > > @@ -554,7 +550,7 @@ void qdisc_destroy(struct Qdisc *qdisc)
> > >  	if (qdisc->parent)
> > >  		list_del(&qdisc->list);
> > >
> > > -	call_rcu(&qdisc->q_rcu, __qdisc_destroy);
> > > +	__qdisc_destroy(qdisc);
> > >  }
> > >  EXPORT_SYMBOL(qdisc_destroy);
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 12:50                                                     ` Denys Fedoryshchenko
@ 2008-08-17 13:03                                                       ` Denys Fedoryshchenko
  2008-08-17 13:36                                                         ` Jarek Poplawski
  2008-08-17 13:16                                                       ` Jarek Poplawski
  1 sibling, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17 13:03 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

All patches applied.
On normal shaper i got just this warning mentioned before.
On alternative shaper (it is not using TBF anymore, it is using flow 
classifier and many HTB classes) - it is panicing. 

After applying all rules, most probably it is happened when some traffic 
passed over there. I can't be absolutely sure in that, but it looks like at 
that time there is no actions by iproute2/tc done. Just traffic.

This is same PC where WARNING appeared which i mention in previous mail. So 
maybe it is result of that warning, no idea... 

Here is latest panic (alternative shaper):
Aug 17 15:58:49 194.146.153.146 [ 2495.110453] BUG: unable to handle kernel
Aug 17 15:58:49 194.146.153.146 NULL pointer dereference
Aug 17 15:58:49 194.146.153.146 at 0000000c
Aug 17 15:58:49 194.146.153.146 [ 2495.110453] IP:
Aug 17 15:58:49 194.146.153.146 [<c01e065e>] _raw_spin_trylock+0x6/0x3a
Aug 17 15:58:49 194.146.153.146 [ 2495.110453] *pde = 00000000
Aug 17 15:58:49 194.146.153.146
Aug 17 15:58:49 194.146.153.146 [ 2495.110453] Oops: 0000 [#1]
Aug 17 15:58:49 194.146.153.146 SMP
Aug 17 15:58:49 194.146.153.146
Aug 17 15:58:49 194.146.153.146 [ 2495.110453] Modules linked in:
Aug 17 15:58:49 194.146.153.146 cls_flow
Aug 17 15:58:49 194.146.153.146 act_police
Aug 17 15:58:49 194.146.153.146 sch_ingress
Aug 17 15:58:49 194.146.153.146 sch_prio
Aug 17 15:58:49 194.146.153.146 netconsole
Aug 17 15:58:49 194.146.153.146 configfs
Aug 17 15:58:49 194.146.153.146 cls_u32
Aug 17 15:58:49 194.146.153.146 em_meta
Aug 17 15:58:49 194.146.153.146 cls_basic
Aug 17 15:58:49 194.146.153.146 xt_dscp
Aug 17 15:58:49 194.146.153.146 xt_DSCP
Aug 17 15:58:49 194.146.153.146 ipt_REJECT
Aug 17 15:58:49 194.146.153.146 ts_bm
Aug 17 15:58:49 194.146.153.146 xt_string
Aug 17 15:58:49 194.146.153.146 ipt_ttl
Aug 17 15:58:49 194.146.153.146 ifb
Aug 17 15:58:49 194.146.153.146 cls_fw
Aug 17 15:58:49 194.146.153.146 sch_tbf
Aug 17 15:58:49 194.146.153.146 sch_htb
Aug 17 15:58:49 194.146.153.146 act_ipt
Aug 17 15:58:49 194.146.153.146 act_mirred
Aug 17 15:58:49 194.146.153.146 xt_MARK
Aug 17 15:58:49 194.146.153.146 pppoe
Aug 17 15:58:49 194.146.153.146 pppox
Aug 17 15:58:49 194.146.153.146 ppp_generic
Aug 17 15:58:49 194.146.153.146 slhc
Aug 17 15:58:49 194.146.153.146 ipt_MASQUERADE
Aug 17 15:58:49 194.146.153.146 xt_TCPMSS
Aug 17 15:58:49 194.146.153.146 xt_mark
Aug 17 15:58:49 194.146.153.146 xt_tcpudp
Aug 17 15:58:49 194.146.153.146 iptable_mangle
Aug 17 15:58:49 194.146.153.146 iptable_nat
Aug 17 15:58:49 194.146.153.146 nf_nat
Aug 17 15:58:49 194.146.153.146 nf_conntrack_ipv4Aug 17 15:58:49 
194.146.153.146 iptable_filter
Aug 17 15:58:49 194.146.153.146 ip_tables
Aug 17 15:58:49 194.146.153.146 x_tables
Aug 17 15:58:49 194.146.153.146 8021q
Aug 17 15:58:49 194.146.153.146 garp
Aug 17 15:58:49 194.146.153.146 stp
Aug 17 15:58:49 194.146.153.146 llc
Aug 17 15:58:49 194.146.153.146 tun
Aug 17 15:58:49 194.146.153.146 loop
Aug 17 15:58:49 194.146.153.146 tulip
Aug 17 15:58:50 194.146.153.146 r8169
Aug 17 15:58:50 194.146.153.146 sky2
Aug 17 15:58:50 194.146.153.146 via_velocity
Aug 17 15:58:50 194.146.153.146 via_rhine
Aug 17 15:58:50 194.146.153.146 sis900
Aug 17 15:58:50 194.146.153.146 ne2k_pci
Aug 17 15:58:50 194.146.153.146 8390
Aug 17 15:58:50 194.146.153.146 skge
Aug 17 15:58:50 194.146.153.146 tg3
Aug 17 15:58:50 194.146.153.146 libphy
Aug 17 15:58:50 194.146.153.146 8139too
Aug 17 15:58:50 194.146.153.146 e1000
Aug 17 15:58:50 194.146.153.146 e100
Aug 17 15:58:50 194.146.153.146 mtdblock
Aug 17 15:58:50 194.146.153.146 mtd_blkdevs
Aug 17 15:58:50 194.146.153.146 aic79xx
Aug 17 15:58:50 194.146.153.146 mptsas
Aug 17 15:58:50 194.146.153.146 scsi_transport_sas
Aug 17 15:58:50 194.146.153.146 mptspi
Aug 17 15:58:50 194.146.153.146 scsi_transport_spi
Aug 17 15:58:50 194.146.153.146 mptscsih
Aug 17 15:58:50 194.146.153.146 mptbase
Aug 17 15:58:50 194.146.153.146 sata_via
Aug 17 15:58:50 194.146.153.146 sata_uliAug 17 15:58:50 194.146.153.146 libata
Aug 17 15:58:50 194.146.153.146 dock
Aug 17 15:58:50 194.146.153.146 usbhid
Aug 17 15:58:50 194.146.153.146 uhci_hcd
Aug 17 15:58:50 194.146.153.146 ehci_hcd
Aug 17 15:58:50 194.146.153.146 ohci_hcd
Aug 17 15:58:50 194.146.153.146 usbcore
Aug 17 15:58:50 194.146.153.146
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] Pid: 0, comm: swapper Not 
tainted (2.6.27-rc3-git3-build-0031 #9)
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] EIP: 0060:[<c01e065e>] EFLAGS: 
00010246 CPU: 0
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] EIP is at 
_raw_spin_trylock+0x6/0x3a
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] EAX: 0000000c EBX: 0000000c 
ECX: c01295d1 EDX: 0000000c
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] ESI: f6c0b024 EDI: 0000000c 
EBP: c0861fb0 ESP: c0861fac
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  DS: 007b ES: 007b FS: 00d8 GS: 
0000 SS: 0068
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] Process swapper (pid: 0, 
ti=c0861000 task=c03833b4 task.ti=c03a9000)
Aug 17 15:58:50 194.146.153.146
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] Stack:
Aug 17 15:58:50 194.146.153.146 0000000c
Aug 17 15:58:50 194.146.153.146 c0861fbc
Aug 17 15:58:50 194.146.153.146 c02bdae3
Aug 17 15:58:50 194.146.153.146 f6c0b000
Aug 17 15:58:50 194.146.153.146 c0861fd4
Aug 17 15:58:50 194.146.153.146 c025fd72
Aug 17 15:58:50 194.146.153.146 00000000
Aug 17 15:58:50 194.146.153.146 c039ea88
Aug 17 15:58:50 194.146.153.146
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]
Aug 17 15:58:50 194.146.153.146 c085a554
Aug 17 15:58:50 194.146.153.146 00000001
Aug 17 15:58:50 194.146.153.146 c0861ff8
Aug 17 15:58:50 194.146.153.146 c012626bAug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 net_tx_action+0x9c/0xd8
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c012626b>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 __do_softirq+0x7b/0xf4
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c01261f0>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 __do_softirq+0x0/0xf4
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c01060b3>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 do_softirq+0x65/0xb6
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c014a264>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 handle_edge_irq+0x0/0x109
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c0125e90>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 irq_exit+0x44/0x79
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c0106038>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 do_IRQ+0xae/0xc4
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c0104288>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 common_interrupt+0x28/0x30
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c013007b>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 task_session_nr_ns+0x6/0x30
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c0108d8a>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 mwait_idle+0x39/0x43
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c01029ee>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 cpu_idle+0xbf/0xe1
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c02b1156>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 rest_init+0x4e/0x50
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  =======================
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] Code:Aug 17 15:58:50 
194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 f8
Aug 17 15:58:50 194.146.153.146 e8
Aug 17 15:58:50 194.146.153.146 2c
Aug 17 15:58:50 194.146.153.146 b5
Aug 17 15:58:50 194.146.153.146 04
Aug 17 15:58:50 194.146.153.146 00
Aug 17 15:58:50 194.146.153.146 eb
Aug 17 15:58:50 194.146.153.146 07
Aug 17 15:58:50 194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 d8
Aug 17 15:58:50 194.146.153.146 e8
Aug 17 15:58:50 194.146.153.146 83
Aug 17 15:58:50 194.146.153.146 b8
Aug 17 15:58:50 194.146.153.146 04
Aug 17 15:58:50 194.146.153.146 00
Aug 17 15:58:50 194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 f0
Aug 17 15:58:50 194.146.153.146 5e
Aug 17 15:58:50 194.146.153.146 5f
Aug 17 15:58:50 194.146.153.146 5b
Aug 17 15:58:50 194.146.153.146 5e
Aug 17 15:58:50 194.146.153.146 5f
Aug 17 15:58:50 194.146.153.146 5d
Aug 17 15:58:50 194.146.153.146 c3
Aug 17 15:58:50 194.146.153.146 55
Aug 17 15:58:50 194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 c2
Aug 17 15:58:50 194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 e5
Aug 17 15:58:50 194.146.153.146 53
Aug 17 15:58:50 194.146.153.146
Aug 17 15:58:50 194.146.153.146 8b
Aug 17 15:58:50 194.146.153.146 02
Aug 17 15:58:50 194.146.153.146 38
Aug 17 15:58:50 194.146.153.146 e0
Aug 17 15:58:50 194.146.153.146 75
Aug 17 15:58:50 194.146.153.146 0a
Aug 17 15:58:50 194.146.153.146 66
Aug 17 15:58:50 194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 c3
Aug 17 15:58:50 194.146.153.146 fe
Aug 17 15:58:50 194.146.153.146 c7
Aug 17 15:58:50 194.146.153.146 f0
Aug 17 15:58:50 194.146.153.146 66
Aug 17 15:58:50 194.146.153.146 0f
Aug 17 15:58:50 194.146.153.146 b1
Aug 17 15:58:50 194.146.153.146 1a
Aug 17 15:58:50 194.146.153.146 0f
Aug 17 15:58:50 194.146.153.146 94
Aug 17 15:58:50 194.146.153.146 c3
Aug 17 15:58:50 194.146.153.146 0f
Aug 17 15:58:50 194.146.153.146
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] EIP: [<c01e065e>]
Aug 17 15:58:50 194.146.153.146 _raw_spin_trylock+0x6/0x3a
Aug 17 15:58:50 194.146.153.146 SS:ESP 0068:c0861fac
Aug 17 15:58:50 194.146.153.146 [ 2495.117172] Kernel panic - not syncing: 
Fatal exception in interrupt
Aug 17 15:58:50 194.146.153.146 [ 2495.117264] Rebooting in 5 seconds..

Aug 17 15:58:50 194.146.153.146 f3
Aug 17 15:58:50 194.146.153.146 ff
Aug 17 15:58:50 194.146.153.146 85
Aug 17 15:58:50 194.146.153.146 c0
Aug 17 15:58:50 194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 c6
Aug 17 15:58:50 194.146.153.146 74
Aug 17 15:58:50 194.146.153.146 0d
Aug 17 15:58:50 194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 03
Aug 17 15:58:50 194.146.153.146 89
Aug 17 15:58:50 194.146.153.146 da

Aug 17 15:58:50 194.146.153.146 c085d700
Aug 17 15:58:50 194.146.153.146 c085d700
Aug 17 15:58:50 194.146.153.146 00000009
Aug 17 15:58:50 194.146.153.146 00000000
Aug 17 15:58:50 194.146.153.146
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]
Aug 17 15:58:50 194.146.153.146 c03a9f44
Aug 17 15:58:50 194.146.153.146 c03a9000
Aug 17 15:58:50 194.146.153.146 c01261f0
Aug 17 15:58:50 194.146.153.146 c03a9f54
Aug 17 15:58:50 194.146.153.146 c01060b3
Aug 17 15:58:50 194.146.153.146
Aug 17 15:58:50 194.146.153.146 [ 2495.110453] Call Trace:
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c02bdae3>]
Aug 17 15:58:50 194.146.153.146 ?
Aug 17 15:58:50 194.146.153.146 __lock_text_start+0xb/0x38
Aug 17 15:58:50 194.146.153.146 [ 2495.110453]  [<c025fd72>]

Aug 17 15:58:50 194.146.153.146 sata_svw
Aug 17 15:58:50 194.146.153.146 sata_sis
Aug 17 15:58:50 194.146.153.146 sata_sil24
Aug 17 15:58:50 194.146.153.146 sata_sil
Aug 17 15:58:50 194.146.153.146 sata_mv
Aug 17 15:58:50 194.146.153.146 ahci
Aug 17 15:58:50 194.146.153.146 pata_via
Aug 17 15:58:50 194.146.153.146 pata_sis
Aug 17 15:58:50 194.146.153.146 pata_serverworks
Aug 17 15:58:50 194.146.153.146 pata_oldpiix
Aug 17 15:58:50 194.146.153.146 pata_mpiix
Aug 17 15:58:50 194.146.153.146 pata_marvell
Aug 17 15:58:50 194.146.153.146 pata_ali
Aug 17 15:58:50 194.146.153.146 ata_generic
Aug 17 15:58:50 194.146.153.146 ata_piix

Aug 17 15:58:49 194.146.153.146 nf_conntrack
Aug 17 15:58:49 194.146.153.146 rtc_cmos
Aug 17 15:58:49 194.146.153.146 rtc_core
Aug 17 15:58:49 194.146.153.146 rtc_lib

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 12:50                                                     ` Denys Fedoryshchenko
  2008-08-17 13:03                                                       ` Denys Fedoryshchenko
@ 2008-08-17 13:16                                                       ` Jarek Poplawski
  1 sibling, 0 replies; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17 13:16 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sun, Aug 17, 2008 at 03:50:01PM +0300, Denys Fedoryshchenko wrote:
> On Sunday 17 August 2008, Jarek Poplawski wrote:
> 
> > Denys, I would like to be sure: do you mean here the last patch
> > attached below? So, the alternative shaper works with previous
> > patches, but not this one? If so, than no need to test this patch
> > anymore, only confirm this, please.
> No.
> Alternative shaper didn't work without applying latest patch (just i had old 
> kernel, and i decide just to try alternative shaper).
> 
> Latest warning with all patches applied. I will test alternative shaper with 
> all patches applied now.
> 

I think it would be better to test first only with patches:
take #3, patch #1, patch #2 (and of these 2 from earlier thread).

This last patch, let's call it patch #4, can wait.

If there are oopses please try to add to them this one:

Jarek P.

--- (patch #5)

 net/sched/sch_api.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
index c25465e..5fdd1e2 100644
--- a/net/sched/sch_api.c
+++ b/net/sched/sch_api.c
@@ -694,6 +694,9 @@ static int qdisc_graft(struct net_device *dev, struct Qdisc *parent,
 
 		err = -EINVAL;
 
+		if (dev->flags & IFF_UP)
+			dev_deactivate(dev);
+
 		if (cops) {
 			unsigned long cl = cops->get(parent, classid);
 			if (cl) {
@@ -703,6 +706,9 @@ static int qdisc_graft(struct net_device *dev, struct Qdisc *parent,
 		}
 		if (!err)
 			notify_and_destroy(skb, n, classid, old, new);
+
+		if (dev->flags & IFF_UP)
+			dev_activate(dev);
 	}
 	return err;
 }

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 13:03                                                       ` Denys Fedoryshchenko
@ 2008-08-17 13:36                                                         ` Jarek Poplawski
  2008-08-17 19:36                                                           ` Denys Fedoryshchenko
  0 siblings, 1 reply; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17 13:36 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sun, Aug 17, 2008 at 04:03:22PM +0300, Denys Fedoryshchenko wrote:
> All patches applied.
> On normal shaper i got just this warning mentioned before.
> On alternative shaper (it is not using TBF anymore, it is using flow 
> classifier and many HTB classes) - it is panicing. 
> 
> After applying all rules, most probably it is happened when some traffic 
> passed over there. I can't be absolutely sure in that, but it looks like at 
> that time there is no actions by iproute2/tc done. Just traffic.
> 
> This is same PC where WARNING appeared which i mention in previous mail. So 
> maybe it is result of that warning, no idea... 

BTW, I'll have to have a break for today, so I'll be able to respond
only tomorrow. Your testing is very valuable and this all needs
rethinking, but as before, I doubt current -rc is good choice for
production.

Cheers,
Jarek P.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 13:36                                                         ` Jarek Poplawski
@ 2008-08-17 19:36                                                           ` Denys Fedoryshchenko
  2008-08-17 21:47                                                             ` Jarek Poplawski
  0 siblings, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17 19:36 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

I am lost in patches :-)
When you will be back, let me know if order and set of patches is correct.
Here is what i am using:

http://www.nuclearcat.com/patches/00-prevfixes1.patch
http://www.nuclearcat.com/patches/00-prevfixes2.patch
http://www.nuclearcat.com/patches/01-take3.patch
http://www.nuclearcat.com/patches/02-fix1.patch
http://www.nuclearcat.com/patches/03-fix_patch_2.patch

And thanks for your help!
I cannot take risk and put to wide production, since it is crashing sometimes, 
but i guess in few days we can make it ready for production.

On Sunday 17 August 2008, Jarek Poplawski wrote:
> On Sun, Aug 17, 2008 at 04:03:22PM +0300, Denys Fedoryshchenko wrote:
> > All patches applied.
> > On normal shaper i got just this warning mentioned before.
> > On alternative shaper (it is not using TBF anymore, it is using flow
> > classifier and many HTB classes) - it is panicing.
> >
> > After applying all rules, most probably it is happened when some traffic
> > passed over there. I can't be absolutely sure in that, but it looks like
> > at that time there is no actions by iproute2/tc done. Just traffic.
> >
> > This is same PC where WARNING appeared which i mention in previous mail.
> > So maybe it is result of that warning, no idea...
>
> BTW, I'll have to have a break for today, so I'll be able to respond
> only tomorrow. Your testing is very valuable and this all needs
> rethinking, but as before, I doubt current -rc is good choice for
> production.
>
> Cheers,
> Jarek P.



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 19:36                                                           ` Denys Fedoryshchenko
@ 2008-08-17 21:47                                                             ` Jarek Poplawski
  2008-08-17 22:02                                                               ` David Miller
  2008-08-17 22:21                                                               ` Denys Fedoryshchenko
  0 siblings, 2 replies; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17 21:47 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Sun, Aug 17, 2008 at 10:36:17PM +0300, Denys Fedoryshchenko wrote:
> I am lost in patches :-)
> When you will be back, let me know if order and set of patches is correct.
> Here is what i am using:
> 
> http://www.nuclearcat.com/patches/00-prevfixes1.patch
> http://www.nuclearcat.com/patches/00-prevfixes2.patch
> http://www.nuclearcat.com/patches/01-take3.patch
> http://www.nuclearcat.com/patches/02-fix1.patch
> http://www.nuclearcat.com/patches/03-fix_patch_2.patch
> 
> And thanks for your help!
> I cannot take risk and put to wide production, since it is crashing sometimes, 
> but i guess in few days we can make it ready for production.
> 

OK, so please try to test with all these patches above plus one new
patch below.

Thanks,
Jarek P.

(04-fix_patch_3.patch)
---

 net/sched/sch_api.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
index c25465e..c8dc72e 100644
--- a/net/sched/sch_api.c
+++ b/net/sched/sch_api.c
@@ -638,9 +638,9 @@ static void notify_and_destroy(struct sk_buff *skb, struct nlmsghdr *n, u32 clid
 		qdisc_notify(skb, n, clid, old, new);
 
 	if (old) {
-		spin_lock_bh(&old->q.lock);
+		sch_tree_lock(old);
 		qdisc_destroy(old);
-		spin_unlock_bh(&old->q.lock);
+		sch_tree_unlock(old);
 	}
 }
 

^ permalink raw reply related	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 21:47                                                             ` Jarek Poplawski
@ 2008-08-17 22:02                                                               ` David Miller
  2008-08-17 22:21                                                               ` Denys Fedoryshchenko
  1 sibling, 0 replies; 38+ messages in thread
From: David Miller @ 2008-08-17 22:02 UTC (permalink / raw)
  To: jarkao2; +Cc: denys, netdev

From: Jarek Poplawski <jarkao2@gmail.com>
Date: Sun, 17 Aug 2008 23:47:52 +0200

> @@ -638,9 +638,9 @@ static void notify_and_destroy(struct sk_buff *skb, struct nlmsghdr *n, u32 clid
>  		qdisc_notify(skb, n, clid, old, new);
>  
>  	if (old) {
> -		spin_lock_bh(&old->q.lock);
> +		sch_tree_lock(old);
>  		qdisc_destroy(old);
> -		spin_unlock_bh(&old->q.lock);
> +		sch_tree_unlock(old);
>  	}
>  }
>  

Ok, this makes sense, for the classful qdisc case.

If we really wanted to be super-careful, we could remember the root
qdisc at the top of qdisc_graft() and pass that into
notify_and_destroy().


^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 21:47                                                             ` Jarek Poplawski
  2008-08-17 22:02                                                               ` David Miller
@ 2008-08-17 22:21                                                               ` Denys Fedoryshchenko
  2008-08-17 22:32                                                                 ` Jarek Poplawski
  1 sibling, 1 reply; 38+ messages in thread
From: Denys Fedoryshchenko @ 2008-08-17 22:21 UTC (permalink / raw)
  To: Jarek Poplawski; +Cc: netdev

Short testing - TBF is working fine. No warnings, no oops, nothing yet. 
Before - intensive changes was causing oops easily. I will test for 1 day 
more with TBF.

P.S. New shaper (based on flow classifier + HTB classes, instead of prio+tbf) 
crashing very fast on loaded server. Probably it is another bug.
Here is message:
http://www.nuclearcat.com/files/crash-newshaper.txt

On Monday 18 August 2008, Jarek Poplawski wrote:
> On Sun, Aug 17, 2008 at 10:36:17PM +0300, Denys Fedoryshchenko wrote:
> > I am lost in patches :-)
> > When you will be back, let me know if order and set of patches is
> > correct. Here is what i am using:
> >
> > http://www.nuclearcat.com/patches/00-prevfixes1.patch
> > http://www.nuclearcat.com/patches/00-prevfixes2.patch
> > http://www.nuclearcat.com/patches/01-take3.patch
> > http://www.nuclearcat.com/patches/02-fix1.patch
> > http://www.nuclearcat.com/patches/03-fix_patch_2.patch
> >
> > And thanks for your help!
> > I cannot take risk and put to wide production, since it is crashing
> > sometimes, but i guess in few days we can make it ready for production.
>
> OK, so please try to test with all these patches above plus one new
> patch below.
>
> Thanks,
> Jarek P.
>
> (04-fix_patch_3.patch)
> ---
>
>  net/sched/sch_api.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
> index c25465e..c8dc72e 100644
> --- a/net/sched/sch_api.c
> +++ b/net/sched/sch_api.c
> @@ -638,9 +638,9 @@ static void notify_and_destroy(struct sk_buff *skb,
> struct nlmsghdr *n, u32 clid qdisc_notify(skb, n, clid, old, new);
>
>  	if (old) {
> -		spin_lock_bh(&old->q.lock);
> +		sch_tree_lock(old);
>  		qdisc_destroy(old);
> -		spin_unlock_bh(&old->q.lock);
> +		sch_tree_unlock(old);
>  	}
>  }
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head
  2008-08-17 22:21                                                               ` Denys Fedoryshchenko
@ 2008-08-17 22:32                                                                 ` Jarek Poplawski
  0 siblings, 0 replies; 38+ messages in thread
From: Jarek Poplawski @ 2008-08-17 22:32 UTC (permalink / raw)
  To: Denys Fedoryshchenko; +Cc: netdev

On Mon, Aug 18, 2008 at 01:21:40AM +0300, Denys Fedoryshchenko wrote:
> Short testing - TBF is working fine. No warnings, no oops, nothing yet. 
> Before - intensive changes was causing oops easily. I will test for 1 day 
> more with TBF.
> 
> P.S. New shaper (based on flow classifier + HTB classes, instead of prio+tbf) 
> crashing very fast on loaded server. Probably it is another bug.
> Here is message:
> http://www.nuclearcat.com/files/crash-newshaper.txt

I'll try to look at this "tomorrow".

Thanks,
Jarek P.

> 
> On Monday 18 August 2008, Jarek Poplawski wrote:
> > On Sun, Aug 17, 2008 at 10:36:17PM +0300, Denys Fedoryshchenko wrote:
> > > I am lost in patches :-)
> > > When you will be back, let me know if order and set of patches is
> > > correct. Here is what i am using:
> > >
> > > http://www.nuclearcat.com/patches/00-prevfixes1.patch
> > > http://www.nuclearcat.com/patches/00-prevfixes2.patch
> > > http://www.nuclearcat.com/patches/01-take3.patch
> > > http://www.nuclearcat.com/patches/02-fix1.patch
> > > http://www.nuclearcat.com/patches/03-fix_patch_2.patch
> > >
> > > And thanks for your help!
> > > I cannot take risk and put to wide production, since it is crashing
> > > sometimes, but i guess in few days we can make it ready for production.
> >
> > OK, so please try to test with all these patches above plus one new
> > patch below.
> >
> > Thanks,
> > Jarek P.
> >
> > (04-fix_patch_3.patch)
> > ---
> >
> >  net/sched/sch_api.c |    4 ++--
> >  1 files changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
> > index c25465e..c8dc72e 100644
> > --- a/net/sched/sch_api.c
> > +++ b/net/sched/sch_api.c
> > @@ -638,9 +638,9 @@ static void notify_and_destroy(struct sk_buff *skb,
> > struct nlmsghdr *n, u32 clid qdisc_notify(skb, n, clid, old, new);
> >
> >  	if (old) {
> > -		spin_lock_bh(&old->q.lock);
> > +		sch_tree_lock(old);
> >  		qdisc_destroy(old);
> > -		spin_unlock_bh(&old->q.lock);
> > +		sch_tree_unlock(old);
> >  	}
> >  }
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

^ permalink raw reply	[flat|nested] 38+ messages in thread

end of thread, other threads:[~2008-08-17 22:31 UTC | newest]

Thread overview: 38+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-08-15 19:09 panic 2.6.27-rc3-git2, qdisc_dequeue_head Denys Fedoryshchenko
2008-08-15 20:16 ` Jarek Poplawski
2008-08-15 20:36   ` Denys Fedoryshchenko
2008-08-15 20:54     ` Denys Fedoryshchenko
2008-08-16  8:55       ` Jarek Poplawski
2008-08-16  9:13         ` Jarek Poplawski
2008-08-16 10:03           ` Denys Fedoryshchenko
2008-08-16 10:05             ` Denys Fedoryshchenko
2008-08-16 12:27               ` Jarek Poplawski
2008-08-16 12:24                 ` Denys Fedoryshchenko
2008-08-16 12:34                   ` Jarek Poplawski
2008-08-16 13:13                     ` Denys Fedoryshchenko
2008-08-16 14:27                       ` Jarek Poplawski
2008-08-16 15:28                         ` Denys Fedoryshchenko
2008-08-16 15:59                           ` Jarek Poplawski
2008-08-16 18:15                             ` Denys Fedoryshchenko
2008-08-16 18:52                               ` Jarek Poplawski
2008-08-16 19:22                                 ` Denys Fedoryshchenko
2008-08-16 19:47                                   ` Jarek Poplawski
2008-08-17  9:02                                     ` Denys Fedoryshchenko
2008-08-17  9:34                                       ` Jarek Poplawski
2008-08-17  9:35                                         ` Denys Fedoryshchenko
2008-08-17  9:52                                           ` Jarek Poplawski
2008-08-17 10:01                                             ` Denys Fedoryshchenko
2008-08-17 10:31                                               ` Jarek Poplawski
2008-08-17 10:55                                               ` Jarek Poplawski
2008-08-17 12:06                                                 ` Denys Fedoryshchenko
2008-08-17 12:18                                                   ` Denys Fedoryshchenko
2008-08-17 12:38                                                   ` Jarek Poplawski
2008-08-17 12:50                                                     ` Denys Fedoryshchenko
2008-08-17 13:03                                                       ` Denys Fedoryshchenko
2008-08-17 13:36                                                         ` Jarek Poplawski
2008-08-17 19:36                                                           ` Denys Fedoryshchenko
2008-08-17 21:47                                                             ` Jarek Poplawski
2008-08-17 22:02                                                               ` David Miller
2008-08-17 22:21                                                               ` Denys Fedoryshchenko
2008-08-17 22:32                                                                 ` Jarek Poplawski
2008-08-17 13:16                                                       ` Jarek Poplawski

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).