Netdev List
 help / color / mirror / Atom feed
* Re: [PATCH] net/8021q/vlan_dev.c - Use print_mac
From: Joe Perches @ 2008-02-18 18:31 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: David Miller, bruno-L9ZBdB2wSWtl57MIdRCFDg,
	netdev-u79uwXL29TY76Z2rM5mHXA, jgarzik-e+AXbWqSrlAAvxtiuMwx3w,
	linux-wireless-u79uwXL29TY76Z2rM5mHXA,
	linville-2XuSBdqkA4R54TAoqtyWWQ
In-Reply-To: <47B9A20C.10304-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>

On Mon, 2008-02-18 at 16:19 +0100, Patrick McHardy wrote:
> > @@ -404,11 +405,8 @@ static int vlan_dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev)
> >  
> >  	pr_debug("%s: about to send skb: %p to dev: %s\n",
> >  		__FUNCTION__, skb, skb->dev->name);
> > -	pr_debug("  " MAC_FMT " " MAC_FMT " %4hx %4hx %4hx\n",
> > -		 veth->h_dest[0], veth->h_dest[1], veth->h_dest[2],
> > -		 veth->h_dest[3], veth->h_dest[4], veth->h_dest[5],
> > -		 veth->h_source[0], veth->h_source[1], veth->h_source[2],
> > -		 veth->h_source[3], veth->h_source[4], veth->h_source[5],
> > +	pr_debug("  %s %s %4hx %4hx %4hx\n",
> > +		 print_mac(mac, veth->h_dest), print_mac(mac2, veth->h_source),
> This results in print_mac getting called twice per packet even without
> debugging. Whats the problem with MAC_FMT?

It's just a consistency thing.
It identifies code where MAC addresses are used.
an allyesconfig is a bit smaller (~.1%).
pr_debug is a noop when not debugging, print_mac is optimized away.

^ permalink raw reply

* Re: Linux 2.6.24.1  - kernel does not boot; IRQ trouble?
From: Chris Rankin @ 2008-02-18 19:42 UTC (permalink / raw)
  To: Stephen Hemminger, Andrew Morton; +Cc: linux-acpi, linux-kernel, netdev
In-Reply-To: <20080218085333.3d32b626@extreme>

--- Stephen Hemminger <shemminger@linux-foundation.org> wrote:
> > > sysfs: duplicate filename 'bridge' can not be created
> > > WARNING: at fs/sysfs/dir.c:424 sysfs_add_one()
> > > Pid: 1, comm: swapper Not tainted 2.6.24.1 #1
> > >  [<c0105020>] show_trace_log_lvl+0x1a/0x2f
> > >  [<c0105990>] show_trace+0x12/0x14
> > >  [<c010613d>] dump_stack+0x6c/0x72
> > >  [<c01991bf>] sysfs_add_one+0x57/0xbc
> > >  [<c0199e41>] sysfs_create_link+0xc2/0x10d
> > >  [<c01bae9a>] pci_bus_add_devices+0xbd/0x103
> > >  [<c034016c>] pci_legacy_init+0x56/0xe3
> > >  [<c03274e1>] kernel_init+0x157/0x2c3
> > >  [<c0104c83>] kernel_thread_helper+0x7/0x10
> > >  =======================
> > > pci 0000:00:01.0: Error creating sysfs bridge symlink, continuing...
> > > sysfs: duplicate filename 'bridge' can not be created
> > > WARNING: at fs/sysfs/dir.c:424 sysfs_add_one()
> > > Pid: 1, comm: swapper Not tainted 2.6.24.1 #1
> > >  [<c0105020>] show_trace_log_lvl+0x1a/0x2f
> > >  [<c0105990>] show_trace+0x12/0x14
> > >  [<c010613d>] dump_stack+0x6c/0x72
> > >  [<c01991bf>] sysfs_add_one+0x57/0xbc
> > >  [<c0199e41>] sysfs_create_link+0xc2/0x10d
> > >  [<c01bae9a>] pci_bus_add_devices+0xbd/0x103
> > >  [<c01bae82>] pci_bus_add_devices+0xa5/0x103
> > >  [<c034016c>] pci_legacy_init+0x56/0xe3
> > >  [<c03274e1>] kernel_init+0x157/0x2c3
> > >  [<c0104c83>] kernel_thread_helper+0x7/0x10
> > >  =======================
> > 
> > I have a vague feeling that this was fixed, perhaps in 2.6.24.x?
> 
> Never heard of this, what is the initialization script that causes this?
> Also do you have the SYSFS_DEPRECATED option configured? that caused issues
> with regular network drivers.

Yes, SYSFS_DEPRECATED is enabled. And the init scripts are from Fedora 8.

Cheers,
Chris






      __________________________________________________________
Sent from Yahoo! Mail - a smarter inbox http://uk.mail.yahoo.com


^ permalink raw reply

* Re: keyboard dead with 45b5035
From: Rafael J. Wysocki @ 2008-02-18 19:50 UTC (permalink / raw)
  To: Pierre Ossman; +Cc: Laszlo Attila Toth, netdev, LKML
In-Reply-To: <20080218185305.48a67587@mjolnir.drzeus.cx>

On Monday, 18 of February 2008, Pierre Ossman wrote:
> The patch "[RTNETLINK]: Send a single notification on device state changes." kills (at least)
> the keyboard here. Everything seems to work fine in single user mode, but when init starts
> spawning of logins, the keyboard goes bye-bye. Even the power button is ignored. :/ 

Please try with the patch from http://lkml.org/lkml/2008/2/18/331 .

Thanks,
Rafael

^ permalink raw reply

* Re: Linux 2.6.24.1  - kernel does not boot; IRQ trouble?
From: Stephen Hemminger @ 2008-02-18 20:06 UTC (permalink / raw)
  To: Chris Rankin; +Cc: Andrew Morton, linux-acpi, linux-kernel, netdev
In-Reply-To: <395400.45093.qm@web52907.mail.re2.yahoo.com>

On Mon, 18 Feb 2008 19:42:25 +0000 (GMT)
Chris Rankin <rankincj@yahoo.com> wrote:

> --- Stephen Hemminger <shemminger@linux-foundation.org> wrote:
> > > > sysfs: duplicate filename 'bridge' can not be created
> > > > WARNING: at fs/sysfs/dir.c:424 sysfs_add_one()
> > > > Pid: 1, comm: swapper Not tainted 2.6.24.1 #1
> > > >  [<c0105020>] show_trace_log_lvl+0x1a/0x2f
> > > >  [<c0105990>] show_trace+0x12/0x14
> > > >  [<c010613d>] dump_stack+0x6c/0x72
> > > >  [<c01991bf>] sysfs_add_one+0x57/0xbc
> > > >  [<c0199e41>] sysfs_create_link+0xc2/0x10d
> > > >  [<c01bae9a>] pci_bus_add_devices+0xbd/0x103
> > > >  [<c034016c>] pci_legacy_init+0x56/0xe3
> > > >  [<c03274e1>] kernel_init+0x157/0x2c3
> > > >  [<c0104c83>] kernel_thread_helper+0x7/0x10
> > > >  =======================
> > > > pci 0000:00:01.0: Error creating sysfs bridge symlink, continuing...
> > > > sysfs: duplicate filename 'bridge' can not be created
> > > > WARNING: at fs/sysfs/dir.c:424 sysfs_add_one()
> > > > Pid: 1, comm: swapper Not tainted 2.6.24.1 #1
> > > >  [<c0105020>] show_trace_log_lvl+0x1a/0x2f
> > > >  [<c0105990>] show_trace+0x12/0x14
> > > >  [<c010613d>] dump_stack+0x6c/0x72
> > > >  [<c01991bf>] sysfs_add_one+0x57/0xbc
> > > >  [<c0199e41>] sysfs_create_link+0xc2/0x10d
> > > >  [<c01bae9a>] pci_bus_add_devices+0xbd/0x103
> > > >  [<c01bae82>] pci_bus_add_devices+0xa5/0x103
> > > >  [<c034016c>] pci_legacy_init+0x56/0xe3
> > > >  [<c03274e1>] kernel_init+0x157/0x2c3
> > > >  [<c0104c83>] kernel_thread_helper+0x7/0x10
> > > >  =======================
> > > 
> > > I have a vague feeling that this was fixed, perhaps in 2.6.24.x?
> > 
> > Never heard of this, what is the initialization script that causes this?
> > Also do you have the SYSFS_DEPRECATED option configured? that caused issues
> > with regular network drivers.
> 
> Yes, SYSFS_DEPRECATED is enabled. And the init scripts are from Fedora 8.

There was a bug (fixed in 2.6.24) that had to do with sysfs_create_link
and SYSFS_DEPRECATED probably there is a similar problem with directories. 


^ permalink raw reply

* [PATCH 1/1] sis190: read the mac address from the eeprom first
From: Francois Romieu @ 2008-02-18 20:20 UTC (permalink / raw)
  To: Jeff Garzik; +Cc: netdev, Andrew Morton, dsd, Gabriel A. Devenyi

Reading a serie of zero from the cmos sram area do not work
well with is_valid_ether_addr(). Let's read the mac address
from the eeprom first as it seems more reliable.

Fix for http://bugzilla.kernel.org/show_bug.cgi?id=9831

Signed-off-by: Francois Romieu <romieu@fr.zoreil.com>
---
 drivers/net/sis190.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/net/sis190.c b/drivers/net/sis190.c
index 202fdf3..20745fd 100644
--- a/drivers/net/sis190.c
+++ b/drivers/net/sis190.c
@@ -1633,13 +1633,18 @@ static inline void sis190_init_rxfilter(struct net_device *dev)
 static int __devinit sis190_get_mac_addr(struct pci_dev *pdev, 
 					 struct net_device *dev)
 {
-	u8 from;
+	int rc;
+
+	rc = sis190_get_mac_addr_from_eeprom(pdev, dev);
+	if (rc < 0) {
+		u8 reg;
 
-	pci_read_config_byte(pdev, 0x73, &from);
+		pci_read_config_byte(pdev, 0x73, &reg);
 
-	return (from & 0x00000001) ?
-		sis190_get_mac_addr_from_apc(pdev, dev) :
-		sis190_get_mac_addr_from_eeprom(pdev, dev);
+		if (reg & 0x00000001)
+			rc = sis190_get_mac_addr_from_apc(pdev, dev);
+	}
+	return rc;
 }
 
 static void sis190_set_speed_auto(struct net_device *dev)
-- 
1.5.3.3


^ permalink raw reply related

* Re: keyboard dead with 45b5035
From: Pierre Ossman @ 2008-02-18 20:50 UTC (permalink / raw)
  To: Rafael J. Wysocki; +Cc: Laszlo Attila Toth, netdev, LKML
In-Reply-To: <200802182050.01964.rjw@sisk.pl>

On Mon, 18 Feb 2008 20:50:01 +0100
"Rafael J. Wysocki" <rjw@sisk.pl> wrote:

> On Monday, 18 of February 2008, Pierre Ossman wrote:
> > The patch "[RTNETLINK]: Send a single notification on device state changes." kills (at least)
> > the keyboard here. Everything seems to work fine in single user mode, but when init starts
> > spawning of logins, the keyboard goes bye-bye. Even the power button is ignored. :/ 
> 
> Please try with the patch from http://lkml.org/lkml/2008/2/18/331 .
> 

That solved it.

I wonder if that's also why modprobe tends to wedge up with the new USB announce thingy... Tomorrow's debugging will tell.

-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org

^ permalink raw reply

* Re: [PATCH] net/8021q/vlan_dev.c - Use print_mac
From: David Miller @ 2008-02-18 20:55 UTC (permalink / raw)
  To: kaber; +Cc: joe, bruno, netdev, jgarzik, linux-wireless, linville
In-Reply-To: <47B9A20C.10304@trash.net>

From: Patrick McHardy <kaber@trash.net>
Date: Mon, 18 Feb 2008 16:19:40 +0100

> Joe Perches wrote:
> > On Fri, 2008-02-15 at 02:58 -0800, David Miller wrote:
> >> From: Bruno Randolf <bruno@thinktube.com>
> >> Date: Fri, 15 Feb 2008 19:48:05 +0900
> >>> is there any chance to include a macro like this for printing mac
> >> addresses?
> >>> its advantage is that it can be used without the need to declare
> >> buffers for
> >>> print_mac(), for example:
> >> We specifically removed this sort of thing, please don't
> >> add it back.
> 
> Why?

We converted the entire tree over the print_mac(), and since
the MAC_FMT stuff was therefore no longer used we could
remove it.

Some references slipped back in somehow, and thus MAC_FMT
did too.

There is no reason to keep around a global interface for
_one_ user when that user can use the recommended interface
just as equally as the rest of the tree which we converted.

This is a pr_debug() statement we're talking about here.
:-)

^ permalink raw reply

* Re: [PATCH] net/8021q/vlan_dev.c - Use print_mac
From: Patrick McHardy @ 2008-02-18 21:17 UTC (permalink / raw)
  To: David Miller
  Cc: joe-6d6DIl74uiNBDgjK7y7TUQ, bruno-L9ZBdB2wSWtl57MIdRCFDg,
	netdev-u79uwXL29TY76Z2rM5mHXA, jgarzik-e+AXbWqSrlAAvxtiuMwx3w,
	linux-wireless-u79uwXL29TY76Z2rM5mHXA,
	linville-2XuSBdqkA4R54TAoqtyWWQ
In-Reply-To: <20080218.125525.192686382.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>

David Miller wrote:
> From: Patrick McHardy <kaber-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
> Date: Mon, 18 Feb 2008 16:19:40 +0100
>
>   
>> Joe Perches wrote:
>>     
>>>
>>>> We specifically removed this sort of thing, please don't
>>>> add it back.
>>>>         
>> Why?
>>     
>
> We converted the entire tree over the print_mac(), and since
> the MAC_FMT stuff was therefore no longer used we could
> remove it.
>
> Some references slipped back in somehow, and thus MAC_FMT
> did too.
>
> There is no reason to keep around a global interface for
> _one_ user when that user can use the recommended interface
> just as equally as the rest of the tree which we converted.
>
> This is a pr_debug() statement we're talking about here.
> :-)
>   

The way pr_debug is implemented it still results in two function
calls per packet since the compiler doesn't know that it doesn't
have visible side-effects besides modifying the (unused) buffer.
I confirmed this using codiff.

^ permalink raw reply

* [PATCH] net: fix kernel-doc warnings in header files
From: Randy Dunlap @ 2008-02-18 21:26 UTC (permalink / raw)
  To: netdev; +Cc: davem

From: Randy Dunlap <randy.dunlap@oracle.com>

Add missing structure kernel-doc descriptions to sock.h & skbuff.h
to fix kernel-doc warnings.

(I think that Stephen H. sent a similar patch, but I can't find it.
I just want to kill the warnings, with either patch.)

Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
---
 include/linux/skbuff.h |    2 ++
 include/net/sock.h     |    1 +
 2 files changed, 3 insertions(+)

--- linux-2625-rc1g4-kdoc.orig/include/linux/skbuff.h
+++ linux-2625-rc1g4-kdoc/include/linux/skbuff.h
@@ -232,6 +232,8 @@ typedef unsigned char *sk_buff_data_t;
  *	@mark: Generic packet mark
  *	@nfct: Associated connection, if any
  *	@ipvs_property: skbuff is owned by ipvs
+ *	@peeked: this packet has been seen already, so stats have been
+ *		done for it, don't do them again
  *	@nf_trace: netfilter packet trace flag
  *	@nfctinfo: Relationship of this skb to the connection
  *	@nfct_reasm: netfilter conntrack re-assembly pointer
--- linux-2625-rc1g4-kdoc.orig/include/net/sock.h
+++ linux-2625-rc1g4-kdoc/include/net/sock.h
@@ -180,6 +180,7 @@ struct sock_common {
   *	@sk_sndmsg_off: cached offset for sendmsg
   *	@sk_send_head: front of stuff to transmit
   *	@sk_security: used by security modules
+  *	@sk_mark: generic packet mark
   *	@sk_write_pending: a write to stream socket waits to start
   *	@sk_state_change: callback to indicate change in the state of the sock
   *	@sk_data_ready: callback to indicate there is data to be processed

^ permalink raw reply

* Re: [PATCH] net/8021q/vlan_dev.c - Use print_mac
From: Patrick McHardy @ 2008-02-18 21:31 UTC (permalink / raw)
  To: Joe Perches
  Cc: David Miller, bruno-L9ZBdB2wSWtl57MIdRCFDg,
	netdev-u79uwXL29TY76Z2rM5mHXA, jgarzik-e+AXbWqSrlAAvxtiuMwx3w,
	linux-wireless-u79uwXL29TY76Z2rM5mHXA,
	linville-2XuSBdqkA4R54TAoqtyWWQ
In-Reply-To: <1203359462.7181.80.camel@localhost>

Joe Perches wrote:
> On Mon, 2008-02-18 at 16:19 +0100, Patrick McHardy wrote:
>   
>>> @@ -404,11 +405,8 @@ static int vlan_dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev)
>>>  
>>>  	pr_debug("%s: about to send skb: %p to dev: %s\n",
>>>  		__FUNCTION__, skb, skb->dev->name);
>>> -	pr_debug("  " MAC_FMT " " MAC_FMT " %4hx %4hx %4hx\n",
>>> -		 veth->h_dest[0], veth->h_dest[1], veth->h_dest[2],
>>> -		 veth->h_dest[3], veth->h_dest[4], veth->h_dest[5],
>>> -		 veth->h_source[0], veth->h_source[1], veth->h_source[2],
>>> -		 veth->h_source[3], veth->h_source[4], veth->h_source[5],
>>> +	pr_debug("  %s %s %4hx %4hx %4hx\n",
>>> +		 print_mac(mac, veth->h_dest), print_mac(mac2, veth->h_source),
>>>       
>> This results in print_mac getting called twice per packet even without
>> debugging. Whats the problem with MAC_FMT?
>>     
>
> It's just a consistency thing.
> It identifies code where MAC addresses are used.
>   
> an allyesconfig is a bit smaller (~.1%).
> pr_debug is a noop when not debugging, print_mac is optimized away.
>   

No its not, which I also stated in the commit message that restored
it.

0x0000000060244313 <vlan_dev_hard_start_xmit+433>:      callq  
0x60161dbd <print_mac>
0x0000000060244318 <vlan_dev_hard_start_xmit+438>:      lea    
-0x50(%rbp),%rdi
0x000000006024431c <vlan_dev_hard_start_xmit+442>:      mov    %r15,%rsi
0x000000006024431f <vlan_dev_hard_start_xmit+445>:      callq  
0x60161dbd <print_mac>

^ permalink raw reply

* Re: ipv6 debugging
From: Ferenc Wagner @ 2008-02-18 21:46 UTC (permalink / raw)
  To: Jorge Boncompte [DTI2]; +Cc: netdev
In-Reply-To: <3740A925537C481CAA40DE393FA0D00E@intranet.dti2.net>

"Jorge Boncompte [DTI2]" <jorge@dti2.net> writes:

> Ferenc Wagner <wferi@niif.hu> writes:
>
>> I'm kindly asking for some debugging tips with the following problem:
>> a machine is running Linux 2.6.24.2, several 802.1q VLAN-s over
>> active/backup bonding over two physical interfaces.  Everything is
>> allright, except for after a reboot, there's no IPv6, while IPv4
>> works.  The router's ARP(6) table is empty, the machine doesn't answer
>> ping6.  However, if I start tcpdump -i bond0 ip6, everything is
>> allright again.  There are some indications that after some period
>> without IPv6 traffic, the same can happen again.  Are there known
>> issues which can exhibit themselves like this?  Other very similar
>> setups don't show this erratic behaviour.
>>
>> I know that the above doesn't give a fully detailed picture, but
>> thought that I'd better ask before taking the setup into pieces.
>
> This sounds to me like the same problem that I was having with
> OSPF, I think ARP(6) uses multicast ethernet address too. Can you try
> if the patch below, that I sent Patrick McHardy some days ago, fixes
> your problem?

Hi Jorge,

Thank you very much!  Your patch indeed fixes my problem.  I hope the
fix will make it into a stable release soon!
-- 
Regards,
Feri.

^ permalink raw reply

* [PATCH][IPROUTE] tc filters usage fixes
From: Jarek Poplawski @ 2008-02-18 22:10 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: netdev

A few usage description fixes of tc filters for some minimal
consistency (FILTER_KIND because of QDISC_KIND).


Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>

---

 tc/f_basic.c   |    4 ++--
 tc/f_rsvp.c    |    2 +-
 tc/f_u32.c     |    2 +-
 tc/tc_filter.c |    6 +++---
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tc/f_basic.c b/tc/f_basic.c
index ad41633..cf2650b 100644
--- a/tc/f_basic.c
+++ b/tc/f_basic.c
@@ -30,8 +30,8 @@ static void explain(void)
 	fprintf(stderr, "Usage: ... basic [ match EMATCH_TREE ] [ police POLICE_SPEC ]\n");
 	fprintf(stderr, "                 [ action ACTION_SPEC ] [ classid CLASSID ]\n");
 	fprintf(stderr, "\n");
-	fprintf(stderr, "Where: SELECTOR := SAMPLE SAMPLE ...\n");
-	fprintf(stderr, "       FILTERID := X:Y:Z\n");
+	fprintf(stderr, "Where:\n");
+	fprintf(stderr, "       CLASSID := X:Y:Z\n");
 	fprintf(stderr, "\nNOTE: CLASSID is parsed as hexadecimal input.\n");
 }
 
diff --git a/tc/f_rsvp.c b/tc/f_rsvp.c
index 7e1e6d9..8f92e8f 100644
--- a/tc/f_rsvp.c
+++ b/tc/f_rsvp.c
@@ -33,7 +33,7 @@ static void explain(void)
 	fprintf(stderr, "Where: GPI := { flowlabel NUMBER | spi/ah SPI | spi/esp SPI |\n");
 	fprintf(stderr, "                u{8|16|32} NUMBER mask MASK at OFFSET}\n");
 	fprintf(stderr, "       POLICE_SPEC := ... look at TBF\n");
-	fprintf(stderr, "       FILTERID := X:Y\n");
+	fprintf(stderr, "       CLASSID := X:Y\n");
 	fprintf(stderr, "\nNOTE: CLASSID is parsed as hexadecimal input.\n");
 }
 
diff --git a/tc/f_u32.c b/tc/f_u32.c
index 9bc4bb5..957b1b1 100644
--- a/tc/f_u32.c
+++ b/tc/f_u32.c
@@ -38,7 +38,7 @@ static void explain(void)
 	fprintf(stderr, "Where: SELECTOR := SAMPLE SAMPLE ...\n");
 	fprintf(stderr, "       SAMPLE := { ip | ip6 | udp | tcp | icmp |"
 		" u{32|16|8} | mark } SAMPLE_ARGS [divisor DIVISOR]\n");
-	fprintf(stderr, "       FILTERID := X:Y:Z\n");
+	fprintf(stderr, "       CLASSID := X:Y:Z\n");
 	fprintf(stderr, "\nNOTE: CLASSID is parsed at hexadecimal input.\n");
 }
 
diff --git a/tc/tc_filter.c b/tc/tc_filter.c
index d70c656..eb74f89 100644
--- a/tc/tc_filter.c
+++ b/tc/tc_filter.c
@@ -33,12 +33,12 @@ static void usage(void)
 	fprintf(stderr, "Usage: tc filter [ add | del | change | replace | show ] dev STRING\n");
 	fprintf(stderr, "       [ pref PRIO ] protocol PROTO\n");
 	fprintf(stderr, "       [ estimator INTERVAL TIME_CONSTANT ]\n");
-	fprintf(stderr, "       [ root | classid CLASSID ] [ handle FILTERID ]\n");
-	fprintf(stderr, "       [ [ FILTER_TYPE ] [ help | OPTIONS ] ]\n");
+	fprintf(stderr, "       [ root | parent CLASSID ] [ handle FILTERID ]\n");
+	fprintf(stderr, "       [ [ FILTER_KIND ] [ help | OPTIONS ] ]\n");
 	fprintf(stderr, "\n");
 	fprintf(stderr, "       tc filter show [ dev STRING ] [ root | parent CLASSID ]\n");
 	fprintf(stderr, "Where:\n");
-	fprintf(stderr, "FILTER_TYPE := { rsvp | u32 | fw | route | etc. }\n");
+	fprintf(stderr, "FILTER_KIND := { rsvp | u32 | fw | route | etc. }\n");
 	fprintf(stderr, "FILTERID := ... format depends on classifier, see there\n");
 	fprintf(stderr, "OPTIONS := ... try tc filter add <desired FILTER_KIND> help\n");
 	return;

^ permalink raw reply related

* Re: [PATCH][PPPOL2TP]: Fix SMP oops in pppol2tp driver
From: James Chapman @ 2008-02-18 22:09 UTC (permalink / raw)
  To: Jarek Poplawski, David Miller; +Cc: netdev
In-Reply-To: <20080214130016.GA2583@ff.dom.local>

Jarek Poplawski wrote:
> Hi,
> 
> It seems, this nice report is still uncomplete: could you check if
> there could have been something more yet?

Unfortunately the ISP's syslog stops. But I've been able to borrow two Quad Xeon boxes and have reproduced the problem.

Here's a new version of the patch. The patch avoids disabling irqs and fixes the sk_dst_get() usage that DaveM mentioned. But even with this patch, lockdep still complains if hundreds of ppp sessions are inserted into a tunnel as rapidly as possible (lockdep trace is below). I can stop these errors by wrapping the call to ppp_input() in pppol2tp_recv_dequeue_skb() with local_irq_save/restore. What is a better fix?

btw, I'm not sending this from my usual mail client so the patch will probably be mangled; I'm sending it only for feedback / discussion at this stage.

Index: linux-2.6.24.2/drivers/net/pppol2tp.c
===================================================================
--- linux-2.6.24.2.orig/drivers/net/pppol2tp.c
+++ linux-2.6.24.2/drivers/net/pppol2tp.c
@@ -302,14 +302,14 @@ pppol2tp_session_find(struct pppol2tp_tu
 	struct pppol2tp_session *session;
 	struct hlist_node *walk;
 
-	read_lock(&tunnel->hlist_lock);
+	read_lock_bh(&tunnel->hlist_lock);
 	hlist_for_each_entry(session, walk, session_list, hlist) {
 		if (session->tunnel_addr.s_session == session_id) {
-			read_unlock(&tunnel->hlist_lock);
+			read_unlock_bh(&tunnel->hlist_lock);
 			return session;
 		}
 	}
-	read_unlock(&tunnel->hlist_lock);
+	read_unlock_bh(&tunnel->hlist_lock);
 
 	return NULL;
 }
@@ -320,14 +320,14 @@ static struct pppol2tp_tunnel *pppol2tp_
 {
 	struct pppol2tp_tunnel *tunnel = NULL;
 
-	read_lock(&pppol2tp_tunnel_list_lock);
+	read_lock_bh(&pppol2tp_tunnel_list_lock);
 	list_for_each_entry(tunnel, &pppol2tp_tunnel_list, list) {
 		if (tunnel->stats.tunnel_id == tunnel_id) {
-			read_unlock(&pppol2tp_tunnel_list_lock);
+			read_unlock_bh(&pppol2tp_tunnel_list_lock);
 			return tunnel;
 		}
 	}
-	read_unlock(&pppol2tp_tunnel_list_lock);
+	read_unlock_bh(&pppol2tp_tunnel_list_lock);
 
 	return NULL;
 }
@@ -344,7 +344,7 @@ static void pppol2tp_recv_queue_skb(stru
 	struct sk_buff *skbp;
 	u16 ns = PPPOL2TP_SKB_CB(skb)->ns;
 
-	spin_lock(&session->reorder_q.lock);
+	spin_lock_bh(&session->reorder_q.lock);
 	skb_queue_walk(&session->reorder_q, skbp) {
 		if (PPPOL2TP_SKB_CB(skbp)->ns > ns) {
 			__skb_insert(skb, skbp->prev, skbp, &session->reorder_q);
@@ -360,7 +360,7 @@ static void pppol2tp_recv_queue_skb(stru
 	__skb_queue_tail(&session->reorder_q, skb);
 
 out:
-	spin_unlock(&session->reorder_q.lock);
+	spin_unlock_bh(&session->reorder_q.lock);
 }
 
 /* Dequeue a single skb.
@@ -442,7 +442,7 @@ static void pppol2tp_recv_dequeue(struct
 	 * expect to send up next, dequeue it and any other
 	 * in-sequence packets behind it.
 	 */
-	spin_lock(&session->reorder_q.lock);
+	spin_lock_bh(&session->reorder_q.lock);
 	skb_queue_walk_safe(&session->reorder_q, skb, tmp) {
 		if (time_after(jiffies, PPPOL2TP_SKB_CB(skb)->expires)) {
 			session->stats.rx_seq_discards++;
@@ -469,13 +469,13 @@ static void pppol2tp_recv_dequeue(struct
 				goto out;
 			}
 		}
-		spin_unlock(&session->reorder_q.lock);
+		spin_unlock_bh(&session->reorder_q.lock);
 		pppol2tp_recv_dequeue_skb(session, skb);
-		spin_lock(&session->reorder_q.lock);
+		spin_lock_bh(&session->reorder_q.lock);
 	}
 
 out:
-	spin_unlock(&session->reorder_q.lock);
+	spin_unlock_bh(&session->reorder_q.lock);
 }
 
 /* Internal receive frame. Do the real work of receiving an L2TP data frame
@@ -964,6 +964,7 @@ static int pppol2tp_xmit(struct ppp_chan
 	static const u8 ppph[2] = { 0xff, 0x03 };
 	struct sock *sk = (struct sock *) chan->private;
 	struct sock *sk_tun;
+	struct rtable *rt;
 	int hdr_len;
 	struct pppol2tp_session *session;
 	struct pppol2tp_tunnel *tunnel;
@@ -1057,8 +1058,32 @@ static int pppol2tp_xmit(struct ppp_chan
 	nf_reset(skb);
 
 	/* Get routing info from the tunnel socket */
-	dst_release(skb->dst);
-	skb->dst = sk_dst_get(sk_tun);
+	rt = (struct rtable *)__sk_dst_check(sk_tun, 0);
+	if (rt == NULL) {
+		struct flowi fl = { .oif = sk_tun->sk_bound_dev_if,
+				    .nl_u = { .ip4_u =
+					      { .daddr = inet->daddr,
+						.saddr = inet->saddr,
+						.tos = inet->tos } },
+				    .proto = sk_tun->sk_protocol,
+				    .uli_u = { .ports =
+					       { .sport = inet->sport,
+						 .dport = inet->dport } } };
+		security_sk_classify_flow(sk_tun, &fl);
+		rc = ip_route_output_flow(&rt, &fl, sk_tun, 1);
+		if (rc) {
+			if (rc == -ENETUNREACH)
+				IP_INC_STATS_BH(IPSTATS_MIB_OUTNOROUTES);
+			goto abort;
+		}
+
+		sk_dst_set(sk_tun, dst_clone(&rt->u.dst));
+	}
+
+	write_lock_bh(&sk_tun->sk_dst_lock);
+	skb->dst = __sk_dst_get(sk_tun);
+	dst_hold(skb->dst);
+	write_unlock_bh(&sk_tun->sk_dst_lock);
 	skb_orphan(skb);
 	skb->sk = sk_tun;
 
@@ -1106,7 +1131,7 @@ static void pppol2tp_tunnel_closeall(str
 	PRINTK(tunnel->debug, PPPOL2TP_MSG_CONTROL, KERN_INFO,
 	       "%s: closing all sessions...\n", tunnel->name);
 
-	write_lock(&tunnel->hlist_lock);
+	write_lock_bh(&tunnel->hlist_lock);
 	for (hash = 0; hash < PPPOL2TP_HASH_SIZE; hash++) {
 again:
 		hlist_for_each_safe(walk, tmp, &tunnel->session_hlist[hash]) {
@@ -1126,7 +1151,7 @@ again:
 			 * disappear as we're jumping between locks.
 			 */
 			sock_hold(sk);
-			write_unlock(&tunnel->hlist_lock);
+			write_unlock_bh(&tunnel->hlist_lock);
 			lock_sock(sk);
 
 			if (sk->sk_state & (PPPOX_CONNECTED | PPPOX_BOUND)) {
@@ -1148,11 +1173,11 @@ again:
 			 * list so we are guaranteed to make forward
 			 * progress.
 			 */
-			write_lock(&tunnel->hlist_lock);
+			write_lock_bh(&tunnel->hlist_lock);
 			goto again;
 		}
 	}
-	write_unlock(&tunnel->hlist_lock);
+	write_unlock_bh(&tunnel->hlist_lock);
 }
 
 /* Really kill the tunnel.
@@ -1161,9 +1186,9 @@ again:
 static void pppol2tp_tunnel_free(struct pppol2tp_tunnel *tunnel)
 {
 	/* Remove from socket list */
-	write_lock(&pppol2tp_tunnel_list_lock);
+	write_lock_bh(&pppol2tp_tunnel_list_lock);
 	list_del_init(&tunnel->list);
-	write_unlock(&pppol2tp_tunnel_list_lock);
+	write_unlock_bh(&pppol2tp_tunnel_list_lock);
 
 	atomic_dec(&pppol2tp_tunnel_count);
 	kfree(tunnel);
@@ -1239,9 +1264,9 @@ static void pppol2tp_session_destruct(st
 				/* Delete the session socket from the
 				 * hash
 				 */
-				write_lock(&tunnel->hlist_lock);
+				write_lock_bh(&tunnel->hlist_lock);
 				hlist_del_init(&session->hlist);
-				write_unlock(&tunnel->hlist_lock);
+				write_unlock_bh(&tunnel->hlist_lock);
 
 				atomic_dec(&pppol2tp_session_count);
 			}
@@ -1386,9 +1411,9 @@ static struct sock *pppol2tp_prepare_tun
 
 	/* Add tunnel to our list */
 	INIT_LIST_HEAD(&tunnel->list);
-	write_lock(&pppol2tp_tunnel_list_lock);
+	write_lock_bh(&pppol2tp_tunnel_list_lock);
 	list_add(&tunnel->list, &pppol2tp_tunnel_list);
-	write_unlock(&pppol2tp_tunnel_list_lock);
+	write_unlock_bh(&pppol2tp_tunnel_list_lock);
 	atomic_inc(&pppol2tp_tunnel_count);
 
 	/* Bump the reference count. The tunnel context is deleted
@@ -1546,14 +1571,15 @@ static int pppol2tp_connect(struct socke
 	session->mtu = session->mru = 1500 - PPPOL2TP_HEADER_OVERHEAD;
 
 	/* If PMTU discovery was enabled, use the MTU that was discovered */
-	dst = sk_dst_get(sk);
+	read_lock_bh(&tunnel_sock->sk_dst_lock);
+	dst = __sk_dst_get(sk);
 	if (dst != NULL) {
-		u32 pmtu = dst_mtu(__sk_dst_get(sk));
+		u32 pmtu = dst_mtu(dst);
 		if (pmtu != 0)
 			session->mtu = session->mru = pmtu -
 				PPPOL2TP_HEADER_OVERHEAD;
-		dst_release(dst);
 	}
+	read_unlock_bh(&tunnel_sock->sk_dst_lock);
 
 	/* Special case: if source & dest session_id == 0x0000, this socket is
 	 * being created to manage the tunnel. Don't add the session to the
@@ -1593,11 +1619,11 @@ static int pppol2tp_connect(struct socke
 	sk->sk_user_data = session;
 
 	/* Add session to the tunnel's hash list */
-	write_lock(&tunnel->hlist_lock);
+	write_lock_bh(&tunnel->hlist_lock);
 	hlist_add_head(&session->hlist,
 		       pppol2tp_session_id_hash(tunnel,
 						session->tunnel_addr.s_session));
-	write_unlock(&tunnel->hlist_lock);
+	write_unlock_bh(&tunnel->hlist_lock);
 
 	atomic_inc(&pppol2tp_session_count);
 
@@ -2199,7 +2225,7 @@ static struct pppol2tp_session *next_ses
 	int next = 0;
 	int i;
 
-	read_lock(&tunnel->hlist_lock);
+	read_lock_bh(&tunnel->hlist_lock);
 	for (i = 0; i < PPPOL2TP_HASH_SIZE; i++) {
 		hlist_for_each_entry(session, walk, &tunnel->session_hlist[i], hlist) {
 			if (curr == NULL) {
@@ -2217,7 +2243,7 @@ static struct pppol2tp_session *next_ses
 		}
 	}
 out:
-	read_unlock(&tunnel->hlist_lock);
+	read_unlock_bh(&tunnel->hlist_lock);
 	if (!found)
 		session = NULL;
 
@@ -2228,13 +2254,13 @@ static struct pppol2tp_tunnel *next_tunn
 {
 	struct pppol2tp_tunnel *tunnel = NULL;
 
-	read_lock(&pppol2tp_tunnel_list_lock);
+	read_lock_bh(&pppol2tp_tunnel_list_lock);
 	if (list_is_last(&curr->list, &pppol2tp_tunnel_list)) {
 		goto out;
 	}
 	tunnel = list_entry(curr->list.next, struct pppol2tp_tunnel, list);
 out:
-	read_unlock(&pppol2tp_tunnel_list_lock);
+	read_unlock_bh(&pppol2tp_tunnel_list_lock);
 
 	return tunnel;
 }

......................................................
Here's the lockdep trace (long).

Feb 18 20:56:18 localhost kernel: 
Feb 18 20:56:18 localhost kernel: ======================================================
Feb 18 20:56:18 localhost kernel: [ INFO: soft-safe -> soft-unsafe lock order detected ]
Feb 18 20:56:18 localhost kernel: 2.6.24.2 #1
Feb 18 20:56:18 localhost kernel: ------------------------------------------------------
Feb 18 20:56:18 localhost kernel: pppd/3241 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
Feb 18 20:56:18 localhost kernel:  (&pch->downl){-...}, at: [<f8c566db>] ppp_push+0x63/0x50d [ppp_generic]
Feb 18 20:56:18 localhost kernel: 
Feb 18 20:56:18 localhost kernel: and this task is already holding:
Feb 18 20:56:18 localhost kernel:  (&ppp->wlock){-...}, at: [<f8c56f10>] ppp_xmit_process+0x15/0x5a1 [ppp_generic]
Feb 18 20:56:18 localhost kernel: which would create a new lock dependency:
Feb 18 20:56:18 localhost kernel:  (&ppp->wlock){-...} -> (&pch->downl){-...}
Feb 18 20:56:18 localhost kernel: 
Feb 18 20:56:18 localhost kernel: but this new dependency connects a soft-irq-safe lock:
Feb 18 20:56:18 localhost kernel:  (&pch->upl){-.-+}
Feb 18 20:56:18 localhost kernel: ... which became soft-irq-safe at:
Feb 18 20:56:18 localhost kernel:   [<c0448013>] check_usage_backwards+0x19/0x41
Feb 18 20:56:18 localhost kernel:   [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:18 localhost kernel:   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:18 localhost kernel:   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:18 localhost kernel:   [<f8c5940d>] ppp_input+0x45/0xef [ppp_generic]
Feb 18 20:56:18 localhost kernel:   [<c061b373>] _read_lock_bh+0x2e/0x39
Feb 18 20:56:18 localhost kernel:   [<f8c5940d>] ppp_input+0x45/0xef [ppp_generic]
Feb 18 20:56:18 localhost kernel:   [<f8c5940d>] ppp_input+0x45/0xef [ppp_generic]
Feb 18 20:56:18 localhost kernel:   [<f8c60263>] pppol2tp_recv_core+0x75c/0x82d [pppol2tp]
Feb 18 20:56:18 localhost kernel:   [<f8c6037a>] pppol2tp_udp_encap_recv+0x46/0x65 [pppol2tp]
Feb 18 20:56:18 localhost kernel:   [<c061b171>] _read_unlock+0x14/0x1c
Feb 18 20:56:18 localhost kernel:   [<c05ef9df>] udp_queue_rcv_skb+0xba/0x259
Feb 18 20:56:18 localhost kernel:   [<c05efffe>] __udp4_lib_rcv+0x480/0x758
Feb 18 20:56:18 localhost kernel:   [<c05d2c4b>] ip_local_deliver_finish+0x13f/0x1f8
Feb 18 20:56:18 localhost kernel:   [<c05d2b3a>] ip_local_deliver_finish+0x2e/0x1f8
Feb 18 20:56:18 localhost kernel:   [<c05d2ad2>] ip_rcv_finish+0x2fe/0x338
Feb 18 20:56:18 localhost kernel:   [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:18 localhost kernel:   [<c05d2dc6>] ip_rcv+0x0/0x237
Feb 18 20:56:18 localhost kernel:   [<c05b5681>] netif_receive_skb+0x373/0x3d4
Feb 18 20:56:18 localhost kernel:   [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:18 localhost kernel:   [<f89877c7>] e1000_clean_rx_irq+0x374/0x44a [e1000]
Feb 18 20:56:18 localhost kernel:   [<f8987453>] e1000_clean_rx_irq+0x0/0x44a [e1000]
Feb 18 20:56:18 localhost kernel:   [<f8984f13>] e1000_clean+0x63/0x203 [e1000]
Feb 18 20:56:19 localhost kernel:   [<c05b77f0>] net_rx_action+0xbc/0x1b3
Feb 18 20:56:19 localhost kernel:   [<c05b7782>] net_rx_action+0x4e/0x1b3
Feb 18 20:56:19 localhost kernel:   [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:19 localhost kernel:   [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:19 localhost kernel:   [<ffffffff>] 0xffffffff
Feb 18 20:56:19 localhost kernel: 
Feb 18 20:56:19 localhost kernel: to a soft-irq-unsafe lock:
Feb 18 20:56:19 localhost kernel:  (&sk->sk_dst_lock){----}
Feb 18 20:56:19 localhost kernel: ... which became soft-irq-unsafe at:
Feb 18 20:56:19 localhost kernel: ...  [<c0449046>] __lock_acquire+0x48b/0xbf1
Feb 18 20:56:19 localhost kernel:   [<c0448594>] mark_held_locks+0x39/0x53
Feb 18 20:56:19 localhost kernel:   [<c043083a>] local_bh_enable+0x10e/0x115
Feb 18 20:56:19 localhost kernel:   [<c05da996>] inet_csk_get_port+0xc1/0x1cb
Feb 18 20:56:19 localhost kernel:   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:19 localhost kernel:   [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:56:19 localhost kernel:   [<c061b2c7>] _write_lock+0x29/0x34
Feb 18 20:56:19 localhost kernel:   [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:56:19 localhost kernel:   [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:56:19 localhost kernel:   [<c05f66da>] inet_listen+0x3b/0x5e
Feb 18 20:56:19 localhost kernel:   [<c05ab68f>] sys_listen+0x43/0x5f
Feb 18 20:56:19 localhost kernel:   [<c05acba8>] sys_socketcall+0xbd/0x261
Feb 18 20:56:19 localhost kernel:   [<c0404ead>] sysenter_past_esp+0x9a/0xa5
Feb 18 20:56:19 localhost kernel:   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:19 localhost kernel:   [<c0404e72>] sysenter_past_esp+0x5f/0xa5
Feb 18 20:56:19 localhost kernel:   [<ffffffff>] 0xffffffff
Feb 18 20:56:19 localhost kernel: 
Feb 18 20:56:19 localhost kernel: other info that might help us debug this:
Feb 18 20:56:19 localhost kernel: 
Feb 18 20:56:19 localhost kernel: 1 lock held by pppd/3241:
Feb 18 20:56:19 localhost kernel:  #0:  (&ppp->wlock){-...}, at: [<f8c56f10>] ppp_xmit_process+0x15/0x5a1 [ppp_generic]
Feb 18 20:56:19 localhost kernel: 
Feb 18 20:56:19 localhost kernel: the soft-irq-safe lock's dependencies:
Feb 18 20:56:19 localhost kernel: -> (&pch->upl){-.-+} ops: 13 {
Feb 18 20:56:19 localhost kernel:    initial-use  at:
Feb 18 20:56:19 localhost kernel:                         [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:56:19 localhost kernel:                         [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:19 localhost kernel:                         [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:19 localhost kernel:                         [<f8c57b7c>] ppp_ioctl+0x4df/0xc06 [ppp_generic]
Feb 18 20:56:19 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:19 localhost kernel:                         [<f8c57ba1>] ppp_ioctl+0x504/0xc06 [ppp_generic]
Feb 18 20:56:19 localhost kernel:                         [<c061b300>] _write_lock_bh+0x2e/0x39
Feb 18 20:56:20 localhost kernel:                         [<f8c57ba1>] ppp_ioctl+0x504/0xc06 [ppp_generic]
Feb 18 20:56:20 localhost kernel:                         [<f8c57ba1>] ppp_ioctl+0x504/0xc06 [ppp_generic]
Feb 18 20:56:20 localhost kernel:                         [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:20 localhost kernel:                         [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:20 localhost kernel:                         [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:20 localhost kernel:                         [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:20 localhost kernel:                         [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:56:20 localhost kernel:                         [<c061ae17>] __down_failed+0x7/0xc
Feb 18 20:56:20 localhost kernel:                         [<c0488854>] do_ioctl+0x4c/0x62
Feb 18 20:56:20 localhost kernel:                         [<c0488aa1>] vfs_ioctl+0x237/0x249
Feb 18 20:56:20 localhost kernel:                         [<c0488af8>] sys_ioctl+0x45/0x5d
Feb 18 20:56:20 localhost kernel:                         [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:56:20 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:56:20 localhost kernel:    hardirq-on-W at:
Feb 18 20:56:20 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:20 localhost kernel:                         [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:56:20 localhost kernel:                         [<c0449027>] __lock_acquire+0x46c/0xbf1
Feb 18 20:56:20 localhost kernel:                         [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:20 localhost kernel:                         [<f8c57b7c>] ppp_ioctl+0x4df/0xc06 [ppp_generic]
Feb 18 20:56:20 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:20 localhost kernel:                         [<f8c57ba1>] ppp_ioctl+0x504/0xc06 [ppp_generic]
Feb 18 20:56:20 localhost kernel:                         [<c061b300>] _write_lock_bh+0x2e/0x39
Feb 18 20:56:20 localhost kernel:                         [<f8c57ba1>] ppp_ioctl+0x504/0xc06 [ppp_generic]
Feb 18 20:56:20 localhost kernel:                         [<f8c57ba1>] ppp_ioctl+0x504/0xc06 [ppp_generic]
Feb 18 20:56:20 localhost kernel:                         [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:20 localhost kernel:                         [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:20 localhost kernel:                         [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:20 localhost kernel:                         [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:20 localhost kernel:                         [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:56:20 localhost kernel:                         [<c061ae17>] __down_failed+0x7/0xc
Feb 18 20:56:20 localhost kernel:                         [<c0488854>] do_ioctl+0x4c/0x62
Feb 18 20:56:20 localhost kernel:                         [<c0488aa1>] vfs_ioctl+0x237/0x249
Feb 18 20:56:20 localhost kernel:                         [<c0488af8>] sys_ioctl+0x45/0x5d
Feb 18 20:56:20 localhost kernel:                         [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:56:20 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:56:20 localhost kernel:    in-softirq-R at:
Feb 18 20:56:20 localhost kernel:                         [<c0448013>] check_usage_backwards+0x19/0x41
Feb 18 20:56:20 localhost kernel:                         [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:20 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:20 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:20 localhost kernel:                         [<f8c5940d>] ppp_input+0x45/0xef [ppp_generic]
Feb 18 20:56:21 localhost kernel:                         [<c061b373>] _read_lock_bh+0x2e/0x39
Feb 18 20:56:21 localhost kernel:                         [<f8c5940d>] ppp_input+0x45/0xef [ppp_generic]
Feb 18 20:56:21 localhost kernel:                         [<f8c5940d>] ppp_input+0x45/0xef [ppp_generic]
Feb 18 20:56:21 localhost kernel:                         [<f8c60263>] pppol2tp_recv_core+0x75c/0x82d [pppol2tp]
Feb 18 20:56:21 localhost kernel:                         [<f8c6037a>] pppol2tp_udp_encap_recv+0x46/0x65 [pppol2tp]
Feb 18 20:56:21 localhost kernel:                         [<c061b171>] _read_unlock+0x14/0x1c
Feb 18 20:56:21 localhost kernel:                         [<c05ef9df>] udp_queue_rcv_skb+0xba/0x259
Feb 18 20:56:21 localhost kernel:                         [<c05efffe>] __udp4_lib_rcv+0x480/0x758
Feb 18 20:56:21 localhost kernel:                         [<c05d2c4b>] ip_local_deliver_finish+0x13f/0x1f8
Feb 18 20:56:21 localhost kernel:                         [<c05d2b3a>] ip_local_deliver_finish+0x2e/0x1f8
Feb 18 20:56:21 localhost kernel:                         [<c05d2ad2>] ip_rcv_finish+0x2fe/0x338
Feb 18 20:56:21 localhost kernel:                         [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:21 localhost kernel:                         [<c05d2dc6>] ip_rcv+0x0/0x237
Feb 18 20:56:21 localhost kernel:                         [<c05b5681>] netif_receive_skb+0x373/0x3d4
Feb 18 20:56:21 localhost kernel:                         [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:21 localhost kernel:                         [<f89877c7>] e1000_clean_rx_irq+0x374/0x44a [e1000]
Feb 18 20:56:21 localhost kernel:                         [<f8987453>] e1000_clean_rx_irq+0x0/0x44a [e1000]
Feb 18 20:56:21 localhost kernel:                         [<f8984f13>] e1000_clean+0x63/0x203 [e1000]
Feb 18 20:56:21 localhost kernel:                         [<c05b77f0>] net_rx_action+0xbc/0x1b3
Feb 18 20:56:21 localhost kernel:                         [<c05b7782>] net_rx_action+0x4e/0x1b3
Feb 18 20:56:21 localhost kernel:                         [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:21 localhost kernel:                         [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:21 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:56:21 localhost kernel:    hardirq-on-R at:
Feb 18 20:56:21 localhost kernel:                         [<c0448771>] trace_hardirqs_on+0x10c/0x14c
Feb 18 20:56:21 localhost kernel:                         [<c0449001>] __lock_acquire+0x446/0xbf1
Feb 18 20:56:21 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:21 localhost kernel:                         [<f8c5834b>] ppp_channel_push+0x71/0x90 [ppp_generic]
Feb 18 20:56:21 localhost kernel:                         [<c061b373>] _read_lock_bh+0x2e/0x39
Feb 18 20:56:21 localhost kernel:                         [<f8c5834b>] ppp_channel_push+0x71/0x90 [ppp_generic]
Feb 18 20:56:21 localhost kernel:                         [<f8c5834b>] ppp_channel_push+0x71/0x90 [ppp_generic]
Feb 18 20:56:21 localhost kernel:                         [<f8c593bd>] ppp_write+0xd0/0xdb [ppp_generic]
Feb 18 20:56:21 localhost kernel:                         [<f8c592ed>] ppp_write+0x0/0xdb [ppp_generic]
Feb 18 20:56:21 localhost kernel:                         [<c047e6d2>] vfs_write+0xa1/0x14d
Feb 18 20:56:21 localhost kernel:                         [<c047ecfd>] sys_write+0x41/0x67
Feb 18 20:56:21 localhost kernel:                         [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:56:21 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:56:21 localhost kernel:  }
Feb 18 20:56:21 localhost kernel:  ... key      at: [<f8c5dd20>] __key.29741+0x0/0xffffb822 [ppp_generic]
Feb 18 20:56:21 localhost kernel:  -> (&ppp->wlock){-...} ops: 11 {
Feb 18 20:56:22 localhost kernel:     initial-use  at:
Feb 18 20:56:22 localhost kernel:                           [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:22 localhost kernel:                           [<f8c57ba1>] ppp_ioctl+0x504/0xc06 [ppp_generic]
Feb 18 20:56:22 localhost kernel:                           [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:22 localhost kernel:                           [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:56:22 localhost kernel:                           [<c061b293>] _spin_lock_bh+0x2e/0x39
Feb 18 20:56:22 localhost kernel:                           [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:56:22 localhost kernel:                           [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:56:22 localhost kernel:                           [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:22 localhost kernel:                           [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:22 localhost kernel:                           [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:22 localhost kernel:                           [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:22 localhost kernel:                           [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:56:22 localhost kernel:                           [<c061ae17>] __down_failed+0x7/0xc
Feb 18 20:56:22 localhost kernel:                           [<c0488854>] do_ioctl+0x4c/0x62
Feb 18 20:56:22 localhost kernel:                           [<c0488aa1>] vfs_ioctl+0x237/0x249
Feb 18 20:56:22 localhost kernel:                           [<c0488af8>] sys_ioctl+0x45/0x5d
Feb 18 20:56:22 localhost kernel:                           [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:56:22 localhost kernel:                           [<ffffffff>] 0xffffffff
Feb 18 20:56:22 localhost kernel:     hardirq-on-W at:
Feb 18 20:56:22 localhost kernel:                           [<c0449027>] __lock_acquire+0x46c/0xbf1
Feb 18 20:56:22 localhost kernel:                           [<f8c57ba1>] ppp_ioctl+0x504/0xc06 [ppp_generic]
Feb 18 20:56:22 localhost kernel:                           [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:22 localhost kernel:                           [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:56:22 localhost kernel:                           [<c061b293>] _spin_lock_bh+0x2e/0x39
Feb 18 20:56:22 localhost kernel:                           [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:56:22 localhost kernel:                           [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:56:22 localhost kernel:                           [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:22 localhost kernel:                           [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:22 localhost kernel:                           [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:22 localhost kernel:                           [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:22 localhost kernel:                           [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:56:22 localhost kernel:                           [<c061ae17>] __down_failed+0x7/0xc
Feb 18 20:56:22 localhost kernel:                           [<c0488854>] do_ioctl+0x4c/0x62
Feb 18 20:56:22 localhost kernel:                           [<c0488aa1>] vfs_ioctl+0x237/0x249
Feb 18 20:56:22 localhost kernel:                           [<c0488af8>] sys_ioctl+0x45/0x5d
Feb 18 20:56:22 localhost kernel:                           [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:56:22 localhost kernel:                           [<ffffffff>] 0xffffffff
Feb 18 20:56:22 localhost kernel:   }
Feb 18 20:56:22 localhost kernel:   ... key      at: [<f8c5dd08>] __key.29980+0x0/0xffffb83a [ppp_generic]
Feb 18 20:56:22 localhost kernel:   -> (&ppp->rlock){-+..} ops: 5 {
Feb 18 20:56:22 localhost kernel:      initial-use  at:
Feb 18 20:56:22 localhost kernel:                             [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:23 localhost kernel:                             [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:56:23 localhost kernel:                             [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:23 localhost kernel:                             [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:56:23 localhost kernel:                             [<c061b293>] _spin_lock_bh+0x2e/0x39
Feb 18 20:56:23 localhost kernel:                             [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:56:23 localhost kernel:                             [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:56:23 localhost kernel:                             [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:23 localhost kernel:                             [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:23 localhost kernel:                             [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:23 localhost kernel:                             [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:23 localhost kernel:                             [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:56:23 localhost kernel:                             [<c061ae17>] __down_failed+0x7/0xc
Feb 18 20:56:23 localhost kernel:                             [<c0488854>] do_ioctl+0x4c/0x62
Feb 18 20:56:23 localhost kernel:                             [<c0488aa1>] vfs_ioctl+0x237/0x249
Feb 18 20:56:23 localhost kernel:                             [<c0488af8>] sys_ioctl+0x45/0x5d
Feb 18 20:56:23 localhost kernel:                             [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:56:23 localhost kernel:                             [<ffffffff>] 0xffffffff
Feb 18 20:56:23 localhost kernel:      in-softirq-W at:
Feb 18 20:56:23 localhost kernel:                             [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:56:23 localhost kernel:                             [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:23 localhost kernel:                             [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:23 localhost kernel:                             [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:23 localhost kernel:                             [<f8c59473>] ppp_input+0xab/0xef [ppp_generic]
Feb 18 20:56:23 localhost kernel:                             [<c061b293>] _spin_lock_bh+0x2e/0x39
Feb 18 20:56:23 localhost kernel:                             [<f8c59473>] ppp_input+0xab/0xef [ppp_generic]
Feb 18 20:56:23 localhost kernel:                             [<f8c59473>] ppp_input+0xab/0xef [ppp_generic]
Feb 18 20:56:23 localhost kernel:                             [<f8c60263>] pppol2tp_recv_core+0x75c/0x82d [pppol2tp]
Feb 18 20:56:23 localhost kernel:                             [<f8c6037a>] pppol2tp_udp_encap_recv+0x46/0x65 [pppol2tp]
Feb 18 20:56:23 localhost kernel:                             [<c061b171>] _read_unlock+0x14/0x1c
Feb 18 20:56:23 localhost kernel:                             [<c05ef9df>] udp_queue_rcv_skb+0xba/0x259
Feb 18 20:56:23 localhost kernel:                             [<c05efffe>] __udp4_lib_rcv+0x480/0x758
Feb 18 20:56:23 localhost kernel:                             [<c05d2c4b>] ip_local_deliver_finish+0x13f/0x1f8
Feb 18 20:56:23 localhost kernel:                             [<c05d2b3a>] ip_local_deliver_finish+0x2e/0x1f8
Feb 18 20:56:23 localhost kernel:                             [<c05d2ad2>] ip_rcv_finish+0x2fe/0x338
Feb 18 20:56:23 localhost kernel:                             [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:23 localhost kernel:                             [<c05d2dc6>] ip_rcv+0x0/0x237
Feb 18 20:56:23 localhost kernel:                             [<c05b5681>] netif_receive_skb+0x373/0x3d4
Feb 18 20:56:23 localhost kernel:                             [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:23 localhost kernel:                             [<f89877c7>] e1000_clean_rx_irq+0x374/0x44a [e1000]
Feb 18 20:56:23 localhost kernel:                             [<f8987453>] e1000_clean_rx_irq+0x0/0x44a [e1000]
Feb 18 20:56:23 localhost kernel:                             [<f8984f13>] e1000_clean+0x63/0x203 [e1000]
Feb 18 20:56:24 localhost kernel:                             [<c05b77f0>] net_rx_action+0xbc/0x1b3
Feb 18 20:56:24 localhost kernel:                             [<c05b7782>] net_rx_action+0x4e/0x1b3
Feb 18 20:56:24 localhost kernel:                             [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:24 localhost kernel:                             [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:24 localhost kernel:                             [<ffffffff>] 0xffffffff
Feb 18 20:56:24 localhost kernel:      hardirq-on-W at:
Feb 18 20:56:24 localhost kernel:                             [<c0449027>] __lock_acquire+0x46c/0xbf1
Feb 18 20:56:24 localhost kernel:                             [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:56:24 localhost kernel:                             [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:24 localhost kernel:                             [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:56:24 localhost kernel:                             [<c061b293>] _spin_lock_bh+0x2e/0x39
Feb 18 20:56:24 localhost kernel:                             [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:56:24 localhost kernel:                             [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:56:24 localhost kernel:                             [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:24 localhost kernel:                             [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:24 localhost kernel:                             [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:24 localhost kernel:                             [<c061b008>] __down+0x82/0xb8
Feb 18 20:56:24 localhost kernel:                             [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:56:24 localhost kernel:                             [<c061ae17>] __down_failed+0x7/0xc
Feb 18 20:56:24 localhost kernel:                             [<c0488854>] do_ioctl+0x4c/0x62
Feb 18 20:56:24 localhost kernel:                             [<c0488aa1>] vfs_ioctl+0x237/0x249
Feb 18 20:56:24 localhost kernel:                             [<c0488af8>] sys_ioctl+0x45/0x5d
Feb 18 20:56:24 localhost kernel:                             [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:56:24 localhost kernel:                             [<ffffffff>] 0xffffffff
Feb 18 20:56:24 localhost kernel:    }
Feb 18 20:56:24 localhost kernel:    ... key      at: [<f8c5dd10>] __key.29979+0x0/0xffffb832 [ppp_generic]
Feb 18 20:56:24 localhost kernel:    -> (&list->lock#7){.+..} ops: 31 {
Feb 18 20:56:24 localhost kernel:       initial-use  at:
Feb 18 20:56:24 localhost kernel:                               [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:24 localhost kernel:                               [<c046b874>] __do_fault+0x31b/0x35d
Feb 18 20:56:24 localhost kernel:                               [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:24 localhost kernel:                               [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:24 localhost kernel:                               [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:24 localhost kernel:                               [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:24 localhost kernel:                               [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:24 localhost kernel:                               [<f8c5939f>] ppp_write+0xb2/0xdb [ppp_generic]
Feb 18 20:56:24 localhost kernel:                               [<f8c592ed>] ppp_write+0x0/0xdb [ppp_generic]
Feb 18 20:56:24 localhost kernel:                               [<c047e6d2>] vfs_write+0xa1/0x14d
Feb 18 20:56:24 localhost kernel:                               [<c047ecfd>] sys_write+0x41/0x67
Feb 18 20:56:24 localhost kernel:                               [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:56:24 localhost kernel:                               [<ffffffff>] 0xffffffff
Feb 18 20:56:24 localhost kernel:       in-softirq-W at:
Feb 18 20:56:24 localhost kernel:                               [<c0446ab0>] save_trace+0x37/0x8b
Feb 18 20:56:25 localhost kernel:                               [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:25 localhost kernel:                               [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:25 localhost kernel:                               [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:25 localhost kernel:                               [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:25 localhost kernel:                               [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:25 localhost kernel:                               [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:25 localhost kernel:                               [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:25 localhost kernel:                               [<f8c59433>] ppp_input+0x6b/0xef [ppp_generic]
Feb 18 20:56:25 localhost kernel:                               [<f8c60263>] pppol2tp_recv_core+0x75c/0x82d [pppol2tp]
Feb 18 20:56:25 localhost kernel:                               [<f8c6037a>] pppol2tp_udp_encap_recv+0x46/0x65 [pppol2tp]
Feb 18 20:56:25 localhost kernel:                               [<c061b171>] _read_unlock+0x14/0x1c
Feb 18 20:56:25 localhost kernel:                               [<c05ef9df>] udp_queue_rcv_skb+0xba/0x259
Feb 18 20:56:25 localhost kernel:                               [<c05efffe>] __udp4_lib_rcv+0x480/0x758
Feb 18 20:56:25 localhost kernel:                               [<c05d2c4b>] ip_local_deliver_finish+0x13f/0x1f8
Feb 18 20:56:25 localhost kernel:                               [<c05d2b3a>] ip_local_deliver_finish+0x2e/0x1f8
Feb 18 20:56:25 localhost kernel:                               [<c05d2ad2>] ip_rcv_finish+0x2fe/0x338
Feb 18 20:56:25 localhost kernel:                               [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:25 localhost kernel:                               [<c05d2dc6>] ip_rcv+0x0/0x237
Feb 18 20:56:25 localhost kernel:                               [<c05b5681>] netif_receive_skb+0x373/0x3d4
Feb 18 20:56:25 localhost kernel:                               [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:25 localhost kernel:                               [<f89877c7>] e1000_clean_rx_irq+0x374/0x44a [e1000]
Feb 18 20:56:25 localhost kernel:                               [<f8987453>] e1000_clean_rx_irq+0x0/0x44a [e1000]
Feb 18 20:56:25 localhost kernel:                               [<f8984f13>] e1000_clean+0x63/0x203 [e1000]
Feb 18 20:56:25 localhost kernel:                               [<c05b77f0>] net_rx_action+0xbc/0x1b3
Feb 18 20:56:25 localhost kernel:                               [<c05b7782>] net_rx_action+0x4e/0x1b3
Feb 18 20:56:25 localhost kernel:                               [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:25 localhost kernel:                               [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:25 localhost kernel:                               [<ffffffff>] 0xffffffff
Feb 18 20:56:25 localhost kernel:     }
Feb 18 20:56:25 localhost kernel:     ... key      at: [<f8c5dd18>] __key.21159+0x0/0xffffb82a [ppp_generic]
Feb 18 20:56:25 localhost kernel:    ... acquired at:
Feb 18 20:56:25 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:56:25 localhost kernel:    [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:25 localhost kernel:    [<c0446ab0>] save_trace+0x37/0x8b
Feb 18 20:56:25 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:25 localhost kernel:    [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:25 localhost kernel:    [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:25 localhost kernel:    [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:25 localhost kernel:    [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:56:25 localhost kernel:    [<f8c589f2>] ppp_receive_nonmp_frame+0x4e6/0x70a [ppp_generic]
Feb 18 20:56:25 localhost kernel:    [<f8c59473>] ppp_input+0xab/0xef [ppp_generic]
Feb 18 20:56:26 localhost kernel:    [<f8c59487>] ppp_input+0xbf/0xef [ppp_generic]
Feb 18 20:56:26 localhost kernel:    [<f8c60263>] pppol2tp_recv_core+0x75c/0x82d [pppol2tp]
Feb 18 20:56:26 localhost kernel:    [<f8c6037a>] pppol2tp_udp_encap_recv+0x46/0x65 [pppol2tp]
Feb 18 20:56:26 localhost kernel:    [<c061b171>] _read_unlock+0x14/0x1c
Feb 18 20:56:26 localhost kernel:    [<c05ef9df>] udp_queue_rcv_skb+0xba/0x259
Feb 18 20:56:26 localhost kernel:    [<c05efffe>] __udp4_lib_rcv+0x480/0x758
Feb 18 20:56:26 localhost kernel:    [<c05d2c4b>] ip_local_deliver_finish+0x13f/0x1f8
Feb 18 20:56:26 localhost kernel:    [<c05d2b3a>] ip_local_deliver_finish+0x2e/0x1f8
Feb 18 20:56:26 localhost kernel:    [<c05d2ad2>] ip_rcv_finish+0x2fe/0x338
Feb 18 20:56:26 localhost kernel:    [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:26 localhost kernel:    [<c05d2dc6>] ip_rcv+0x0/0x237
Feb 18 20:56:26 localhost kernel:    [<c05b5681>] netif_receive_skb+0x373/0x3d4
Feb 18 20:56:26 localhost kernel:    [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:56:26 localhost kernel:    [<f89877c7>] e1000_clean_rx_irq+0x374/0x44a [e1000]
Feb 18 20:56:26 localhost kernel:    [<f8987453>] e1000_clean_rx_irq+0x0/0x44a [e1000]
Feb 18 20:56:26 localhost kernel:    [<f8984f13>] e1000_clean+0x63/0x203 [e1000]
Feb 18 20:56:26 localhost kernel:    [<c05b77f0>] net_rx_action+0xbc/0x1b3
Feb 18 20:56:26 localhost kernel:    [<c05b7782>] net_rx_action+0x4e/0x1b3
Feb 18 20:56:26 localhost kernel:    [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:26 localhost kernel:    [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:26 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:56:26 localhost kernel: 
Feb 18 20:56:26 localhost kernel:    -> (&q->lock){++..} ops: 468037 {
Feb 18 20:56:26 localhost kernel:       initial-use  at:
Feb 18 20:56:26 localhost kernel:                               [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:56:26 localhost kernel:                               [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:26 localhost kernel:                               [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:56:26 localhost kernel:                               [<c0448594>] mark_held_locks+0x39/0x53
Feb 18 20:56:26 localhost kernel:                               [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:26 localhost kernel:                               [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:26 localhost kernel:                               [<c0619856>] wait_for_common+0x2a/0xe0
Feb 18 20:56:26 localhost kernel:                               [<c061b4a0>] _spin_lock_irq+0x2f/0x3a
Feb 18 20:56:26 localhost kernel:                               [<c0619856>] wait_for_common+0x2a/0xe0
Feb 18 20:56:26 localhost kernel:                               [<c0619856>] wait_for_common+0x2a/0xe0
Feb 18 20:56:26 localhost kernel:                               [<c043d4b2>] kthread_create+0x70/0xa4
Feb 18 20:56:26 localhost kernel:                               [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:26 localhost kernel:                               [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:56:26 localhost kernel:                               [<c04265bc>] migration_call+0x49/0x364
Feb 18 20:56:26 localhost kernel:                               [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:26 localhost kernel:                               [<c075e01b>] migration_init+0x19/0x40
Feb 18 20:56:26 localhost kernel:                               [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:27 localhost kernel:                               [<c074e39e>] kernel_init+0x55/0x2af
Feb 18 20:56:27 localhost kernel:                               [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:56:27 localhost kernel:                               [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:27 localhost kernel:                               [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:27 localhost kernel:                               [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:27 localhost kernel:                               [<ffffffff>] 0xffffffff
Feb 18 20:56:27 localhost kernel:       in-hardirq-W at:
Feb 18 20:56:27 localhost kernel:                               [<c04480df>] check_usage_forwards+0x19/0x41
Feb 18 20:56:27 localhost kernel:                               [<c0446ab0>] save_trace+0x37/0x8b
Feb 18 20:56:27 localhost kernel:                               [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:56:27 localhost kernel:                               [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:27 localhost kernel:                               [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:27 localhost kernel:                               [<c04235da>] complete+0x12/0x44
Feb 18 20:56:27 localhost kernel:                               [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:27 localhost kernel:                               [<c04235da>] complete+0x12/0x44
Feb 18 20:56:27 localhost kernel:                               [<c04235da>] complete+0x12/0x44
Feb 18 20:56:27 localhost kernel:                               [<c058446e>] i8042_aux_test_irq+0x44/0x5c
Feb 18 20:56:27 localhost kernel:                               [<c045bcde>] handle_IRQ_event+0x13/0x3d
Feb 18 20:56:27 localhost kernel:                               [<c045cd38>] handle_edge_irq+0xc2/0xff
Feb 18 20:56:27 localhost kernel:                               [<c045cc76>] handle_edge_irq+0x0/0xff
Feb 18 20:56:27 localhost kernel:                               [<c04071b4>] do_IRQ+0xac/0xd4
Feb 18 20:56:27 localhost kernel:                               [<ffffffff>] 0xffffffff
Feb 18 20:56:27 localhost kernel:       in-softirq-W at:
Feb 18 20:56:27 localhost kernel:                               [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:27 localhost kernel:                               [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:27 localhost kernel:                               [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:27 localhost kernel:                               [<c04235da>] complete+0x12/0x44
Feb 18 20:56:27 localhost kernel:                               [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:27 localhost kernel:                               [<c04235da>] complete+0x12/0x44
Feb 18 20:56:27 localhost kernel:                               [<c04235da>] complete+0x12/0x44
Feb 18 20:56:27 localhost kernel:                               [<c0430b61>] tasklet_action+0x23/0xa4
Feb 18 20:56:27 localhost kernel:                               [<c043b647>] __rcu_process_callbacks+0xfc/0x16e
Feb 18 20:56:27 localhost kernel:                               [<c043b6d1>] rcu_process_callbacks+0x18/0x30
Feb 18 20:56:27 localhost kernel:                               [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:56:27 localhost kernel:                               [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:27 localhost kernel:                               [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:27 localhost kernel:                               [<ffffffff>] 0xffffffff
Feb 18 20:56:27 localhost kernel:     }
Feb 18 20:56:27 localhost kernel:     ... key      at: [<c0804610>] __key.17392+0x0/0x8
Feb 18 20:56:27 localhost kernel:     -> (&rq->rq_lock_key){++..} ops: 138094 {
Feb 18 20:56:28 localhost kernel:        initial-use  at:
Feb 18 20:56:28 localhost kernel:                                 [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:28 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:28 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:28 localhost kernel:                                 [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:28 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:28 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:28 localhost kernel:                                 [<c075e3c9>] sched_init+0x27d/0x292
Feb 18 20:56:28 localhost kernel:                                 [<c074e756>] start_kernel+0x15e/0x327
Feb 18 20:56:28 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:56:28 localhost kernel:        in-hardirq-W at:
Feb 18 20:56:28 localhost kernel:                                 [<c04480a2>] find_usage_forwards+0x67/0x8b
Feb 18 20:56:28 localhost kernel:                                 [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:56:28 localhost kernel:                                 [<c0442f77>] clocksource_get_next+0xa/0x3f
Feb 18 20:56:28 localhost kernel:                                 [<c0442fa6>] clocksource_get_next+0x39/0x3f
Feb 18 20:56:28 localhost kernel:                                 [<c044204f>] update_wall_time+0x58d/0x71c
Feb 18 20:56:28 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:28 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:28 localhost kernel:                                 [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:28 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:28 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:28 localhost kernel:                                 [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:56:28 localhost kernel:                                 [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:56:28 localhost kernel:                                 [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:56:28 localhost kernel:                                 [<c0407b80>] timer_interrupt+0x44/0x4a
Feb 18 20:56:28 localhost kernel:                                 [<c045bcde>] handle_IRQ_event+0x13/0x3d
Feb 18 20:56:28 localhost kernel:                                 [<c045d17e>] handle_level_irq+0x78/0xbe
Feb 18 20:56:28 localhost kernel:                                 [<c045d106>] handle_level_irq+0x0/0xbe
Feb 18 20:56:28 localhost kernel:                                 [<c04071b4>] do_IRQ+0xac/0xd4
Feb 18 20:56:28 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:56:28 localhost kernel:        in-softirq-W at:
Feb 18 20:56:28 localhost kernel:                                 [<c04eb4c4>] __next_cpu+0x12/0x21
Feb 18 20:56:28 localhost kernel:                                 [<c0422625>] find_busiest_group+0x204/0x5f3
Feb 18 20:56:28 localhost kernel:                                 [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:28 localhost kernel:                                 [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:28 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:28 localhost kernel:                                 [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:56:28 localhost kernel:                                 [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:28 localhost kernel:                                 [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:56:28 localhost kernel:                                 [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:56:28 localhost kernel:                                 [<c04250c8>] try_to_wake_up+0x19/0x3c5
Feb 18 20:56:29 localhost kernel:                                 [<c0448771>] trace_hardirqs_on+0x10c/0x14c
Feb 18 20:56:29 localhost kernel:                                 [<c0430afa>] __do_softirq+0xc9/0xde
Feb 18 20:56:29 localhost kernel:                                 [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:29 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:56:29 localhost kernel:      }
Feb 18 20:56:29 localhost kernel:      ... key      at: [<c2a22488>] 0xc2a22488
Feb 18 20:56:29 localhost kernel:      -> (&rq->rq_lock_key#2){++..} ops: 382046 {
Feb 18 20:56:29 localhost kernel:         initial-use  at:
Feb 18 20:56:29 localhost kernel:                                   [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:29 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:29 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:29 localhost kernel:                                   [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:29 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:29 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:29 localhost kernel:                                   [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:56:29 localhost kernel:                                   [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:56:29 localhost kernel:                                   [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:56:29 localhost kernel:                                   [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:56:29 localhost kernel:                                   [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:56:29 localhost kernel:                                   [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:56:29 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:29 localhost kernel:                                   [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:29 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:29 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:29 localhost kernel:                                   [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:56:29 localhost kernel:                                   [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:56:29 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:29 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:29 localhost kernel:                                   [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:56:29 localhost kernel:                                   [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:56:29 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:29 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:29 localhost kernel:                                   [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:29 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:29 localhost kernel:         in-hardirq-W at:
Feb 18 20:56:29 localhost kernel:                                   [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:56:29 localhost kernel:                                   [<c040a9d6>] save_stack_trace+0x20/0x3a
Feb 18 20:56:29 localhost kernel:                                   [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:56:29 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:29 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:29 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:30 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:30 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:30 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:30 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:30 localhost kernel:                                   [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:56:30 localhost kernel:                                   [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:56:30 localhost kernel:                                   [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:56:30 localhost kernel:                                   [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:56:30 localhost kernel:                                   [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:56:30 localhost kernel:                                   [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:56:30 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:30 localhost kernel:                                   [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:56:30 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:30 localhost kernel:                                   [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:56:30 localhost kernel:                                   [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:56:30 localhost kernel:                                   [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:56:30 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:30 localhost kernel:         in-softirq-W at:
Feb 18 20:56:30 localhost kernel:                                   [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:30 localhost kernel:                                   [<c04eb4c4>] __next_cpu+0x12/0x21
Feb 18 20:56:30 localhost kernel:                                   [<c0422625>] find_busiest_group+0x204/0x5f3
Feb 18 20:56:30 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:30 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:30 localhost kernel:                                   [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:56:30 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:30 localhost kernel:                                   [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:56:30 localhost kernel:                                   [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:56:30 localhost kernel:                                   [<c0433efb>] process_timeout+0x0/0x5
Feb 18 20:56:30 localhost kernel:                                   [<c04250c8>] try_to_wake_up+0x19/0x3c5
Feb 18 20:56:30 localhost kernel:                                   [<c0433d59>] run_timer_softirq+0x113/0x188
Feb 18 20:56:30 localhost kernel:                                   [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:56:30 localhost kernel:                                   [<c0433efb>] process_timeout+0x0/0x5
Feb 18 20:56:30 localhost kernel:                                   [<c0448771>] trace_hardirqs_on+0x10c/0x14c
Feb 18 20:56:30 localhost kernel:                                   [<c0433efb>] process_timeout+0x0/0x5
Feb 18 20:56:30 localhost kernel:                                   [<c0433d67>] run_timer_softirq+0x121/0x188
Feb 18 20:56:30 localhost kernel:                                   [<c0433efb>] process_timeout+0x0/0x5
Feb 18 20:56:30 localhost kernel:                                   [<c0448771>] trace_hardirqs_on+0x10c/0x14c
Feb 18 20:56:30 localhost kernel:                                   [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:30 localhost kernel:                                   [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:30 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:30 localhost kernel:       }
Feb 18 20:56:31 localhost kernel:       ... key      at: [<c2a2b488>] 0xc2a2b488
Feb 18 20:56:31 localhost kernel:       -> (&rq->rq_lock_key#3){++..} ops: 250064 {
Feb 18 20:56:31 localhost kernel:          initial-use  at:
Feb 18 20:56:31 localhost kernel:                                     [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:31 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:31 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:31 localhost kernel:                                     [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:31 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:31 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:31 localhost kernel:                                     [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:56:31 localhost kernel:                                     [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:56:31 localhost kernel:                                     [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:56:31 localhost kernel:                                     [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:56:31 localhost kernel:                                     [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:56:31 localhost kernel:                                     [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:56:31 localhost kernel:                                     [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:31 localhost kernel:                                     [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:31 localhost kernel:                                     [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:31 localhost kernel:                                     [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:31 localhost kernel:                                     [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:56:31 localhost kernel:                                     [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:56:31 localhost kernel:                                     [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:31 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:31 localhost kernel:                                     [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:56:31 localhost kernel:                                     [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:56:31 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:31 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:31 localhost kernel:                                     [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:31 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:56:31 localhost kernel:          in-hardirq-W at:
Feb 18 20:56:31 localhost kernel:                                     [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:56:31 localhost kernel:                                     [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:31 localhost kernel:                                     [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:31 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:31 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:31 localhost kernel:                                     [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:31 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:31 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:31 localhost kernel:                                     [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:56:31 localhost kernel:                                     [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:56:31 localhost kernel:                                     [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:56:32 localhost kernel:                                     [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:56:32 localhost kernel:                                     [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:56:32 localhost kernel:                                     [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:56:32 localhost kernel:                                     [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:32 localhost kernel:                                     [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:56:32 localhost kernel:                                     [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:32 localhost kernel:                                     [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:56:32 localhost kernel:                                     [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:56:32 localhost kernel:                                     [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:56:32 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:56:32 localhost kernel:          in-softirq-W at:
Feb 18 20:56:32 localhost kernel:                                     [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:56:32 localhost kernel:                                     [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:32 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:32 localhost kernel:                                     [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:56:32 localhost kernel:                                     [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:32 localhost kernel:                                     [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:56:32 localhost kernel:                                     [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:56:32 localhost kernel:                                     [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:56:32 localhost kernel:                                     [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:56:32 localhost kernel:                                     [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:56:32 localhost kernel:                                     [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:32 localhost kernel:                                     [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:32 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:56:32 localhost kernel:        }
Feb 18 20:56:32 localhost kernel:        ... key      at: [<c2a34488>] 0xc2a34488
Feb 18 20:56:32 localhost kernel:        -> (&rq->rq_lock_key#4){++..} ops: 233519 {
Feb 18 20:56:32 localhost kernel:           initial-use  at:
Feb 18 20:56:32 localhost kernel:                                       [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:32 localhost kernel:                                       [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:32 localhost kernel:                                       [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:32 localhost kernel:                                       [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:32 localhost kernel:                                       [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:32 localhost kernel:                                       [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:32 localhost kernel:                                       [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:56:32 localhost kernel:                                       [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:56:32 localhost kernel:                                       [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:56:32 localhost kernel:                                       [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:56:32 localhost kernel:                                       [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:56:32 localhost kernel:                                       [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:56:32 localhost kernel:                                       [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:32 localhost kernel:                                       [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:33 localhost kernel:                                       [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:33 localhost kernel:                                       [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:33 localhost kernel:                                       [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:56:33 localhost kernel:                                       [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:56:33 localhost kernel:                                       [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:33 localhost kernel:                                       [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:33 localhost kernel:                                       [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:56:33 localhost kernel:                                       [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:56:33 localhost kernel:                                       [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:33 localhost kernel:                                       [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:33 localhost kernel:                                       [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:33 localhost kernel:                                       [<ffffffff>] 0xffffffff
Feb 18 20:56:33 localhost kernel:           in-hardirq-W at:
Feb 18 20:56:33 localhost kernel:                                       [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:56:33 localhost kernel:                                       [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:33 localhost kernel:                                       [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:33 localhost kernel:                                       [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:33 localhost kernel:                                       [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:33 localhost kernel:                                       [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:33 localhost kernel:                                       [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:33 localhost kernel:                                       [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:33 localhost kernel:                                       [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:56:33 localhost kernel:                                       [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:56:33 localhost kernel:                                       [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:56:33 localhost kernel:                                       [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:56:33 localhost kernel:                                       [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:56:33 localhost kernel:                                       [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:56:33 localhost kernel:                                       [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:33 localhost kernel:                                       [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:56:33 localhost kernel:                                       [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:33 localhost kernel:                                       [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:56:33 localhost kernel:                                       [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:56:33 localhost kernel:                                       [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:56:33 localhost kernel:                                       [<ffffffff>] 0xffffffff
Feb 18 20:56:33 localhost kernel:           in-softirq-W at:
Feb 18 20:56:33 localhost kernel:                                       [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:33 localhost kernel:                                       [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:33 localhost kernel:                                       [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:33 localhost kernel:                                       [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:33 localhost kernel:                                       [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:33 localhost kernel:                                       [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:33 localhost kernel:                                       [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:56:34 localhost kernel:                                       [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:56:34 localhost kernel:                                       [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:56:34 localhost kernel:                                       [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:34 localhost kernel:                                       [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:34 localhost kernel:                                       [<ffffffff>] 0xffffffff
Feb 18 20:56:34 localhost kernel:         }
Feb 18 20:56:34 localhost kernel:         ... key      at: [<c2a3d488>] 0xc2a3d488
Feb 18 20:56:34 localhost kernel:        ... acquired at:
Feb 18 20:56:34 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:56:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:34 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:34 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:34 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:56:34 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:34 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:56:34 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:34 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:56:34 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:56:34 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:34 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:56:34 localhost kernel: 
Feb 18 20:56:34 localhost kernel:       ... acquired at:
Feb 18 20:56:34 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:56:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:34 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:34 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:34 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:56:34 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:34 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:56:34 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:34 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:56:34 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:56:34 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:34 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:56:35 localhost kernel: 
Feb 18 20:56:35 localhost kernel:       -> (&rq->rq_lock_key#4){++..} ops: 233519 {
Feb 18 20:56:35 localhost kernel:          initial-use  at:
Feb 18 20:56:35 localhost kernel:                                     [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:35 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:35 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:35 localhost kernel:                                     [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:35 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:35 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:35 localhost kernel:                                     [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:56:35 localhost kernel:                                     [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:56:35 localhost kernel:                                     [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:56:35 localhost kernel:                                     [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:56:35 localhost kernel:                                     [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:56:35 localhost kernel:                                     [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:56:35 localhost kernel:                                     [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:35 localhost kernel:                                     [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:35 localhost kernel:                                     [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:35 localhost kernel:                                     [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:35 localhost kernel:                                     [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:56:35 localhost kernel:                                     [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:56:35 localhost kernel:                                     [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:35 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:35 localhost kernel:                                     [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:56:35 localhost kernel:                                     [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:56:35 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:35 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:35 localhost kernel:                                     [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:35 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:56:35 localhost kernel:          in-hardirq-W at:
Feb 18 20:56:35 localhost kernel:                                     [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:56:35 localhost kernel:                                     [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:35 localhost kernel:                                     [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:35 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:35 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:35 localhost kernel:                                     [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:35 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:35 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:35 localhost kernel:                                     [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:56:35 localhost kernel:                                     [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:56:35 localhost kernel:                                     [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:56:36 localhost kernel:                                     [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:56:36 localhost kernel:                                     [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:56:36 localhost kernel:                                     [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:56:36 localhost kernel:                                     [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:36 localhost kernel:                                     [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:56:36 localhost kernel:                                     [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:36 localhost kernel:                                     [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:56:36 localhost kernel:                                     [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:56:36 localhost kernel:                                     [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:56:36 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:56:36 localhost kernel:          in-softirq-W at:
Feb 18 20:56:36 localhost kernel:                                     [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:36 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:36 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:36 localhost kernel:                                     [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:36 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:36 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:36 localhost kernel:                                     [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:56:36 localhost kernel:                                     [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:56:36 localhost kernel:                                     [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:56:36 localhost kernel:                                     [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:36 localhost kernel:                                     [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:36 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:56:36 localhost kernel:        }
Feb 18 20:56:36 localhost kernel:        ... key      at: [<c2a3d488>] 0xc2a3d488
Feb 18 20:56:36 localhost kernel:       ... acquired at:
Feb 18 20:56:36 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:56:36 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:36 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:36 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:36 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:36 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:36 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:36 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:56:36 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:36 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:56:36 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:36 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:56:36 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:56:36 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:36 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:56:37 localhost kernel: 
Feb 18 20:56:37 localhost kernel:      ... acquired at:
Feb 18 20:56:37 localhost kernel:    [<c0446b6a>] add_lock_to_list+0x66/0x89
Feb 18 20:56:37 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:56:37 localhost kernel:    [<c0619307>] schedule+0x22e/0x6ec
Feb 18 20:56:37 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:37 localhost kernel:    [<c0619307>] schedule+0x22e/0x6ec
Feb 18 20:56:37 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:37 localhost kernel:    [<c0619307>] schedule+0x22e/0x6ec
Feb 18 20:56:37 localhost kernel:    [<c0619307>] schedule+0x22e/0x6ec
Feb 18 20:56:37 localhost kernel:    [<c0619a92>] schedule_timeout+0x13/0x8d
Feb 18 20:56:37 localhost kernel:    [<c0448594>] mark_held_locks+0x39/0x53
Feb 18 20:56:37 localhost kernel:    [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:56:37 localhost kernel:    [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:37 localhost kernel:    [<c06198d2>] wait_for_common+0xa6/0xe0
Feb 18 20:56:37 localhost kernel:    [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:56:37 localhost kernel:    [<c0439c5c>] call_usermodehelper_exec+0x7a/0xbe
Feb 18 20:56:37 localhost kernel:    [<c04ecbd6>] kobject_uevent_env+0x348/0x37c
Feb 18 20:56:37 localhost kernel:    [<c055f4c0>] device_add+0x318/0x54c
Feb 18 20:56:37 localhost kernel:    [<c055fa67>] device_create+0x7a/0x9a
Feb 18 20:56:37 localhost kernel:    [<c053c13e>] tty_register_device+0xb5/0xbd
Feb 18 20:56:37 localhost kernel:    [<c04efab1>] vsnprintf+0x440/0x47c
Feb 18 20:56:37 localhost kernel:    [<c0448594>] mark_held_locks+0x39/0x53
Feb 18 20:56:37 localhost kernel:    [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:37 localhost kernel:    [<c042c9c6>] register_console+0x4b/0x1fe
Feb 18 20:56:37 localhost kernel:    [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:37 localhost kernel:    [<c055aff0>] serial8250_pm+0x0/0xb2
Feb 18 20:56:37 localhost kernel:    [<c0558795>] uart_change_pm+0x23/0x2b
Feb 18 20:56:37 localhost kernel:    [<c055998a>] uart_add_one_port+0x29d/0x2f6
Feb 18 20:56:37 localhost kernel:    [<c05623c9>] platform_device_add+0xee/0x11c
Feb 18 20:56:37 localhost kernel:    [<c055ed94>] device_initialize+0x7b/0x93
Feb 18 20:56:37 localhost kernel:    [<c05624ee>] platform_device_alloc+0x40/0x51
Feb 18 20:56:37 localhost kernel:    [<c076a545>] serial8250_init+0xca/0x110
Feb 18 20:56:37 localhost kernel:    [<c074e491>] kernel_init+0x148/0x2af
Feb 18 20:56:37 localhost kernel:    [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:37 localhost kernel:    [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:37 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:37 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:56:37 localhost kernel: 
Feb 18 20:56:37 localhost kernel:      -> (&rq->rq_lock_key#4){++..} ops: 233519 {
Feb 18 20:56:37 localhost kernel:         initial-use  at:
Feb 18 20:56:38 localhost kernel:                                   [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:38 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:38 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:38 localhost kernel:                                   [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:38 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:38 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:38 localhost kernel:                                   [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:56:38 localhost kernel:                                   [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:56:38 localhost kernel:                                   [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:56:38 localhost kernel:                                   [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:56:38 localhost kernel:                                   [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:56:38 localhost kernel:                                   [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:56:38 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:38 localhost kernel:                                   [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:38 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:38 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:38 localhost kernel:                                   [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:56:38 localhost kernel:                                   [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:56:38 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:38 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:38 localhost kernel:                                   [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:56:38 localhost kernel:                                   [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:56:38 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:38 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:38 localhost kernel:                                   [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:38 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:38 localhost kernel:         in-hardirq-W at:
Feb 18 20:56:38 localhost kernel:                                   [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:56:38 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:38 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:38 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:38 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:38 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:38 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:38 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:38 localhost kernel:                                   [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:56:38 localhost kernel:                                   [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:56:38 localhost kernel:                                   [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:56:38 localhost kernel:                                   [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:56:38 localhost kernel:                                   [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:56:38 localhost kernel:                                   [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:56:39 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:39 localhost kernel:                                   [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:56:39 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:39 localhost kernel:                                   [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:56:39 localhost kernel:                                   [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:56:39 localhost kernel:                                   [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:56:39 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:39 localhost kernel:         in-softirq-W at:
Feb 18 20:56:39 localhost kernel:                                   [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:39 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:39 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:39 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:39 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:39 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:39 localhost kernel:                                   [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:56:39 localhost kernel:                                   [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:56:39 localhost kernel:                                   [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:56:39 localhost kernel:                                   [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:39 localhost kernel:                                   [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:39 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:39 localhost kernel:       }
Feb 18 20:56:39 localhost kernel:       ... key      at: [<c2a3d488>] 0xc2a3d488
Feb 18 20:56:39 localhost kernel:      ... acquired at:
Feb 18 20:56:39 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:56:39 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:39 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:39 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:39 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:39 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:39 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:56:39 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:56:39 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:39 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:56:39 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:56:39 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:56:39 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:56:39 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:39 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:56:39 localhost kernel: 
Feb 18 20:56:39 localhost kernel:      -> (&rq->rq_lock_key#3){++..} ops: 250064 {
Feb 18 20:56:39 localhost kernel:         initial-use  at:
Feb 18 20:56:40 localhost kernel:                                   [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:40 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:40 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:40 localhost kernel:                                   [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:40 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:40 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:40 localhost kernel:                                   [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:56:40 localhost kernel:                                   [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:56:40 localhost kernel:                                   [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:56:40 localhost kernel:                                   [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:56:40 localhost kernel:                                   [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:56:40 localhost kernel:                                   [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:56:40 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:40 localhost kernel:                                   [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:40 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:40 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:40 localhost kernel:                                   [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:56:40 localhost kernel:                                   [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:56:40 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:40 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:40 localhost kernel:                                   [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:56:40 localhost kernel:                                   [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:56:40 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:40 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:56:40 localhost kernel:                                   [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:56:40 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:40 localhost kernel:         in-hardirq-W at:
Feb 18 20:56:40 localhost kernel:                                   [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:56:40 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:40 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:56:40 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:40 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:40 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:40 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:40 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:56:40 localhost kernel:                                   [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:56:40 localhost kernel:                                   [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:56:40 localhost kernel:                                   [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:56:40 localhost kernel:                                   [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:56:40 localhost kernel:                                   [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:56:40 localhost kernel:                                   [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:56:41 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:41 localhost kernel:                                   [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:56:41 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:56:41 localhost kernel:                                   [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:56:41 localhost kernel:                                   [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:56:41 localhost kernel:                                   [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:56:41 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:41 localhost kernel:         in-softirq-W at:
Feb 18 20:56:41 localhost kernel:                                   [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:56:41 localhost kernel:                                   [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:56:41 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:41 localhost kernel:                                   [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:56:41 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:56:41 localhost kernel:                                   [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:56:41 localhost kernel:                                   [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:56:41 localhost kernel:                                   [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:56:41 localhost kernel:                                   [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:56:41 localhost kernel:                                   [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:56:41 localhost kernel:                                   [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:56:41 localhost kernel:                                   [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:56:41 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:56:41 localhost kernel:       }
Feb 18 20:56:41 localhost kernel:       ... key      at: [<c2a34488>] 0xc2a34488
Feb 18 20:56:41 localhost kernel:       -> (&rq->rq_lock_key#4){++..} ops: 233519 {
Feb 18 20:56:41 localhost kernel:          initial-use  at:
Feb 18 20:56:41 localhost kernel:                                     [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:56:41 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:56:41 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:41 localhost kernel:                                     [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:56:41 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:41 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:56:41 localhost kernel:                                     [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:56:41 localhost kernel:                                     [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:56:41 localhost kernel:                                     [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:56:41 localhost kernel:                                     [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:56:41 localhost kernel:                                     [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:56:41 localhost kernel:                                     [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:56:41 localhost kernel:                                     [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:42 localhost kernel:                                     [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:56:43 localhost kernel:                                     [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:56:48 localhost kernel:                                     [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:56:53 localhost kernel:                                     [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:56:58 localhost kernel:                                     [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:03 localhost kernel:                                     [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:09 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:13 localhost kernel:                                     [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:57:18 localhost kernel:                                     [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:57:21 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:25 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:26 localhost kernel:                                     [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:27 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:57:27 localhost kernel:          in-hardirq-W at:
Feb 18 20:57:27 localhost kernel:                                     [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:57:27 localhost kernel:                                     [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:27 localhost kernel:                                     [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:27 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:27 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:27 localhost kernel:                                     [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:27 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:27 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:27 localhost kernel:                                     [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:57:27 localhost kernel:                                     [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:57:27 localhost kernel:                                     [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:57:27 localhost kernel:                                     [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:57:27 localhost kernel:                                     [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:57:27 localhost kernel:                                     [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:57:27 localhost kernel:                                     [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:27 localhost kernel:                                     [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:57:27 localhost kernel:                                     [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:27 localhost kernel:                                     [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:57:27 localhost kernel:                                     [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:57:27 localhost kernel:                                     [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:57:27 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:57:27 localhost kernel:          in-softirq-W at:
Feb 18 20:57:27 localhost kernel:                                     [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:27 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:27 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:27 localhost kernel:                                     [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:27 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:                                     [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:57:28 localhost kernel:                                     [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:57:28 localhost kernel:                                     [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:57:28 localhost kernel:                                     [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:28 localhost kernel:                                     [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:28 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:57:28 localhost kernel:        }
Feb 18 20:57:28 localhost kernel:        ... key      at: [<c2a3d488>] 0xc2a3d488
Feb 18 20:57:28 localhost kernel:       ... acquired at:
Feb 18 20:57:28 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:28 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:28 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:28 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:57:28 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:28 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:57:28 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:28 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:57:28 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:28 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:28 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:28 localhost kernel: 
Feb 18 20:57:28 localhost kernel:      ... acquired at:
Feb 18 20:57:28 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:28 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:28 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:28 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:28 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:57:28 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:28 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:57:28 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:28 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:57:28 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:28 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:29 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:29 localhost kernel: 
Feb 18 20:57:29 localhost kernel:     ... acquired at:
Feb 18 20:57:29 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:29 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:29 localhost kernel:    [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:29 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:29 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:29 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:29 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:29 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:29 localhost kernel:    [<c04250c8>] try_to_wake_up+0x19/0x3c5
Feb 18 20:57:29 localhost kernel:    [<c0421394>] __wake_up_common+0x32/0x5c
Feb 18 20:57:29 localhost kernel:    [<c04235fe>] complete+0x36/0x44
Feb 18 20:57:29 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:29 localhost kernel:    [<c043d502>] kthread+0x1c/0x5e
Feb 18 20:57:29 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:29 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:29 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:29 localhost kernel: 
Feb 18 20:57:29 localhost kernel:     -> (&rq->rq_lock_key#2){++..} ops: 382046 {
Feb 18 20:57:29 localhost kernel:        initial-use  at:
Feb 18 20:57:29 localhost kernel:                                 [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:29 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:29 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:29 localhost kernel:                                 [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:29 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:29 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:29 localhost kernel:                                 [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:57:29 localhost kernel:                                 [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:57:29 localhost kernel:                                 [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:57:29 localhost kernel:                                 [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:57:29 localhost kernel:                                 [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:57:29 localhost kernel:                                 [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:57:29 localhost kernel:                                 [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:29 localhost kernel:                                 [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:29 localhost kernel:                                 [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:29 localhost kernel:                                 [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:29 localhost kernel:                                 [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:57:29 localhost kernel:                                 [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:29 localhost kernel:                                 [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:29 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:30 localhost kernel:                                 [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:57:30 localhost kernel:                                 [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:57:30 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:30 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:30 localhost kernel:                                 [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:30 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:30 localhost kernel:        in-hardirq-W at:
Feb 18 20:57:30 localhost kernel:                                 [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:57:30 localhost kernel:                                 [<c040a9d6>] save_stack_trace+0x20/0x3a
Feb 18 20:57:30 localhost kernel:                                 [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:57:30 localhost kernel:                                 [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:30 localhost kernel:                                 [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:30 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:30 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:30 localhost kernel:                                 [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:30 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:30 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:30 localhost kernel:                                 [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:57:30 localhost kernel:                                 [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:57:30 localhost kernel:                                 [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:57:30 localhost kernel:                                 [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:57:30 localhost kernel:                                 [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:57:30 localhost kernel:                                 [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:57:30 localhost kernel:                                 [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:30 localhost kernel:                                 [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:57:30 localhost kernel:                                 [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:30 localhost kernel:                                 [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:57:30 localhost kernel:                                 [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:57:30 localhost kernel:                                 [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:57:30 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:30 localhost kernel:        in-softirq-W at:
Feb 18 20:57:30 localhost kernel:                                 [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:30 localhost kernel:                                 [<c04eb4c4>] __next_cpu+0x12/0x21
Feb 18 20:57:30 localhost kernel:                                 [<c0422625>] find_busiest_group+0x204/0x5f3
Feb 18 20:57:30 localhost kernel:                                 [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:30 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:30 localhost kernel:                                 [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:30 localhost kernel:                                 [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:30 localhost kernel:                                 [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:30 localhost kernel:                                 [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:30 localhost kernel:                                 [<c0433efb>] process_timeout+0x0/0x5
Feb 18 20:57:31 localhost kernel:                                 [<c04250c8>] try_to_wake_up+0x19/0x3c5
Feb 18 20:57:31 localhost kernel:                                 [<c0433d59>] run_timer_softirq+0x113/0x188
Feb 18 20:57:31 localhost kernel:                                 [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:31 localhost kernel:                                 [<c0433efb>] process_timeout+0x0/0x5
Feb 18 20:57:31 localhost kernel:                                 [<c0448771>] trace_hardirqs_on+0x10c/0x14c
Feb 18 20:57:31 localhost kernel:                                 [<c0433efb>] process_timeout+0x0/0x5
Feb 18 20:57:31 localhost kernel:                                 [<c0433d67>] run_timer_softirq+0x121/0x188
Feb 18 20:57:31 localhost kernel:                                 [<c0433efb>] process_timeout+0x0/0x5
Feb 18 20:57:31 localhost kernel:                                 [<c0448771>] trace_hardirqs_on+0x10c/0x14c
Feb 18 20:57:31 localhost kernel:                                 [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:31 localhost kernel:                                 [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:31 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:31 localhost kernel:      }
Feb 18 20:57:31 localhost kernel:      ... key      at: [<c2a2b488>] 0xc2a2b488
Feb 18 20:57:31 localhost kernel:      -> (&rq->rq_lock_key#3){++..} ops: 250064 {
Feb 18 20:57:31 localhost kernel:         initial-use  at:
Feb 18 20:57:31 localhost kernel:                                   [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:31 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:31 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:31 localhost kernel:                                   [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:31 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:31 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:31 localhost kernel:                                   [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:57:31 localhost kernel:                                   [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:57:31 localhost kernel:                                   [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:57:31 localhost kernel:                                   [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:57:31 localhost kernel:                                   [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:57:31 localhost kernel:                                   [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:57:31 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:31 localhost kernel:                                   [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:31 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:31 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:31 localhost kernel:                                   [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:57:31 localhost kernel:                                   [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:31 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:31 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:31 localhost kernel:                                   [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:57:31 localhost kernel:                                   [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:57:31 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:31 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:31 localhost kernel:                                   [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:32 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:32 localhost kernel:         in-hardirq-W at:
Feb 18 20:57:32 localhost kernel:                                   [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:57:32 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:32 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:32 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:32 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:32 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:32 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:32 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:32 localhost kernel:                                   [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:57:32 localhost kernel:                                   [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:57:32 localhost kernel:                                   [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:57:32 localhost kernel:                                   [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:57:32 localhost kernel:                                   [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:57:32 localhost kernel:                                   [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:57:32 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:32 localhost kernel:                                   [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:57:32 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:32 localhost kernel:                                   [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:57:32 localhost kernel:                                   [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:57:32 localhost kernel:                                   [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:57:32 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:32 localhost kernel:         in-softirq-W at:
Feb 18 20:57:32 localhost kernel:                                   [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:57:32 localhost kernel:                                   [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:32 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:32 localhost kernel:                                   [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:57:32 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:32 localhost kernel:                                   [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:57:32 localhost kernel:                                   [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:57:32 localhost kernel:                                   [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:57:32 localhost kernel:                                   [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:57:32 localhost kernel:                                   [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:57:32 localhost kernel:                                   [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:32 localhost kernel:                                   [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:32 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:32 localhost kernel:       }
Feb 18 20:57:32 localhost kernel:       ... key      at: [<c2a34488>] 0xc2a34488
Feb 18 20:57:32 localhost kernel:       -> (&rq->rq_lock_key#4){++..} ops: 233519 {
Feb 18 20:57:32 localhost kernel:          initial-use  at:
Feb 18 20:57:33 localhost kernel:                                     [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:33 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:33 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:33 localhost kernel:                                     [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:33 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:33 localhost kernel:                                     [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:33 localhost kernel:                                     [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:57:33 localhost kernel:                                     [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:57:33 localhost kernel:                                     [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:57:33 localhost kernel:                                     [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:57:33 localhost kernel:                                     [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:57:33 localhost kernel:                                     [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:57:33 localhost kernel:                                     [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:33 localhost kernel:                                     [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:33 localhost kernel:                                     [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:33 localhost kernel:                                     [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:33 localhost kernel:                                     [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:57:33 localhost kernel:                                     [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:33 localhost kernel:                                     [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:33 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:33 localhost kernel:                                     [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:57:33 localhost kernel:                                     [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:57:33 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:33 localhost kernel:                                     [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:33 localhost kernel:                                     [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:33 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:57:33 localhost kernel:          in-hardirq-W at:
Feb 18 20:57:33 localhost kernel:                                     [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:57:33 localhost kernel:                                     [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:33 localhost kernel:                                     [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:33 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:33 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:33 localhost kernel:                                     [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:33 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:33 localhost kernel:                                     [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:33 localhost kernel:                                     [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:57:33 localhost kernel:                                     [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:57:33 localhost kernel:                                     [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:57:33 localhost kernel:                                     [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:57:33 localhost kernel:                                     [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:57:33 localhost kernel:                                     [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:57:33 localhost kernel:                                     [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:33 localhost kernel:                                     [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:57:34 localhost kernel:                                     [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:34 localhost kernel:                                     [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:57:34 localhost kernel:                                     [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:57:34 localhost kernel:                                     [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:57:34 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:57:34 localhost kernel:          in-softirq-W at:
Feb 18 20:57:34 localhost kernel:                                     [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:34 localhost kernel:                                     [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:34 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:34 localhost kernel:                                     [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:34 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:34 localhost kernel:                                     [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:34 localhost kernel:                                     [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:57:34 localhost kernel:                                     [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:57:34 localhost kernel:                                     [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:57:34 localhost kernel:                                     [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:34 localhost kernel:                                     [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:34 localhost kernel:                                     [<ffffffff>] 0xffffffff
Feb 18 20:57:34 localhost kernel:        }
Feb 18 20:57:34 localhost kernel:        ... key      at: [<c2a3d488>] 0xc2a3d488
Feb 18 20:57:34 localhost kernel:       ... acquired at:
Feb 18 20:57:34 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:34 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:34 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:34 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:57:34 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:34 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:57:34 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:34 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:57:34 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:34 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:34 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:34 localhost kernel: 
Feb 18 20:57:34 localhost kernel:      ... acquired at:
Feb 18 20:57:34 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:34 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:34 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:35 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:35 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:35 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:35 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:57:35 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:35 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:57:35 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:35 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:57:35 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:35 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:35 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:35 localhost kernel: 
Feb 18 20:57:35 localhost kernel:      -> (&rq->rq_lock_key#4){++..} ops: 233519 {
Feb 18 20:57:35 localhost kernel:         initial-use  at:
Feb 18 20:57:35 localhost kernel:                                   [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:35 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:35 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:35 localhost kernel:                                   [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:35 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:35 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:35 localhost kernel:                                   [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:57:35 localhost kernel:                                   [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:57:35 localhost kernel:                                   [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:57:35 localhost kernel:                                   [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:57:35 localhost kernel:                                   [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:57:35 localhost kernel:                                   [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:57:35 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:35 localhost kernel:                                   [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:35 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:35 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:35 localhost kernel:                                   [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:57:35 localhost kernel:                                   [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:35 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:35 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:35 localhost kernel:                                   [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:57:35 localhost kernel:                                   [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:57:35 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:35 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:35 localhost kernel:                                   [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:35 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:35 localhost kernel:         in-hardirq-W at:
Feb 18 20:57:35 localhost kernel:                                   [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:57:36 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:36 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:36 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:36 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:36 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:36 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:36 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:36 localhost kernel:                                   [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:57:36 localhost kernel:                                   [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:57:36 localhost kernel:                                   [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:57:36 localhost kernel:                                   [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:57:36 localhost kernel:                                   [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:57:36 localhost kernel:                                   [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:57:36 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:36 localhost kernel:                                   [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:57:36 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:36 localhost kernel:                                   [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:57:36 localhost kernel:                                   [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:57:36 localhost kernel:                                   [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:57:36 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:36 localhost kernel:         in-softirq-W at:
Feb 18 20:57:36 localhost kernel:                                   [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:36 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:36 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:36 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:36 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:36 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:36 localhost kernel:                                   [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:57:36 localhost kernel:                                   [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:57:36 localhost kernel:                                   [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:57:36 localhost kernel:                                   [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:36 localhost kernel:                                   [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:36 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:36 localhost kernel:       }
Feb 18 20:57:36 localhost kernel:       ... key      at: [<c2a3d488>] 0xc2a3d488
Feb 18 20:57:36 localhost kernel:      ... acquired at:
Feb 18 20:57:36 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:36 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:36 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:36 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:37 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:37 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:37 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:37 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:57:37 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:37 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:57:37 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:37 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:57:37 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:37 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:37 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:37 localhost kernel: 
Feb 18 20:57:37 localhost kernel:     ... acquired at:
Feb 18 20:57:37 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:37 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:37 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:37 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:37 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:37 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:37 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:37 localhost kernel:    [<c04250c8>] try_to_wake_up+0x19/0x3c5
Feb 18 20:57:37 localhost kernel:    [<c0421394>] __wake_up_common+0x32/0x5c
Feb 18 20:57:37 localhost kernel:    [<c04235fe>] complete+0x36/0x44
Feb 18 20:57:37 localhost kernel:    [<c043aad8>] worker_thread+0x0/0xc4
Feb 18 20:57:37 localhost kernel:    [<c043d502>] kthread+0x1c/0x5e
Feb 18 20:57:37 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:37 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:37 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:37 localhost kernel: 
Feb 18 20:57:37 localhost kernel:     -> (&rq->rq_lock_key#3){++..} ops: 250064 {
Feb 18 20:57:37 localhost kernel:        initial-use  at:
Feb 18 20:57:37 localhost kernel:                                 [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:37 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:37 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:37 localhost kernel:                                 [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:37 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:37 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:37 localhost kernel:                                 [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:57:37 localhost kernel:                                 [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:57:37 localhost kernel:                                 [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:57:38 localhost kernel:                                 [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:57:38 localhost kernel:                                 [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:57:38 localhost kernel:                                 [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:57:38 localhost kernel:                                 [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:38 localhost kernel:                                 [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:38 localhost kernel:                                 [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:38 localhost kernel:                                 [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:38 localhost kernel:                                 [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:57:38 localhost kernel:                                 [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:38 localhost kernel:                                 [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:38 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:38 localhost kernel:                                 [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:57:38 localhost kernel:                                 [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:57:38 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:38 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:38 localhost kernel:                                 [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:38 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:38 localhost kernel:        in-hardirq-W at:
Feb 18 20:57:38 localhost kernel:                                 [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:57:38 localhost kernel:                                 [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:38 localhost kernel:                                 [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:38 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:38 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:38 localhost kernel:                                 [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:38 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:38 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:38 localhost kernel:                                 [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:57:38 localhost kernel:                                 [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:57:38 localhost kernel:                                 [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:57:38 localhost kernel:                                 [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:57:38 localhost kernel:                                 [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:57:38 localhost kernel:                                 [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:57:38 localhost kernel:                                 [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:38 localhost kernel:                                 [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:57:38 localhost kernel:                                 [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:38 localhost kernel:                                 [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:57:38 localhost kernel:                                 [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:57:38 localhost kernel:                                 [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:57:38 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:38 localhost kernel:        in-softirq-W at:
Feb 18 20:57:38 localhost kernel:                                 [<c044816c>] mark_lock+0x65/0x454
Feb 18 20:57:38 localhost kernel:                                 [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:39 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:39 localhost kernel:                                 [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:57:39 localhost kernel:                                 [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:39 localhost kernel:                                 [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:57:39 localhost kernel:                                 [<c042313a>] double_rq_lock+0x29/0x40
Feb 18 20:57:39 localhost kernel:                                 [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:57:39 localhost kernel:                                 [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:57:39 localhost kernel:                                 [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:57:39 localhost kernel:                                 [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:39 localhost kernel:                                 [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:39 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:39 localhost kernel:      }
Feb 18 20:57:39 localhost kernel:      ... key      at: [<c2a34488>] 0xc2a34488
Feb 18 20:57:39 localhost kernel:      -> (&rq->rq_lock_key#4){++..} ops: 233519 {
Feb 18 20:57:39 localhost kernel:         initial-use  at:
Feb 18 20:57:39 localhost kernel:                                   [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:39 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:39 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:39 localhost kernel:                                   [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:39 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:39 localhost kernel:                                   [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:39 localhost kernel:                                   [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:57:39 localhost kernel:                                   [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:57:39 localhost kernel:                                   [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:57:39 localhost kernel:                                   [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:57:39 localhost kernel:                                   [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:57:39 localhost kernel:                                   [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:57:39 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:39 localhost kernel:                                   [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:39 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:39 localhost kernel:                                   [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:39 localhost kernel:                                   [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:57:39 localhost kernel:                                   [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:39 localhost kernel:                                   [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:39 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:39 localhost kernel:                                   [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:57:39 localhost kernel:                                   [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:57:39 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:39 localhost kernel:                                   [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:39 localhost kernel:                                   [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:39 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:40 localhost kernel:         in-hardirq-W at:
Feb 18 20:57:40 localhost kernel:                                   [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:57:40 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:40 localhost kernel:                                   [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:40 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:40 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:40 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:40 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:40 localhost kernel:                                   [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:40 localhost kernel:                                   [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:57:40 localhost kernel:                                   [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:57:40 localhost kernel:                                   [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:57:40 localhost kernel:                                   [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:57:40 localhost kernel:                                   [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:57:40 localhost kernel:                                   [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:57:40 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:40 localhost kernel:                                   [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:57:40 localhost kernel:                                   [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:40 localhost kernel:                                   [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:57:40 localhost kernel:                                   [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:57:40 localhost kernel:                                   [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:57:40 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:40 localhost kernel:         in-softirq-W at:
Feb 18 20:57:40 localhost kernel:                                   [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:40 localhost kernel:                                   [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:40 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:40 localhost kernel:                                   [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:40 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:40 localhost kernel:                                   [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:40 localhost kernel:                                   [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:57:40 localhost kernel:                                   [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:57:40 localhost kernel:                                   [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:57:40 localhost kernel:                                   [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:40 localhost kernel:                                   [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:40 localhost kernel:                                   [<ffffffff>] 0xffffffff
Feb 18 20:57:40 localhost kernel:       }
Feb 18 20:57:40 localhost kernel:       ... key      at: [<c2a3d488>] 0xc2a3d488
Feb 18 20:57:40 localhost kernel:      ... acquired at:
Feb 18 20:57:40 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:40 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:40 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:40 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:41 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:41 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:41 localhost kernel:    [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:41 localhost kernel:    [<c04231b1>] __migrate_task+0x45/0xc0
Feb 18 20:57:41 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:41 localhost kernel:    [<c0428988>] migration_thread+0x178/0x1d2
Feb 18 20:57:41 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:41 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:57:41 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:41 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:41 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:41 localhost kernel: 
Feb 18 20:57:41 localhost kernel:     ... acquired at:
Feb 18 20:57:41 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:41 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:41 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:41 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:41 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:41 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:41 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:41 localhost kernel:    [<c04250c8>] try_to_wake_up+0x19/0x3c5
Feb 18 20:57:41 localhost kernel:    [<c0421394>] __wake_up_common+0x32/0x5c
Feb 18 20:57:41 localhost kernel:    [<c04235fe>] complete+0x36/0x44
Feb 18 20:57:41 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:41 localhost kernel:    [<c0428996>] migration_thread+0x186/0x1d2
Feb 18 20:57:41 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:41 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:57:41 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:41 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:41 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:41 localhost kernel: 
Feb 18 20:57:41 localhost kernel:     -> (&rq->rq_lock_key#4){++..} ops: 233519 {
Feb 18 20:57:41 localhost kernel:        initial-use  at:
Feb 18 20:57:41 localhost kernel:                                 [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:41 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:41 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:41 localhost kernel:                                 [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:41 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:41 localhost kernel:                                 [<c0423559>] init_idle+0x77/0xa1
Feb 18 20:57:41 localhost kernel:                                 [<c042b8d9>] fork_idle+0x45/0x4d
Feb 18 20:57:41 localhost kernel:                                 [<c04191c1>] do_boot_cpu+0x3d/0x4b9
Feb 18 20:57:42 localhost kernel:                                 [<c041a8a3>] setup_local_APIC+0x9e/0x28f
Feb 18 20:57:42 localhost kernel:                                 [<c041aa84>] setup_local_APIC+0x27f/0x28f
Feb 18 20:57:42 localhost kernel:                                 [<c075a26f>] verify_local_APIC+0x89/0x137
Feb 18 20:57:42 localhost kernel:                                 [<c07593e0>] native_smp_prepare_cpus+0x34b/0x4a3
Feb 18 20:57:42 localhost kernel:                                 [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:42 localhost kernel:                                 [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:42 localhost kernel:                                 [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:42 localhost kernel:                                 [<c04284c5>] set_cpus_allowed+0x85/0x8d
Feb 18 20:57:42 localhost kernel:                                 [<c0427e1d>] finish_task_switch+0x50/0xbb
Feb 18 20:57:42 localhost kernel:                                 [<c061b505>] _spin_unlock_irq+0x20/0x23
Feb 18 20:57:42 localhost kernel:                                 [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:42 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:42 localhost kernel:                                 [<c074e399>] kernel_init+0x50/0x2af
Feb 18 20:57:42 localhost kernel:                                 [<c0404f5b>] restore_nocheck+0x12/0x15
Feb 18 20:57:42 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:42 localhost kernel:                                 [<c074e349>] kernel_init+0x0/0x2af
Feb 18 20:57:42 localhost kernel:                                 [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:42 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:42 localhost kernel:        in-hardirq-W at:
Feb 18 20:57:42 localhost kernel:                                 [<c0448fc0>] __lock_acquire+0x405/0xbf1
Feb 18 20:57:42 localhost kernel:                                 [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:42 localhost kernel:                                 [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:42 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:42 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:42 localhost kernel:                                 [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:42 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:42 localhost kernel:                                 [<c0427ec7>] scheduler_tick+0x3f/0x19d
Feb 18 20:57:42 localhost kernel:                                 [<c0434365>] update_process_times+0x3a/0x44
Feb 18 20:57:42 localhost kernel:                                 [<c04449ea>] tick_periodic+0x63/0x6d
Feb 18 20:57:42 localhost kernel:                                 [<c0444a0b>] tick_handle_periodic+0x17/0x5c
Feb 18 20:57:42 localhost kernel:                                 [<c0445962>] tick_nohz_stop_sched_tick+0x294/0x2a1
Feb 18 20:57:42 localhost kernel:                                 [<c041af93>] smp_apic_timer_interrupt+0x6f/0x80
Feb 18 20:57:42 localhost kernel:                                 [<c04059e1>] apic_timer_interrupt+0x29/0x38
Feb 18 20:57:42 localhost kernel:                                 [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:42 localhost kernel:                                 [<c04059eb>] apic_timer_interrupt+0x33/0x38
Feb 18 20:57:42 localhost kernel:                                 [<c0403c9b>] default_idle+0x0/0x54
Feb 18 20:57:42 localhost kernel:                                 [<c041007b>] cyrix_get_arr+0xbc/0x126
Feb 18 20:57:42 localhost kernel:                                 [<c0403cd8>] default_idle+0x3d/0x54
Feb 18 20:57:42 localhost kernel:                                 [<c0403583>] cpu_idle+0x9f/0xc0
Feb 18 20:57:42 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:42 localhost kernel:        in-softirq-W at:
Feb 18 20:57:43 localhost kernel:                                 [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:43 localhost kernel:                                 [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:43 localhost kernel:                                 [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:43 localhost kernel:                                 [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:43 localhost kernel:                                 [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:43 localhost kernel:                                 [<c0423141>] double_rq_lock+0x30/0x40
Feb 18 20:57:43 localhost kernel:                                 [<c0425611>] rebalance_domains+0x182/0x393
Feb 18 20:57:43 localhost kernel:                                 [<c0429203>] run_rebalance_domains+0x28/0x99
Feb 18 20:57:43 localhost kernel:                                 [<c0430b8e>] tasklet_action+0x50/0xa4
Feb 18 20:57:43 localhost kernel:                                 [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:43 localhost kernel:                                 [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:43 localhost kernel:                                 [<ffffffff>] 0xffffffff
Feb 18 20:57:43 localhost kernel:      }
Feb 18 20:57:43 localhost kernel:      ... key      at: [<c2a3d488>] 0xc2a3d488
Feb 18 20:57:43 localhost kernel:     ... acquired at:
Feb 18 20:57:43 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:43 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:43 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:43 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:43 localhost kernel:    [<c061b25a>] _spin_lock+0x29/0x34
Feb 18 20:57:43 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:43 localhost kernel:    [<c0423379>] task_rq_lock+0x2d/0x50
Feb 18 20:57:43 localhost kernel:    [<c04250c8>] try_to_wake_up+0x19/0x3c5
Feb 18 20:57:43 localhost kernel:    [<c0421394>] __wake_up_common+0x32/0x5c
Feb 18 20:57:43 localhost kernel:    [<c04235fe>] complete+0x36/0x44
Feb 18 20:57:43 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:43 localhost kernel:    [<c0428996>] migration_thread+0x186/0x1d2
Feb 18 20:57:43 localhost kernel:    [<c0428810>] migration_thread+0x0/0x1d2
Feb 18 20:57:43 localhost kernel:    [<c043d51e>] kthread+0x38/0x5e
Feb 18 20:57:43 localhost kernel:    [<c043d4e6>] kthread+0x0/0x5e
Feb 18 20:57:43 localhost kernel:    [<c0405b83>] kernel_thread_helper+0x7/0x10
Feb 18 20:57:43 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:43 localhost kernel: 
Feb 18 20:57:43 localhost kernel:    ... acquired at:
Feb 18 20:57:43 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:43 localhost kernel:    [<c042366f>] __wake_up+0x18/0x42
Feb 18 20:57:43 localhost kernel:    [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:43 localhost kernel:    [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:57:43 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:43 localhost kernel:    [<c042366f>] __wake_up+0x18/0x42
Feb 18 20:57:43 localhost kernel:    [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:44 localhost kernel:    [<c042366f>] __wake_up+0x18/0x42
Feb 18 20:57:44 localhost kernel:    [<c042366f>] __wake_up+0x18/0x42
Feb 18 20:57:44 localhost kernel:    [<f8c58a2b>] ppp_receive_nonmp_frame+0x51f/0x70a [ppp_generic]
Feb 18 20:57:44 localhost kernel:    [<f8c59487>] ppp_input+0xbf/0xef [ppp_generic]
Feb 18 20:57:44 localhost kernel:    [<f8c60263>] pppol2tp_recv_core+0x75c/0x82d [pppol2tp]
Feb 18 20:57:44 localhost kernel:    [<f8c6037a>] pppol2tp_udp_encap_recv+0x46/0x65 [pppol2tp]
Feb 18 20:57:44 localhost kernel:    [<c061b171>] _read_unlock+0x14/0x1c
Feb 18 20:57:44 localhost kernel:    [<c05ef9df>] udp_queue_rcv_skb+0xba/0x259
Feb 18 20:57:44 localhost kernel:    [<c05efffe>] __udp4_lib_rcv+0x480/0x758
Feb 18 20:57:44 localhost kernel:    [<c05d2c4b>] ip_local_deliver_finish+0x13f/0x1f8
Feb 18 20:57:44 localhost kernel:    [<c05d2b3a>] ip_local_deliver_finish+0x2e/0x1f8
Feb 18 20:57:44 localhost kernel:    [<c05d2ad2>] ip_rcv_finish+0x2fe/0x338
Feb 18 20:57:44 localhost kernel:    [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:57:44 localhost kernel:    [<c05d2dc6>] ip_rcv+0x0/0x237
Feb 18 20:57:44 localhost kernel:    [<c05b5681>] netif_receive_skb+0x373/0x3d4
Feb 18 20:57:44 localhost kernel:    [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:57:44 localhost kernel:    [<f89877c7>] e1000_clean_rx_irq+0x374/0x44a [e1000]
Feb 18 20:57:44 localhost kernel:    [<f8987453>] e1000_clean_rx_irq+0x0/0x44a [e1000]
Feb 18 20:57:44 localhost kernel:    [<f8984f13>] e1000_clean+0x63/0x203 [e1000]
Feb 18 20:57:44 localhost kernel:    [<c05b77f0>] net_rx_action+0xbc/0x1b3
Feb 18 20:57:44 localhost kernel:    [<c05b7782>] net_rx_action+0x4e/0x1b3
Feb 18 20:57:44 localhost kernel:    [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:44 localhost kernel:    [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:44 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:44 localhost kernel: 
Feb 18 20:57:44 localhost kernel:   ... acquired at:
Feb 18 20:57:44 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:44 localhost kernel:    [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:57:44 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:44 localhost kernel:    [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:57:44 localhost kernel:    [<c061b293>] _spin_lock_bh+0x2e/0x39
Feb 18 20:57:44 localhost kernel:    [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:57:44 localhost kernel:    [<f8c57bc0>] ppp_ioctl+0x523/0xc06 [ppp_generic]
Feb 18 20:57:44 localhost kernel:    [<c061b008>] __down+0x82/0xb8
Feb 18 20:57:44 localhost kernel:    [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:44 localhost kernel:    [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:44 localhost kernel:    [<c061b008>] __down+0x82/0xb8
Feb 18 20:57:44 localhost kernel:    [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:57:44 localhost kernel:    [<c061ae17>] __down_failed+0x7/0xc
Feb 18 20:57:44 localhost kernel:    [<c0488854>] do_ioctl+0x4c/0x62
Feb 18 20:57:44 localhost kernel:    [<c0488aa1>] vfs_ioctl+0x237/0x249
Feb 18 20:57:45 localhost kernel:    [<c0488af8>] sys_ioctl+0x45/0x5d
Feb 18 20:57:45 localhost kernel:    [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:57:45 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:45 localhost kernel: 
Feb 18 20:57:45 localhost kernel:   -> (&list->lock#7){.+..} ops: 31 {
Feb 18 20:57:45 localhost kernel:      initial-use  at:
Feb 18 20:57:45 localhost kernel:                             [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:45 localhost kernel:                             [<c046b874>] __do_fault+0x31b/0x35d
Feb 18 20:57:45 localhost kernel:                             [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:45 localhost kernel:                             [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:57:45 localhost kernel:                             [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:45 localhost kernel:                             [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:57:45 localhost kernel:                             [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:57:45 localhost kernel:                             [<f8c5939f>] ppp_write+0xb2/0xdb [ppp_generic]
Feb 18 20:57:45 localhost kernel:                             [<f8c592ed>] ppp_write+0x0/0xdb [ppp_generic]
Feb 18 20:57:45 localhost kernel:                             [<c047e6d2>] vfs_write+0xa1/0x14d
Feb 18 20:57:45 localhost kernel:                             [<c047ecfd>] sys_write+0x41/0x67
Feb 18 20:57:45 localhost kernel:                             [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:57:45 localhost kernel:                             [<ffffffff>] 0xffffffff
Feb 18 20:57:45 localhost kernel:      in-softirq-W at:
Feb 18 20:57:45 localhost kernel:                             [<c0446ab0>] save_trace+0x37/0x8b
Feb 18 20:57:45 localhost kernel:                             [<c0448fdf>] __lock_acquire+0x424/0xbf1
Feb 18 20:57:45 localhost kernel:                             [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:45 localhost kernel:                             [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:45 localhost kernel:                             [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:57:45 localhost kernel:                             [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:45 localhost kernel:                             [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:57:45 localhost kernel:                             [<c05b0641>] skb_queue_tail+0x11/0x2d
Feb 18 20:57:45 localhost kernel:                             [<f8c59433>] ppp_input+0x6b/0xef [ppp_generic]
Feb 18 20:57:45 localhost kernel:                             [<f8c60263>] pppol2tp_recv_core+0x75c/0x82d [pppol2tp]
Feb 18 20:57:45 localhost kernel:                             [<f8c6037a>] pppol2tp_udp_encap_recv+0x46/0x65 [pppol2tp]
Feb 18 20:57:45 localhost kernel:                             [<c061b171>] _read_unlock+0x14/0x1c
Feb 18 20:57:45 localhost kernel:                             [<c05ef9df>] udp_queue_rcv_skb+0xba/0x259
Feb 18 20:57:45 localhost kernel:                             [<c05efffe>] __udp4_lib_rcv+0x480/0x758
Feb 18 20:57:45 localhost kernel:                             [<c05d2c4b>] ip_local_deliver_finish+0x13f/0x1f8
Feb 18 20:57:45 localhost kernel:                             [<c05d2b3a>] ip_local_deliver_finish+0x2e/0x1f8
Feb 18 20:57:45 localhost kernel:                             [<c05d2ad2>] ip_rcv_finish+0x2fe/0x338
Feb 18 20:57:45 localhost kernel:                             [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:57:45 localhost kernel:                             [<c05d2dc6>] ip_rcv+0x0/0x237
Feb 18 20:57:45 localhost kernel:                             [<c05b5681>] netif_receive_skb+0x373/0x3d4
Feb 18 20:57:45 localhost kernel:                             [<c05b5402>] netif_receive_skb+0xf4/0x3d4
Feb 18 20:57:45 localhost kernel:                             [<f89877c7>] e1000_clean_rx_irq+0x374/0x44a [e1000]
Feb 18 20:57:45 localhost kernel:                             [<f8987453>] e1000_clean_rx_irq+0x0/0x44a [e1000]
Feb 18 20:57:46 localhost kernel:                             [<f8984f13>] e1000_clean+0x63/0x203 [e1000]
Feb 18 20:57:46 localhost kernel:                             [<c05b77f0>] net_rx_action+0xbc/0x1b3
Feb 18 20:57:46 localhost kernel:                             [<c05b7782>] net_rx_action+0x4e/0x1b3
Feb 18 20:57:46 localhost kernel:                             [<c0430a9a>] __do_softirq+0x69/0xde
Feb 18 20:57:46 localhost kernel:                             [<c040709f>] do_softirq+0x5e/0xc7
Feb 18 20:57:46 localhost kernel:                             [<ffffffff>] 0xffffffff
Feb 18 20:57:46 localhost kernel:    }
Feb 18 20:57:46 localhost kernel:    ... key      at: [<f8c5dd18>] __key.21159+0x0/0xffffb82a [ppp_generic]
Feb 18 20:57:46 localhost kernel:   ... acquired at:
Feb 18 20:57:46 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:46 localhost kernel:    [<c05b06e1>] skb_dequeue+0xf/0x3f
Feb 18 20:57:46 localhost kernel:    [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:46 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:46 localhost kernel:    [<c05b06e1>] skb_dequeue+0xf/0x3f
Feb 18 20:57:46 localhost kernel:    [<c061b55d>] _spin_lock_irqsave+0x32/0x41
Feb 18 20:57:46 localhost kernel:    [<c05b06e1>] skb_dequeue+0xf/0x3f
Feb 18 20:57:46 localhost kernel:    [<c05b06e1>] skb_dequeue+0xf/0x3f
Feb 18 20:57:46 localhost kernel:    [<f8c57424>] ppp_xmit_process+0x529/0x5a1 [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<f8c5834b>] ppp_channel_push+0x71/0x90 [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<c061b373>] _read_lock_bh+0x2e/0x39
Feb 18 20:57:46 localhost kernel:    [<f8c5835a>] ppp_channel_push+0x80/0x90 [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<f8c593bd>] ppp_write+0xd0/0xdb [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<f8c592ed>] ppp_write+0x0/0xdb [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<c047e6d2>] vfs_write+0xa1/0x14d
Feb 18 20:57:46 localhost kernel:    [<c047ecfd>] sys_write+0x41/0x67
Feb 18 20:57:46 localhost kernel:    [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:57:46 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:46 localhost kernel: 
Feb 18 20:57:46 localhost kernel:  ... acquired at:
Feb 18 20:57:46 localhost kernel:    [<c04495c0>] __lock_acquire+0xa05/0xbf1
Feb 18 20:57:46 localhost kernel:    [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:46 localhost kernel:    [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<c061b293>] _spin_lock_bh+0x2e/0x39
Feb 18 20:57:46 localhost kernel:    [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<f8c57bb5>] ppp_ioctl+0x518/0xc06 [ppp_generic]
Feb 18 20:57:46 localhost kernel:    [<c061b008>] __down+0x82/0xb8
Feb 18 20:57:46 localhost kernel:    [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:46 localhost kernel:    [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:46 localhost kernel:    [<c061b008>] __down+0x82/0xb8
Feb 18 20:57:46 localhost kernel:    [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:57:47 localhost kernel:    [<c061ae17>] __down_failed+0x7/0xc
Feb 18 20:57:47 localhost kernel:    [<c0488854>] do_ioctl+0x4c/0x62
Feb 18 20:57:47 localhost kernel:    [<c0488aa1>] vfs_ioctl+0x237/0x249
Feb 18 20:57:47 localhost kernel:    [<c0488af8>] sys_ioctl+0x45/0x5d
Feb 18 20:57:47 localhost kernel:    [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:57:47 localhost kernel:    [<ffffffff>] 0xffffffff
Feb 18 20:57:47 localhost kernel: 
Feb 18 20:57:47 localhost kernel: 
Feb 18 20:57:47 localhost kernel: the soft-irq-unsafe lock's dependencies:
Feb 18 20:57:47 localhost kernel: -> (&sk->sk_dst_lock){----} ops: 284 {
Feb 18 20:57:47 localhost kernel:    initial-use  at:
Feb 18 20:57:47 localhost kernel:                         [<c0449059>] __lock_acquire+0x49e/0xbf1
Feb 18 20:57:47 localhost kernel:                         [<c0448594>] mark_held_locks+0x39/0x53
Feb 18 20:57:47 localhost kernel:                         [<c043083a>] local_bh_enable+0x10e/0x115
Feb 18 20:57:47 localhost kernel:                         [<c05da996>] inet_csk_get_port+0xc1/0x1cb
Feb 18 20:57:47 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:47 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:47 localhost kernel:                         [<c061b2c7>] _write_lock+0x29/0x34
Feb 18 20:57:47 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:47 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:47 localhost kernel:                         [<c05f66da>] inet_listen+0x3b/0x5e
Feb 18 20:57:47 localhost kernel:                         [<c05ab68f>] sys_listen+0x43/0x5f
Feb 18 20:57:47 localhost kernel:                         [<c05acba8>] sys_socketcall+0xbd/0x261
Feb 18 20:57:47 localhost kernel:                         [<c0404ead>] sysenter_past_esp+0x9a/0xa5
Feb 18 20:57:47 localhost kernel:                         [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:47 localhost kernel:                         [<c0404e72>] sysenter_past_esp+0x5f/0xa5
Feb 18 20:57:47 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:57:47 localhost kernel:    softirq-on-W at:
Feb 18 20:57:47 localhost kernel:                         [<c0449046>] __lock_acquire+0x48b/0xbf1
Feb 18 20:57:47 localhost kernel:                         [<c0448594>] mark_held_locks+0x39/0x53
Feb 18 20:57:47 localhost kernel:                         [<c043083a>] local_bh_enable+0x10e/0x115
Feb 18 20:57:47 localhost kernel:                         [<c05da996>] inet_csk_get_port+0xc1/0x1cb
Feb 18 20:57:47 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:47 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:47 localhost kernel:                         [<c061b2c7>] _write_lock+0x29/0x34
Feb 18 20:57:47 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:47 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:47 localhost kernel:                         [<c05f66da>] inet_listen+0x3b/0x5e
Feb 18 20:57:47 localhost kernel:                         [<c05ab68f>] sys_listen+0x43/0x5f
Feb 18 20:57:47 localhost kernel:                         [<c05acba8>] sys_socketcall+0xbd/0x261
Feb 18 20:57:47 localhost kernel:                         [<c0404ead>] sysenter_past_esp+0x9a/0xa5
Feb 18 20:57:47 localhost kernel:                         [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:48 localhost kernel:                         [<c0404e72>] sysenter_past_esp+0x5f/0xa5
Feb 18 20:57:48 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:57:48 localhost kernel:    hardirq-on-W at:
Feb 18 20:57:48 localhost kernel:                         [<c0449027>] __lock_acquire+0x46c/0xbf1
Feb 18 20:57:48 localhost kernel:                         [<c0448594>] mark_held_locks+0x39/0x53
Feb 18 20:57:48 localhost kernel:                         [<c043083a>] local_bh_enable+0x10e/0x115
Feb 18 20:57:48 localhost kernel:                         [<c05da996>] inet_csk_get_port+0xc1/0x1cb
Feb 18 20:57:48 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:48 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:48 localhost kernel:                         [<c061b2c7>] _write_lock+0x29/0x34
Feb 18 20:57:48 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:48 localhost kernel:                         [<c05da85d>] inet_csk_listen_start+0x75/0xed
Feb 18 20:57:48 localhost kernel:                         [<c05f66da>] inet_listen+0x3b/0x5e
Feb 18 20:57:48 localhost kernel:                         [<c05ab68f>] sys_listen+0x43/0x5f
Feb 18 20:57:48 localhost kernel:                         [<c05acba8>] sys_socketcall+0xbd/0x261
Feb 18 20:57:48 localhost kernel:                         [<c0404ead>] sysenter_past_esp+0x9a/0xa5
Feb 18 20:57:48 localhost kernel:                         [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:48 localhost kernel:                         [<c0404e72>] sysenter_past_esp+0x5f/0xa5
Feb 18 20:57:48 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:57:48 localhost kernel:    softirq-on-R at:
Feb 18 20:57:48 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:48 localhost kernel:                         [<c0449046>] __lock_acquire+0x48b/0xbf1
Feb 18 20:57:48 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:48 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:48 localhost kernel:                         [<c05ada91>] sk_dst_check+0x18/0x105
Feb 18 20:57:48 localhost kernel:                         [<c061b3a7>] _read_lock+0x29/0x34
Feb 18 20:57:48 localhost kernel:                         [<c05ada91>] sk_dst_check+0x18/0x105
Feb 18 20:57:48 localhost kernel:                         [<c05ada91>] sk_dst_check+0x18/0x105
Feb 18 20:57:48 localhost kernel:                         [<f8a6a9b1>] ipv6_chk_addr+0x8a/0x96 [ipv6]
Feb 18 20:57:48 localhost kernel:                         [<f8a649b0>] ip6_sk_dst_lookup+0x2d/0x17a [ipv6]
Feb 18 20:57:48 localhost kernel:                         [<c0489d61>] __pollwait+0x0/0xac
Feb 18 20:57:48 localhost kernel:                         [<f8a75e6d>] udpv6_sendmsg+0x4c1/0x8a1 [ipv6]
Feb 18 20:57:48 localhost kernel:                         [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:57:48 localhost last message repeated 2 times
Feb 18 20:57:48 localhost kernel:                         [<c05f5fc7>] inet_sendmsg+0x3b/0x45
Feb 18 20:57:48 localhost kernel:                         [<c05ab8cd>] sock_sendmsg+0xc9/0xe4
Feb 18 20:57:48 localhost kernel:                         [<c043d5d5>] autoremove_wake_function+0x0/0x35
Feb 18 20:57:48 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:48 localhost kernel:                         [<c04f05f4>] copy_from_user+0x32/0x5e
Feb 18 20:57:49 localhost kernel:                         [<c04f05f4>] copy_from_user+0x32/0x5e
Feb 18 20:57:49 localhost kernel:                         [<c05aba7a>] sys_sendmsg+0x192/0x1f7
Feb 18 20:57:49 localhost kernel:                         [<c048f764>] file_update_time+0x22/0x6a
Feb 18 20:57:49 localhost kernel:                         [<c0483d90>] pipe_write+0x392/0x3ef
Feb 18 20:57:49 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:49 localhost kernel:                         [<c043d5d5>] autoremove_wake_function+0x0/0x35
Feb 18 20:57:49 localhost kernel:                         [<c05acd2b>] sys_socketcall+0x240/0x261
Feb 18 20:57:49 localhost kernel:                         [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:57:49 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:57:49 localhost kernel:    hardirq-on-R at:
Feb 18 20:57:49 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:49 localhost kernel:                         [<c0449001>] __lock_acquire+0x446/0xbf1
Feb 18 20:57:49 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:49 localhost kernel:                         [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:49 localhost kernel:                         [<c05ada91>] sk_dst_check+0x18/0x105
Feb 18 20:57:49 localhost kernel:                         [<c061b3a7>] _read_lock+0x29/0x34
Feb 18 20:57:49 localhost kernel:                         [<c05ada91>] sk_dst_check+0x18/0x105
Feb 18 20:57:49 localhost kernel:                         [<c05ada91>] sk_dst_check+0x18/0x105
Feb 18 20:57:49 localhost kernel:                         [<f8a6a9b1>] ipv6_chk_addr+0x8a/0x96 [ipv6]
Feb 18 20:57:49 localhost kernel:                         [<f8a649b0>] ip6_sk_dst_lookup+0x2d/0x17a [ipv6]
Feb 18 20:57:49 localhost kernel:                         [<c0489d61>] __pollwait+0x0/0xac
Feb 18 20:57:49 localhost kernel:                         [<f8a75e6d>] udpv6_sendmsg+0x4c1/0x8a1 [ipv6]
Feb 18 20:57:49 localhost kernel:                         [<c0425474>] default_wake_function+0x0/0x8
Feb 18 20:57:49 localhost last message repeated 2 times
Feb 18 20:57:49 localhost kernel:                         [<c05f5fc7>] inet_sendmsg+0x3b/0x45
Feb 18 20:57:49 localhost kernel:                         [<c05ab8cd>] sock_sendmsg+0xc9/0xe4
Feb 18 20:57:49 localhost kernel:                         [<c043d5d5>] autoremove_wake_function+0x0/0x35
Feb 18 20:57:49 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:49 localhost kernel:                         [<c04f05f4>] copy_from_user+0x32/0x5e
Feb 18 20:57:49 localhost kernel:                         [<c04f05f4>] copy_from_user+0x32/0x5e
Feb 18 20:57:49 localhost kernel:                         [<c05aba7a>] sys_sendmsg+0x192/0x1f7
Feb 18 20:57:49 localhost kernel:                         [<c048f764>] file_update_time+0x22/0x6a
Feb 18 20:57:49 localhost kernel:                         [<c0483d90>] pipe_write+0x392/0x3ef
Feb 18 20:57:49 localhost kernel:                         [<c0449764>] __lock_acquire+0xba9/0xbf1
Feb 18 20:57:49 localhost kernel:                         [<c043d5d5>] autoremove_wake_function+0x0/0x35
Feb 18 20:57:49 localhost kernel:                         [<c05acd2b>] sys_socketcall+0x240/0x261
Feb 18 20:57:49 localhost kernel:                         [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:57:49 localhost kernel:                         [<ffffffff>] 0xffffffff
Feb 18 20:57:49 localhost kernel:  }
Feb 18 20:57:49 localhost kernel:  ... key      at: [<c0a3dc28>] __key.35996+0x0/0x8
Feb 18 20:57:49 localhost kernel: 
Feb 18 20:57:49 localhost kernel: stack backtrace:
Feb 18 20:57:50 localhost kernel: Pid: 3241, comm: pppd Not tainted 2.6.24.2 #1
Feb 18 20:57:50 localhost kernel:  [<c0448b1e>] check_usage+0x24e/0x258
Feb 18 20:57:50 localhost kernel:  [<c0449555>] __lock_acquire+0x99a/0xbf1
Feb 18 20:57:50 localhost kernel:  [<c044981c>] lock_acquire+0x70/0x8a
Feb 18 20:57:50 localhost kernel:  [<f8c566db>] ppp_push+0x63/0x50d [ppp_generic]
Feb 18 20:57:50 localhost kernel:  [<c061b293>] _spin_lock_bh+0x2e/0x39
Feb 18 20:57:50 localhost kernel:  [<f8c566db>] ppp_push+0x63/0x50d [ppp_generic]
Feb 18 20:57:50 localhost kernel:  [<f8c566db>] ppp_push+0x63/0x50d [ppp_generic]
Feb 18 20:57:50 localhost kernel:  [<c061b622>] _spin_unlock_irqrestore+0x34/0x39
Feb 18 20:57:50 localhost kernel:  [<c0448771>] trace_hardirqs_on+0x10c/0x14c
Feb 18 20:57:50 localhost kernel:  [<f8c573f1>] ppp_xmit_process+0x4f6/0x5a1 [ppp_generic]
Feb 18 20:57:50 localhost kernel:  [<c0448787>] trace_hardirqs_on+0x122/0x14c
Feb 18 20:57:50 localhost kernel:  [<f8c593b4>] ppp_write+0xc7/0xdb [ppp_generic]
Feb 18 20:57:50 localhost kernel:  [<f8c592ed>] ppp_write+0x0/0xdb [ppp_generic]
Feb 18 20:57:50 localhost kernel:  [<c047e6d2>] vfs_write+0xa1/0x14d
Feb 18 20:57:50 localhost kernel:  [<c047ecfd>] sys_write+0x41/0x67
Feb 18 20:57:50 localhost kernel:  [<c0404efa>] syscall_call+0x7/0xb
Feb 18 20:57:50 localhost kernel:  =======================

> 
> Thanks,
> Jarek P.
> 
> On Tue, Feb 12, 2008 at 10:58:21AM +0000, James Chapman wrote:
> ...
>> Here is a trace from when we had _bh locks.
>>
>> Feb  5 16:26:32  ======================================================
>> Feb  5 16:26:32  [ INFO: soft-safe -> soft-unsafe lock order detected ]
>> Feb  5 16:26:32  2.6.24-core2 #1
>> Feb  5 16:26:32  ------------------------------------------------------
>> Feb  5 16:26:32  pppd/3224 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
> ...
>> Feb  5 16:26:32  [<f8d7b84d>] e1000_clean+0x5d/0x290 [e1000]
>> Feb  5 16:26:32  [<c039d580>] net_rx_action+0x1a0/0x2a0
>> Feb  5 16:26:32  [<c039d43f>] net_rx_action+0x5f/0x2a0
>> Feb  5 16:26:32  [<c0131e72>] __do_softirq+0x92/0x120
>> Feb  5 16:26:32  [<c0131f78>] do_softirq+0x78/0x80
>> Feb  5 16:26:32  [<c010b15a>] do_IRQ+0x4a/0xa0
>> Feb  5 16:26:32  [<c0127af0>] finish_task_switch+0x0/0xc0
>> Feb  5 16:26:32  [<c0108dcc>] common_interrupt+0x24/0x34
>> Feb  5 16:26:32  [<c0108dd6>] common_interrupt+0x2e/0x34
>> Feb  5 16:26:32  [<c01062d6>] mwait_idle_with_hints+0x46/0x60
>> Feb  5 16:26:32  [<c0106550>] mwait_idle+0x0/0x20
>> Feb  5 16:26:32  [<c0106694>] cpu_idle+0x74/0xe0
>> Feb  5 16:26:32  [<c0536a9a>] start_kernel+0x30a/0x3a0

-- 
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development


^ permalink raw reply

* Re: [PATCH 1/3] Support arbitrary initial TCP timestamps
From: Glenn Griffin @ 2008-02-18 22:46 UTC (permalink / raw)
  To: David Miller; +Cc: ggriffin.kernel, netdev, linux-kernel
In-Reply-To: <20080217.233330.157255657.davem@davemloft.net>

> Adding yet another member to the already bloated tcp_sock structure to
> implement this is too high a cost.

Yes, I was worried that would be deemed too high of a cost, but it was
the most efficient way I could think to accomplish what I wanted.

> I would instead prefer that there be some global random number
> calculated when the first TCP socket is created, and use that as a
> global offset.  You can even recompute it every few hours if you
> like.

That would work fine if my mine purpose was to randomize the tcp
timestamp to mitigate the leak in information regarding uptime, but
despite the brief description, that's only a side effect of what I
intended to do.  What I wanted was a way to be able to choose an initial
tcp timestamp for a particular connection that was not tied directly to
jiffies.

The two patches following this show my intended use case.  I intend to
enhance syncookie support to allow it to support advanced tcp options
(sack and window scaling).  Normally syncookies encode the bare minimum
state of a connection in the ISN they choose, but the 32bit ISN isn't
enough to encode advanced tcp options so you are left with a working but
crippled tcp stack during a synflood attack.  If in addition to choosing
an ISN we are able to choose an initial tcp timestamp, we are then able
to encode an additional 32 bits of information that can contain more of
the advanced tcp options.

This stems from a discussion about implementing IPv6 support for
syncookies, and the main concern being that syncookies disabled too many
valuable tcp features to be relevant on modern systems.  Many people
stood in opposition to that statement, but it did not seem as though a
general consensus was reached.  http://lkml.org/lkml/2008/2/4/396

I'm always open to alternatives.

--Glenn

^ permalink raw reply

* TG3 network data corruption regression 2.6.24/2.6.23.4
From: Tony Battersby @ 2008-02-18 22:41 UTC (permalink / raw)
  To: Michael Chan, Herbert Xu, David S. Miller, netdev
  Cc: Greg Kroah-Hartman, linux-kernel

I am experiencing network data corruption with a 3Com 3C996B-T NIC
(Broadcom NetXtreme BCM5701; driver tg3.ko).  I have identified the
following patch as the trigger:

commit fb93134dfc2a6e6fbedc7c270a31da03fce88db9
Author: Herbert Xu <herbert@gondor.apana.org.au>
Date:   Wed Nov 14 15:45:21 2007 -0800

    [TCP]: Fix size calculation in sk_stream_alloc_pskb
   
    We round up the header size in sk_stream_alloc_pskb so that
    TSO packets get zero tail room.  Unfortunately this rounding
    up is not coordinated with the select_size() function used by
    TCP to calculate the second parameter of sk_stream_alloc_pskb.
   
    As a result, we may allocate more than a page of data in the
    non-TSO case when exactly one page is desired.
   
    In fact, rounding up the head room is detrimental in the non-TSO
    case because it makes memory that would otherwise be available to
    the payload head room.  TSO doesn't need this either, all it wants
    is the guarantee that there is no tail room.
   
    So this patch fixes this by adjusting the skb_reserve call so that
    exactly the requested amount (which all callers have calculated in
    a precise way) is made available as tail room.
   
    Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
    Signed-off-by: David S. Miller <davem@davemloft.net>

This patch was included in 2.6.24 and 2.6.23.4 -stable.  I am
experiencing data corruption with kernels 2.6.23.4 - 2.6.23.16, 2.6.24 -
2.6.24.2, and 2.6.25-rc2-git1.  I have verified that reverting the above
patch (by hand) makes the data corruption go away on all affected
kernels (note that in 2.6.25 the function is sk_stream_alloc_skb() in
net/ipv4/tcp.c rather than sk_stream_alloc_pskb() in include/net/sock.h).

(Also note that when testing 2.6.23 - 2.6.23.4, I had to apply the
individual patch "TG3: Fix performance regression on 5705." from 2.6.23.5.)

I do not get data corruption when substituting a SysKonnect 9D21 NIC
(which also uses the tg3.ko driver) or a Intel PRO/1000 82546GB NIC
(which uses the e1000.ko driver).

In addition to the 3Com NIC, my computer has a SCSI HBA with an attached
tape drive.  The network data corruption happens only when reading from
or writing to the tape drive.  I have tried both a LSI MPT Fusion
Ultra320 SCSI HBA (mptspi.ko) and a LSI 53c1010 Ultra160 HBA
(sym53c8xx.ko) with the same results.  The NIC and SCSI HBA are on
separate PCI-X buses and do not share IRQs.  I am using two completely
separate test programs to access the SCSI tape drive and test network
data integrity, so one would expect no interaction between the two tests
other than CPU scheduling and DMA bandwidth.  There is no disk I/O
generated by either test program.

The test program that I am using to debug this problem does the following:

Computer A (kernel 2.6.24.2; 3Com 3C996B-T NIC):
   malloc a 64 KB buf aligned to a 4 KB boundary
   loop {
      fill 64 KB buf with count data pattern
      send(64 KB, MSG_MORE) <--- eventually sends corrupted data
   }
   (SCSI tape drive test program runs separately in the background)

Computer B (kernel 2.6.12):
   malloc a 64 KB buf aligned to a 4 KB boundary
   loop {
      recv(64 KB, MSG_WAITALL)
      verify count data pattern in 64 KB buf
   }

After running for a few seconds, the verify on computer B detects data
corruption in the last 4 bytes of the 64 KB buffer.  The last 48 bytes
of the corrupted 64 KB buffer look like this:

D0 D1 D2 D3 | D4 D5 D6 D7 | D8 D9 DA DB | DC DD DE DF
E0 E1 E2 E3 | E4 E5 E6 E7 | E8 E9 EA EB | EC ED EE EF
F0 F1 F2 F3 | F4 F5 F6 F7 | F8 F9 FA FB | F4 F5 F6 F7

The last 4 bytes should be "FC FD FE FF" but instead are corrupted to
"F4 F5 F6 F7", a sequence which came earlier in the data stream.  The
data corruption always occurs at this same buffer offset and with the
same 4 earlier bytes duplicated.  However, it occurs on a different
iteration of the send()/recv() loop each time the test is run.

When I reverse the test so that Computer A does recv() and Computer B
does send(), the test passes with no data corruption.  Therefore, it
appears that the data corruption happens on send() but not recv().

The motherboard that I am using is a Commell LV-672.  This motherboard
has a PCI-express x16 slot but no PCI-X slots.  To plug in the PCI-X NIC
and SCSI HBA, I am using a SuperMicro CSE-RR2UE-AX riser card which
plugs into the PCI-express slot on the motherboard and provides 3 PCI-X
slots (two slots together on one PCI-X bus and one slot on its own PCI-X
bus).  The data corruption happens with every combination of the 2 cards
in the 3 slots.

I assume that the above patch is just exposing some way in which the tg3
driver or the BCM5701 chip are broken.  For now, I am just reverting the
above patch for kernels that I use until a better solution is
forthcoming.  I expect that this problem will be difficult for other
developers to reproduce, but I can test any patches that anyone wants to
send me.  [ In the meantime, should we revert the patch for 2.6.23.x and
2.6.24.x -stable, or wait for a fix to tg3? ]

I am not sure if it is relevant, but I am also getting the following
messages sometimes during testing:

Clocksource tsc unstable (delta = 64002086 ns)
Time: pit clocksource has been installed.

This seems bogus because the CPU is a Intel Pentium 4 with
HyperThreading, so the tsc should be reliable.

---

network MTU = 1500

lspci
00:00.0 Host bridge: Intel Corporation 82915G/P/GV/GL/PL/910GL Memory
Controller Hub (rev 0e)
00:01.0 PCI bridge: Intel Corporation 82915G/P/GV/GL/PL/910GL PCI
Express Root Port (rev 0e)
00:02.0 VGA compatible controller: Intel Corporation 82915G/GV/910GL
Integrated Graphics Controller (rev 0e)
00:1d.0 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB UHCI #1 (rev 04)
00:1d.1 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB UHCI #2 (rev 04)
00:1d.2 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB UHCI #3 (rev 04)
00:1d.3 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB UHCI #4 (rev 04)
00:1d.7 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB2 EHCI Controller (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d4)
00:1f.0 ISA bridge: Intel Corporation 82801FB/FR (ICH6/ICH6R) LPC
Interface Bridge (rev 04)
00:1f.1 IDE interface: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) IDE Controller (rev 04)
00:1f.3 SMBus: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family)
SMBus Controller (rev 04)
01:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge
A (rev 09)
01:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge
B (rev 09)
02:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X
Fusion-MPT Dual Ultra320 SCSI (rev 08)
03:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701
Gigabit Ethernet (rev 15)
04:0d.0 FireWire (IEEE 1394): Agere Systems FW323 (rev 61)

cat /proc/interrupts
           CPU0       CPU1      
  0:         89          0   IO-APIC-edge      timer
  1:         78          0   IO-APIC-edge      i8042
  3:         17          0   IO-APIC-edge      serial
  8:          0          0   IO-APIC-edge      rtc
  9:          0          0   IO-APIC-fasteoi   acpi
 12:          5          0   IO-APIC-edge      i8042
 14:        465          0   IO-APIC-edge      ide0
 16:          0          0   IO-APIC-fasteoi   uhci_hcd:usb5
 17:     149220          0   IO-APIC-fasteoi   eth0
 18:      10007          0   IO-APIC-fasteoi   uhci_hcd:usb4, ioc0
 19:         29          0   IO-APIC-fasteoi   uhci_hcd:usb3
 23:          2          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
NMI:          0          0   Non-maskable interrupts
LOC:       7457      10023   Local timer interrupts
RES:       1962      14316   Rescheduling interrupts
CAL:         40         49   function call interrupts
TLB:         39         76   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0
(eth0 == tg3; ioc0 == LSI SCSI HBA)

ifconfig
eth0      Link encap:Ethernet  HWaddr 00:02:A5:E7:3C:2D 
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:77198 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3488350 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5403873 (5.1 MiB)  TX bytes:1000276920 (953.9 MiB)
          Interrupt:17

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

information from ethtool

driver: tg3
version: 3.87
firmware-version:
bus-info: 0000:03:01.0

Settings for eth0:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Half 1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Half 1000baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: g
    Wake-on: g
    Current message level: 0x000000ff (255)
    Link detected: yes

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off

NIC statistics:
     rx_octets: 5403873
     rx_fragments: 0
     rx_ucast_packets: 77197
     rx_mcast_packets: 0
     rx_bcast_packets: 1
     rx_fcs_errors: 0
     rx_align_errors: 0
     rx_xon_pause_rcvd: 0
     rx_xoff_pause_rcvd: 0
     rx_mac_ctrl_rcvd: 0
     rx_xoff_entered: 0
     rx_frame_too_long_errors: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_in_length_errors: 0
     rx_out_length_errors: 0
     rx_64_or_less_octet_packets: 2
     rx_65_to_127_octet_packets: 77196
     rx_128_to_255_octet_packets: 0
     rx_256_to_511_octet_packets: 0
     rx_512_to_1023_octet_packets: 0
     rx_1024_to_1522_octet_packets: 0
     rx_1523_to_2047_octet_packets: 0
     rx_2048_to_4095_octet_packets: 0
     rx_4096_to_8191_octet_packets: 0
     rx_8192_to_9022_octet_packets: 0
     tx_octets: 1000276920
     tx_collisions: 0
     tx_xon_sent: 0
     tx_xoff_sent: 0
     tx_flow_control: 0
     tx_mac_errors: 0
     tx_single_collisions: 0
     tx_mult_collisions: 0
     tx_deferred: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     tx_collide_2times: 0
     tx_collide_3times: 0
     tx_collide_4times: 0
     tx_collide_5times: 0
     tx_collide_6times: 0
     tx_collide_7times: 0
     tx_collide_8times: 0
     tx_collide_9times: 0
     tx_collide_10times: 0
     tx_collide_11times: 0
     tx_collide_12times: 0
     tx_collide_13times: 0
     tx_collide_14times: 0
     tx_collide_15times: 0
     tx_ucast_packets: 3488350
     tx_mcast_packets: 0
     tx_bcast_packets: 0
     tx_carrier_sense_errors: 0
     tx_discards: 0
     tx_errors: 0
     dma_writeq_full: 0
     dma_write_prioq_full: 0
     rxbds_empty: 0
     rx_discards: 0
     rx_errors: 0
     rx_threshold_hit: 11
     dma_readq_full: 2188114
     dma_read_prioq_full: 162588
     tx_comp_queue_full: 0
     ring_set_send_prod_index: 2901128
     ring_status_update: 218885
     nic_irqs: 146494
     nic_avoided_irqs: 72391
     nic_tx_threshold_hit: 103584

Tony Battersby
Cybernetics


^ permalink raw reply

* Re: [PATCH][PPPOL2TP]: Fix SMP oops in pppol2tp driver
From: Jarek Poplawski @ 2008-02-18 23:01 UTC (permalink / raw)
  To: James Chapman; +Cc: David Miller, netdev
In-Reply-To: <47BA0214.40703@katalix.com>

On Mon, Feb 18, 2008 at 10:09:24PM +0000, James Chapman wrote:
> Jarek Poplawski wrote:
>> Hi,
>>
>> It seems, this nice report is still uncomplete: could you check if
>> there could have been something more yet?
>
> Unfortunately the ISP's syslog stops. But I've been able to borrow two
> Quad Xeon boxes and have reproduced the problem.
>
> Here's a new version of the patch. The patch avoids disabling irqs and
> fixes the sk_dst_get() usage that DaveM mentioned. But even with this
> patch, lockdep still complains if hundreds of ppp sessions are inserted
> into a tunnel as rapidly as possible (lockdep trace is below). I can
> stop these errors by wrapping the call to ppp_input() in
> pppol2tp_recv_dequeue_skb() with local_irq_save/restore. What is a
> better fix?

Hmm... This is a really long report and quite a bit different from
the previous one. I need some time for this. BTW: you sent before a
lockdep report with hlist_lock problem. I think this could be fixed
in some independent patch to make this all more readable. Are all
the other changes in this current patch only because of this or
previous lockdep report or for some other reasons (or reports) yet?

Regards,
Jarek P.

^ permalink raw reply

* [PATCH v2][IPROUTE] tc filters usage fixes
From: Jarek Poplawski @ 2008-02-18 23:26 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: netdev


CLASSID := X:Y:Z ==> X:Y in f_basic is changed here and no change for
f_u32 (it has both CLASSID and FILTERID mentioned).

-----------------> (take 2)

A few usage description fixes of tc filters for some minimal
consistency (FILTER_KIND because of QDISC_KIND).


Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>

---

 tc/f_basic.c   |    4 ++--
 tc/f_rsvp.c    |    2 +-
 tc/tc_filter.c |    6 +++---
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/tc/f_basic.c b/tc/f_basic.c
index ad41633..d8a42d9 100644
--- a/tc/f_basic.c
+++ b/tc/f_basic.c
@@ -30,8 +30,8 @@ static void explain(void)
 	fprintf(stderr, "Usage: ... basic [ match EMATCH_TREE ] [ police POLICE_SPEC ]\n");
 	fprintf(stderr, "                 [ action ACTION_SPEC ] [ classid CLASSID ]\n");
 	fprintf(stderr, "\n");
-	fprintf(stderr, "Where: SELECTOR := SAMPLE SAMPLE ...\n");
-	fprintf(stderr, "       FILTERID := X:Y:Z\n");
+	fprintf(stderr, "Where:\n");
+	fprintf(stderr, "       CLASSID := X:Y\n");
 	fprintf(stderr, "\nNOTE: CLASSID is parsed as hexadecimal input.\n");
 }
 
diff --git a/tc/f_rsvp.c b/tc/f_rsvp.c
index 7e1e6d9..8f92e8f 100644
--- a/tc/f_rsvp.c
+++ b/tc/f_rsvp.c
@@ -33,7 +33,7 @@ static void explain(void)
 	fprintf(stderr, "Where: GPI := { flowlabel NUMBER | spi/ah SPI | spi/esp SPI |\n");
 	fprintf(stderr, "                u{8|16|32} NUMBER mask MASK at OFFSET}\n");
 	fprintf(stderr, "       POLICE_SPEC := ... look at TBF\n");
-	fprintf(stderr, "       FILTERID := X:Y\n");
+	fprintf(stderr, "       CLASSID := X:Y\n");
 	fprintf(stderr, "\nNOTE: CLASSID is parsed as hexadecimal input.\n");
 }
 
diff --git a/tc/tc_filter.c b/tc/tc_filter.c
index d70c656..eb74f89 100644
--- a/tc/tc_filter.c
+++ b/tc/tc_filter.c
@@ -33,12 +33,12 @@ static void usage(void)
 	fprintf(stderr, "Usage: tc filter [ add | del | change | replace | show ] dev STRING\n");
 	fprintf(stderr, "       [ pref PRIO ] protocol PROTO\n");
 	fprintf(stderr, "       [ estimator INTERVAL TIME_CONSTANT ]\n");
-	fprintf(stderr, "       [ root | classid CLASSID ] [ handle FILTERID ]\n");
-	fprintf(stderr, "       [ [ FILTER_TYPE ] [ help | OPTIONS ] ]\n");
+	fprintf(stderr, "       [ root | parent CLASSID ] [ handle FILTERID ]\n");
+	fprintf(stderr, "       [ [ FILTER_KIND ] [ help | OPTIONS ] ]\n");
 	fprintf(stderr, "\n");
 	fprintf(stderr, "       tc filter show [ dev STRING ] [ root | parent CLASSID ]\n");
 	fprintf(stderr, "Where:\n");
-	fprintf(stderr, "FILTER_TYPE := { rsvp | u32 | fw | route | etc. }\n");
+	fprintf(stderr, "FILTER_KIND := { rsvp | u32 | fw | route | etc. }\n");
 	fprintf(stderr, "FILTERID := ... format depends on classifier, see there\n");
 	fprintf(stderr, "OPTIONS := ... try tc filter add <desired FILTER_KIND> help\n");
 	return;

^ permalink raw reply related

* [PATCH v2.6.25] gianfar: don't pass NULL dev ptr to DMA ops
From: Andy Fleming @ 2008-02-18 23:24 UTC (permalink / raw)
  To: jeff, netdev; +Cc: Becky Bruce, Becky Bruce

From: Becky Bruce <bgill@freescale.com>

Change all dma op invocations in gianfar.c to actually pass in the
device pointer.  Currently, the value is ignored, but it will be
used going forward as we implement archdata for 32-bit powerpc.

Signed-off-by: Becky Bruce <becky.bruce@freescale.com>
Acked-by: Andy Fleming <afleming@freescale.com>
---
 drivers/net/gianfar.c |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/net/gianfar.c b/drivers/net/gianfar.c
index 0431e9e..9a5160b 100644
--- a/drivers/net/gianfar.c
+++ b/drivers/net/gianfar.c
@@ -605,7 +605,7 @@ void stop_gfar(struct net_device *dev)
 
 	free_skb_resources(priv);
 
-	dma_free_coherent(NULL,
+	dma_free_coherent(&dev->dev,
 			sizeof(struct txbd8)*priv->tx_ring_size
 			+ sizeof(struct rxbd8)*priv->rx_ring_size,
 			priv->tx_bd_base,
@@ -626,7 +626,7 @@ static void free_skb_resources(struct gfar_private *priv)
 	for (i = 0; i < priv->tx_ring_size; i++) {
 
 		if (priv->tx_skbuff[i]) {
-			dma_unmap_single(NULL, txbdp->bufPtr,
+			dma_unmap_single(&priv->dev->dev, txbdp->bufPtr,
 					txbdp->length,
 					DMA_TO_DEVICE);
 			dev_kfree_skb_any(priv->tx_skbuff[i]);
@@ -643,7 +643,7 @@ static void free_skb_resources(struct gfar_private *priv)
 	if(priv->rx_skbuff != NULL) {
 		for (i = 0; i < priv->rx_ring_size; i++) {
 			if (priv->rx_skbuff[i]) {
-				dma_unmap_single(NULL, rxbdp->bufPtr,
+				dma_unmap_single(&priv->dev->dev, rxbdp->bufPtr,
 						priv->rx_buffer_size,
 						DMA_FROM_DEVICE);
 
@@ -708,7 +708,7 @@ int startup_gfar(struct net_device *dev)
 	gfar_write(&regs->imask, IMASK_INIT_CLEAR);
 
 	/* Allocate memory for the buffer descriptors */
-	vaddr = (unsigned long) dma_alloc_coherent(NULL,
+	vaddr = (unsigned long) dma_alloc_coherent(&dev->dev,
 			sizeof (struct txbd8) * priv->tx_ring_size +
 			sizeof (struct rxbd8) * priv->rx_ring_size,
 			&addr, GFP_KERNEL);
@@ -919,7 +919,7 @@ err_irq_fail:
 rx_skb_fail:
 	free_skb_resources(priv);
 tx_skb_fail:
-	dma_free_coherent(NULL,
+	dma_free_coherent(&dev->dev,
 			sizeof(struct txbd8)*priv->tx_ring_size
 			+ sizeof(struct rxbd8)*priv->rx_ring_size,
 			priv->tx_bd_base,
@@ -1053,7 +1053,7 @@ static int gfar_start_xmit(struct sk_buff *skb, struct net_device *dev)
 
 	/* Set buffer length and pointer */
 	txbdp->length = skb->len;
-	txbdp->bufPtr = dma_map_single(NULL, skb->data,
+	txbdp->bufPtr = dma_map_single(&dev->dev, skb->data,
 			skb->len, DMA_TO_DEVICE);
 
 	/* Save the skb pointer so we can free it later */
@@ -1332,7 +1332,7 @@ struct sk_buff * gfar_new_skb(struct net_device *dev, struct rxbd8 *bdp)
 	 */
 	skb_reserve(skb, alignamount);
 
-	bdp->bufPtr = dma_map_single(NULL, skb->data,
+	bdp->bufPtr = dma_map_single(&dev->dev, skb->data,
 			priv->rx_buffer_size, DMA_FROM_DEVICE);
 
 	bdp->length = 0;
-- 
1.5.4.23.gef5b9


^ permalink raw reply related

* Re: [PATCH][IPROUTE] tc filters usage fixes
From: Jarek Poplawski @ 2008-02-18 23:32 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: netdev
In-Reply-To: <20080218221043.GA6373@ami.dom.local>

Jarek Poplawski wrote, On 02/18/2008 11:10 PM:

> A few usage description fixes of tc filters for some minimal
> consistency (FILTER_KIND because of QDISC_KIND).
> 
> 
> Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>

Don't apply: I've sent 2nd version of this patch.

Sorry,
Jarek P.



^ permalink raw reply

* Re: [PATCH] Add IPv6 support to TCP SYN cookies
From: Glenn Griffin @ 2008-02-18 23:45 UTC (permalink / raw)
  To: ggriffin.kernel
  Cc: dada1, johnpol, alan, andi, netdev, linux-kernel, yoshfuji
In-Reply-To: <20080212.030741.132208013.yoshfuji@linux-ipv6.org>

I've posted a series of patches that I believe address Andi's concerns
about syncookies not supporting valuable tcp options (primarily SACK,
and window scaling).  The premise being if the client support tcp
timestamps we can encode the additional tcp options in the initial
timestamp we send back to the client, and they will be echo'd back to us
in the ack.  Anyone interested have a look, and provide any suggestions
you may have.  The new patches are a superset of this patch, so if they
are accepted this is one obsolete.

Support arbitrary initial TCP timestamps
http://lkml.org/lkml/2008/2/15/244

Enable the use of TCP options with syncookies
http://lkml.org/lkml/2008/2/15/245

Add IPv6 Support to TCP SYN cookies
http://lkml.org/lkml/2008/2/15/246

--Glenn

^ permalink raw reply

* Re: TG3 network data corruption regression 2.6.24/2.6.23.4
From: Michael Chan @ 2008-02-19  0:32 UTC (permalink / raw)
  To: Tony Battersby
  Cc: Herbert Xu, David Miller, netdev, Greg Kroah-Hartman,
	linux-kernel
In-Reply-To: <47BA0984.2070306@cybernetics.com>

On Mon, 2008-02-18 at 17:41 -0500, Tony Battersby wrote:
> I am experiencing network data corruption with a 3Com 3C996B-T NIC
> (Broadcom NetXtreme BCM5701; driver tg3.ko).  I have identified the
> following patch as the trigger:

Assuming this problem is unique to the 5701, I'm not sure how it is
exposed by Herbert's patch.  One thing unique on the 5701 is that it
double-copies all RX packets so that the data starts at offset 2, but
that's quite unrelated to the patch below.

> 
> commit fb93134dfc2a6e6fbedc7c270a31da03fce88db9
> Author: Herbert Xu <herbert@gondor.apana.org.au>
> Date:   Wed Nov 14 15:45:21 2007 -0800
> 
>     [TCP]: Fix size calculation in sk_stream_alloc_pskb
>    
> 

> I do not get data corruption when substituting a SysKonnect 9D21 NIC
> (which also uses the tg3.ko driver)

What Broadcom chip is on the Syskonnect card?




^ permalink raw reply

* Re: TG3 network data corruption regression 2.6.24/2.6.23.4
From: David Miller @ 2008-02-19  0:35 UTC (permalink / raw)
  To: mchan; +Cc: tonyb, herbert, netdev, gregkh, linux-kernel
In-Reply-To: <1203381120.13495.78.camel@dell>

From: "Michael Chan" <mchan@broadcom.com>
Date: Mon, 18 Feb 2008 16:32:00 -0800

> On Mon, 2008-02-18 at 17:41 -0500, Tony Battersby wrote:
> > I am experiencing network data corruption with a 3Com 3C996B-T NIC
> > (Broadcom NetXtreme BCM5701; driver tg3.ko).  I have identified the
> > following patch as the trigger:
> 
> Assuming this problem is unique to the 5701, I'm not sure how it is
> exposed by Herbert's patch.  One thing unique on the 5701 is that it
> double-copies all RX packets so that the data starts at offset 2, but
> that's quite unrelated to the patch below.

One consequence of Herbert's change is that the chip will see a
different datastream.  The initial skb->data linear area will be
smaller, and the transition to the fragmented area of pages will be
quicker.

^ permalink raw reply

* Re: [PATCH] net/8021q/vlan_dev.c - Use print_mac
From: David Miller @ 2008-02-19  0:43 UTC (permalink / raw)
  To: kaber-dcUjhNyLwpNeoWH0uzbU5w
  Cc: joe-6d6DIl74uiNBDgjK7y7TUQ, bruno-L9ZBdB2wSWtl57MIdRCFDg,
	netdev-u79uwXL29TY76Z2rM5mHXA, jgarzik-e+AXbWqSrlAAvxtiuMwx3w,
	linux-wireless-u79uwXL29TY76Z2rM5mHXA,
	linville-2XuSBdqkA4R54TAoqtyWWQ
In-Reply-To: <47B9F5E7.3020905-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>

From: Patrick McHardy <kaber-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
Date: Mon, 18 Feb 2008 22:17:27 +0100

> The way pr_debug is implemented it still results in two function
> calls per packet since the compiler doesn't know that it doesn't
> have visible side-effects besides modifying the (unused) buffer.
> I confirmed this using codiff.

That's a bug.

I think we can fix this easily by using __attribute_const_
on the print_mac() declaration.  Let me play with that.

^ permalink raw reply

* Re: [PATCH] net/8021q/vlan_dev.c - Use print_mac
From: David Miller @ 2008-02-19  0:50 UTC (permalink / raw)
  To: kaber-dcUjhNyLwpNeoWH0uzbU5w
  Cc: joe-6d6DIl74uiNBDgjK7y7TUQ, bruno-L9ZBdB2wSWtl57MIdRCFDg,
	netdev-u79uwXL29TY76Z2rM5mHXA, jgarzik-e+AXbWqSrlAAvxtiuMwx3w,
	linux-wireless-u79uwXL29TY76Z2rM5mHXA,
	linville-2XuSBdqkA4R54TAoqtyWWQ
In-Reply-To: <20080218.164305.67586867.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>

From: David Miller <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
Date: Mon, 18 Feb 2008 16:43:05 -0800 (PST)

> I think we can fix this easily by using __attribute_const_
> on the print_mac() declaration.  Let me play with that.

Actually it seems the 'pure' attribute is more important
here.  Although it's not semantically a perfect match,
what we need to tell the compiler is basically that:

1) the return value depends upon the inputs
2) if the input is not used, it's safe to avoid the call

and 'pure' accomplishes that without any unwanted side-effects.

I think this will not result in any unwanted over-optimization.
Because if the inputs change in any way GCC has to emit the
call.

Any objections?

commit 8f789c48448aed74fe1c07af76de8f04adacec7d
Author: David S. Miller <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
Date:   Mon Feb 18 16:50:22 2008 -0800

    [NET]: Elminate spurious print_mac() calls.
    
    Patrick McHardy notes that print_mac() can get invoked
    even if the result it unused (f.e. as an argument to
    pr_debug() when DEBUG is not defined).
    
    Mark this function as "__pure" to eliminate this problem.
    
    Signed-off-by: David S. Miller <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>

diff --git a/include/linux/if_ether.h b/include/linux/if_ether.h
index 7a1e011..42dc6a3 100644
--- a/include/linux/if_ether.h
+++ b/include/linux/if_ether.h
@@ -129,7 +129,7 @@ extern ssize_t sysfs_format_mac(char *buf, const unsigned char *addr, int len);
 /*
  *	Display a 6 byte device address (MAC) in a readable format.
  */
-extern char *print_mac(char *buf, const unsigned char *addr);
+extern __pure char *print_mac(char *buf, const unsigned char *addr);
 #define MAC_BUF_SIZE	18
 #define DECLARE_MAC_BUF(var) char var[MAC_BUF_SIZE] __maybe_unused
 

^ permalink raw reply related

* Re: TG3 network data corruption regression 2.6.24/2.6.23.4
From: Michael Chan @ 2008-02-19  1:04 UTC (permalink / raw)
  To: David Miller; +Cc: tonyb, herbert, netdev, gregkh, linux-kernel
In-Reply-To: <20080218.163554.74130592.davem@davemloft.net>

On Mon, 2008-02-18 at 16:35 -0800, David Miller wrote:

> One consequence of Herbert's change is that the chip will see a
> different datastream.  The initial skb->data linear area will be
> smaller, and the transition to the fragmented area of pages will be
> quicker.
> 

I see.  Perhaps when we get to the end of the data-stream, there is a
tiny frag that the chip cannot handle.  That's the only thing I can
think of.

Please try this patch to see if the problem goes away.  This will
disable SG on 5701 so we always get linear SKBs.

diff --git a/drivers/net/tg3.c b/drivers/net/tg3.c
index db606b6..bb37e76 100644
--- a/drivers/net/tg3.c
+++ b/drivers/net/tg3.c
@@ -12717,6 +12717,9 @@ static int __devinit tg3_init_one(struct pci_dev *pdev,
 	} else
 		tp->tg3_flags &= ~TG3_FLAG_RX_CHECKSUMS;
 
+	if (GET_ASIC_REV(tp->pci_chip_rev_id) == ASIC_REV_5701)
+		dev->features &= ~(NETIF_F_IP_CSUM | NETIF_F_SG);
+
 	/* flow control autonegotiation is default behavior */
 	tp->tg3_flags |= TG3_FLAG_PAUSE_AUTONEG;
 	tp->link_config.flowctrl = TG3_FLOW_CTRL_TX | TG3_FLOW_CTRL_RX;



^ permalink raw reply related


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox