From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: kernel panic, inconsistent lock, vanilla 3.3.4, 32bit, tcp Date: Sun, 29 Apr 2012 09:32:42 +0200 Message-ID: <1335684762.2900.69.camel@edumazet-glaptop> References: <04009a9cbb27bf97cb7d86612191b751@visp.net.lb> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: netdev@vger.kernel.org, Glauber Costa To: Denys Fedoryshchenko Return-path: Received: from mail-we0-f174.google.com ([74.125.82.174]:43209 "EHLO mail-we0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752222Ab2D2Hcs (ORCPT ); Sun, 29 Apr 2012 03:32:48 -0400 Received: by wejx9 with SMTP id x9so1235633wej.19 for ; Sun, 29 Apr 2012 00:32:46 -0700 (PDT) In-Reply-To: <04009a9cbb27bf97cb7d86612191b751@visp.net.lb> Sender: netdev-owner@vger.kernel.org List-ID: On Sun, 2012-04-29 at 00:45 +0300, Denys Fedoryshchenko wrote: > Hi > > Just got repeatable kernel panic on loaded proxy server > > Here is the log: > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.903868] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.903880] > ================================= > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.903890] [ INFO: inconsistent > lock state ] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.903903] 3.3.4-build-0061 #8 > Not tainted > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904133] > --------------------------------- > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904344] inconsistent > {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage. > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] squid/1603 > [HC0[0]:SC0[0]:HE1:SE1] takes: > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] (key#3){+.?...}, at: > [] __percpu_counter_sum+0xd/0x58 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] {IN-SOFTIRQ-W} state > was registered at: > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > __lock_acquire+0x284/0xc26 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > lock_acquire+0x71/0x85 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > _raw_spin_lock+0x33/0x40 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > __percpu_counter_add+0x58/0x7c > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > sk_clone_lock+0x1e5/0x200 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > inet_csk_clone_lock+0xe/0x78 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > tcp_create_openreq_child+0x1b/0x404 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > tcp_v4_syn_recv_sock+0x32/0x1c1 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > tcp_check_req+0x1fd/0x2d7 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > tcp_v4_do_rcv+0xab/0x194 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > tcp_v4_rcv+0x3b3/0x5cc > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > ip_local_deliver_finish+0x13a/0x1e9 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > NF_HOOK.clone.11+0x46/0x4d > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > ip_local_deliver+0x41/0x45 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > ip_rcv_finish+0x31a/0x33c > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > NF_HOOK.clone.11+0x46/0x4d > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > ip_rcv+0x201/0x23e > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > __netif_receive_skb+0x319/0x368 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > netif_receive_skb+0x4e/0x7d > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > napi_skb_finish+0x1e/0x34 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > napi_gro_receive+0x20/0x24 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > e1000_receive_skb+0x3f/0x45 [e1000e] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > e1000_clean_rx_irq+0x1f9/0x284 [e1000e] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > e1000_clean+0x62/0x1f4 [e1000e] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > net_rx_action+0x90/0x160 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > __do_softirq+0x7b/0x118 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] irq event stamp: > 156915469 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] hardirqs last enabled > at (156915469): [] __slab_alloc.clone.58.clone.63+0xc4/0x2de > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] hardirqs last disabled > at (156915468): [] __slab_alloc.clone.58.clone.63+0x22/0x2de > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] softirqs last enabled > at (156915466): [] lock_sock_nested+0x64/0x6c > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] softirqs last disabled > at (156915464): [] _raw_spin_lock_bh+0xe/0x45 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] other info that might > help us debug this: > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] Possible unsafe > locking scenario: > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] CPU0 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] ---- > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] lock(key#3); > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] lock(key#3); > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] *** DEADLOCK *** > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] 1 lock held by > squid/1603: > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] #0: > (sk_lock-AF_INET){+.+.+.}, at: [] lock_sock+0xa/0xc > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] stack backtrace: > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] Pid: 1603, comm: squid > Not tainted 3.3.4-build-0061 #8 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] Call Trace: > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > printk+0x18/0x1d > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > valid_state+0x1f6/0x201 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > mark_lock+0xd1/0x1bb > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > mark_lock+0x26/0x1bb > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > check_usage_forwards+0x77/0x77 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > __lock_acquire+0x2f8/0xc26 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > mark_held_locks+0x5d/0x7b > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > trace_hardirqs_on+0xb/0xd > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > __lock_acquire+0x4d4/0xc26 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > lock_acquire+0x71/0x85 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > __percpu_counter_sum+0xd/0x58 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > _raw_spin_lock+0x33/0x40 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > __percpu_counter_sum+0xd/0x58 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > __percpu_counter_sum+0xd/0x58 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > __sk_mem_schedule+0xdd/0x1c7 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > __alloc_skb+0x76/0x100 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > sk_wmem_schedule+0x21/0x2d > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > sk_stream_alloc_skb+0x42/0xaa > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > tcp_sendmsg+0x18f/0x68b > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > ip_fast_csum+0x30/0x30 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > inet_sendmsg+0x53/0x5a > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > sock_aio_write+0xd2/0xda > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > mark_lock+0x26/0x1bb > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > do_sync_write+0x9f/0xd9 > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > file_free_rcu+0x2f/0x2f > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > vfs_write+0x8f/0xab > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > fget_light+0x75/0x7c > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > sys_write+0x3d/0x5e > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] > syscall_call+0x7/0xb > Apr 29 00:38:17 10.0.15.2 kernel: [ 762.904542] [] ? > rp_sidt+0x41/0x83 > > > --- > Denys Fedoryshchenko Seems to be related to Glauber Costa work sk_memory_allocated_add() seems to assume __percpu_counter_add() is irq safe (truth is : it is not safe)