From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Yongjun Date: Wed, 03 Feb 2010 09:51:56 +0000 Subject: Re: [PATCH] sctp: avoid irq lock inversion while call sk->sk_data_ready() Message-Id: <4B69473C.6060100@cn.fujitsu.com> List-Id: References: <4B6903D6.8070106@cn.fujitsu.com> In-Reply-To: <4B6903D6.8070106@cn.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: linux-sctp@vger.kernel.org > > sk->sk_data_ready() of sctp socket can be called from both BH and non-BH > contexts, but the default sk->sk_data_ready(), sock_def_readable(), can > not be used in this case. Therefore, we have to make a new function > sctp_data_ready() to grab sk->sk_data_ready() with BH disabling. > > ============================> [ INFO: possible irq lock inversion dependency detected ] > 2.6.33-rc6 #129 > --------------------------------------------------------- > sctp_darn/1517 just changed the state of lock: > (clock-AF_INET){++.?..}, at: [] sock_def_readable+0x20/0x80 > but this lock took another, SOFTIRQ-unsafe lock in the past: > (slock-AF_INET){+.-...} > > and interrupts could create inverse lock ordering between them. > > other info that might help us debug this: > 1 lock held by sctp_darn/1517: > #0: (sk_lock-AF_INET){+.+.+.}, at: [] sctp_sendmsg+0x23d/0xc00 [sctp] > The full lockdep output message is: ============================[ INFO: possible irq lock inversion dependency detected ] 2.6.33-rc6 #129 --------------------------------------------------------- sctp_darn/1517 just changed the state of lock: (clock-AF_INET){++.?..}, at: [] sock_def_readable+0x20/0x80 but this lock took another, SOFTIRQ-unsafe lock in the past: (slock-AF_INET){+.-...} and interrupts could create inverse lock ordering between them. other info that might help us debug this: 1 lock held by sctp_darn/1517: #0: (sk_lock-AF_INET){+.+.+.}, at: [] sctp_sendmsg+0x23d/0xc00 [sctp] the shortest dependencies between 2nd lock and 1st lock: -> (slock-AF_INET){+.-...} ops: 0 { HARDIRQ-ON-W at: [] __lock_acquire+0x9af/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_spin_lock_bh+0x3d/0x50 [] lock_sock_nested+0x33/0xf0 [] tcp_close+0x1a/0x390 [] inet_release+0x3b/0x60 [] sock_release+0x20/0x70 [] sock_close+0x17/0x30 [] __fput+0xfb/0x200 [] fput+0x1d/0x30 [] filp_close+0x4c/0x80 [] sys_close+0x77/0xc0 [] sysenter_do_call+0x12/0x32 IN-SOFTIRQ-W at: [] __lock_acquire+0x993/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_spin_lock+0x38/0x50 [] udp_queue_rcv_skb+0xee/0x2c0 [] __udp4_lib_rcv+0x1bf/0x6a0 [] udp_rcv+0x17/0x20 [] ip_local_deliver_finish+0xdf/0x2c0 [] ip_local_deliver+0x8f/0xa0 [] ip_rcv_finish+0xdb/0x3c0 [] ip_rcv+0x206/0x2c0 [] netif_receive_skb+0x34f/0x570 [] pcnet32_poll+0x27e/0x7a0 [pcnet32] [] net_rx_action+0x150/0x230 [] __do_softirq+0xa0/0x1c0 INITIAL USE at: [] __lock_acquire+0x36f/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_spin_lock_bh+0x3d/0x50 [] lock_sock_nested+0x33/0xf0 [] tcp_close+0x1a/0x390 [] inet_release+0x3b/0x60 [] sock_release+0x20/0x70 [] sock_close+0x17/0x30 [] __fput+0xfb/0x200 [] fput+0x1d/0x30 [] filp_close+0x4c/0x80 [] sys_close+0x77/0xc0 [] sysenter_do_call+0x12/0x32 } ... key at: [] af_family_slock_keys+0x10/0x140 ... acquired at: [] __lock_acquire+0x1172/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_write_lock_bh+0x3d/0x50 [] sk_common_release+0x2d/0xb0 [] sctp_close+0xe8/0x1f0 [sctp] [] inet_release+0x3b/0x60 [] sock_release+0x20/0x70 [] sock_close+0x17/0x30 [] __fput+0xfb/0x200 [] fput+0x1d/0x30 [] filp_close+0x4c/0x80 [] sys_close+0x77/0xc0 [] sysenter_do_call+0x12/0x32 -> (clock-AF_INET){++.?..} ops: 0 { HARDIRQ-ON-W at: [] __lock_acquire+0x9af/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_write_lock_bh+0x3d/0x50 [] tcp_close+0xf9/0x390 [] inet_release+0x3b/0x60 [] sock_release+0x20/0x70 [] sock_close+0x17/0x30 [] __fput+0xfb/0x200 [] fput+0x1d/0x30 [] filp_close+0x4c/0x80 [] sys_close+0x77/0xc0 [] sysenter_do_call+0x12/0x32 HARDIRQ-ON-R at: [] __lock_acquire+0x186/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_read_lock+0x38/0x50 [] sock_def_write_space+0x1c/0xb0 [] sock_wfree+0x4a/0x60 [] skb_release_head_state+0x45/0xc0 [] __kfree_skb+0x10/0x90 [] net_tx_action+0x59/0x140 [] __do_softirq+0xa0/0x1c0 IN-SOFTIRQ-R at: [] __lock_acquire+0x993/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_read_lock+0x38/0x50 [] sock_def_write_space+0x1c/0xb0 [] sock_wfree+0x4a/0x60 [] skb_release_head_state+0x45/0xc0 [] __kfree_skb+0x10/0x90 [] net_tx_action+0x59/0x140 [] __do_softirq+0xa0/0x1c0 SOFTIRQ-ON-R at: [] __lock_acquire+0x9d4/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_read_lock+0x38/0x50 [] sock_def_readable+0x20/0x80 [] sctp_ulpq_tail_event+0x134/0x210 [sctp] [] sctp_side_effects+0x8ee/0x10c0 [sctp] [] sctp_do_sm+0xb0/0x1c0 [sctp] [] sctp_primitive_ABORT+0x42/0x50 [sctp] [] sctp_sendmsg+0x492/0xc00 [sctp] [] inet_sendmsg+0x2e/0x60 [] sock_sendmsg+0xe7/0x110 [] sys_sendmsg+0x113/0x230 [] sys_socketcall+0xeb/0x2a0 [] sysenter_do_call+0x12/0x32 INITIAL USE at: [] __lock_acquire+0x36f/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_write_lock_bh+0x3d/0x50 [] tcp_close+0xf9/0x390 [] inet_release+0x3b/0x60 [] sock_release+0x20/0x70 [] sock_close+0x17/0x30 [] __fput+0xfb/0x200 [] fput+0x1d/0x30 [] filp_close+0x4c/0x80 [] sys_close+0x77/0xc0 [] sysenter_do_call+0x12/0x32 } ... key at: [] af_callback_keys+0x10/0x128 ... acquired at: [] check_usage_backwards+0x8b/0xd0 [] mark_lock+0x1ba/0x5c0 [] __lock_acquire+0x9d4/0x1890 [] lock_acquire+0x7f/0xf0 [] _raw_read_lock+0x38/0x50 [] sock_def_readable+0x20/0x80 [] sctp_ulpq_tail_event+0x134/0x210 [sctp] [] sctp_side_effects+0x8ee/0x10c0 [sctp] [] sctp_do_sm+0xb0/0x1c0 [sctp] [] sctp_primitive_ABORT+0x42/0x50 [sctp] [] sctp_sendmsg+0x492/0xc00 [sctp] [] inet_sendmsg+0x2e/0x60 [] sock_sendmsg+0xe7/0x110 [] sys_sendmsg+0x113/0x230 [] sys_socketcall+0xeb/0x2a0 [] sysenter_do_call+0x12/0x32 stack backtrace: Pid: 1517, comm: sctp_darn Not tainted 2.6.33-rc6 #129 Call Trace: [] ? printk+0x1d/0x21 [] print_irq_inversion_bug.clone.0+0xfe/0x110 [] check_usage_backwards+0x8b/0xd0 [] mark_lock+0x1ba/0x5c0 [] ? string+0x33/0xe0 [] ? check_usage_backwards+0x0/0xd0 [] __lock_acquire+0x9d4/0x1890 [] ? string+0x33/0xe0 [] ? _raw_spin_lock_irqsave+0x1b/0x60 [] ? _raw_spin_unlock_irqrestore+0x4f/0x60 [] ? trace_hardirqs_off+0xb/0x10 [] ? _raw_spin_unlock_irqrestore+0x4f/0x60 [] ? release_console_sem+0x1ef/0x240 [] lock_acquire+0x7f/0xf0 [] ? sock_def_readable+0x20/0x80 [] _raw_read_lock+0x38/0x50 [] ? sock_def_readable+0x20/0x80 [] sock_def_readable+0x20/0x80 [] sctp_ulpq_tail_event+0x134/0x210 [sctp] [] sctp_side_effects+0x8ee/0x10c0 [sctp] [] ? _raw_spin_lock_irqsave+0x1b/0x60 [] sctp_do_sm+0xb0/0x1c0 [sctp] [] ? release_console_sem+0x1ef/0x240 [] sctp_primitive_ABORT+0x42/0x50 [sctp] [] sctp_sendmsg+0x492/0xc00 [sctp] [] inet_sendmsg+0x2e/0x60 [] sock_sendmsg+0xe7/0x110 [] ? trace_hardirqs_off+0xb/0x10 [] ? might_fault+0x50/0xa0 [] ? might_fault+0x50/0xa0 [] ? might_fault+0x96/0xa0 [] ? might_fault+0x50/0xa0 [] ? _copy_from_user+0x3d/0x130 [] sys_sendmsg+0x113/0x230 [] ? release_sock+0xd7/0xe0 [] ? trace_hardirqs_on+0xb/0x10 [] ? local_bh_enable_ip+0x68/0xd0 [] ? sctp_getsockopt+0x9c/0x1010 [sctp] [] ? lock_release_non_nested+0x59/0x2f0 [] ? trace_hardirqs_on+0xb/0x10 [] ? put_ldisc+0x3e/0xc0 [] ? might_fault+0x50/0xa0 [] ? might_fault+0x50/0xa0 [] sys_socketcall+0xeb/0x2a0 [] ? sysenter_exit+0xf/0x16 [] sysenter_do_call+0x12/0x32