Beatrice Barbe reported a reproducible crash after creating large
numbers of nested GRE tunnels and then pinging with the source address
forced.  I was able to reproduce this using net-2.6.  I'm attaching the
kernel config I used and a script to reproduce this based on the script
she provided.  The magic number of tunnels to create is apparently 37.

With lockdep enabled, I get the following output:

=============================================
[ INFO: possible recursive locking detected ]
2.6.36-rc7-00040-gb0057c5 #5
---------------------------------------------
ping/2199 is trying to acquire lock:
 (_xmit_IPGRE){+.....}, at: [<c1139968>] dev_queue_xmit+0x37e/0x454

but task is already holding lock:
 (_xmit_IPGRE){+.....}, at: [<c1139968>] dev_queue_xmit+0x37e/0x454

other info that might help us debug this:
4 locks held by ping/2199:
 #0:  (sk_lock-AF_INET){+.+.+.}, at: [<c1168c46>] raw_sendmsg+0x590/0x64c
 #1:  (rcu_read_lock_bh){.+....}, at: [<c11395ea>] dev_queue_xmit+0x0/0x454
 #2:  (_xmit_IPGRE){+.....}, at: [<c1139968>] dev_queue_xmit+0x37e/0x454
 #3:  (rcu_read_lock_bh){.+....}, at: [<c11395ea>] dev_queue_xmit+0x0/0x454

stack backtrace:
Pid: 2199, comm: ping Not tainted 2.6.36-rc7-00040-gb0057c5 #5
Call Trace:
 [<c1187b3c>] ? printk+0xf/0x13
 [<c103a942>] __lock_acquire+0xbda/0x1311
 [<c103a32b>] ? __lock_acquire+0x5c3/0x1311
 [<c103b0d2>] lock_acquire+0x59/0x77
 [<c1139968>] ? dev_queue_xmit+0x37e/0x454
 [<c11898b4>] _raw_spin_lock+0x1b/0x2a
 [<c1139968>] ? dev_queue_xmit+0x37e/0x454
 [<c1139968>] dev_queue_xmit+0x37e/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c1151382>] ? ip_append_data+0x536/0x7dc
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c1151851>] ? ip_generic_getfrag+0x0/0x8a
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c1150dff>] ip_push_pending_frames+0x260/0x2ad
 [<c1168c85>] raw_sendmsg+0x5cf/0x64c
 [<c11708ad>] inet_sendmsg+0x46/0x4f
 [<c112cea9>] sock_sendmsg+0xa4/0xba
 [<c105897d>] ? might_fault+0x35/0x6f
 [<c105897d>] ? might_fault+0x35/0x6f
 [<c1134614>] ? verify_iovec+0x3e/0x6a
 [<c112d2af>] sys_sendmsg+0x149/0x196
 [<c104b079>] ? unlock_page+0x3f/0x42
 [<c103b176>] ? lock_release_non_nested+0x86/0x221
 [<c105897d>] ? might_fault+0x35/0x6f
 [<c105897d>] ? might_fault+0x35/0x6f
 [<c112e287>] sys_socketcall+0x146/0x18b
 [<c10cb5c8>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1189f5d>] syscall_call+0x7/0xb
------------[ cut here ]------------
WARNING: at kernel/softirq.c:143 local_bh_enable_ip+0x39/0xa5()
Hardware name: Bochs
Pid: 2199, comm: ping Not tainted 2.6.36-rc7-00040-gb0057c5 #5
Call Trace:
 [<c101a092>] warn_slowpath_common+0x60/0x75
 [<c101e534>] ? local_bh_enable_ip+0x39/0xa5
 [<c114b993>] ? rt_intern_hash+0x4da/0x4f9
 [<c101a0b6>] warn_slowpath_null+0xf/0x13
 [<c101e534>] local_bh_enable_ip+0x39/0xa5
 [<c1189d4e>] _raw_spin_unlock_bh+0x25/0x28
 [<c114b993>] rt_intern_hash+0x4da/0x4f9
 [<c114c1b8>] __ip_route_output_key+0x806/0x860
 [<c114c220>] ip_route_output_flow+0xe/0x3e
 [<c114c25c>] ip_route_output_key+0xc/0xe
 [<c11793d6>] ipgre_tunnel_xmit+0x1ac/0x757
 [<c1139968>] ? dev_queue_xmit+0x37e/0x454
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c11798e2>] ipgre_tunnel_xmit+0x6b8/0x757
 [<c114a188>] ? ip_rt_update_pmtu+0x0/0x60
 [<c11394fc>] dev_hard_start_xmit+0x33a/0x428
 [<c1139987>] dev_queue_xmit+0x39d/0x454
 [<c1151382>] ? ip_append_data+0x536/0x7dc
 [<c115292e>] ip_finish_output+0x29d/0x2c7
 [<c1151851>] ? ip_generic_getfrag+0x0/0x8a
 [<c11529e2>] ip_output+0x8a/0x8f
 [<c1150b9c>] ip_local_out+0x50/0x53
 [<c1150dff>] ip_push_pending_frames+0x260/0x2ad
 [<c1168c85>] raw_sendmsg+0x5cf/0x64c
 [<c11708ad>] inet_sendmsg+0x46/0x4f
 [<c112cea9>] sock_sendmsg+0xa4/0xba
 [<c105897d>] ? might_fault+0x35/0x6f
 [<c105897d>] ? might_fault+0x35/0x6f
 [<c1134614>] ? verify_iovec+0x3e/0x6a
 [<c112d2af>] sys_sendmsg+0x149/0x196
 [<c104b079>] ? unlock_page+0x3f/0x42
 [<c103b176>] ? lock_release_non_nested+0x86/0x221
 [<c105897d>] ? might_fault+0x35/0x6f
 [<c105897d>] ? might_fault+0x35/0x6f
 [<c112e287>] sys_socketcall+0x146/0x18b
 [<c10cb5c8>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1189f5d>] syscall_call+0x7/0xb
 <IRQ> 

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.