Re: lockdep false positive? -- firewire-core transaction timer vs. scsi-core host lock

[Yong Zhang <yong.zhang0@gmail.com>]

On Mon, Aug 16, 2010 at 08:42:34PM +0200, Stefan Richter wrote:
> Hi, I just saw the following (twice, with a reboot in between) with
> kernel 2.6.35 plus firewire driver patched to their state in 2.6.36-rc1,
> i.e. basically with a kernel according to commit e78483c5.
> 
> I suspect it might also happen without firewire patches, as 2.6.35
> already contains the change where firewire-core's per-transaction timer
> was introduced which shows up in the following log.

Cc'ing Thomas.

I suspect it's introduced by commit 5c40cbfefa828208c671e2f58789e4dd04f79563
which call del_timer_sync() in softirq.

comments on del_timer_sync() say "It must not be called from interrupt contexts."

Thanks,
Yong
> 
> It happened while I was testing hotplugging of FireWire disks with
> several ones on a bus together with several FireWire 800 repeaters,
> which sometimes act up so that reconnect does not succeed immediately.
> Perhaps such reconnection troubles are necessary to provoke this lockdep
> report.
> 
> [I only accidentally discovered this on a test PC which do not use very
> frequently.  Unfortunately, I had lockdep disabled on my main PC on
> which I test most of my FireWire hardware for quite a while now.  I
> guess I should enable lockdep there again, but I have a suspicion that I
> won't leave it on for too long.]
> 
> Note how the IN-HARDIRQ-W talks about a SCSI host lock of a libata host,
> whereas the locking in the firewire stack surely does only interact
> (hopefully correctly) with SCSI host locks of firewire-sbp2 host
> instances.
> 
> What does this all mean?  How to act on it?
> 
> =========================================================
> [ INFO: possible irq lock inversion dependency detected ]
> 2.6.35 #1
> ---------------------------------------------------------
> swapper/0 just changed the state of lock:
>  (&t->split_timeout_timer){+.-...}, at: [<c1032228>] run_timer_softirq+0x112/0x21c
> but this lock was taken by another, HARDIRQ-safe lock in the past:
>  (&(shost->host_lock)->rlock){-.-...}
> 
> and interrupts could create inverse lock ordering between them.
> 
> 
> other info that might help us debug this:
> no locks held by swapper/0.
> 
> the shortest dependencies between 2nd lock and 1st lock:
>  -> (&(shost->host_lock)->rlock){-.-...} ops: 123308 {
>     IN-HARDIRQ-W at:
>                           [<c104a080>] __lock_acquire+0x612/0x153e
>                           [<c104b008>] lock_acquire+0x5c/0x73
>                           [<c1268b57>] _raw_spin_lock_irqsave+0x2e/0x3e
>                           [<c11ad508>] scsi_eh_scmd_add+0x25/0x8a
>                           [<c11ad5b7>] scsi_times_out+0x4a/0x64
>                           [<c10e9fca>] blk_rq_timed_out+0xf/0x4b
>                           [<c10ea044>] blk_abort_request+0x3e/0x41
>                           [<c11c43c0>] ata_qc_schedule_eh+0x55/0x67
>                           [<c11b9c4e>] ata_qc_complete+0x59/0x1b6
>                           [<c11c6751>] ata_hsm_qc_complete+0xb5/0xcf
>                           [<c11c6e18>] ata_sff_hsm_move+0x6ad/0x6fa
>                           [<c11c7002>] __ata_sff_port_intr+0x9f/0xad
>                           [<c11c70c9>] ata_bmdma_port_intr+0xb9/0x127
>                           [<c11c71c3>] ata_bmdma_interrupt+0x8c/0x170
>                           [<c1059b58>] handle_IRQ_event+0x1d/0xa2
>                           [<c105b45f>] handle_fasteoi_irq+0x79/0xb2
>                           [<c10045bb>] handle_irq+0x3b/0x48
>                           [<c1003df9>] do_IRQ+0x45/0x9f
>                           [<c1002d2e>] common_interrupt+0x2e/0x34
>                           [<c11f4500>] cpuidle_idle_call+0x6d/0xa1
>                           [<c1001a72>] cpu_idle+0x49/0x77
>                           [<c12594f7>] rest_init+0xbf/0xc4
>                           [<c13c98c9>] start_kernel+0x306/0x30b
>                           [<c13c90b7>] i386_start_kernel+0xb7/0xbf
>     IN-SOFTIRQ-W at:
>                           [<c104a0a1>] __lock_acquire+0x633/0x153e
>                           [<c104b008>] lock_acquire+0x5c/0x73
>                           [<c1268b57>] _raw_spin_lock_irqsave+0x2e/0x3e
>                           [<c11aec72>] scsi_device_unbusy+0x1e/0x8d
>                           [<c11a92fa>] scsi_finish_command+0x22/0xdf
>                           [<c11af291>] scsi_softirq_done+0xf4/0xfc
>                           [<c10e9d9c>] blk_done_softirq+0x5c/0x69
>                           [<c102defb>] __do_softirq+0x8b/0x10a
>                           [<c102dfa5>] do_softirq+0x2b/0x43
>                           [<c102e031>] run_ksoftirqd+0x74/0x15a
>                           [<c103ae19>] kthread+0x61/0x66
>                           [<c1002d3a>] kernel_thread_helper+0x6/0x1a
>     INITIAL USE at:
>                          [<c104a17b>] __lock_acquire+0x70d/0x153e
>                          [<c104b008>] lock_acquire+0x5c/0x73
>                          [<c1268b57>] _raw_spin_lock_irqsave+0x2e/0x3e
>                          [<c11ad5e0>] scsi_schedule_eh+0xf/0x4f
>                          [<c11c4125>] ata_port_schedule_eh+0x37/0x3a
>                          [<c11bd629>] async_port_probe+0x65/0x9c
>                          [<c1040051>] async_thread+0xe7/0x1c8
>                          [<c103ae19>] kthread+0x61/0x66
>                          [<c1002d3a>] kernel_thread_helper+0x6/0x1a
>   }
>   ... key      at: [<c19b0ab4>] __key.30680+0x0/0x8
>   ... acquired at:
>    [<c104b008>] lock_acquire+0x5c/0x73
>    [<c1032a3f>] del_timer_sync+0x29/0x6c
>    [<f84a6ea0>] close_transaction+0x93/0xb4 [firewire_core]
>    [<f84a6f00>] transmit_complete_callback+0x3f/0x43 [firewire_core]
>    [<f8da60ff>] at_context_transmit+0x59b/0x5a3 [firewire_ohci]
>    [<f8da6123>] ohci_send_request+0xd/0xf [firewire_ohci]
>    [<f84a7ce4>] fw_send_request+0x21c/0x224 [firewire_core]
>    [<f8dab307>] 0xf8dab307
>    [<f8dac075>] 0xf8dac075
>    [<c11a962a>] scsi_dispatch_cmd+0x182/0x1f2
>    [<c11ae4ed>] scsi_request_fn+0x359/0x492
>    [<c10e644f>] __generic_unplug_device+0x26/0x29
>    [<c10e9306>] blk_execute_rq_nowait+0x56/0x75
>    [<c10e93cd>] blk_execute_rq+0xa8/0xc7
>    [<c11af359>] scsi_execute+0xc0/0x105
>    [<c11af3f3>] scsi_execute_req+0x55/0x7d
>    [<c11b3d9a>] sd_start_stop_device+0x7b/0x117
>    [<c11b41a7>] sd_shutdown+0x108/0x117
>    [<c11b42d0>] sd_remove+0x42/0x6f
>    [<c11975af>] __device_release_driver+0x5e/0x96
>    [<c119768d>] device_release_driver+0x1a/0x25
>    [<c1196dcc>] bus_remove_device+0x7b/0x92
>    [<c119592a>] device_del+0xfa/0x154
>    [<c11b21ae>] __scsi_remove_device+0x3d/0x7b
>    [<c11b2208>] scsi_remove_device+0x1c/0x27
>    [<f8dab83e>] 0xf8dab83e
>    [<c10f2b89>] kref_put+0x39/0x44
>    [<f8dab0ff>] 0xf8dab0ff
>    [<c11975af>] __device_release_driver+0x5e/0x96
>    [<c119768d>] device_release_driver+0x1a/0x25
>    [<c1196dcc>] bus_remove_device+0x7b/0x92
>    [<c119592a>] device_del+0xfa/0x154
>    [<c119598f>] device_unregister+0xb/0x15
>    [<f84a4eb8>] shutdown_unit+0x8/0xc [firewire_core]
>    [<c1195270>] device_for_each_child+0x2a/0x54
>    [<f84a4f2e>] fw_device_shutdown+0x72/0xa4 [firewire_core]
>    [<c1037f34>] worker_thread+0x16a/0x242
>    [<c103ae19>] kthread+0x61/0x66
>    [<c1002d3a>] kernel_thread_helper+0x6/0x1a
> 
> -> (&t->split_timeout_timer){+.-...} ops: 14281 {
>    HARDIRQ-ON-W at:
>                         [<c104a100>] __lock_acquire+0x692/0x153e
>                         [<c104b008>] lock_acquire+0x5c/0x73
>                         [<c1032291>] run_timer_softirq+0x17b/0x21c
>                         [<c102defb>] __do_softirq+0x8b/0x10a
>                         [<c102dfa5>] do_softirq+0x2b/0x43
>                         [<c102e14f>] irq_exit+0x38/0x74
>                         [<c1003e3d>] do_IRQ+0x89/0x9f
>                         [<c1002d2e>] common_interrupt+0x2e/0x34
>                         [<c11f4500>] cpuidle_idle_call+0x6d/0xa1
>                         [<c1001a72>] cpu_idle+0x49/0x77
>                         [<c12594f7>] rest_init+0xbf/0xc4
>                         [<c13c98c9>] start_kernel+0x306/0x30b
>                         [<c13c90b7>] i386_start_kernel+0xb7/0xbf
>    IN-SOFTIRQ-W at:
>                         [<c104a0a1>] __lock_acquire+0x633/0x153e
>                         [<c104b008>] lock_acquire+0x5c/0x73
>                         [<c1032a3f>] del_timer_sync+0x29/0x6c
>                         [<f84a6ddc>] fw_core_handle_response+0x108/0x139 [firewire_core]
>                         [<f8da5b58>] handle_ar_packet+0x108/0x114 [firewire_ohci]
>                         [<f8da6236>] ar_context_tasklet+0x111/0x11f [firewire_ohci]
>                         [<c102da6d>] tasklet_action+0x8d/0xe0
>                         [<c102defb>] __do_softirq+0x8b/0x10a
>                         [<c102dfa5>] do_softirq+0x2b/0x43
>                         [<c102e14f>] irq_exit+0x38/0x74
>                         [<c1003e3d>] do_IRQ+0x89/0x9f
>                         [<c1002d2e>] common_interrupt+0x2e/0x34
>                         [<c11f4500>] cpuidle_idle_call+0x6d/0xa1
>                         [<c1001a72>] cpu_idle+0x49/0x77
>                         [<c12594f7>] rest_init+0xbf/0xc4
>                         [<c13c98c9>] start_kernel+0x306/0x30b
>                         [<c13c90b7>] i386_start_kernel+0xb7/0xbf
>    INITIAL USE at:
>                        [<c104a17b>] __lock_acquire+0x70d/0x153e
>                        [<c104b008>] lock_acquire+0x5c/0x73
>                        [<c1032a3f>] del_timer_sync+0x29/0x6c
>                        [<f84a6ddc>] fw_core_handle_response+0x108/0x139 [firewire_core]
>                        [<f8da5de2>] at_context_transmit+0x27e/0x5a3 [firewire_ohci]
>                        [<f8da6123>] ohci_send_request+0xd/0xf [firewire_ohci]
>                        [<f84a7ce4>] fw_send_request+0x21c/0x224 [firewire_core]
>                        [<f84a7d71>] fw_run_transaction+0x85/0xc5 [firewire_core]
>                        [<f84a2485>] bm_work+0x16c/0x34b [firewire_core]
>                        [<c1037f34>] worker_thread+0x16a/0x242
>                        [<c103ae19>] kthread+0x61/0x66
>                        [<c1002d3a>] kernel_thread_helper+0x6/0x1a
>  }
>  ... key      at: [<f84a97d0>] __key.22471+0x0/0xffffe611 [firewire_core]
>  ... acquired at:
>    [<c10494ba>] check_usage_backwards+0x94/0x9f
>    [<c1048b60>] mark_lock+0x2c0/0x4cd
>    [<c104a100>] __lock_acquire+0x692/0x153e
>    [<c104b008>] lock_acquire+0x5c/0x73
>    [<c1032291>] run_timer_softirq+0x17b/0x21c
>    [<c102defb>] __do_softirq+0x8b/0x10a
>    [<c102dfa5>] do_softirq+0x2b/0x43
>    [<c102e14f>] irq_exit+0x38/0x74
>    [<c1003e3d>] do_IRQ+0x89/0x9f
>    [<c1002d2e>] common_interrupt+0x2e/0x34
>    [<c11f4500>] cpuidle_idle_call+0x6d/0xa1
>    [<c1001a72>] cpu_idle+0x49/0x77
>    [<c12594f7>] rest_init+0xbf/0xc4
>    [<c13c98c9>] start_kernel+0x306/0x30b
>    [<c13c90b7>] i386_start_kernel+0xb7/0xbf
> 
> 
> stack backtrace:
> Pid: 0, comm: swapper Not tainted 2.6.35 #1
> Call Trace:
>  [<c1266701>] ? printk+0xf/0x16
>  [<c104937c>] print_irq_inversion_bug+0xeb/0xf6
>  [<c10494ba>] check_usage_backwards+0x94/0x9f
>  [<c1048b60>] mark_lock+0x2c0/0x4cd
>  [<c1049426>] ? check_usage_backwards+0x0/0x9f
>  [<c104a100>] __lock_acquire+0x692/0x153e
>  [<c126910a>] ? _raw_spin_unlock_irqrestore+0x54/0x58
>  [<c10fbc1f>] ? debug_object_deactivate+0xa1/0xcc
>  [<c104b008>] lock_acquire+0x5c/0x73
>  [<c1032228>] ? run_timer_softirq+0x112/0x21c
>  [<c1032291>] run_timer_softirq+0x17b/0x21c
>  [<c1032228>] ? run_timer_softirq+0x112/0x21c
>  [<f84a77ce>] ? split_transaction_timeout_callback+0x0/0x8b [firewire_core]
>  [<c102defb>] __do_softirq+0x8b/0x10a
>  [<c102dfa5>] do_softirq+0x2b/0x43
>  [<c102e14f>] irq_exit+0x38/0x74
>  [<c1003e3d>] do_IRQ+0x89/0x9f
>  [<c1002d2e>] common_interrupt+0x2e/0x34
>  [<c1140f6d>] ? acpi_idle_enter_simple+0x122/0x14e
>  [<c11f4500>] cpuidle_idle_call+0x6d/0xa1
>  [<c1001a72>] cpu_idle+0x49/0x77
>  [<c12594f7>] rest_init+0xbf/0xc4
>  [<c13c98c9>] start_kernel+0x306/0x30b
>  [<c13c90b7>] i386_start_kernel+0xb7/0xbf
> 
> 
> -- 
> Stefan Richter
> -=====-==-=- =--- =----
> http://arcgraph.de/sr/
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/