From: Jason Gunthorpe <jgg@nvidia.com>
To: Joerg Roedel <joro@8bytes.org>, Kevin Tian <kevin.tian@intel.com>,
Robin Murphy <robin.murphy@arm.com>,
Lu Baolu <baolu.lu@linux.intel.com>
Cc: iommu@lists.linux.dev
Subject: lockdep splat with intel iommu
Date: Thu, 27 Oct 2022 10:59:04 -0300 [thread overview]
Message-ID: <Y1qOqBXgll9qa7Hm@nvidia.com> (raw)
I'm seeing this at every boot, on v6.1-rc2
I've just realized it means lockdep is turned off for the actual
testing I want to do. Very annoying
Lu, is there a solution yet? I saw some patches for a different
lockdep splat but not this one.
[ 0.604492] ======================================================
[ 0.604838] WARNING: possible circular locking dependency detected
[ 0.605221] 6.1.0-rc2+ #24 Not tainted
[ 0.605432] ------------------------------------------------------
[ 0.605734] swapper/0/1 is trying to acquire lock:
[ 0.605985] ffff8880069ab8a0 (&md->mutex){+.+.}-{3:3}, at: msi_get_virq+0x99/0x150
[ 0.606281]
[ 0.606281] but task is already holding lock:
[ 0.606281] ffffffff82676890 (cpu_hotplug_lock){++++}-{0:0}, at: init_vqs+0x6db/0x940
[ 0.606281]
[ 0.606281] which lock already depends on the new lock.
[ 0.606281]
[ 0.606281]
[ 0.606281] the existing dependency chain (in reverse order) is:
[ 0.606281]
[ 0.606281] -> #3 (cpu_hotplug_lock){++++}-{0:0}:
[ 0.606281] cpus_read_lock+0x36/0xb0
[ 0.606281] __cpuhp_state_add_instance+0x16/0x40
[ 0.606281] iova_domain_init_rcaches+0x22c/0x260
[ 0.606281] iommu_setup_dma_ops+0x15c/0x790
[ 0.606281] intel_iommu_probe_finalize+0x30/0x40
[ 0.606281] bus_iommu_probe+0x47b/0x4c0
[ 0.606281] iommu_device_register+0x135/0x210
[ 0.606281] intel_iommu_init+0x498/0x6a5
[ 0.606281] pci_iommu_init+0x16/0x3b
[ 0.606281] do_one_initcall+0x167/0x3e0
[ 0.606281] do_initcall_level+0xc0/0xd4
[ 0.606281] do_initcalls+0x4b/0x7c
[ 0.606281] do_basic_setup+0x49/0x50
[ 0.606281] kernel_init_freeable+0xd0/0x10a
[ 0.606281] kernel_init+0x18/0x190
[ 0.606281] ret_from_fork+0x1f/0x30
[ 0.606281]
[ 0.606281] -> #2 (dmar_global_lock){++++}-{3:3}:
[ 0.606281] down_read+0x37/0x50
[ 0.606281] intel_irq_remapping_alloc+0x10a/0xca0
[ 0.606281] irq_domain_alloc_irqs_parent+0x64/0x80
[ 0.606281] mp_irqdomain_alloc+0xe8/0x420
[ 0.606281] __irq_domain_alloc_irqs+0x22f/0x5b0
[ 0.606281] alloc_isa_irq_from_domain+0x1cd/0x240
[ 0.606281] mp_map_pin_to_irq+0x2d1/0x490
[ 0.606281] setup_IO_APIC_irqs+0xec/0x17b
[ 0.606281] setup_IO_APIC+0xb7/0x193
[ 0.606281] apic_bsp_setup+0x21/0x2f
[ 0.606281] apic_intr_mode_init+0x78/0x7b
[ 0.606281] x86_late_time_init+0x3a/0x4b
[ 0.606281] start_kernel+0x28f/0x36d
[ 0.606281] x86_64_start_reservations+0x2a/0x2c
[ 0.606281] x86_64_start_kernel+0x7c/0x81
[ 0.606281] secondary_startup_64_no_verify+0xce/0xdb
[ 0.606281]
[ 0.606281] -> #1 (irq_domain_mutex){+.+.}-{3:3}:
[ 0.606281] __mutex_lock_common+0xf7/0x1410
[ 0.606281] mutex_lock_nested+0x1b/0x30
[ 0.606281] __irq_domain_alloc_irqs+0x1fd/0x5b0
[ 0.606281] __msi_domain_alloc_irqs+0x30d/0x730
[ 0.606281] msi_domain_alloc_irqs_descs_locked+0xad/0x160
[ 0.606281] pci_msi_setup_msi_irqs+0x54/0x60
[ 0.606281] __pci_enable_msix_range+0x755/0xaa0
[ 0.606281] pci_alloc_irq_vectors_affinity+0x80/0x120
[ 0.606281] vp_find_vqs_msix+0x294/0x710
[ 0.606281] vp_find_vqs+0x6c/0x2b0
[ 0.606281] vp_modern_find_vqs+0x16/0x80
[ 0.606281] init_vqs+0x3bb/0x6d0
[ 0.606281] virtcons_probe+0x1ed/0x5f0
[ 0.606281] virtio_dev_probe+0x4d6/0x5c0
[ 0.606281] really_probe+0x1d5/0x590
[ 0.606281] __driver_probe_device+0xea/0x130
[ 0.606281] driver_probe_device+0x4e/0x250
[ 0.606281] __driver_attach+0x19f/0x2f0
[ 0.606281] bus_for_each_dev+0xb3/0xf0
[ 0.606281] driver_attach+0x26/0x30
[ 0.606281] bus_add_driver+0x1d8/0x300
[ 0.606281] driver_register+0x11e/0x1f0
[ 0.606281] register_virtio_driver+0x4c/0x60
[ 0.606281] virtio_console_init+0x6e/0xe1
[ 0.606281] do_one_initcall+0x167/0x3e0
[ 0.606281] do_initcall_level+0xc0/0xd4
[ 0.606281] do_initcalls+0x4b/0x7c
[ 0.606281] do_basic_setup+0x49/0x50
[ 0.606281] kernel_init_freeable+0xd0/0x10a
[ 0.606281] kernel_init+0x18/0x190
[ 0.606281] ret_from_fork+0x1f/0x30
[ 0.606281]
[ 0.606281] -> #0 (&md->mutex){+.+.}-{3:3}:
[ 0.606281] __lock_acquire+0x2092/0x3c80
[ 0.606281] lock_acquire+0x1b5/0x300
[ 0.606281] __mutex_lock_common+0xf7/0x1410
[ 0.606281] mutex_lock_nested+0x1b/0x30
[ 0.606281] msi_get_virq+0x99/0x150
[ 0.606281] pci_irq_vector+0x3d/0x70
[ 0.606281] vp_set_vq_affinity+0xb4/0x120
[ 0.606281] virtnet_set_affinity+0x20b/0x2b0
[ 0.606281] init_vqs+0x6e3/0x940
[ 0.606281] virtnet_probe+0xcd2/0x1540
[ 0.606281] virtio_dev_probe+0x4d6/0x5c0
[ 0.606281] really_probe+0x1d5/0x590
[ 0.606281] __driver_probe_device+0xea/0x130
[ 0.606281] driver_probe_device+0x4e/0x250
[ 0.606281] __driver_attach+0x19f/0x2f0
[ 0.606281] bus_for_each_dev+0xb3/0xf0
[ 0.606281] driver_attach+0x26/0x30
[ 0.606281] bus_add_driver+0x1d8/0x300
[ 0.606281] driver_register+0x11e/0x1f0
[ 0.606281] register_virtio_driver+0x4c/0x60
[ 0.606281] virtio_net_driver_init+0x71/0x9b
[ 0.606281] do_one_initcall+0x167/0x3e0
[ 0.606281] do_initcall_level+0xc0/0xd4
[ 0.606281] do_initcalls+0x4b/0x7c
[ 0.606281] do_basic_setup+0x49/0x50
[ 0.606281] kernel_init_freeable+0xd0/0x10a
[ 0.606281] kernel_init+0x18/0x190
[ 0.606281] ret_from_fork+0x1f/0x30
[ 0.606281]
[ 0.606281] other info that might help us debug this:
[ 0.606281]
[ 0.606281] Chain exists of:
[ 0.606281] &md->mutex --> dmar_global_lock --> cpu_hotplug_lock
[ 0.606281]
[ 0.606281] Possible unsafe locking scenario:
[ 0.606281]
[ 0.606281] CPU0 CPU1
[ 0.606281] ---- ----
[ 0.606281] lock(cpu_hotplug_lock);
[ 0.606281] lock(dmar_global_lock);
[ 0.606281] lock(cpu_hotplug_lock);
[ 0.606281] lock(&md->mutex);
[ 0.606281]
[ 0.606281] *** DEADLOCK ***
[ 0.606281]
[ 0.606281] 2 locks held by swapper/0/1:
[ 0.606281] #0: ffff8880066f5170 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x194/0x2f0
[ 0.606281] #1: ffffffff82676890 (cpu_hotplug_lock){++++}-{0:0}, at: init_vqs+0x6db/0x940
[ 0.606281]
[ 0.606281] stack backtrace:
[ 0.606281] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.1.0-rc2+ #24
[ 0.606281] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[ 0.606281] Call Trace:
[ 0.606281] <TASK>
[ 0.606281] dump_stack_lvl+0x72/0xa2
[ 0.606281] dump_stack+0x10/0x12
[ 0.606281] print_circular_bug+0x110/0x120
[ 0.606281] check_noncircular+0x18f/0x1a0
[ 0.606281] __lock_acquire+0x2092/0x3c80
[ 0.606281] ? kfree+0x12a/0x190
[ 0.606281] ? kasan_set_track+0x52/0x60
[ 0.606281] ? kasan_set_track+0x3d/0x60
[ 0.606281] ? lock_is_held_type+0x8e/0x120
[ 0.606281] ? kfree+0x12a/0x190
[ 0.606281] lock_acquire+0x1b5/0x300
[ 0.606281] ? msi_get_virq+0x99/0x150
[ 0.606281] __mutex_lock_common+0xf7/0x1410
[ 0.606281] ? msi_get_virq+0x99/0x150
[ 0.606281] ? msi_get_virq+0x99/0x150
[ 0.606281] ? request_threaded_irq+0x178/0x1d0
[ 0.606281] ? lockdep_hardirqs_on_prepare+0x1cb/0x310
[ 0.606281] mutex_lock_nested+0x1b/0x30
[ 0.606281] msi_get_virq+0x99/0x150
[ 0.606281] pci_irq_vector+0x3d/0x70
[ 0.606281] vp_set_vq_affinity+0xb4/0x120
[ 0.606281] virtnet_set_affinity+0x20b/0x2b0
[ 0.606281] init_vqs+0x6e3/0x940
[ 0.606281] virtnet_probe+0xcd2/0x1540
[ 0.606281] ? virtqueue_enable_cb_delayed+0x1e1/0x320
[ 0.606281] virtio_dev_probe+0x4d6/0x5c0
[ 0.606281] really_probe+0x1d5/0x590
[ 0.606281] __driver_probe_device+0xea/0x130
[ 0.606281] driver_probe_device+0x4e/0x250
[ 0.606281] __driver_attach+0x19f/0x2f0
[ 0.606281] ? driver_attach+0x30/0x30
[ 0.606281] bus_for_each_dev+0xb3/0xf0
[ 0.606281] driver_attach+0x26/0x30
[ 0.606281] bus_add_driver+0x1d8/0x300
[ 0.606281] driver_register+0x11e/0x1f0
[ 0.606281] register_virtio_driver+0x4c/0x60
[ 0.606281] virtio_net_driver_init+0x71/0x9b
[ 0.606281] ? blackhole_netdev_init+0x9c/0x9c
[ 0.606281] do_one_initcall+0x167/0x3e0
[ 0.606281] ? kasan_set_track+0x52/0x60
[ 0.606281] ? kasan_set_track+0x3d/0x60
[ 0.606281] ? kasan_save_alloc_info+0x1f/0x30
[ 0.606281] ? __kasan_kmalloc+0x81/0x90
[ 0.606281] ? __kmalloc+0xa5/0x150
[ 0.606281] ? do_initcalls+0x28/0x7c
[ 0.606281] ? do_basic_setup+0x49/0x50
[ 0.606281] ? kernel_init_freeable+0xd0/0x10a
[ 0.606281] ? kernel_init+0x18/0x190
[ 0.606281] ? ret_from_fork+0x1f/0x30
[ 0.606281] ? mark_lock+0xb3/0x220
[ 0.606281] ? lock_is_held_type+0x8e/0x120
[ 0.606281] ? lock_is_held_type+0x8e/0x120
[ 0.606281] ? skip_spaces+0x31/0x50
[ 0.606281] ? next_arg+0x1ee/0x200
[ 0.606281] ? strlen+0x21/0x40
[ 0.606281] ? parse_args+0x8f/0x4f0
[ 0.606281] ? rcu_read_lock_sched_held+0x44/0xa0
[ 0.606281] do_initcall_level+0xc0/0xd4
[ 0.606281] do_initcalls+0x4b/0x7c
[ 0.606281] do_basic_setup+0x49/0x50
[ 0.606281] kernel_init_freeable+0xd0/0x10a
[ 0.606281] ? rest_init+0x1c0/0x1c0
[ 0.606281] kernel_init+0x18/0x190
[ 0.606281] ? rest_init+0x1c0/0x1c0
[ 0.606281] ret_from_fork+0x1f/0x30
[ 0.606281] </TASK>
next reply other threads:[~2022-10-27 13:59 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-27 13:59 Jason Gunthorpe [this message]
2022-10-29 8:22 ` lockdep splat with intel iommu Baolu Lu
2022-10-29 16:06 ` Jason Gunthorpe
2022-10-30 8:41 ` Baolu Lu
2022-10-31 12:11 ` Jason Gunthorpe
2023-03-24 21:01 ` Jason Gunthorpe
2023-03-25 8:49 ` Baolu Lu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y1qOqBXgll9qa7Hm@nvidia.com \
--to=jgg@nvidia.com \
--cc=baolu.lu@linux.intel.com \
--cc=iommu@lists.linux.dev \
--cc=joro@8bytes.org \
--cc=kevin.tian@intel.com \
--cc=robin.murphy@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox