* kernel BUG at kernel/locking/rtmutex.c:1059 @ 2017-08-30 21:25 Mart van de Wege 2017-08-31 10:59 ` Pratyush Patel 2017-08-31 14:52 ` Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) Sebastian Andrzej Siewior 0 siblings, 2 replies; 14+ messages in thread From: Mart van de Wege @ 2017-08-30 21:25 UTC (permalink / raw) To: linux-rt-users Hi, As of v4.11.12-rt8 my laptop runs into trouble booting up, throwing a kernel BUG when starting up bluetooth. Bisecting *seems* to lead back to commit 6fb87cb1181786dc88432c6c6093a02a897e5789, 'Add the hotplug rework including all its side stories'. The issue persists up to v4.11.12-rt10 Stacktrace: Aug 29 17:28:04 localhost kernel: [ 46.483810] ------------[ cut here ]------------ Aug 29 17:28:04 localhost kernel: [ 46.483812] kernel BUG at kernel/locking/rtmutex.c:1059! Aug 29 17:28:04 localhost kernel: [ 46.483816] invalid opcode: 0000 [#1] PREEMPT SMP Aug 29 17:28:04 localhost kernel: [ 46.483817] Modules linked in: cmac bnep msr fuse intel_rapl intel_powerclamp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel uvcvideo videobuf2_vmalloc videobuf2_memops video buf2_v4l2 videobuf2_core videodev pcbc media aesni_intel btusb joydev evdev aes_x86_64 btrtl crypto_simd glue_helper serio_raw cryptd intel_cstate intel_uncore hci_uart intel_rapl_perf btbcm efi_pstore arc4 snd_hda_codec_hdmi snd_hda_code c_realtek snd_hda_codec_generic btqca btintel iwlmvm snd_hda_intel mac80211 snd_hda_codec pcspkr snd_hda_core iwlwifi i915 snd_hwdep snd_pcm_oss iTCO_wdt sg efivars iTCO_vendor_support rtsx_pci_ms memstick wmi bluetooth crc16 drm_kms_help er snd_mixer_oss snd_pcm snd_timer mei_me snd cfg80211 drm soundcore mei rfkill i2c_algo_bit intel_lpss_acpi intel_lpss video battery Aug 29 17:28:04 localhost kernel: [ 46.483870] ac acpi_pad button intel_pch_thermal tpm_crb shpchp binfmt_misc nls_ascii nls_cp437 vfat fat coretemp ecryptfs cbc encrypted_keys parport_pc ppdev lp parport loop dm_crypt dm_mod sunrpc ef ivarfs ip_tables x_tables autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod sr_mod cdrom sd_mod mmc_block rtsx_pci_sdmmc mmc_c ore ahci libahci xhci_pci xhci_hcd libata rtsx_pci r8169 crc32c_intel psmouse mfd_core mii i2c_i801 scsi_mod usbcore thermal i2c_hid hid Aug 29 17:28:04 localhost kernel: [ 46.483945] CPU: 3 PID: 227 Comm: kworker/u9:0 Not tainted 4.11.12-rt8+ #6 Aug 29 17:28:04 localhost kernel: [ 46.483947] Hardware name: Notebook W94_95_97JU/W94_95_97JU, BIOS 5.11 10/21/2015 Aug 29 17:28:04 localhost kernel: [ 46.483988] Workqueue: hci0 hci_rx_work [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.483990] task: ffff935b9a506000 task.stack: ffff9edcc3398000 Aug 29 17:28:04 localhost kernel: [ 46.483996] RIP: 0010:rt_spin_lock_slowlock_locked+0x26e/0x2d0 Aug 29 17:28:04 localhost kernel: [ 46.483998] RSP: 0018:ffff9edcc339bb20 EFLAGS: 00010046 Aug 29 17:28:04 localhost kernel: [ 46.484000] RAX: ffff935b9a506000 RBX: ffffffffc0b13268 RCX: 0000000000000001 Aug 29 17:28:04 localhost kernel: [ 46.484002] RDX: 0000000000000000 RSI: ffff935b9a506000 RDI: ffffffffc0b13268 Aug 29 17:28:04 localhost kernel: [ 46.484003] RBP: 0000000000000246 R08: ffff935b9a506000 R09: ffffffffc0b13280 Aug 29 17:28:04 localhost kernel: [ 46.484004] R10: ffff935b9a506001 R11: 0000000000000001 R12: ffff935b9a506000 Aug 29 17:28:04 localhost kernel: [ 46.484005] R13: ffff9edcc339bb68 R14: 0000000000000246 R15: ffff935b8f0de800 Aug 29 17:28:04 localhost kernel: [ 46.484008] FS: 0000000000000000(0000) GS:ffff935bb3d80000(0000) knlGS:0000000000000000 Aug 29 17:28:04 localhost kernel: [ 46.484009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 29 17:28:04 localhost kernel: [ 46.484011] CR2: 0000009b9614e800 CR3: 0000000418c0e000 CR4: 00000000003406e0 Aug 29 17:28:04 localhost kernel: [ 46.484012] Call Trace: Aug 29 17:28:04 localhost kernel: [ 46.484019] ? __slab_alloc.isra.76+0x72/0xb0 Aug 29 17:28:04 localhost kernel: [ 46.484023] ? rt_spin_lock_slowlock+0x50/0x80 Aug 29 17:28:04 localhost kernel: [ 46.484029] ? __kmalloc_reserve.isra.35+0x2e/0x80 Aug 29 17:28:04 localhost kernel: [ 46.484056] ? hci_send_to_channel+0x2c/0xe0 [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.484080] ? hci_send_monitor_ctrl_event+0x17a/0x1b0 [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.484108] ? mgmt_send_event+0x104/0x110 [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.484131] ? mgmt_set_class_of_dev_complete+0xb6/0x150 [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.484152] ? hci_cmd_complete_evt+0x15c5/0x2f40 [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.484172] ? hci_event_packet+0x13b2/0x2cd0 [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.484178] ? cpuacct_charge+0x5f/0x80 Aug 29 17:28:04 localhost kernel: [ 46.484181] ? unpin_current_cpu+0x37/0x90 Aug 29 17:28:04 localhost kernel: [ 46.484187] ? migrate_enable+0x230/0x380 Aug 29 17:28:04 localhost kernel: [ 46.484206] ? hci_rx_work+0x192/0x370 [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.484222] ? hci_rx_work+0x192/0x370 [bluetooth] Aug 29 17:28:04 localhost kernel: [ 46.484226] ? process_one_work+0x174/0x480 Aug 29 17:28:04 localhost kernel: [ 46.484230] ? worker_thread+0x4a/0x4d0 Aug 29 17:28:04 localhost kernel: [ 46.484233] ? process_one_work+0x480/0x480 Aug 29 17:28:04 localhost kernel: [ 46.484237] ? kthread+0x118/0x130 Aug 29 17:28:04 localhost kernel: [ 46.484241] ? kthread_create_on_node+0x70/0x70 Aug 29 17:28:04 localhost kernel: [ 46.484244] ? do_group_exit+0x47/0xc0 Aug 29 17:28:04 localhost kernel: [ 46.484247] ? ret_from_fork+0x25/0x30 Aug 29 17:28:04 localhost kernel: [ 46.484250] Code: ff ff ff 7f 0f 85 4b fe ff ff 41 8b 74 24 08 85 f6 0f 85 3e fe ff ff 49 8b 04 24 f6 c4 02 0f 84 31 fe ff ff e9 27 fe ff ff 0f 0b <0f> 0b a9 ff ff ff 7f 0f 85 20 ff ff ff 41 8b 46 08 8 5 c0 0f 85 Aug 29 17:28:04 localhost kernel: [ 46.484286] RIP: rt_spin_lock_slowlock_locked+0x26e/0x2d0 RSP: ffff9edcc339bb20 Aug 29 17:28:04 localhost kernel: [ 46.561380] ---[ end trace 0000000000000002 ]--- -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at kernel/locking/rtmutex.c:1059 2017-08-30 21:25 kernel BUG at kernel/locking/rtmutex.c:1059 Mart van de Wege @ 2017-08-31 10:59 ` Pratyush Patel 2017-08-31 14:54 ` Sebastian Andrzej Siewior 2017-08-31 14:52 ` Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) Sebastian Andrzej Siewior 1 sibling, 1 reply; 14+ messages in thread From: Pratyush Patel @ 2017-08-31 10:59 UTC (permalink / raw) To: Mart van de Wege; +Cc: linux-rt-users Hello, I too faced a similar issue on startup when I ran v4.11.12-rt10 on a virtual QEMU-KVM setup with Arch Linux. The kernel panics when the Locking API test suite runs during the startup. I'm not sure if these could be related. Stacktrace: [ 0.000000] recursive read-lock: | [ 0.000000] ------------[ cut here ]------------ [ 0.000000] kernel BUG at kernel/locking/rtmutex.c:1059! [ 0.000000] invalid opcode: 0000 [#1] PREEMPT SMP [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.11.12-rt10 #2 [ 0.000000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014 [ 0.000000] task: ffffffff82055500 task.stack: ffffffff82000000 [ 0.000000] RIP: 0010:rt_spin_lock_slowlock_locked+0x28f/0x2c0 [ 0.000000] RSP: 0000:ffffffff82003d50 EFLAGS: 00010046 [ 0.000000] RAX: ffffffff82055500 RBX: ffffffff820f7d00 RCX: 0000000000000001 [ 0.000000] RDX: 0000000000000000 RSI: ffffffff82055501 RDI: ffffffff820f7d18 [ 0.000000] RBP: ffffffff82003d90 R08: 0000000000000000 R09: 0000000000000000 [ 0.000000] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff82003da0 [ 0.000000] R13: ffffffff82055500 R14: ffffffff813e5450 R15: 0000000000000286 [ 0.000000] FS: 0000000000000000(0000) GS:ffff88003de00000(0000) knlGS:0000000000000000 [ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.000000] CR2: ffff88000330f000 CR3: 0000000002050000 CR4: 00000000000006b0 [ 0.000000] Call Trace: [ 0.000000] ? AA_mutex+0x20/0x20 [ 0.000000] ? AA_mutex+0x20/0x20 [ 0.000000] rt_spin_lock_slowlock+0x58/0x80 [ 0.000000] __rt_spin_lock+0xe/0x10 [ 0.000000] rt_read_lock+0x3b/0x50 [ 0.000000] ? rlock_AA1+0x1c/0x20 [ 0.000000] rlock_AA1+0x1c/0x20 [ 0.000000] dotest+0x3c/0x694 [ 0.000000] locking_selftest+0x699/0xfd0 [ 0.000000] start_kernel+0x2d1/0x3ed [ 0.000000] x86_64_start_reservations+0x2a/0x2c [ 0.000000] x86_64_start_kernel+0x178/0x18b [ 0.000000] start_cpu+0x14/0x14 [ 0.000000] ? start_cpu+0x14/0x14 [ 0.000000] Code: 85 35 ff ff ff 48 c7 c2 78 b4 e5 81 be 5b 03 00 00 48 c7 c7 21 b0 e5 81 c6 05 c1 14 67 00 01 e8 48 f3 67 ff e9 11 ff ff ff 0f 0b <0f> 0b 0f 0b 48 8b 43 50 48 3b 58 38 75 18 49 39 c4 0f 85 66 ff [ 0.000000] RIP: rt_spin_lock_slowlock_locked+0x28f/0x2c0 RSP: ffffffff82003d50 [ 0.000000] ---[ end trace 0000000000000001 ]--- [ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task! - Pratyush On Thu, Aug 31, 2017 at 2:55 AM, Mart van de Wege <mvdwege@gmail.com> wrote: > Hi, > > As of v4.11.12-rt8 my laptop runs into trouble booting up, throwing a > kernel BUG when starting up bluetooth. Bisecting *seems* to lead back to > commit 6fb87cb1181786dc88432c6c6093a02a897e5789, 'Add the hotplug rework > including all its side stories'. > > The issue persists up to v4.11.12-rt10 > > Stacktrace: > > Aug 29 17:28:04 localhost kernel: [ 46.483810] ------------[ cut here ]------------ > Aug 29 17:28:04 localhost kernel: [ 46.483812] kernel BUG at kernel/locking/rtmutex.c:1059! > Aug 29 17:28:04 localhost kernel: [ 46.483816] invalid opcode: 0000 [#1] PREEMPT SMP > Aug 29 17:28:04 localhost kernel: [ 46.483817] Modules linked in: cmac bnep msr fuse intel_rapl intel_powerclamp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel uvcvideo videobuf2_vmalloc videobuf2_memops video > buf2_v4l2 videobuf2_core videodev pcbc media aesni_intel btusb joydev evdev aes_x86_64 btrtl crypto_simd glue_helper serio_raw cryptd intel_cstate intel_uncore hci_uart intel_rapl_perf btbcm efi_pstore arc4 snd_hda_codec_hdmi snd_hda_code > c_realtek snd_hda_codec_generic btqca btintel iwlmvm snd_hda_intel mac80211 snd_hda_codec pcspkr snd_hda_core iwlwifi i915 snd_hwdep snd_pcm_oss iTCO_wdt sg efivars iTCO_vendor_support rtsx_pci_ms memstick wmi bluetooth crc16 drm_kms_help > er snd_mixer_oss snd_pcm snd_timer mei_me snd cfg80211 drm soundcore mei rfkill i2c_algo_bit intel_lpss_acpi intel_lpss video battery > Aug 29 17:28:04 localhost kernel: [ 46.483870] ac acpi_pad button intel_pch_thermal tpm_crb shpchp binfmt_misc nls_ascii nls_cp437 vfat fat coretemp ecryptfs cbc encrypted_keys parport_pc ppdev lp parport loop dm_crypt dm_mod sunrpc ef > ivarfs ip_tables x_tables autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod sr_mod cdrom sd_mod mmc_block rtsx_pci_sdmmc mmc_c > ore ahci libahci xhci_pci xhci_hcd libata rtsx_pci r8169 crc32c_intel psmouse mfd_core mii i2c_i801 scsi_mod usbcore thermal i2c_hid hid > Aug 29 17:28:04 localhost kernel: [ 46.483945] CPU: 3 PID: 227 Comm: kworker/u9:0 Not tainted 4.11.12-rt8+ #6 > Aug 29 17:28:04 localhost kernel: [ 46.483947] Hardware name: Notebook W94_95_97JU/W94_95_97JU, BIOS 5.11 10/21/2015 > Aug 29 17:28:04 localhost kernel: [ 46.483988] Workqueue: hci0 hci_rx_work [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.483990] task: ffff935b9a506000 task.stack: ffff9edcc3398000 > Aug 29 17:28:04 localhost kernel: [ 46.483996] RIP: 0010:rt_spin_lock_slowlock_locked+0x26e/0x2d0 > Aug 29 17:28:04 localhost kernel: [ 46.483998] RSP: 0018:ffff9edcc339bb20 EFLAGS: 00010046 > Aug 29 17:28:04 localhost kernel: [ 46.484000] RAX: ffff935b9a506000 RBX: ffffffffc0b13268 RCX: 0000000000000001 > Aug 29 17:28:04 localhost kernel: [ 46.484002] RDX: 0000000000000000 RSI: ffff935b9a506000 RDI: ffffffffc0b13268 > Aug 29 17:28:04 localhost kernel: [ 46.484003] RBP: 0000000000000246 R08: ffff935b9a506000 R09: ffffffffc0b13280 > Aug 29 17:28:04 localhost kernel: [ 46.484004] R10: ffff935b9a506001 R11: 0000000000000001 R12: ffff935b9a506000 > Aug 29 17:28:04 localhost kernel: [ 46.484005] R13: ffff9edcc339bb68 R14: 0000000000000246 R15: ffff935b8f0de800 > Aug 29 17:28:04 localhost kernel: [ 46.484008] FS: 0000000000000000(0000) GS:ffff935bb3d80000(0000) knlGS:0000000000000000 > Aug 29 17:28:04 localhost kernel: [ 46.484009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > Aug 29 17:28:04 localhost kernel: [ 46.484011] CR2: 0000009b9614e800 CR3: 0000000418c0e000 CR4: 00000000003406e0 > Aug 29 17:28:04 localhost kernel: [ 46.484012] Call Trace: > Aug 29 17:28:04 localhost kernel: [ 46.484019] ? __slab_alloc.isra.76+0x72/0xb0 > Aug 29 17:28:04 localhost kernel: [ 46.484023] ? rt_spin_lock_slowlock+0x50/0x80 > Aug 29 17:28:04 localhost kernel: [ 46.484029] ? __kmalloc_reserve.isra.35+0x2e/0x80 > Aug 29 17:28:04 localhost kernel: [ 46.484056] ? hci_send_to_channel+0x2c/0xe0 [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.484080] ? hci_send_monitor_ctrl_event+0x17a/0x1b0 [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.484108] ? mgmt_send_event+0x104/0x110 [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.484131] ? mgmt_set_class_of_dev_complete+0xb6/0x150 [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.484152] ? hci_cmd_complete_evt+0x15c5/0x2f40 [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.484172] ? hci_event_packet+0x13b2/0x2cd0 [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.484178] ? cpuacct_charge+0x5f/0x80 > Aug 29 17:28:04 localhost kernel: [ 46.484181] ? unpin_current_cpu+0x37/0x90 > Aug 29 17:28:04 localhost kernel: [ 46.484187] ? migrate_enable+0x230/0x380 > Aug 29 17:28:04 localhost kernel: [ 46.484206] ? hci_rx_work+0x192/0x370 [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.484222] ? hci_rx_work+0x192/0x370 [bluetooth] > Aug 29 17:28:04 localhost kernel: [ 46.484226] ? process_one_work+0x174/0x480 > Aug 29 17:28:04 localhost kernel: [ 46.484230] ? worker_thread+0x4a/0x4d0 > Aug 29 17:28:04 localhost kernel: [ 46.484233] ? process_one_work+0x480/0x480 > Aug 29 17:28:04 localhost kernel: [ 46.484237] ? kthread+0x118/0x130 > Aug 29 17:28:04 localhost kernel: [ 46.484241] ? kthread_create_on_node+0x70/0x70 > Aug 29 17:28:04 localhost kernel: [ 46.484244] ? do_group_exit+0x47/0xc0 > Aug 29 17:28:04 localhost kernel: [ 46.484247] ? ret_from_fork+0x25/0x30 > Aug 29 17:28:04 localhost kernel: [ 46.484250] Code: ff ff ff 7f 0f 85 4b fe ff ff 41 8b 74 24 08 85 f6 0f 85 3e fe ff ff 49 8b 04 24 f6 c4 02 0f 84 31 fe ff ff e9 27 fe ff ff 0f 0b <0f> 0b a9 ff ff ff 7f 0f 85 20 ff ff ff 41 8b 46 08 8 > 5 c0 0f 85 > Aug 29 17:28:04 localhost kernel: [ 46.484286] RIP: rt_spin_lock_slowlock_locked+0x26e/0x2d0 RSP: ffff9edcc339bb20 > Aug 29 17:28:04 localhost kernel: [ 46.561380] ---[ end trace 0000000000000002 ]--- > > -- > "We will need a longer wall when the revolution comes." > --- AJS, quoting an uncertain source. > -- > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at kernel/locking/rtmutex.c:1059 2017-08-31 10:59 ` Pratyush Patel @ 2017-08-31 14:54 ` Sebastian Andrzej Siewior 2017-08-31 16:02 ` Pratyush Patel 0 siblings, 1 reply; 14+ messages in thread From: Sebastian Andrzej Siewior @ 2017-08-31 14:54 UTC (permalink / raw) To: Pratyush Patel; +Cc: Mart van de Wege, linux-rt-users On 2017-08-31 16:29:54 [+0530], Pratyush Patel wrote: > Hello, Hi, > I too faced a similar issue on startup when I ran v4.11.12-rt10 on a > virtual QEMU-KVM setup with Arch Linux. The kernel panics when the > Locking API test suite runs during the startup. I'm not sure if these > could be related. This recursive locking is done by the self-test with the same result. I suggest you disable it. Sebastian ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at kernel/locking/rtmutex.c:1059 2017-08-31 14:54 ` Sebastian Andrzej Siewior @ 2017-08-31 16:02 ` Pratyush Patel 0 siblings, 0 replies; 14+ messages in thread From: Pratyush Patel @ 2017-08-31 16:02 UTC (permalink / raw) To: Sebastian Andrzej Siewior; +Cc: Mart van de Wege, linux-rt-users On Thu, Aug 31, 2017 at 8:24 PM, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > On 2017-08-31 16:29:54 [+0530], Pratyush Patel wrote: >> Hello, > Hi, > >> I too faced a similar issue on startup when I ran v4.11.12-rt10 on a >> virtual QEMU-KVM setup with Arch Linux. The kernel panics when the >> Locking API test suite runs during the startup. I'm not sure if these >> could be related. > > This recursive locking is done by the self-test with the same result. I > suggest you disable it. > It works after disabling the self-test and also after switching CONFIG_RWLOCK_RT_READER_BIASED=y. Thanks, Pratyush ^ permalink raw reply [flat|nested] 14+ messages in thread
* Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) 2017-08-30 21:25 kernel BUG at kernel/locking/rtmutex.c:1059 Mart van de Wege 2017-08-31 10:59 ` Pratyush Patel @ 2017-08-31 14:52 ` Sebastian Andrzej Siewior 2017-08-31 14:56 ` Thomas Gleixner ` (2 more replies) 1 sibling, 3 replies; 14+ messages in thread From: Sebastian Andrzej Siewior @ 2017-08-31 14:52 UTC (permalink / raw) To: Mart van de Wege, Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth, linux-kernel Cc: linux-rt-users, tglx, Peter Zijlstra On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote: > Hi, Hi, > The issue persists up to v4.11.12-rt10 Does CONFIG_RWLOCK_RT_READER_BIASED=y make it go away? > Stacktrace: > > Aug 29 17:28:04 localhost kernel: [ 46.483810] ------------[ cut here ]------------ > Aug 29 17:28:04 localhost kernel: [ 46.483812] kernel BUG at kernel/locking/rtmutex.c:1059! this is a deadlock on RT however !RT has a hidden problem which not yelled at by lockdep. We have the following call path: | hci_send_monitor_ctrl_event() | read_lock(&hci_sk_list.lock); | | hci_send_to_channel() | read_lock(&hci_sk_list.lock); So both functions acquire the same read_lock. If a write comes along between the first read_lock() and second read_lock() then we have a deadlock because the write_lock() will lock down further readers from acquiring the read-lock until the writer completed its task. This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add support for sending MGMT commands and events to monitor"). Sebastian ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) 2017-08-31 14:52 ` Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) Sebastian Andrzej Siewior @ 2017-08-31 14:56 ` Thomas Gleixner 2017-09-01 8:51 ` Mart van de Wege 2017-09-21 13:51 ` [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel() Sebastian Andrzej Siewior 2 siblings, 0 replies; 14+ messages in thread From: Thomas Gleixner @ 2017-08-31 14:56 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Mart van de Wege, Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth, linux-kernel, linux-rt-users, Peter Zijlstra On Thu, 31 Aug 2017, Sebastian Andrzej Siewior wrote: > On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote: > > Hi, > Hi, > > > The issue persists up to v4.11.12-rt10 > Does > CONFIG_RWLOCK_RT_READER_BIASED=y > make it go away? > > > Stacktrace: > > > > Aug 29 17:28:04 localhost kernel: [ 46.483810] ------------[ cut here ]------------ > > Aug 29 17:28:04 localhost kernel: [ 46.483812] kernel BUG at kernel/locking/rtmutex.c:1059! > > this is a deadlock on RT however !RT has a hidden problem which not > yelled at by lockdep. We have the following call path: > > | hci_send_monitor_ctrl_event() > | read_lock(&hci_sk_list.lock); > | > | hci_send_to_channel() > | read_lock(&hci_sk_list.lock); > > So both functions acquire the same read_lock. If a write comes along > between the first read_lock() and second read_lock() then we have a > deadlock because the write_lock() will lock down further readers from > acquiring the read-lock until the writer completed its task. > > This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add > support for sending MGMT commands and events to monitor"). Just for clarification: This is a mainline issue because qrlock based rwlocks are writer fair, while the traditional rwlocks are reader biased. Lockdep does not yell about it yet, because the wrapper function use read-recursive mode. Thanks, tglx ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) 2017-08-31 14:52 ` Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) Sebastian Andrzej Siewior 2017-08-31 14:56 ` Thomas Gleixner @ 2017-09-01 8:51 ` Mart van de Wege 2017-09-02 7:27 ` Mart van de Wege 2017-09-21 13:51 ` [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel() Sebastian Andrzej Siewior 2 siblings, 1 reply; 14+ messages in thread From: Mart van de Wege @ 2017-09-01 8:51 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth, linux-kernel, linux-rt-users, tglx, Peter Zijlstra On Thu, 31 Aug 2017 16:52:12 +0200 Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote: > > Hi, > Hi, > > > The issue persists up to v4.11.12-rt10 > Does > CONFIG_RWLOCK_RT_READER_BIASED=y > make it go away? > Yes, that does make it go away. -rt8 now boots cleanly, and works fine. Building -rt10 now to be sure. > > Stacktrace: > > > > Aug 29 17:28:04 localhost kernel: [ 46.483810] ------------[ cut > > here ]------------ Aug 29 17:28:04 localhost kernel: [ 46.483812] > > kernel BUG at kernel/locking/rtmutex.c:1059! > > this is a deadlock on RT however !RT has a hidden problem which not > yelled at by lockdep. We have the following call path: > > | hci_send_monitor_ctrl_event() > | read_lock(&hci_sk_list.lock); > | > | hci_send_to_channel() > | read_lock(&hci_sk_list.lock); > > So both functions acquire the same read_lock. If a write comes along > between the first read_lock() and second read_lock() then we have a > deadlock because the write_lock() will lock down further readers from > acquiring the read-lock until the writer completed its task. > > This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add > support for sending MGMT commands and events to monitor"). > > Sebastian ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) @ 2017-09-02 7:27 ` Mart van de Wege 0 siblings, 0 replies; 14+ messages in thread From: Mart van de Wege @ 2017-09-02 7:27 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth, linux-kernel, linux-rt-users, tglx, Peter Zijlstra On Fri, 1 Sep 2017 10:51:07 +0200 Mart van de Wege <mvdwege@gmail.com> wrote: > On Thu, 31 Aug 2017 16:52:12 +0200 > Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > > On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote: > > > Hi, > > Hi, > > > > > The issue persists up to v4.11.12-rt10 > > Does > > CONFIG_RWLOCK_RT_READER_BIASED=y > > make it go away? > > > Yes, that does make it go away. -rt8 now boots cleanly, and works > fine. Building -rt10 now to be sure. > Confirm that as of 4.11.12-rt11 the deadlock still goes away when enabling CONFIG_RWLOCK_RT_READER_BIASED Mart ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) @ 2017-09-02 7:27 ` Mart van de Wege 0 siblings, 0 replies; 14+ messages in thread From: Mart van de Wege @ 2017-09-02 7:27 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth, linux-kernel, linux-rt-users, tglx, Peter Zijlstra On Fri, 1 Sep 2017 10:51:07 +0200 Mart van de Wege <mvdwege@gmail.com> wrote: > On Thu, 31 Aug 2017 16:52:12 +0200 > Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > > On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote: > > > Hi, > > Hi, > > > > > The issue persists up to v4.11.12-rt10 > > Does > > CONFIG_RWLOCK_RT_READER_BIASED=y > > make it go away? > > > Yes, that does make it go away. -rt8 now boots cleanly, and works > fine. Building -rt10 now to be sure. > Confirm that as of 4.11.12-rt11 the deadlock still goes away when enabling CONFIG_RWLOCK_RT_READER_BIASED Mart ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) @ 2017-09-02 7:27 ` Mart van de Wege 0 siblings, 0 replies; 14+ messages in thread From: Mart van de Wege @ 2017-09-02 7:27 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA, linux-rt-users-u79uwXL29TY76Z2rM5mHXA, tglx-hfZtesqFncYOwBW4kG4KsQ, Peter Zijlstra On Fri, 1 Sep 2017 10:51:07 +0200 Mart van de Wege <mvdwege-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > On Thu, 31 Aug 2017 16:52:12 +0200 > Sebastian Andrzej Siewior <bigeasy-hfZtesqFncYOwBW4kG4KsQ@public.gmane.org> wrote: > > > On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote: > > > Hi, > > Hi, > > > > > The issue persists up to v4.11.12-rt10 > > Does > > CONFIG_RWLOCK_RT_READER_BIASED=y > > make it go away? > > > Yes, that does make it go away. -rt8 now boots cleanly, and works > fine. Building -rt10 now to be sure. > Confirm that as of 4.11.12-rt11 the deadlock still goes away when enabling CONFIG_RWLOCK_RT_READER_BIASED Mart ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel() 2017-08-31 14:52 ` Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) Sebastian Andrzej Siewior 2017-08-31 14:56 ` Thomas Gleixner 2017-09-01 8:51 ` Mart van de Wege @ 2017-09-21 13:51 ` Sebastian Andrzej Siewior 2017-09-24 14:53 ` Mart van de Wege 2017-10-30 8:05 ` Marcel Holtmann 2 siblings, 2 replies; 14+ messages in thread From: Sebastian Andrzej Siewior @ 2017-09-21 13:51 UTC (permalink / raw) To: Mart van de Wege, Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth, linux-kernel Cc: linux-rt-users, tglx, Peter Zijlstra Mart reported a deadlock in -RT in the call path: hci_send_monitor_ctrl_event() -> hci_send_to_channel() because both functions acquire the same read lock hci_sk_list.lock. This is also a mainline issue because the qrwlock implementation is writer fair (the traditional rwlock implementation is reader biased). To avoid the deadlock there is now __hci_send_to_channel() which expects the readlock to be held. Cc: Marcel Holtmann <marcel@holtmann.org> Cc: Johan Hedberg <johan.hedberg@intel.com> Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT commands and events to monitor") Reported-by: Mart van de Wege <mvdwege@gmail.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- net/bluetooth/hci_sock.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c index 638bf0e1a2e3..f1ee820d871c 100644 --- a/net/bluetooth/hci_sock.c +++ b/net/bluetooth/hci_sock.c @@ -251,15 +251,13 @@ void hci_send_to_sock(struct hci_dev *hdev, struct sk_buff *skb) } /* Send frame to sockets with specific channel */ -void hci_send_to_channel(unsigned short channel, struct sk_buff *skb, - int flag, struct sock *skip_sk) +static void __hci_send_to_channel(unsigned short channel, struct sk_buff *skb, + int flag, struct sock *skip_sk) { struct sock *sk; BT_DBG("channel %u len %d", channel, skb->len); - read_lock(&hci_sk_list.lock); - sk_for_each(sk, &hci_sk_list.head) { struct sk_buff *nskb; @@ -285,6 +283,13 @@ void hci_send_to_channel(unsigned short channel, struct sk_buff *skb, kfree_skb(nskb); } +} + +void hci_send_to_channel(unsigned short channel, struct sk_buff *skb, + int flag, struct sock *skip_sk) +{ + read_lock(&hci_sk_list.lock); + __hci_send_to_channel(channel, skb, flag, skip_sk); read_unlock(&hci_sk_list.lock); } @@ -388,8 +393,8 @@ void hci_send_monitor_ctrl_event(struct hci_dev *hdev, u16 event, hdr->index = index; hdr->len = cpu_to_le16(skb->len - HCI_MON_HDR_SIZE); - hci_send_to_channel(HCI_CHANNEL_MONITOR, skb, - HCI_SOCK_TRUSTED, NULL); + __hci_send_to_channel(HCI_CHANNEL_MONITOR, skb, + HCI_SOCK_TRUSTED, NULL); kfree_skb(skb); } -- 2.14.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel() @ 2017-09-24 14:53 ` Mart van de Wege 0 siblings, 0 replies; 14+ messages in thread From: Mart van de Wege @ 2017-09-24 14:53 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth, linux-kernel, linux-rt-users, tglx, Peter Zijlstra Confirm that 4.11.12-rt14 with this fix boots normally. On Thu, 21 Sep 2017 15:51:23 +0200 Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > Mart reported a deadlock in -RT in the call path: > hci_send_monitor_ctrl_event() -> hci_send_to_channel() > > because both functions acquire the same read lock hci_sk_list.lock. > This is also a mainline issue because the qrwlock implementation is > writer fair (the traditional rwlock implementation is reader biased). > > To avoid the deadlock there is now __hci_send_to_channel() which > expects the readlock to be held. > > Cc: Marcel Holtmann <marcel@holtmann.org> > Cc: Johan Hedberg <johan.hedberg@intel.com> > Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT > commands and events to monitor") Reported-by: Mart van de Wege > <mvdwege@gmail.com> Signed-off-by: Sebastian Andrzej Siewior > <bigeasy@linutronix.de> --- > net/bluetooth/hci_sock.c | 17 +++++++++++------ > 1 file changed, 11 insertions(+), 6 deletions(-) > > diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c > index 638bf0e1a2e3..f1ee820d871c 100644 > --- a/net/bluetooth/hci_sock.c > +++ b/net/bluetooth/hci_sock.c > @@ -251,15 +251,13 @@ void hci_send_to_sock(struct hci_dev *hdev, > struct sk_buff *skb) } > > /* Send frame to sockets with specific channel */ > -void hci_send_to_channel(unsigned short channel, struct sk_buff *skb, > - int flag, struct sock *skip_sk) > +static void __hci_send_to_channel(unsigned short channel, struct > sk_buff *skb, > + int flag, struct sock *skip_sk) > { > struct sock *sk; > > BT_DBG("channel %u len %d", channel, skb->len); > > - read_lock(&hci_sk_list.lock); > - > sk_for_each(sk, &hci_sk_list.head) { > struct sk_buff *nskb; > > @@ -285,6 +283,13 @@ void hci_send_to_channel(unsigned short channel, > struct sk_buff *skb, kfree_skb(nskb); > } > > +} > + > +void hci_send_to_channel(unsigned short channel, struct sk_buff *skb, > + int flag, struct sock *skip_sk) > +{ > + read_lock(&hci_sk_list.lock); > + __hci_send_to_channel(channel, skb, flag, skip_sk); > read_unlock(&hci_sk_list.lock); > } > > @@ -388,8 +393,8 @@ void hci_send_monitor_ctrl_event(struct hci_dev > *hdev, u16 event, hdr->index = index; > hdr->len = cpu_to_le16(skb->len - HCI_MON_HDR_SIZE); > > - hci_send_to_channel(HCI_CHANNEL_MONITOR, skb, > - HCI_SOCK_TRUSTED, NULL); > + __hci_send_to_channel(HCI_CHANNEL_MONITOR, skb, > + HCI_SOCK_TRUSTED, NULL); > kfree_skb(skb); > } > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel() @ 2017-09-24 14:53 ` Mart van de Wege 0 siblings, 0 replies; 14+ messages in thread From: Mart van de Wege @ 2017-09-24 14:53 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Marcel Holtmann, Johan Hedberg, Gustavo Padovan, linux-bluetooth-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA, linux-rt-users-u79uwXL29TY76Z2rM5mHXA, tglx-hfZtesqFncYOwBW4kG4KsQ, Peter Zijlstra Confirm that 4.11.12-rt14 with this fix boots normally. On Thu, 21 Sep 2017 15:51:23 +0200 Sebastian Andrzej Siewior <bigeasy-hfZtesqFncYOwBW4kG4KsQ@public.gmane.org> wrote: > Mart reported a deadlock in -RT in the call path: > hci_send_monitor_ctrl_event() -> hci_send_to_channel() > > because both functions acquire the same read lock hci_sk_list.lock. > This is also a mainline issue because the qrwlock implementation is > writer fair (the traditional rwlock implementation is reader biased). > > To avoid the deadlock there is now __hci_send_to_channel() which > expects the readlock to be held. > > Cc: Marcel Holtmann <marcel-kz+m5ild9QBg9hUCZPvPmw@public.gmane.org> > Cc: Johan Hedberg <johan.hedberg-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> > Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT > commands and events to monitor") Reported-by: Mart van de Wege > <mvdwege-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> Signed-off-by: Sebastian Andrzej Siewior > <bigeasy-hfZtesqFncYOwBW4kG4KsQ@public.gmane.org> --- > net/bluetooth/hci_sock.c | 17 +++++++++++------ > 1 file changed, 11 insertions(+), 6 deletions(-) > > diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c > index 638bf0e1a2e3..f1ee820d871c 100644 > --- a/net/bluetooth/hci_sock.c > +++ b/net/bluetooth/hci_sock.c > @@ -251,15 +251,13 @@ void hci_send_to_sock(struct hci_dev *hdev, > struct sk_buff *skb) } > > /* Send frame to sockets with specific channel */ > -void hci_send_to_channel(unsigned short channel, struct sk_buff *skb, > - int flag, struct sock *skip_sk) > +static void __hci_send_to_channel(unsigned short channel, struct > sk_buff *skb, > + int flag, struct sock *skip_sk) > { > struct sock *sk; > > BT_DBG("channel %u len %d", channel, skb->len); > > - read_lock(&hci_sk_list.lock); > - > sk_for_each(sk, &hci_sk_list.head) { > struct sk_buff *nskb; > > @@ -285,6 +283,13 @@ void hci_send_to_channel(unsigned short channel, > struct sk_buff *skb, kfree_skb(nskb); > } > > +} > + > +void hci_send_to_channel(unsigned short channel, struct sk_buff *skb, > + int flag, struct sock *skip_sk) > +{ > + read_lock(&hci_sk_list.lock); > + __hci_send_to_channel(channel, skb, flag, skip_sk); > read_unlock(&hci_sk_list.lock); > } > > @@ -388,8 +393,8 @@ void hci_send_monitor_ctrl_event(struct hci_dev > *hdev, u16 event, hdr->index = index; > hdr->len = cpu_to_le16(skb->len - HCI_MON_HDR_SIZE); > > - hci_send_to_channel(HCI_CHANNEL_MONITOR, skb, > - HCI_SOCK_TRUSTED, NULL); > + __hci_send_to_channel(HCI_CHANNEL_MONITOR, skb, > + HCI_SOCK_TRUSTED, NULL); > kfree_skb(skb); > } > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel() 2017-09-21 13:51 ` [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel() Sebastian Andrzej Siewior 2017-09-24 14:53 ` Mart van de Wege @ 2017-10-30 8:05 ` Marcel Holtmann 1 sibling, 0 replies; 14+ messages in thread From: Marcel Holtmann @ 2017-10-30 8:05 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Mart van de Wege, Johan Hedberg, Gustavo F. Padovan, open list:BLUETOOTH DRIVERS, Linux Kernel Mailing List, linux-rt-users, Thomas Gleixner, Peter Zijlstra Hi Sebastian, > Mart reported a deadlock in -RT in the call path: > hci_send_monitor_ctrl_event() -> hci_send_to_channel() > > because both functions acquire the same read lock hci_sk_list.lock. This > is also a mainline issue because the qrwlock implementation is writer > fair (the traditional rwlock implementation is reader biased). > > To avoid the deadlock there is now __hci_send_to_channel() which expects > the readlock to be held. > > Cc: Marcel Holtmann <marcel@holtmann.org> > Cc: Johan Hedberg <johan.hedberg@intel.com> > Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT commands and events to monitor") > Reported-by: Mart van de Wege <mvdwege@gmail.com> > Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> > --- > net/bluetooth/hci_sock.c | 17 +++++++++++------ > 1 file changed, 11 insertions(+), 6 deletions(-) patch has been applied to bluetooth-next tree. Regards Marcel ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2017-10-30 8:05 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-08-30 21:25 kernel BUG at kernel/locking/rtmutex.c:1059 Mart van de Wege 2017-08-31 10:59 ` Pratyush Patel 2017-08-31 14:54 ` Sebastian Andrzej Siewior 2017-08-31 16:02 ` Pratyush Patel 2017-08-31 14:52 ` Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059) Sebastian Andrzej Siewior 2017-08-31 14:56 ` Thomas Gleixner 2017-09-01 8:51 ` Mart van de Wege 2017-09-02 7:27 ` Mart van de Wege 2017-09-02 7:27 ` Mart van de Wege 2017-09-02 7:27 ` Mart van de Wege 2017-09-21 13:51 ` [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel() Sebastian Andrzej Siewior 2017-09-24 14:53 ` Mart van de Wege 2017-09-24 14:53 ` Mart van de Wege 2017-10-30 8:05 ` Marcel Holtmann
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.