From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Bart Van Assche <bart.vanassche@wdc.com>,
Tejun Heo <tj@kernel.org>,
Johannes Thumshirn <jthumshirn@suse.de>,
"Martin K. Petersen" <martin.petersen@oracle.com>
Subject: [PATCH 4.9 100/107] scsi: core: Avoid that SCSI device removal through sysfs triggers a deadlock
Date: Mon, 3 Sep 2018 18:57:04 +0200 [thread overview]
Message-ID: <20180903165642.465724701@linuxfoundation.org> (raw)
In-Reply-To: <20180903165637.293735109@linuxfoundation.org>
4.9-stable review patch. If anyone has any objections, please let me know.
------------------
From: Bart Van Assche <bart.vanassche@wdc.com>
commit 0ee223b2e1f67cb2de9c0e3247c510d846e74d63 upstream.
A long time ago the unfortunate decision was taken to add a self-deletion
attribute to the sysfs SCSI device directory. That decision was unfortunate
because self-deletion is really tricky. We can't drop that attribute
because widely used user space software depends on it, namely the
rescan-scsi-bus.sh script. Hence this patch that avoids that writing into
that attribute triggers a deadlock. See also commit 7973cbd9fbd9 ("[PATCH]
add sysfs attributes to scan and delete scsi_devices").
This patch avoids that self-removal triggers the following deadlock:
======================================================
WARNING: possible circular locking dependency detected
4.18.0-rc2-dbg+ #5 Not tainted
------------------------------------------------------
modprobe/6539 is trying to acquire lock:
000000008323c4cd (kn->count#202){++++}, at: kernfs_remove_by_name_ns+0x45/0x90
but task is already holding lock:
00000000a6ec2c69 (&shost->scan_mutex){+.+.}, at: scsi_remove_host+0x21/0x150 [scsi_mod]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&shost->scan_mutex){+.+.}:
__mutex_lock+0xfe/0xc70
mutex_lock_nested+0x1b/0x20
scsi_remove_device+0x26/0x40 [scsi_mod]
sdev_store_delete+0x27/0x30 [scsi_mod]
dev_attr_store+0x3e/0x50
sysfs_kf_write+0x87/0xa0
kernfs_fop_write+0x190/0x230
__vfs_write+0xd2/0x3b0
vfs_write+0x101/0x270
ksys_write+0xab/0x120
__x64_sys_write+0x43/0x50
do_syscall_64+0x77/0x230
entry_SYSCALL_64_after_hwframe+0x49/0xbe
-> #0 (kn->count#202){++++}:
lock_acquire+0xd2/0x260
__kernfs_remove+0x424/0x4a0
kernfs_remove_by_name_ns+0x45/0x90
remove_files.isra.1+0x3a/0x90
sysfs_remove_group+0x5c/0xc0
sysfs_remove_groups+0x39/0x60
device_remove_attrs+0x82/0xb0
device_del+0x251/0x580
__scsi_remove_device+0x19f/0x1d0 [scsi_mod]
scsi_forget_host+0x37/0xb0 [scsi_mod]
scsi_remove_host+0x9b/0x150 [scsi_mod]
sdebug_driver_remove+0x4b/0x150 [scsi_debug]
device_release_driver_internal+0x241/0x360
device_release_driver+0x12/0x20
bus_remove_device+0x1bc/0x290
device_del+0x259/0x580
device_unregister+0x1a/0x70
sdebug_remove_adapter+0x8b/0xf0 [scsi_debug]
scsi_debug_exit+0x76/0xe8 [scsi_debug]
__x64_sys_delete_module+0x1c1/0x280
do_syscall_64+0x77/0x230
entry_SYSCALL_64_after_hwframe+0x49/0xbe
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&shost->scan_mutex);
lock(kn->count#202);
lock(&shost->scan_mutex);
lock(kn->count#202);
*** DEADLOCK ***
2 locks held by modprobe/6539:
#0: 00000000efaf9298 (&dev->mutex){....}, at: device_release_driver_internal+0x68/0x360
#1: 00000000a6ec2c69 (&shost->scan_mutex){+.+.}, at: scsi_remove_host+0x21/0x150 [scsi_mod]
stack backtrace:
CPU: 10 PID: 6539 Comm: modprobe Not tainted 4.18.0-rc2-dbg+ #5
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014
Call Trace:
dump_stack+0xa4/0xf5
print_circular_bug.isra.34+0x213/0x221
__lock_acquire+0x1a7e/0x1b50
lock_acquire+0xd2/0x260
__kernfs_remove+0x424/0x4a0
kernfs_remove_by_name_ns+0x45/0x90
remove_files.isra.1+0x3a/0x90
sysfs_remove_group+0x5c/0xc0
sysfs_remove_groups+0x39/0x60
device_remove_attrs+0x82/0xb0
device_del+0x251/0x580
__scsi_remove_device+0x19f/0x1d0 [scsi_mod]
scsi_forget_host+0x37/0xb0 [scsi_mod]
scsi_remove_host+0x9b/0x150 [scsi_mod]
sdebug_driver_remove+0x4b/0x150 [scsi_debug]
device_release_driver_internal+0x241/0x360
device_release_driver+0x12/0x20
bus_remove_device+0x1bc/0x290
device_del+0x259/0x580
device_unregister+0x1a/0x70
sdebug_remove_adapter+0x8b/0xf0 [scsi_debug]
scsi_debug_exit+0x76/0xe8 [scsi_debug]
__x64_sys_delete_module+0x1c1/0x280
do_syscall_64+0x77/0x230
entry_SYSCALL_64_after_hwframe+0x49/0xbe
See also https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg54525.html.
Fixes: ac0ece9174ac ("scsi: use device_remove_file_self() instead of device_schedule_callback()")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Johannes Thumshirn <jthumshirn@suse.de>
Cc: <stable@vger.kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
---
drivers/scsi/scsi_sysfs.c | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
--- a/drivers/scsi/scsi_sysfs.c
+++ b/drivers/scsi/scsi_sysfs.c
@@ -709,8 +709,24 @@ static ssize_t
sdev_store_delete(struct device *dev, struct device_attribute *attr,
const char *buf, size_t count)
{
- if (device_remove_file_self(dev, attr))
- scsi_remove_device(to_scsi_device(dev));
+ struct kernfs_node *kn;
+
+ kn = sysfs_break_active_protection(&dev->kobj, &attr->attr);
+ WARN_ON_ONCE(!kn);
+ /*
+ * Concurrent writes into the "delete" sysfs attribute may trigger
+ * concurrent calls to device_remove_file() and scsi_remove_device().
+ * device_remove_file() handles concurrent removal calls by
+ * serializing these and by ignoring the second and later removal
+ * attempts. Concurrent calls of scsi_remove_device() are
+ * serialized. The second and later calls of scsi_remove_device() are
+ * ignored because the first call of that function changes the device
+ * state into SDEV_DEL.
+ */
+ device_remove_file(dev, attr);
+ scsi_remove_device(to_scsi_device(dev));
+ if (kn)
+ sysfs_unbreak_active_protection(kn);
return count;
};
static DEVICE_ATTR(delete, S_IWUSR, NULL, sdev_store_delete);
next prev parent reply other threads:[~2018-09-03 21:36 UTC|newest]
Thread overview: 105+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-03 16:55 [PATCH 4.9 000/107] 4.9.125-stable review Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 001/107] vti6: fix PMTU caching and reporting on xmit Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 002/107] xfrm: fix missing dst_release() after policy blocking lbcast and multicast Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 003/107] xfrm: free skb if nlsk pointer is NULL Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 004/107] mac80211: add stations tied to AP_VLANs during hw reconfig Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 005/107] nl80211: Add a missing break in parse_station_flags Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 006/107] drm/bridge: adv7511: Reset registers on hotplug Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 007/107] scsi: libiscsi: fix possible NULL pointer dereference in case of TMF Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 008/107] drm/imx: imx-ldb: disable LDB on driver bind Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 009/107] drm/imx: imx-ldb: check if channel is enabled before printing warning Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 010/107] usb: gadget: r8a66597: Fix two possible sleep-in-atomic-context bugs in init_controller() Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 011/107] usb: gadget: r8a66597: Fix a possible sleep-in-atomic-context bugs in r8a66597_queue() Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 012/107] usb/phy: fix PPC64 build errors in phy-fsl-usb.c Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 014/107] usb: gadget: f_uac2: fix endianness of struct cntrl_*_lay3 Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 015/107] bpf, ppc64: fix unexpected r0=0 exit path inside bpf_xadd Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 016/107] tools/power turbostat: fix -S on UP systems Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 017/107] net: caif: Add a missing rcu_read_unlock() in caif_flow_cb Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 018/107] qed: Fix possible race for the link state value Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 019/107] qed: Correct Multicast API to reflect existence of 256 approximate buckets Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 020/107] atl1c: reserve min skb headroom Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 021/107] net: prevent ISA drivers from building on PPC32 Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 022/107] can: mpc5xxx_can: check of_iomap return before use Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 023/107] i2c: davinci: Avoid zero value of CLKH Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 024/107] perf/x86/amd/ibs: Dont access non-started event Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 025/107] media: staging: omap4iss: Include asm/cacheflush.h after generic includes Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 026/107] bnx2x: Fix invalid memory access in rss hash config path Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 028/107] net: axienet: Fix double deregister of mdio Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 029/107] x86/boot: Fix if_changed build flip/flop bug Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 030/107] fscache: Allow cancelled operations to be enqueued Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 031/107] cachefiles: Fix refcounting bug in backing-file read monitoring Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 032/107] cachefiles: Wait rather than BUGing on "Unexpected object collision" Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 033/107] selftests/ftrace: Add snapshot and tracing_on test case Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 034/107] zswap: re-check zswap_is_full() after do zswap_shrink() Greg Kroah-Hartman
2018-09-03 16:55 ` [PATCH 4.9 035/107] tools/power turbostat: Read extended processor family from CPUID Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 037/107] enic: handle mtu change for vf properly Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 038/107] arc: [plat-eznps] fix data type errors in platform headers Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 039/107] arc: fix build errors in arc/include/asm/delay.h Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 040/107] arc: fix type warnings in arc/mm/cache.c Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 041/107] squashfs metadata 2: electric boogaloo Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 043/107] drivers: net: lmc: fix case value for target abort error Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 044/107] memcg: remove memcg_cgroup::id from IDR on mem_cgroup_css_alloc() failure Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 045/107] scsi: fcoe: drop frames in ELS LOGO error path Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 046/107] scsi: fcoe: clear FC_RP_STARTED flags when receiving a LOGO Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 047/107] scsi: vmw_pvscsi: Return DID_RESET for status SAM_STAT_COMMAND_TERMINATED Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 048/107] mm/memory.c: check return value of ioremap_prot Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 049/107] sched/sysctl: Check user input value of sysctl_sched_time_avg Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 050/107] Cipso: cipso_v4_optptr enter infinite loop Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 051/107] mei: dont update offset in write Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 052/107] cifs: add missing debug entries for kconfig options Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 053/107] cifs: check kmalloc before use Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 054/107] smb3: enumerating snapshots was leaving part of the data off end Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 055/107] smb3: Do not send SMB3 SET_INFO if nothing changed Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 056/107] smb3: dont request leases in symlink creation and query Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 057/107] kprobes/arm64: Fix %p uses in error messages Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 058/107] arm64: mm: check for upper PAGE_SHIFT bits in pfn_valid() Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 059/107] s390/kvm: fix deadlock when killed by oom Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 060/107] ext4: check for NUL characters in extended attributes name Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 061/107] ext4: sysfs: print ext4_super_block fields as little-endian Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 062/107] ext4: reset error code in ext4_find_entry in fallback Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 063/107] staging: android: ion: fix ION_IOC_{MAP,SHARE} use-after-free Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 064/107] KVM: arm/arm64: Skip updating PTE entry if no change Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 065/107] KVM: arm/arm64: Skip updating PMD " Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 067/107] x86/speculation/l1tf: Fix overflow in l1tf_pfn_limit() on 32bit Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 068/107] x86/speculation/l1tf: Fix off-by-one error when warning that system has too much RAM Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 069/107] x86/speculation/l1tf: Suggest what to do on systems with " Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 070/107] x86/process: Re-export start_thread() Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 072/107] x86/kvm/vmx: Remove duplicate l1d flush definitions Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 073/107] fuse: Dont access pipe->buffers without pipe_lock() Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 074/107] fuse: fix initial parallel dirops Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 075/107] fuse: fix double request_end() Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 076/107] fuse: fix unlocked access to processing queue Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 077/107] fuse: umount should wait for all requests Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 078/107] fuse: Fix oops at process_init_reply() Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 079/107] fuse: Add missed unlock_page() to fuse_readpages_fill() Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 080/107] udl-kms: change down_interruptible to down Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 081/107] udl-kms: handle allocation failure Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 082/107] udl-kms: fix crash due to uninitialized memory Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 083/107] b43legacy/leds: Ensure NUL-termination of LED name string Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 084/107] b43/leds: " Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 085/107] ASoC: dpcm: dont merge format from invalid codec dai Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 086/107] ASoC: sirf: Fix potential NULL pointer dereference Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 088/107] x86/irqflags: Mark native_restore_fl extern inline Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 089/107] x86/spectre: Add missing family 6 check to microcode check Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 090/107] x86/speculation/l1tf: Increase l1tf memory limit for Nehalem+ Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 091/107] x86/entry/64: Wipe KASAN stack shadow before rewind_stack_do_exit() Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 092/107] s390: fix br_r1_trampoline for machines without exrl Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 093/107] s390/qdio: reset old sbal_state flags Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 094/107] s390/numa: move initial setup of node_to_cpumask_map Greg Kroah-Hartman
2018-09-03 16:56 ` [PATCH 4.9 095/107] s390/pci: fix out of bounds access during irq setup Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 096/107] kprobes: Make list and blacklist root user read only Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 097/107] MIPS: Correct the 64-bit DSP accumulator register size Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 098/107] MIPS: lib: Provide MIPS64r6 __multi3() for GCC < 7 Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 099/107] scsi: sysfs: Introduce sysfs_{un,}break_active_protection() Greg Kroah-Hartman
2018-09-03 16:57 ` Greg Kroah-Hartman [this message]
2018-09-03 16:57 ` [PATCH 4.9 101/107] iscsi target: fix session creation failure handling Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 102/107] clk: rockchip: fix clk_i2sout parent selection bits on rk3399 Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 103/107] PM / clk: signedness bug in of_pm_clk_add_clks() Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 104/107] power: generic-adc-battery: fix out-of-bounds write when copying channel properties Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 105/107] power: generic-adc-battery: check for duplicate properties copied from iio channels Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 106/107] cdrom: Fix info leak/OOB read in cdrom_ioctl_drive_status Greg Kroah-Hartman
2018-09-03 16:57 ` [PATCH 4.9 107/107] staging: android: ion: check for kref overflow Greg Kroah-Hartman
2018-09-04 8:22 ` [PATCH 4.9 000/107] 4.9.125-stable review Naresh Kamboju
2018-09-04 19:33 ` Greg Kroah-Hartman
2018-09-04 19:29 ` Shuah Khan
2018-09-04 22:51 ` Guenter Roeck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180903165642.465724701@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=bart.vanassche@wdc.com \
--cc=jthumshirn@suse.de \
--cc=linux-kernel@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=stable@vger.kernel.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).