From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Song Liu <songliubraving@fb.com>,
Alexei Starovoitov <ast@kernel.org>,
Sasha Levin <sashal@kernel.org>,
netdev@vger.kernel.org, bpf@vger.kernel.org
Subject: [PATCH AUTOSEL 5.10 014/217] bpf: Use separate lockdep class for each hashtab
Date: Tue, 22 Dec 2020 21:13:03 -0500 [thread overview]
Message-ID: <20201223021626.2790791-14-sashal@kernel.org> (raw)
In-Reply-To: <20201223021626.2790791-1-sashal@kernel.org>
From: Song Liu <songliubraving@fb.com>
[ Upstream commit c50eb518e262fa06bd334e6eec172eaf5d7a5bd9 ]
If a hashtab is accessed in both NMI and non-NMI contexts, it may cause
deadlock in bucket->lock. LOCKDEP NMI warning highlighted this issue:
./test_progs -t stacktrace
[ 74.828970]
[ 74.828971] ================================
[ 74.828972] WARNING: inconsistent lock state
[ 74.828973] 5.9.0-rc8+ #275 Not tainted
[ 74.828974] --------------------------------
[ 74.828975] inconsistent {INITIAL USE} -> {IN-NMI} usage.
[ 74.828976] taskset/1174 [HC2[2]:SC0[0]:HE0:SE1] takes:
[ 74.828977] ffffc90000ee96b0 (&htab->buckets[i].raw_lock){....}-{2:2}, at: htab_map_update_elem+0x271/0x5a0
[ 74.828981] {INITIAL USE} state was registered at:
[ 74.828982] lock_acquire+0x137/0x510
[ 74.828983] _raw_spin_lock_irqsave+0x43/0x90
[ 74.828984] htab_map_update_elem+0x271/0x5a0
[ 74.828984] 0xffffffffa0040b34
[ 74.828985] trace_call_bpf+0x159/0x310
[ 74.828986] perf_trace_run_bpf_submit+0x5f/0xd0
[ 74.828987] perf_trace_urandom_read+0x1be/0x220
[ 74.828988] urandom_read_nowarn.isra.0+0x26f/0x380
[ 74.828989] vfs_read+0xf8/0x280
[ 74.828989] ksys_read+0xc9/0x160
[ 74.828990] do_syscall_64+0x33/0x40
[ 74.828991] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 74.828992] irq event stamp: 1766
[ 74.828993] hardirqs last enabled at (1765): [<ffffffff82800ace>] asm_exc_page_fault+0x1e/0x30
[ 74.828994] hardirqs last disabled at (1766): [<ffffffff8267df87>] irqentry_enter+0x37/0x60
[ 74.828995] softirqs last enabled at (856): [<ffffffff81043e7c>] fpu__clear+0xac/0x120
[ 74.828996] softirqs last disabled at (854): [<ffffffff81043df0>] fpu__clear+0x20/0x120
[ 74.828997]
[ 74.828998] other info that might help us debug this:
[ 74.828999] Possible unsafe locking scenario:
[ 74.828999]
[ 74.829000] CPU0
[ 74.829001] ----
[ 74.829001] lock(&htab->buckets[i].raw_lock);
[ 74.829003] <Interrupt>
[ 74.829004] lock(&htab->buckets[i].raw_lock);
[ 74.829006]
[ 74.829006] *** DEADLOCK ***
[ 74.829007]
[ 74.829008] 1 lock held by taskset/1174:
[ 74.829008] #0: ffff8883ec3fd020 (&cpuctx_lock){-...}-{2:2}, at: perf_event_task_tick+0x101/0x650
[ 74.829012]
[ 74.829013] stack backtrace:
[ 74.829014] CPU: 0 PID: 1174 Comm: taskset Not tainted 5.9.0-rc8+ #275
[ 74.829015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
[ 74.829016] Call Trace:
[ 74.829016] <NMI>
[ 74.829017] dump_stack+0x9a/0xd0
[ 74.829018] lock_acquire+0x461/0x510
[ 74.829019] ? lock_release+0x6b0/0x6b0
[ 74.829020] ? stack_map_get_build_id_offset+0x45e/0x800
[ 74.829021] ? htab_map_update_elem+0x271/0x5a0
[ 74.829022] ? rcu_read_lock_held_common+0x1a/0x50
[ 74.829022] ? rcu_read_lock_held+0x5f/0xb0
[ 74.829023] _raw_spin_lock_irqsave+0x43/0x90
[ 74.829024] ? htab_map_update_elem+0x271/0x5a0
[ 74.829025] htab_map_update_elem+0x271/0x5a0
[ 74.829026] bpf_prog_1fd9e30e1438d3c5_oncpu+0x9c/0xe88
[ 74.829027] bpf_overflow_handler+0x127/0x320
[ 74.829028] ? perf_event_text_poke_output+0x4d0/0x4d0
[ 74.829029] ? sched_clock_cpu+0x18/0x130
[ 74.829030] __perf_event_overflow+0xae/0x190
[ 74.829030] handle_pmi_common+0x34c/0x470
[ 74.829031] ? intel_pmu_save_and_restart+0x90/0x90
[ 74.829032] ? lock_acquire+0x3f8/0x510
[ 74.829033] ? lock_release+0x6b0/0x6b0
[ 74.829034] intel_pmu_handle_irq+0x11e/0x240
[ 74.829034] perf_event_nmi_handler+0x40/0x60
[ 74.829035] nmi_handle+0x110/0x360
[ 74.829036] ? __intel_pmu_enable_all.constprop.0+0x72/0xf0
[ 74.829037] default_do_nmi+0x6b/0x170
[ 74.829038] exc_nmi+0x106/0x130
[ 74.829038] end_repeat_nmi+0x16/0x55
[ 74.829039] RIP: 0010:__intel_pmu_enable_all.constprop.0+0x72/0xf0
[ 74.829042] Code: 2f 1f 03 48 8d bb b8 0c 00 00 e8 29 09 41 00 48 ...
[ 74.829043] RSP: 0000:ffff8880a604fc90 EFLAGS: 00000002
[ 74.829044] RAX: 000000070000000f RBX: ffff8883ec2195a0 RCX: 000000000000038f
[ 74.829045] RDX: 0000000000000007 RSI: ffffffff82e72c20 RDI: ffff8883ec21a258
[ 74.829046] RBP: 000000070000000f R08: ffffffff8101b013 R09: fffffbfff0a7982d
[ 74.829047] R10: ffffffff853cc167 R11: fffffbfff0a7982c R12: 0000000000000000
[ 74.829049] R13: ffff8883ec3f0af0 R14: ffff8883ec3fd120 R15: ffff8883e9c92098
[ 74.829049] ? intel_pmu_lbr_enable_all+0x43/0x240
[ 74.829050] ? __intel_pmu_enable_all.constprop.0+0x72/0xf0
[ 74.829051] ? __intel_pmu_enable_all.constprop.0+0x72/0xf0
[ 74.829052] </NMI>
[ 74.829053] perf_event_task_tick+0x48d/0x650
[ 74.829054] scheduler_tick+0x129/0x210
[ 74.829054] update_process_times+0x37/0x70
[ 74.829055] tick_sched_handle.isra.0+0x35/0x90
[ 74.829056] tick_sched_timer+0x8f/0xb0
[ 74.829057] __hrtimer_run_queues+0x364/0x7d0
[ 74.829058] ? tick_sched_do_timer+0xa0/0xa0
[ 74.829058] ? enqueue_hrtimer+0x1e0/0x1e0
[ 74.829059] ? recalibrate_cpu_khz+0x10/0x10
[ 74.829060] ? ktime_get_update_offsets_now+0x1a3/0x360
[ 74.829061] hrtimer_interrupt+0x1bb/0x360
[ 74.829062] ? rcu_read_lock_sched_held+0xa1/0xd0
[ 74.829063] __sysvec_apic_timer_interrupt+0xed/0x3d0
[ 74.829064] sysvec_apic_timer_interrupt+0x3f/0xd0
[ 74.829064] ? asm_sysvec_apic_timer_interrupt+0xa/0x20
[ 74.829065] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 74.829066] RIP: 0033:0x7fba18d579b4
[ 74.829068] Code: 74 54 44 0f b6 4a 04 41 83 e1 0f 41 80 f9 ...
[ 74.829069] RSP: 002b:00007ffc9ba69570 EFLAGS: 00000206
[ 74.829071] RAX: 00007fba192084c0 RBX: 00007fba18c24d28 RCX: 00000000000007a4
[ 74.829072] RDX: 00007fba18c30488 RSI: 0000000000000000 RDI: 000000000000037b
[ 74.829073] RBP: 00007fba18ca5760 R08: 00007fba18c248fc R09: 00007fba18c94c30
[ 74.829074] R10: 000000000000002f R11: 0000000000073c30 R12: 00007ffc9ba695e0
[ 74.829075] R13: 00000000000003f3 R14: 00007fba18c21ac8 R15: 00000000000058d6
However, such warning should not apply across multiple hashtabs. The
system will not deadlock if one hashtab is used in NMI, while another
hashtab is used in non-NMI.
Use separate lockdep class for each hashtab, so that we don't get this
false alert.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20201029071925.3103400-2-songliubraving@fb.com
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
kernel/bpf/hashtab.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 1fccba6e88c4e..224e2652a7c64 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -99,6 +99,7 @@ struct bpf_htab {
u32 n_buckets; /* number of hash buckets */
u32 elem_size; /* size of each element in bytes */
u32 hashrnd;
+ struct lock_class_key lockdep_key;
};
/* each htab element is struct htab_elem + key + value */
@@ -136,12 +137,18 @@ static void htab_init_buckets(struct bpf_htab *htab)
{
unsigned i;
+ lockdep_register_key(&htab->lockdep_key);
for (i = 0; i < htab->n_buckets; i++) {
INIT_HLIST_NULLS_HEAD(&htab->buckets[i].head, i);
- if (htab_use_raw_lock(htab))
+ if (htab_use_raw_lock(htab)) {
raw_spin_lock_init(&htab->buckets[i].raw_lock);
- else
+ lockdep_set_class(&htab->buckets[i].raw_lock,
+ &htab->lockdep_key);
+ } else {
spin_lock_init(&htab->buckets[i].lock);
+ lockdep_set_class(&htab->buckets[i].lock,
+ &htab->lockdep_key);
+ }
}
}
@@ -1338,6 +1345,7 @@ static void htab_map_free(struct bpf_map *map)
free_percpu(htab->extra_elems);
bpf_map_area_free(htab->buckets);
+ lockdep_unregister_key(&htab->lockdep_key);
kfree(htab);
}
--
2.27.0
next prev parent reply other threads:[~2020-12-23 2:18 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-23 2:12 [PATCH AUTOSEL 5.10 001/217] soc: aspeed-lpc-ctrl: Fail probe of lpc-ctrl if reserved memory is not aligned Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 002/217] drm/ingenic: Reset pixclock rate when parent clock rate changes Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 003/217] drm/bridge: ti-sn65dsi86: Add retries for link training Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 004/217] drm/amd/display: setup system context in dm_init Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 005/217] drm/amd/display: Fix the display corruption issue on Navi10 Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 006/217] locks: Fix UBSAN undefined behaviour in flock64_to_posix_lock Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 007/217] platform/x86: asus-wmi: Add support for SW_TABLET_MODE on UX360 Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 008/217] tomoyo: fix clang pointer arithmetic warning Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 009/217] HID: hid-input: occasionally report stylus battery even if not changed Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 010/217] drm/amdgpu: change to save bad pages in UMC error interrupt callback Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 011/217] crypto: omap-aes - fix the reference count leak of omap device Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 012/217] drm/amd/display: Do not silently accept DCC for multiplane formats Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 013/217] ASoC: intel: sof_rt5682: Add quirk for Dooly Sasha Levin
2020-12-23 2:13 ` Sasha Levin [this message]
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 015/217] drm/msm: Fix race condition in msm driver with async layer updates Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 016/217] staging: wimax: depends on NET Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 017/217] RDMA/siw: Fix typo of EAGAIN not -EAGAIN in siw_cm_work_handler() Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 018/217] drm/amd/display: Fix compilation error Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 019/217] drm/amd/display: Force prefetch mode to 0 Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 020/217] drm/amd/display: Keep GSL for full updates with planes that flip VSYNC Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 021/217] drm/amd/display: stop top_mgr when type change to non-MST during s3 Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 022/217] drm/amd/display: correct eDP T9 delay Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 023/217] drm/amd/display: Update connector on DSC property change Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 024/217] drm/amd/display: fix recout calculation for left side clip Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 025/217] drm/amdgpu: disable gfxoff if VCN is busy Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 026/217] net: ethernet: ti: am65-cpsw: fix tx csum offload for multi mac mode Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 027/217] net: mscc: ocelot: don't reset the pvid to 0 when deleting it Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 028/217] drm/amdgpu: set LDS_CONFIG=0x20 on Navy Flounder to fix a GPU hang (v2) Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 029/217] scsi: target: Fix cmd_count ref leak Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 030/217] scsi: pm80xx: Make mpi_build_cmd locking consistent Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 031/217] scsi: pm80xx: Make running_req atomic Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 032/217] scsi: pm80xx: Avoid busywait in FW ready check Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 033/217] scsi: pm80xx: Fix pm8001_mpi_get_nvmd_resp() race condition Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 034/217] scsi: ufs: Allow an error return value from ->device_reset() Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 035/217] fcntl: Fix potential deadlock in send_sig{io, urg}() Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 036/217] drm/amdgpu: add missing clock gating info in amdgpu_pm_info Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 037/217] mac80211: don't overwrite QoS TID of injected frames Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 038/217] staging: ks7010: fix missing destroy_workqueue() on error in ks7010_sdio_probe Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 039/217] staging: rtl8192u: fix wrong judgement in rtl8192_rx_isr Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 040/217] mips: ar7: add missing iounmap() on error in ar7_gpio_init Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 041/217] mips: cm: add missing iounmap() on error in mips_cm_probe() Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 042/217] usb: typec: ucsi: Work around PPM losing change information Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 043/217] rcuscale: Prevent hangs for invalid arguments Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 044/217] refscale: " Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 045/217] locktorture: " Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 046/217] torture: Prevent jitter processes from delaying failed run Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 047/217] rcutorture: Prevent hangs for invalid arguments Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 048/217] kcsan: Fix encoding masks and regain address bit Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 049/217] ath10k: fix compilation warning Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 050/217] rsi: Fix TX EAPOL packet handling against iwlwifi AP Sasha Levin
2021-03-02 19:25 ` Marek Vasut
2021-03-04 20:47 ` Sasha Levin
2021-03-04 21:07 ` Marek Vasut
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 051/217] thermal: intel: pch: fix S0ix failure due to PCH temperature above threshold Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 052/217] drm: panel: simple: add missing platform_driver_unregister() in panel_simple_init Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 053/217] drm/bridge: lvds-codec: Use dev_err_probe for error handling Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 054/217] drm/ast: Fixed 1920x1080 sync. polarity issue Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 055/217] s390: make sure vmemmap is top region table entry aligned Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 056/217] s390/pci: remove races against pte updates Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 057/217] s390/trng: set quality to 1024 Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 058/217] Bluetooth: btqca: Add valid le states quirk Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 059/217] Bluetooth: Resume advertising after LE connection Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 060/217] Bluetooth: Move force_bredr_smp debugfs into hci_debugfs_create_bredr Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 061/217] Bluetooth: hidp: use correct wait queue when removing ctrl_wait Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 062/217] binder: change error code from postive to negative in binder_transaction Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 063/217] net: skb_vlan_untag(): don't reset transport offset if set by GRO layer Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 064/217] drm/omap: Fix runtime PM imbalance on error Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 065/217] pinctrl: qcom: Kconfig: Rework PINCTRL_MSM to be a depenency rather then a selected config Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 066/217] ASoC: Fix vaud18 power leakage of mt6359 Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 067/217] mwifiex: pcie: skip cancel_work_sync() on reset failure path Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 068/217] drm/amd/amdgpu: Fix incorrect logic to increment VCN doorbell index Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 069/217] drm/amd/amdgpu: Add rev_id workaround logic for SRIOV setup Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 070/217] ASoC: Intel: sof_sdw: add quirk for new TigerLake-SDCA device Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 071/217] RDMA/core: Postpone uobject cleanup on failure till FD close Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 072/217] MIPS: BMC47xx: fix kconfig dependency bug for BCM47XX_SSB Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 073/217] net: ipconfig: Avoid spurious blank lines in boot log Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 074/217] drm/amd/amdgpu: Update VCN initizalization behvaior Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 075/217] drm/amdgpu: check hive pointer before access Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 076/217] jfs: Fix memleak in dbAdjCtl Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 077/217] r8169: use READ_ONCE in rtl_tx_slots_avail Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201223021626.2790791-14-sashal@kernel.org \
--to=sashal@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=songliubraving@fb.com \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox