public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Prasad Sodagudi <psodagud@codeaurora.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Chenbo Feng <fengc@google.com>,
	Richard Smith <richard-llvm@metafoo.co.uk>,
	Chandler Carruth <chandlerc@gmail.com>,
	Alexei Starovoitov <ast@kernel.org>
Subject: [PATCH 4.14 099/101] kbuild: disable clangs default use of -fmerge-all-constants
Date: Tue, 27 Mar 2018 18:28:11 +0200	[thread overview]
Message-ID: <20180327162756.067317846@linuxfoundation.org> (raw)
In-Reply-To: <20180327162749.993880276@linuxfoundation.org>

4.14-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Daniel Borkmann <daniel@iogearbox.net>

commit 87e0d4f0f37fb0c8c4aeeac46fff5e957738df79 upstream.

Prasad reported that he has seen crashes in BPF subsystem with netd
on Android with arm64 in the form of (note, the taint is unrelated):

  [ 4134.721483] Unable to handle kernel paging request at virtual address 800000001
  [ 4134.820925] Mem abort info:
  [ 4134.901283]   Exception class = DABT (current EL), IL = 32 bits
  [ 4135.016736]   SET = 0, FnV = 0
  [ 4135.119820]   EA = 0, S1PTW = 0
  [ 4135.201431] Data abort info:
  [ 4135.301388]   ISV = 0, ISS = 0x00000021
  [ 4135.359599]   CM = 0, WnR = 0
  [ 4135.470873] user pgtable: 4k pages, 39-bit VAs, pgd = ffffffe39b946000
  [ 4135.499757] [0000000800000001] *pgd=0000000000000000, *pud=0000000000000000
  [ 4135.660725] Internal error: Oops: 96000021 [#1] PREEMPT SMP
  [ 4135.674610] Modules linked in:
  [ 4135.682883] CPU: 5 PID: 1260 Comm: netd Tainted: G S      W       4.14.19+ #1
  [ 4135.716188] task: ffffffe39f4aa380 task.stack: ffffff801d4e0000
  [ 4135.731599] PC is at bpf_prog_add+0x20/0x68
  [ 4135.741746] LR is at bpf_prog_inc+0x20/0x2c
  [ 4135.751788] pc : [<ffffff94ab7ad584>] lr : [<ffffff94ab7ad638>] pstate: 60400145
  [ 4135.769062] sp : ffffff801d4e3ce0
  [...]
  [ 4136.258315] Process netd (pid: 1260, stack limit = 0xffffff801d4e0000)
  [ 4136.273746] Call trace:
  [...]
  [ 4136.442494] 3ca0: ffffff94ab7ad584 0000000060400145 ffffffe3a01bf8f8 0000000000000006
  [ 4136.460936] 3cc0: 0000008000000000 ffffff94ab844204 ffffff801d4e3cf0 ffffff94ab7ad584
  [ 4136.479241] [<ffffff94ab7ad584>] bpf_prog_add+0x20/0x68
  [ 4136.491767] [<ffffff94ab7ad638>] bpf_prog_inc+0x20/0x2c
  [ 4136.504536] [<ffffff94ab7b5d08>] bpf_obj_get_user+0x204/0x22c
  [ 4136.518746] [<ffffff94ab7ade68>] SyS_bpf+0x5a8/0x1a88

Android's netd was basically pinning the uid cookie BPF map in BPF
fs (/sys/fs/bpf/traffic_cookie_uid_map) and later on retrieving it
again resulting in above panic. Issue is that the map was wrongly
identified as a prog! Above kernel was compiled with clang 4.0,
and it turns out that clang decided to merge the bpf_prog_iops and
bpf_map_iops into a single memory location, such that the two i_ops
could then not be distinguished anymore.

Reason for this miscompilation is that clang has the more aggressive
-fmerge-all-constants enabled by default. In fact, clang source code
has a comment about it in lib/AST/ExprConstant.cpp on why it is okay
to do so:

  Pointers with different bases cannot represent the same object.
  (Note that clang defaults to -fmerge-all-constants, which can
  lead to inconsistent results for comparisons involving the address
  of a constant; this generally doesn't matter in practice.)

The issue never appeared with gcc however, since gcc does not enable
-fmerge-all-constants by default and even *explicitly* states in
it's option description that using this flag results in non-conforming
behavior, quote from man gcc:

  Languages like C or C++ require each variable, including multiple
  instances of the same variable in recursive calls, to have distinct
  locations, so using this option results in non-conforming behavior.

There are also various clang bug reports open on that matter [1],
where clang developers acknowledge the non-conforming behavior,
and refer to disabling it with -fno-merge-all-constants. But even
if this gets fixed in clang today, there are already users out there
that triggered this. Thus, fix this issue by explicitly adding
-fno-merge-all-constants to the kernel's Makefile to generically
disable this optimization, since potentially other places in the
kernel could subtly break as well.

Note, there is also a flag called -fmerge-constants (not supported
by clang), which is more conservative and only applies to strings
and it's enabled in gcc's -O/-O2/-O3/-Os optimization levels. In
gcc's code, the two flags -fmerge-{all-,}constants share the same
variable internally, so when disabling it via -fno-merge-all-constants,
then we really don't merge any const data (e.g. strings), and text
size increases with gcc (14,927,214 -> 14,942,646 for vmlinux.o).

  $ gcc -fverbose-asm -O2 foo.c -S -o foo.S
    -> foo.S lists -fmerge-constants under options enabled
  $ gcc -fverbose-asm -O2 -fno-merge-all-constants foo.c -S -o foo.S
    -> foo.S doesn't list -fmerge-constants under options enabled
  $ gcc -fverbose-asm -O2 -fno-merge-all-constants -fmerge-constants foo.c -S -o foo.S
    -> foo.S lists -fmerge-constants under options enabled

Thus, as a workaround we need to set both -fno-merge-all-constants
*and* -fmerge-constants in the Makefile in order for text size to
stay as is.

  [1] https://bugs.llvm.org/show_bug.cgi?id=18538

Reported-by: Prasad Sodagudi <psodagud@codeaurora.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Chenbo Feng <fengc@google.com>
Cc: Richard Smith <richard-llvm@metafoo.co.uk>
Cc: Chandler Carruth <chandlerc@gmail.com>
Cc: linux-kernel@vger.kernel.org
Tested-by: Prasad Sodagudi <psodagud@codeaurora.org>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 Makefile |    9 +++++++++
 1 file changed, 9 insertions(+)

--- a/Makefile
+++ b/Makefile
@@ -805,6 +805,15 @@ KBUILD_CFLAGS += $(call cc-disable-warni
 # disable invalid "can't wrap" optimizations for signed / pointers
 KBUILD_CFLAGS	+= $(call cc-option,-fno-strict-overflow)
 
+# clang sets -fmerge-all-constants by default as optimization, but this
+# is non-conforming behavior for C and in fact breaks the kernel, so we
+# need to disable it here generally.
+KBUILD_CFLAGS	+= $(call cc-option,-fno-merge-all-constants)
+
+# for gcc -fno-merge-all-constants disables everything, but it is fine
+# to have actual conforming behavior enabled.
+KBUILD_CFLAGS	+= $(call cc-option,-fmerge-constants)
+
 # Make sure -fstack-check isn't enabled (like gentoo apparently did)
 KBUILD_CFLAGS  += $(call cc-option,-fno-stack-check,)
 

  parent reply	other threads:[~2018-03-27 16:28 UTC|newest]

Thread overview: 107+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-03-27 16:26 [PATCH 4.14 000/101] 4.14.31-stable review Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 001/101] MIPS: ralink: Remove ralink_halt() Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 002/101] MIPS: ralink: Fix booting on MT7621 Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 003/101] MIPS: lantiq: Fix Danube USB clock Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 004/101] MIPS: lantiq: Enable AHB Bus for USB Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 005/101] MIPS: lantiq: ase: Enable MFD_SYSCON Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 006/101] iio: chemical: ccs811: Corrected firmware boot/application mode transition Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 007/101] iio: st_pressure: st_accel: pass correct platform data to init Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 008/101] iio: adc: meson-saradc: unlock on error in meson_sar_adc_lock() Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 009/101] ALSA: usb-audio: Fix parsing descriptor of UAC2 processing unit Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 010/101] ALSA: aloop: Sync stale timer before release Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 011/101] ALSA: aloop: Fix access to not-yet-ready substream via cable Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 012/101] ALSA: hda - Force polling mode on CFL for fixing codec communication Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 013/101] ALSA: hda/realtek - Fix speaker no sound after system resume Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 014/101] ALSA: hda/realtek - Fix Dell headset Mic cant record Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 015/101] ALSA: hda/realtek - Always immediately update mute LED with pin VREF Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 016/101] mmc: core: Fix tracepoint print of blk_addr and blksz Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 017/101] mmc: core: Disable HPI for certain Micron (Numonyx) eMMC cards Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 018/101] mmc: block: fix updating ext_csd caches on ioctl call Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 019/101] mmc: dw_mmc: Fix the DTO/CTO timeout overflow calculation for 32-bit systems Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 020/101] mmc: dw_mmc: exynos: fix the suspend/resume issue for exynos5433 Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 021/101] mmc: dw_mmc: fix falling from idmac to PIO mode when dw_mci_reset occurs Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 022/101] PCI: Add function 1 DMA alias quirk for Highpoint RocketRAID 644L Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 023/101] ahci: Add PCI-id for the Highpoint Rocketraid 644L card Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 024/101] lockdep: fix fs_reclaim warning Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 025/101] clk: bcm2835: Fix ana->maskX definitions Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 026/101] clk: bcm2835: Protect sections updating shared registers Greg Kroah-Hartman
2018-03-27 16:26 ` [PATCH 4.14 027/101] clk: sunxi-ng: a31: Fix CLK_OUT_* clock ops Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 028/101] RDMA/mlx5: Fix crash while accessing garbage pointer and freed memory Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 029/101] Drivers: hv: vmbus: Fix ring buffer signaling Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 030/101] pinctrl: samsung: Validate alias coming from DT Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 031/101] Bluetooth: btusb: Remove Yoga 920 from the btusb_needs_reset_resume_table Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 032/101] Bluetooth: btusb: Add Dell OptiPlex 3060 to btusb_needs_reset_resume_table Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 033/101] Bluetooth: btusb: Fix quirk for Atheros 1525/QCA6174 Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 034/101] libata: fix length validation of ATAPI-relayed SCSI commands Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 035/101] libata: remove WARN() for DMA or PIO command without data Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 036/101] libata: dont try to pass through NCQ commands to non-NCQ devices Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 037/101] libata: Apply NOLPM quirk to Crucial MX100 512GB SSDs Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 038/101] libata: disable LPM for Crucial BX100 SSD 500GB drive Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 039/101] libata: Enable queued TRIM for Samsung SSD 860 Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 040/101] libata: Apply NOLPM quirk to Crucial M500 480 and 960GB SSDs Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 041/101] libata: Make Crucial BX100 500GB LPM quirk apply to all firmware versions Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 042/101] libata: Modify quirks for MX100 to limit NCQ_TRIM quirk to MU01 version Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 043/101] cgroup: fix rule checking for threaded mode switching Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 044/101] nfsd: remove blocked locks on client teardown Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 045/101] hugetlbfs: check for pgoff value overflow Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 046/101] h8300: remove extraneous __BIG_ENDIAN definition Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 047/101] mm/vmalloc: add interfaces to free unmapped page table Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 048/101] x86/mm: implement free pmd/pte page interfaces Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 049/101] mm/khugepaged.c: convert VM_BUG_ON() to collapse fail Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 050/101] mm/thp: do not wait for lock_page() in deferred_split_scan() Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 051/101] mm/shmem: do not wait for lock_page() in shmem_unused_huge_shrink() Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 052/101] Revert "mm: page_alloc: skip over regions of invalid pfns where possible" Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 053/101] drm/vmwgfx: Fix black screen and device errors when running without fbdev Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 054/101] drm/vmwgfx: Fix a destoy-while-held mutex problem Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 055/101] drm/radeon: Dont turn off DP sink when disconnected Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 056/101] drm: Reject getfb for multi-plane framebuffers Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 057/101] drm: udl: Properly check framebuffer mmap offsets Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 058/101] mm/vmscan: wake up flushers for legacy cgroups too Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 059/101] acpi, numa: fix pxm to online numa node associations Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 060/101] ACPI / watchdog: Fix off-by-one error at resource assignment Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 061/101] libnvdimm, {btt, blk}: do integrity setup before add_disk() Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 062/101] brcmfmac: fix P2P_DEVICE ethernet address generation Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 063/101] rtlwifi: rtl8723be: Fix loss of signal Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 064/101] tracing: probeevent: Fix to support minus offset from symbol Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 065/101] mtdchar: fix usage of mtd_ooblayout_ecc() Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 066/101] mtd: nand: fsl_ifc: Fix nand waitfunc return value Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 067/101] mtd: nand: fsl_ifc: Fix eccstat array overflow for IFC ver >= 2.0.0 Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 068/101] mtd: nand: fsl_ifc: Read ECCSTAT0 and ECCSTAT1 registers for IFC 2.0 Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 069/101] staging: ncpfs: memory corruption in ncp_read_kernel() Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 070/101] can: peak/pcie_fd: fix echo_skb is occupied! bug Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 071/101] can: peak/pcie_fd: remove useless code when interface starts Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 072/101] can: ifi: Repair the error handling Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 073/101] can: ifi: Check core revision upon probe Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 074/101] can: cc770: Fix stalls on rt-linux, remove redundant IRQ ack Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 075/101] can: cc770: Fix queue stall & dropped RTR reply Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 076/101] can: cc770: Fix use after free in cc770_tx_interrupt() Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 077/101] tty: vt: fix up tabstops properly Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 078/101] x86/entry/64: Dont use IST entry for #BP stack Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 079/101] selftests/x86/ptrace_syscall: Fix for yet more glibc interference Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 080/101] x86/vsyscall/64: Use proper accessor to update P4D entry Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 081/101] x86/efi: Free efi_pgd with free_pages() Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 082/101] posix-timers: Protect posix clock array access against speculation Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 083/101] kvm/x86: fix icebp instruction handling Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 084/101] x86/build/64: Force the linker to use 2MB page size Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 085/101] x86/boot/64: Verify alignment of the LOAD segment Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 086/101] drm/syncobj: Stop reusing the same struct file for all syncobj -> fd Greg Kroah-Hartman
2018-03-27 16:27 ` [PATCH 4.14 087/101] perf/x86/intel/uncore: Fix Skylake UPI event format Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 088/101] perf stat: Fix CVS output format for non-supported counters Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 089/101] perf/core: Fix ctx_event_type in ctx_resched() Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 090/101] perf/x86/intel: Dont accidentally clear high bits in bdw_limit_period() Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 091/101] perf/x86/intel/uncore: Fix multi-domain PCI CHA enumeration bug on Skylake servers Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 092/101] iio: ABI: Fix name of timestamp sysfs file Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 093/101] iio: imu: st_lsm6dsx: fix endianness in st_lsm6dsx_read_oneshot() Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 094/101] staging: android: ion: Zero CMA allocated memory Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 095/101] staging: lustre: ptlrpc: kfree used instead of kvfree Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 096/101] usb: xhci: Disable slot even when virt-dev is null Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 097/101] usb: xhci: Fix potential memory leak in xhci_disable_slot() Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 098/101] x86/pkeys/selftests: Rename si_pkey to siginfo_pkey Greg Kroah-Hartman
2018-03-27 16:28 ` Greg Kroah-Hartman [this message]
2018-03-27 16:28 ` [PATCH 4.14 100/101] bpf: skip unnecessary capability check Greg Kroah-Hartman
2018-03-27 16:28 ` [PATCH 4.14 101/101] bpf, x64: increase number of passes Greg Kroah-Hartman
2018-03-27 22:56 ` [PATCH 4.14 000/101] 4.14.31-stable review Shuah Khan
2018-03-27 23:41 ` kernelci.org bot
2018-03-28 13:38 ` Guenter Roeck
2018-03-28 16:21   ` Greg Kroah-Hartman
2018-03-28 18:41 ` Dan Rue

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180327162756.067317846@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=ast@kernel.org \
    --cc=chandlerc@gmail.com \
    --cc=daniel@iogearbox.net \
    --cc=fengc@google.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=psodagud@codeaurora.org \
    --cc=richard-llvm@metafoo.co.uk \
    --cc=stable@vger.kernel.org \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox