xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
From: Xen.org security team <security@xen.org>
To: xen-announce@lists.xen.org, xen-devel@lists.xen.org,
	xen-users@lists.xen.org, oss-security@lists.openwall.com
Cc: "Xen.org security team" <security-team-members@xen.org>
Subject: Xen Security Advisory 226 (CVE-2017-12135) - multiple problems with transitive grants
Date: Tue, 15 Aug 2017 12:05:46 +0000	[thread overview]
Message-ID: <E1dhabS-0006cM-Ce@xenbits.xenproject.org> (raw)

[-- Attachment #1: Type: text/plain, Size: 3673 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

            Xen Security Advisory CVE-2017-12135 / XSA-226
                               version 5

               multiple problems with transitive grants

UPDATES IN VERSION 5
====================

Public release.

ISSUE DESCRIPTION
=================

1) Code to handle copy operations on transitive grants has built in
   retry logic, involving a function reinvoking itself with unchanged
   parameters.  Such use assumes that the compiler would also translate
   this to a so called "tail call" when generating machine code.
   Empirically, this is not commonly the case, allowing for
   theoretically unbounded nesting of such function calls.

2) The reference counting and locking discipline for transitive grants
   is broken.  Concurrent use of the transitive grant can leak
   references on the transitively-referenced grant.

IMPACT
======

A malicious or buggy guest may be able to crash Xen.  Privilege
escalation and information leaks cannot be ruled out.  A malicious or
buggy guest can leak references on grants it has been given, amounting
to a DoS against the grantee.

VULNERABLE SYSTEMS
==================

All versions of Xen are vulnerable.

MITIGATION
==========

There is no known mitigation.

CREDITS
=======

This issue was discovered by Jan Beulich of SUSE.

The security team would also like to thank Amazon for helping to identify that
the problems with transitive grants were deeper than originally believed.

RESOLUTION
==========

Applying the appropriate attached patch works around this issue by disabling
transitive grants by default.

xsa226.patch           xen-unstable, Xen 4.9.x, Xen 4.8.x
xsa226-4.7.patch       Xen 4.7.x
xsa226-4.6.patch       Xen 4.6.x
xsa226-4.5.patch       Xen 4.5.x

$ sha256sum xsa226*
b09e07aaf422ae04a4ece5e2c5b5e54036cfae5b5c632bfc6953a0cacd6f60ff  xsa226.patch
ca8b92b2ff58b87e8bec137a34784cbf11e2820659046df6e1d71e23bf7e7dee  xsa226-4.5.patch
28c7df7edabb91fb2f1fa3fc7d6906bfae75a6e701f1cd335baafaae3e087696  xsa226-4.6.patch
fffcc0a4428723e6aea391ff4f1d27326b5a3763d2308cbde64e6a786502c702  xsa226-4.7.patch
$

(The .meta file is a prototype machine-readable file for describing
which patches are to be applied how.)

DEPLOYMENT DURING EMBARGO
=========================

Deployment of the patches and/or mitigations described above (or
others which are substantially similar) is permitted during the
embargo, even on public-facing systems with untrusted guest users and
administrators.

But: Distribution of updated software is prohibited (except to other
members of the predisclosure list).

Predisclosure list members who wish to deploy significantly different
patches and/or mitigations, please contact the Xen Project Security
Team.


(Note: this during-embargo deployment notice is retained in
post-embargo publicly released Xen Project advisories, even though it
is then no longer applicable.  This is to enable the community to have
oversight of the Xen Project Security Team's decisionmaking.)

For more information about permissible uses of embargoed information,
consult the Xen Project community's agreed Security Policy:
  http://www.xenproject.org/security-policy.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJZkuNKAAoJEIP+FMlX6CvZUHMIALQcTfo00unwBX9RO7lBy4na
LSkFE9yaPtA/pg5RRGo7Nrwl2nIDRc6Xc0ZkhNm0rfi1gnR0htP3jyJXxkXv1sah
jkBP0bZYfWDHRxSdVBbNNn8q0mhuanycFhVuEiu+vmTPKRUTyODkAdAoi/TkY9Iq
XD24clIrjY2xIDO3pKbDTJUZ86rHD0nepHdnnvN2rywyBd2VkJfJWGavqHgs61XX
j9jX0nI4Wcm4nQKx37MBUwwN3oYeEKrzYQY3+AGVKQEWuULP4sWRKhxZaqclCbfd
Cx/9gACwPEORU6bRXE/vzlxn7Ks6yf2tqgNAGCTrZgwW8q3SFNASHzaAM3EXz3w=
=VNkV
-----END PGP SIGNATURE-----

[-- Attachment #2: xsa226.patch --]
[-- Type: application/octet-stream, Size: 4517 bytes --]

From: Andrew Cooper <andrew.cooper3@citrix.com>
Subject: grant_table: Default to v1, and disallow transitive grants

The reference counting and locking discipline for transitive grants is broken.
Their use is therefore declared out of security support.

This is XSA-226.

Transitive grants are expected to be unconditionally available with grant
table v2.  Hiding transitive grants alone is an ABI breakage for the guest.
Modern versions of Linux and the Windows PV drivers use grant table v1, but
older versions did use v2.

In principle, disabling gnttab v2 entirely is the safer way to cause guests to
avoid using transitive grants. However, some older guests which defaulted to
using gnttab v2 don't tolerate falling back from v2 to v1 over migrate.

This patch introduces a new command line option to control grant table
behaviour.  One suboption allows a choice of the maximum grant table version
Xen will allow the guest to use, and defaults to v2.  A different suboption
independently controls whether transitive grants can be used.

The default case is:

    gnttab=max_ver:2

To disable gnttab v2 entirely, use:

    gnttab=max_ver:1

To allow gnttab v2 and transitive grants, use:

    gnttab=max_ver:2,transitive

Reported-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index 4002eab..af079b4 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -868,6 +868,22 @@ Controls EPT related features.
 
 Specify which console gdbstub should use. See **console**.
 
+### gnttab
+> `= List of [ max_ver:<integer>, transitive ]`
+
+> Default: `gnttab=max_ver:2,no-transitive`
+
+Control various aspects of the grant table behaviour available to guests.
+
+* `max_ver` Select the maximum grant table version to offer to guests.  Valid
+version are 1 and 2.
+* `transitive` Permit or disallow the use of transitive grants.  Note that the
+use of grant table v2 without transitive grants is an ABI breakage from the
+guests point of view.
+
+*Warning:*
+Due to XSA-226, the use of transitive grants is outside of security support.
+
 ### gnttab\_max\_frames
 > `= <integer>`
 
diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index ae34547..87131f8 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -50,6 +50,42 @@ integer_param("gnttab_max_nr_frames", max_nr_grant_frames);
 unsigned int __read_mostly max_grant_frames;
 integer_param("gnttab_max_frames", max_grant_frames);
 
+static unsigned int __read_mostly opt_gnttab_max_version = 2;
+static bool __read_mostly opt_transitive_grants;
+
+static void __init parse_gnttab(char *s)
+{
+    char *ss;
+
+    do {
+        ss = strchr(s, ',');
+        if ( ss )
+            *ss = '\0';
+
+        if ( !strncmp(s, "max_ver:", 8) )
+        {
+            long ver = simple_strtol(s + 8, NULL, 10);
+
+            if ( ver >= 1 && ver <= 2 )
+                opt_gnttab_max_version = ver;
+        }
+        else
+        {
+            bool val = !!strncmp(s, "no-", 3);
+
+            if ( !val )
+                s += 3;
+
+            if ( !strcmp(s, "transitive") )
+                opt_transitive_grants = val;
+        }
+
+        s = ss + 1;
+    } while ( ss );
+}
+
+custom_param("gnttab", parse_gnttab);
+
 /* The maximum number of grant mappings is defined as a multiplier of the
  * maximum number of grant table entries. This defines the multiplier used.
  * Pretty arbitrary. [POLICY]
@@ -2191,6 +2227,10 @@ __acquire_grant_for_copy(
         }
         else if ( (shah->flags & GTF_type_mask) == GTF_transitive )
         {
+            if ( !opt_transitive_grants )
+                PIN_FAIL(unlock_out_clear, GNTST_general_error,
+                         "transitive grant disallowed by policy\n");
+
             if ( !allow_transitive )
                 PIN_FAIL(unlock_out_clear, GNTST_general_error,
                          "transitive grant when transitivity not allowed\n");
@@ -3159,7 +3199,10 @@ do_grant_table_op(
     }
     case GNTTABOP_set_version:
     {
-        rc = gnttab_set_version(guest_handle_cast(uop, gnttab_set_version_t));
+        if ( opt_gnttab_max_version == 1 )
+            rc = -ENOSYS; /* Behave as before set_version was introduced. */
+        else
+            rc = gnttab_set_version(guest_handle_cast(uop, gnttab_set_version_t));
         break;
     }
     case GNTTABOP_get_status_frames:

[-- Attachment #3: xsa226-4.5.patch --]
[-- Type: application/octet-stream, Size: 4545 bytes --]

From: Andrew Cooper <andrew.cooper3@citrix.com>
Subject: grant_table: Default to v1, and disallow transitive grants

The reference counting and locking discipline for transitive grants is broken.
Their use is therefore declared out of security support.

This is XSA-226.

Transitive grants are expected to be unconditionally available with grant
table v2.  Hiding transitive grants alone is an ABI breakage for the guest.
Modern versions of Linux and the Windows PV drivers use grant table v1, but
older versions did use v2.

In principle, disabling gnttab v2 entirely is the safer way to cause guests to
avoid using transitive grants. However, some older guests which defaulted to
using gnttab v2 don't tolerate falling back from v2 to v1 over migrate.

This patch introduces a new command line option to control grant table
behaviour.  One suboption allows a choice of the maximum grant table version
Xen will allow the guest to use, and defaults to v2.  A different suboption
independently controls whether transitive grants can be used.

The default case is:

    gnttab=max_ver:2

To disable gnttab v2 entirely, use:

    gnttab=max_ver:1

To allow gnttab v2 and transitive grants, use:

    gnttab=max_ver:2,transitive

Reported-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index 16bfb39..3936316 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -662,6 +662,22 @@ does not provide VM\_ENTRY\_LOAD\_GUEST\_PAT.
 
 Specify the serial parameters for the GDB stub.
 
+### gnttab
+> `= List of [ max_ver:<integer>, transitive ]`
+
+> Default: `gnttab=max_ver:2,no-transitive`
+
+Control various aspects of the grant table behaviour available to guests.
+
+* `max_ver` Select the maximum grant table version to offer to guests.  Valid
+version are 1 and 2.
+* `transitive` Permit or disallow the use of transitive grants.  Note that the
+use of grant table v2 without transitive grants is an ABI breakage from the
+guests point of view.
+
+*Warning:*
+Due to XSA-226, the use of transitive grants is outside of security support.
+
 ### gnttab\_max\_frames
 > `= <integer>`
 
diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index 83a4b9e..c9a6cd9 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -50,6 +50,42 @@ integer_param("gnttab_max_nr_frames", max_nr_grant_frames);
 unsigned int __read_mostly max_grant_frames;
 integer_param("gnttab_max_frames", max_grant_frames);
 
+static unsigned int __read_mostly opt_gnttab_max_version = 2;
+static bool_t __read_mostly opt_transitive_grants;
+
+static void __init parse_gnttab(char *s)
+{
+    char *ss;
+
+    do {
+        ss = strchr(s, ',');
+        if ( ss )
+            *ss = '\0';
+
+        if ( !strncmp(s, "max_ver:", 8) )
+        {
+            long ver = simple_strtol(s + 8, NULL, 10);
+
+            if ( ver >= 1 && ver <= 2 )
+                opt_gnttab_max_version = ver;
+        }
+        else
+        {
+            bool_t val = !!strncmp(s, "no-", 3);
+
+            if ( !val )
+                s += 3;
+
+            if ( !strcmp(s, "transitive") )
+                opt_transitive_grants = val;
+        }
+
+        s = ss + 1;
+    } while ( ss );
+}
+
+custom_param("gnttab", parse_gnttab);
+
 /* The maximum number of grant mappings is defined as a multiplier of the
  * maximum number of grant table entries. This defines the multiplier used.
  * Pretty arbitrary. [POLICY]
@@ -1958,6 +1994,10 @@ __acquire_grant_for_copy(
         trans_gref = gref;
         if ( sha2 && (shah->flags & GTF_type_mask) == GTF_transitive )
         {
+            if ( !opt_transitive_grants )
+                PIN_FAIL(unlock_out_clear, GNTST_general_error,
+                         "transitive grant disallowed by policy\n");
+
             if ( !allow_transitive )
                 PIN_FAIL(unlock_out_clear, GNTST_general_error,
                          "transitive grant when transitivity not allowed\n");
@@ -2741,7 +2781,10 @@ do_grant_table_op(
     }
     case GNTTABOP_set_version:
     {
-        rc = gnttab_set_version(guest_handle_cast(uop, gnttab_set_version_t));
+        if ( opt_gnttab_max_version == 1 )
+            rc = -ENOSYS; /* Behave as before set_version was introduced. */
+        else
+            rc = gnttab_set_version(guest_handle_cast(uop, gnttab_set_version_t));
         break;
     }
     case GNTTABOP_get_status_frames:

[-- Attachment #4: xsa226-4.6.patch --]
[-- Type: application/octet-stream, Size: 4510 bytes --]

From: Andrew Cooper <andrew.cooper3@citrix.com>
Subject: grant_table: Default to v1, and disallow transitive grants

The reference counting and locking discipline for transitive grants is broken.
Their use is therefore declared out of security support.

This is XSA-226.

Transitive grants are expected to be unconditionally available with grant
table v2.  Hiding transitive grants alone is an ABI breakage for the guest.
Modern versions of Linux and the Windows PV drivers use grant table v1, but
older versions did use v2.

In principle, disabling gnttab v2 entirely is the safer way to cause guests to
avoid using transitive grants. However, some older guests which defaulted to
using gnttab v2 don't tolerate falling back from v2 to v1 over migrate.

This patch introduces a new command line option to control grant table
behaviour.  One suboption allows a choice of the maximum grant table version
Xen will allow the guest to use, and defaults to v2.  A different suboption
independently controls whether transitive grants can be used.

The default case is:

    gnttab=max_ver:2

To disable gnttab v2 entirely, use:

    gnttab=max_ver:1

To allow gnttab v2 and transitive grants, use:

    gnttab=max_ver:2,transitive

Reported-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index d99a20a..113bb29 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -733,6 +733,22 @@ Controls EPT related features.
 
 Specify the serial parameters for the GDB stub.
 
+### gnttab
+> `= List of [ max_ver:<integer>, transitive ]`
+
+> Default: `gnttab=max_ver:2,no-transitive`
+
+Control various aspects of the grant table behaviour available to guests.
+
+* `max_ver` Select the maximum grant table version to offer to guests.  Valid
+version are 1 and 2.
+* `transitive` Permit or disallow the use of transitive grants.  Note that the
+use of grant table v2 without transitive grants is an ABI breakage from the
+guests point of view.
+
+*Warning:*
+Due to XSA-226, the use of transitive grants is outside of security support.
+
 ### gnttab\_max\_frames
 > `= <integer>`
 
diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index 20230fb..98845c4 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -50,6 +50,42 @@ integer_param("gnttab_max_nr_frames", max_nr_grant_frames);
 unsigned int __read_mostly max_grant_frames;
 integer_param("gnttab_max_frames", max_grant_frames);
 
+static unsigned int __read_mostly opt_gnttab_max_version = 2;
+static bool_t __read_mostly opt_transitive_grants;
+
+static void __init parse_gnttab(char *s)
+{
+    char *ss;
+
+    do {
+        ss = strchr(s, ',');
+        if ( ss )
+            *ss = '\0';
+
+        if ( !strncmp(s, "max_ver:", 8) )
+        {
+            long ver = simple_strtol(s + 8, NULL, 10);
+
+            if ( ver >= 1 && ver <= 2 )
+                opt_gnttab_max_version = ver;
+        }
+        else
+        {
+            bool_t val = !!strncmp(s, "no-", 3);
+
+            if ( !val )
+                s += 3;
+
+            if ( !strcmp(s, "transitive") )
+                opt_transitive_grants = val;
+        }
+
+        s = ss + 1;
+    } while ( ss );
+}
+
+custom_param("gnttab", parse_gnttab);
+
 /* The maximum number of grant mappings is defined as a multiplier of the
  * maximum number of grant table entries. This defines the multiplier used.
  * Pretty arbitrary. [POLICY]
@@ -2175,6 +2211,10 @@ __acquire_grant_for_copy(
         }
         else if ( (shah->flags & GTF_type_mask) == GTF_transitive )
         {
+            if ( !opt_transitive_grants )
+                PIN_FAIL(unlock_out_clear, GNTST_general_error,
+                         "transitive grant disallowed by policy\n");
+
             if ( !allow_transitive )
                 PIN_FAIL(unlock_out_clear, GNTST_general_error,
                          "transitive grant when transitivity not allowed\n");
@@ -3143,7 +3183,10 @@ do_grant_table_op(
     }
     case GNTTABOP_set_version:
     {
-        rc = gnttab_set_version(guest_handle_cast(uop, gnttab_set_version_t));
+        if ( opt_gnttab_max_version == 1 )
+            rc = -ENOSYS; /* Behave as before set_version was introduced. */
+        else
+            rc = gnttab_set_version(guest_handle_cast(uop, gnttab_set_version_t));
         break;
     }
     case GNTTABOP_get_status_frames:

[-- Attachment #5: xsa226-4.7.patch --]
[-- Type: application/octet-stream, Size: 4521 bytes --]

From: Andrew Cooper <andrew.cooper3@citrix.com>
Subject: grant_table: Default to v1, and disallow transitive grants

The reference counting and locking discipline for transitive grants is broken.
Their use is therefore declared out of security support.

This is XSA-226.

Transitive grants are expected to be unconditionally available with grant
table v2.  Hiding transitive grants alone is an ABI breakage for the guest.
Modern versions of Linux and the Windows PV drivers use grant table v1, but
older versions did use v2.

In principle, disabling gnttab v2 entirely is the safer way to cause guests to
avoid using transitive grants. However, some older guests which defaulted to
using gnttab v2 don't tolerate falling back from v2 to v1 over migrate.

This patch introduces a new command line option to control grant table
behaviour.  One suboption allows a choice of the maximum grant table version
Xen will allow the guest to use, and defaults to v2.  A different suboption
independently controls whether transitive grants can be used.

The default case is:

    gnttab=max_ver:2

To disable gnttab v2 entirely, use:

    gnttab=max_ver:1

To allow gnttab v2 and transitive grants, use:

    gnttab=max_ver:2,transitive

Reported-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index 73f5265..b792abf 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -758,6 +758,22 @@ Controls EPT related features.
 
 Specify which console gdbstub should use. See **console**.
 
+### gnttab
+> `= List of [ max_ver:<integer>, transitive ]`
+
+> Default: `gnttab=max_ver:2,no-transitive`
+
+Control various aspects of the grant table behaviour available to guests.
+
+* `max_ver` Select the maximum grant table version to offer to guests.  Valid
+version are 1 and 2.
+* `transitive` Permit or disallow the use of transitive grants.  Note that the
+use of grant table v2 without transitive grants is an ABI breakage from the
+guests point of view.
+
+*Warning:*
+Due to XSA-226, the use of transitive grants is outside of security support.
+
 ### gnttab\_max\_frames
 > `= <integer>`
 
diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index f06b664..109c552 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -50,6 +50,42 @@ integer_param("gnttab_max_nr_frames", max_nr_grant_frames);
 unsigned int __read_mostly max_grant_frames;
 integer_param("gnttab_max_frames", max_grant_frames);
 
+static unsigned int __read_mostly opt_gnttab_max_version = 2;
+static bool_t __read_mostly opt_transitive_grants;
+
+static void __init parse_gnttab(char *s)
+{
+    char *ss;
+
+    do {
+        ss = strchr(s, ',');
+        if ( ss )
+            *ss = '\0';
+
+        if ( !strncmp(s, "max_ver:", 8) )
+        {
+            long ver = simple_strtol(s + 8, NULL, 10);
+
+            if ( ver >= 1 && ver <= 2 )
+                opt_gnttab_max_version = ver;
+        }
+        else
+        {
+            bool_t val = !!strncmp(s, "no-", 3);
+
+            if ( !val )
+                s += 3;
+
+            if ( !strcmp(s, "transitive") )
+                opt_transitive_grants = val;
+        }
+
+        s = ss + 1;
+    } while ( ss );
+}
+
+custom_param("gnttab", parse_gnttab);
+
 /* The maximum number of grant mappings is defined as a multiplier of the
  * maximum number of grant table entries. This defines the multiplier used.
  * Pretty arbitrary. [POLICY]
@@ -2188,6 +2224,10 @@ __acquire_grant_for_copy(
         }
         else if ( (shah->flags & GTF_type_mask) == GTF_transitive )
         {
+            if ( !opt_transitive_grants )
+                PIN_FAIL(unlock_out_clear, GNTST_general_error,
+                         "transitive grant disallowed by policy\n");
+
             if ( !allow_transitive )
                 PIN_FAIL(unlock_out_clear, GNTST_general_error,
                          "transitive grant when transitivity not allowed\n");
@@ -3156,7 +3196,10 @@ do_grant_table_op(
     }
     case GNTTABOP_set_version:
     {
-        rc = gnttab_set_version(guest_handle_cast(uop, gnttab_set_version_t));
+        if ( opt_gnttab_max_version == 1 )
+            rc = -ENOSYS; /* Behave as before set_version was introduced. */
+        else
+            rc = gnttab_set_version(guest_handle_cast(uop, gnttab_set_version_t));
         break;
     }
     case GNTTABOP_get_status_frames:

[-- Attachment #6: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

             reply	other threads:[~2017-08-15 12:05 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-08-15 12:05 Xen.org security team [this message]
  -- strict thread matches above, loose matches on Subject: below --
2017-08-17 14:34 Xen Security Advisory 226 (CVE-2017-12135) - multiple problems with transitive grants Xen.org security team
2017-08-29 12:04 Xen.org security team

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=E1dhabS-0006cM-Ce@xenbits.xenproject.org \
    --to=security@xen.org \
    --cc=oss-security@lists.openwall.com \
    --cc=security-team-members@xen.org \
    --cc=xen-announce@lists.xen.org \
    --cc=xen-devel@lists.xen.org \
    --cc=xen-users@lists.xen.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).