public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Elladan <elladan@eskimo.com>
To: James Bruce <bruce@andrew.cmu.edu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>,
	"Bill Huey (hui)" <bhuey@lnxw.com>, Andi Kleen <ak@muc.de>,
	Sven-Thorsten Dietrich <sdietrich@mvista.com>,
	Ingo Molnar <mingo@elte.hu>,
	dwalker@mvista.com, hch@infradead.org, akpm@osdl.org,
	linux-kernel@vger.kernel.org
Subject: Re: RT patch acceptance
Date: Tue, 31 May 2005 12:10:39 -0700	[thread overview]
Message-ID: <20050531191039.GV1680@eskimo.com> (raw)
In-Reply-To: <429BA27A.5010406@andrew.cmu.edu>

On Mon, May 30, 2005 at 07:32:10PM -0400, James Bruce wrote:
> Nick Piggin wrote:
> >Sorry James, we were talking about hard realtime. Read the thread.
> 
> hard realtime = mathematically provable maximum latency
> 
> Yes, you'll want a nanokernel for that, you're right.  That's because 
> one has to analyze every line of code, and protect against introduced 
> regressions, which is almost impossible given the pace that Linux-proper 
> is developed.  Then there's the other 95% of applications, for which a 
> "statistical RT" approach such as used in the RT patch suffice.  So 
> arguing for a nanokernel for (provable) hard realtime is orthogonal to 
> the discussion of this patch, and we apparently don't actually disagree.

In the real world, this isn't really possible.  Ideally, you'd like to be able
to offer some proof of correctness for the software, but this isn't actually
going to get you provable maximum latency, because you can't prove the
hardware.  

Even with perfect software, the hardware is subject to cosmic rays, bad design,
etc.  Even if you strongly control the hardware for latency, eg.  turn off
cache and try to make sure everything is measurable, in the end the real proof
that your device does what it says it does is measurement.  If the RTOS
guarantees aren't violated during testing, or at best, in a time period
comparable to the failure rate of the hardware, that's "good enough."

Given that hardware is always subject to failure or flakiness, the more
practical distinction between "hard" and "soft" realtime is whether the failure
rate is measurable or is lost in the noise of other failure modes such as
hardware.  "Soft" RT typically means that the failure rate is measurable but
may be sufficient for particular tasks, and in comparison "hard" means the
software is thought to be correct within your ability to measure.

Certainly there's a lot of value for some applications in trying to control the
software well enough that all the latencies can be understood and characterized
by inspection, but on any sort of consumer commodity hardware system this is
really not going to buy you much.  There are so many potential latencies just
due to wacky hardware that even a "perfect" RTOS is going to be subject to all
sorts of weird latencies and bizarre issues eg. with interrupt routing and CPU
thermal control and the like.  

Showing that the application works as intended is really just going to be a
matter of showing that on a particular system, the latency requirements are met
under load.  Which is exactly a sort of statistical approach.  For almost all
"PC" applications that need realtime, this is exactly what's desired.

And clearly, the ultimate test of any RT system is exactly a "statistical" test
- can it be measured to fail, and if so, why and how often?

For limited embedded applications, a "hard" nanokernel approach can certainly
lead to higher confidence that the device works as intended, but for anything
outside of embedded products it's really not very practical.  Nobody's going to
run their desktop OS under a nanokernel just to make their DVD software work
right.

-J

  parent reply	other threads:[~2005-05-31 19:13 UTC|newest]

Thread overview: 345+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2005-05-23 23:14 RT patch acceptance Daniel Walker
2005-05-24  2:01 ` john cooper
2005-05-24  5:47 ` Christoph Hellwig
2005-05-24  6:45   ` Ingo Molnar
2005-05-24  8:03     ` Nick Piggin
2005-05-24  8:15       ` Ingo Molnar
2005-05-24  8:27         ` Nick Piggin
2005-05-24  9:02           ` Ingo Molnar
2005-05-24  9:14             ` Nick Piggin
2005-05-24 10:13               ` Ingo Molnar
2005-05-24 18:05               ` Daniel Walker
2005-05-24 22:41                 ` Bill Huey
2005-05-24 23:44                   ` Daniel Walker
2005-05-25  0:10                     ` Bill Huey
2005-05-25  0:32                       ` K.R. Foley
2005-05-25  0:45                         ` Lee Revell
2005-05-25  1:32                           ` K.R. Foley
2005-05-25  0:45                       ` Daniel Walker
2005-05-25  0:59                         ` Bill Huey
2005-05-25  1:02                           ` Daniel Walker
2005-05-25  1:43                             ` Andrew Morton
2005-05-25  2:02                               ` Sven Dietrich
2005-05-25  2:20                                 ` Andrew Morton
2005-05-25  2:26                                   ` Lee Revell
2005-05-25  3:24                                     ` Nick Piggin
2005-05-25  3:36                                       ` Sven Dietrich
2005-05-25  5:05                                         ` Nick Piggin
2005-05-25  6:09                                           ` RT patch acceptance (scheduler) Bill Huey
2005-05-25  7:00                                             ` Nick Piggin
2005-05-25  9:27                                               ` Bill Huey
2005-05-25 14:50                                                 ` Nick Piggin
2005-05-25 15:12                                                   ` RT patch acceptance (scheduler) --- QUESTION Brian O'Mahoney
2005-05-25 15:36                                                     ` Steven Rostedt
2005-05-25  6:05                                       ` RT patch acceptance Ingo Molnar
2005-05-25 18:50                                         ` Lee Revell
2005-05-25  3:26                                   ` Gene Heskett
2005-05-25  5:17                                     ` Nick Piggin
2005-05-25 13:20                                       ` Gene Heskett
2005-05-25  6:33                                   ` Ingo Molnar
2005-05-25  7:18                                     ` Nick Piggin
2005-05-25  7:46                                       ` Ingo Molnar
2005-05-25 10:51                                         ` Con Kolivas
2005-05-25  8:29                                     ` Giuseppe Bilotta
2005-05-25 13:23                                     ` Gene Heskett
2005-05-25 17:17                                     ` Andi Kleen
2005-05-25 17:52                                       ` Steven Rostedt
2005-05-25 18:00                                       ` Sven-Thorsten Dietrich
2005-05-26 19:32                                         ` Andi Kleen
2005-05-26 20:11                                           ` Sven-Thorsten Dietrich
2005-05-26 20:27                                             ` Andi Kleen
2005-05-26 20:46                                               ` Sven-Thorsten Dietrich
2005-05-26 23:57                                                 ` john cooper
2005-05-26 21:28                                               ` Bill Huey
2005-05-26 23:44                                               ` john cooper
2005-05-27  5:19                                               ` Nick Piggin
2005-05-27  5:24                                                 ` Nick Piggin
2005-05-27  7:18                                                 ` Ingo Molnar
2005-05-27  8:07                                                   ` Nick Piggin
2005-05-27 15:56                                                     ` K.R. Foley
2005-05-28  3:59                                                   ` Lee Revell
2005-05-27  7:20                                                 ` Sven-Thorsten Dietrich
2005-05-27  8:17                                                   ` Nick Piggin
2005-05-27 10:17                                                     ` Thomas Gleixner
2005-05-27 17:46                                                   ` Karim Yaghmour
2005-05-27 12:08                                                 ` Bill Huey
2005-05-27 12:10                                                   ` Ingo Molnar
2005-05-27 20:36                                                     ` Bill Huey
2005-05-27 12:43                                                   ` Nick Piggin
2005-05-27 23:36                                                     ` Bill Huey
2005-05-28  3:53                                                       ` Nick Piggin
2005-05-28  4:27                                                         ` Lee Revell
2005-05-28  4:43                                                           ` Nick Piggin
2005-05-28  5:53                                                             ` Bill Huey
2005-05-28  5:45                                                         ` Bill Huey
2005-05-28  6:49                                                           ` Nick Piggin
2005-05-29 11:37                                                             ` James Bruce
2005-05-29 12:23                                                               ` Brian O'Mahoney
2005-05-30  9:37                                                               ` Nick Piggin
2005-05-30 13:43                                                                 ` James Bruce
2005-05-30 14:21                                                                   ` Nick Piggin
2005-05-30 22:27                                                                     ` Bill Huey
2005-05-30 22:54                                                                       ` Karim Yaghmour
2005-05-30 23:05                                                                         ` Bill Huey
2005-05-30 23:29                                                                           ` Karim Yaghmour
2005-05-31  1:21                                                                       ` Nick Piggin
2005-05-31  2:09                                                                         ` Bill Huey
2005-05-31  9:12                                                                           ` James Bruce
2005-05-31  9:33                                                                             ` Nick Piggin
2005-05-31 10:23                                                                               ` Bill Huey
2005-05-31 10:48                                                                               ` James Bruce
2005-05-31 11:06                                                                                 ` Nick Piggin
2005-05-31 11:14                                                                                 ` Andi Kleen
2005-05-31 11:31                                                                                   ` Hari N
2005-05-31 16:59                                                                                   ` James Bruce
2005-05-31 12:09                                                                                 ` Esben Nielsen
2005-05-31 16:18                                                                                   ` Steven Rostedt
2005-05-31 16:42                                                                                     ` Esben Nielsen
2005-05-31 17:11                                                                                     ` Andrea Arcangeli
2005-05-31 17:42                                                                                       ` Steven Rostedt
2005-05-31 17:51                                                                                         ` Andrea Arcangeli
2005-05-31 18:29                                                                                           ` Steven Rostedt
2005-05-31 20:54                                                                                             ` Andrea Arcangeli
2005-05-31 21:22                                                                                               ` Steven Rostedt
2005-05-31 21:47                                                                                                 ` Lee Revell
2005-05-31 22:24                                                                                                   ` Andrea Arcangeli
2005-05-31 23:41                                                                                                   ` Steven Rostedt
2005-06-01  0:50                                                                                                     ` Zan Lynx
2005-06-01  2:38                                                                                                     ` Zwane Mwaikambo
2005-05-31 22:15                                                                                                 ` Andrea Arcangeli
2005-05-31 22:33                                                                                                   ` NZG
2005-05-31 23:14                                                                                                     ` Bill Huey
2005-05-31 21:33                                                                                               ` Bill Huey
2005-05-31 23:59                                                                                                 ` Steven Rostedt
2005-06-01  0:57                                                                                             ` Brian O'Mahoney
2005-06-01  6:23                                                                                             ` John Alvord
2005-06-01 11:41                                                                                               ` Steven Rostedt
2005-06-01  0:26                                                                                   ` Peter Chubb
2005-06-01  8:17                                                                                     ` Esben Nielsen
2005-05-31 14:30                                                                                 ` Andrea Arcangeli
2005-05-31 15:07                                                                                   ` Esben Nielsen
2005-05-31 16:11                                                                                     ` Andrea Arcangeli
2005-05-31 18:36                                                                                       ` Paul E. McKenney
2005-05-31 20:45                                                                                         ` Andrea Arcangeli
2005-06-01  1:14                                                                                           ` Karim Yaghmour
2005-06-01  1:30                                                                                             ` Steven Rostedt
2005-06-01  1:50                                                                                               ` Karim Yaghmour
2005-06-01  2:42                                                                                             ` Paul E. McKenney
2005-06-01 12:18                                                                                           ` Paulo Marques
2005-06-01 13:51                                                                                             ` Andrea Arcangeli
2005-06-01 14:19                                                                                               ` Ingo Molnar
2005-06-01 14:32                                                                                                 ` Andrea Arcangeli
2005-06-01 14:39                                                                                                   ` Daniel Walker
2005-06-01 14:45                                                                                                   ` Ingo Molnar
2005-06-01 14:57                                                                                                     ` Andrea Arcangeli
2005-06-01 14:45                                                                                                   ` Esben Nielsen
2005-06-01 15:05                                                                                                     ` Andrea Arcangeli
2005-06-01 15:15                                                                                                       ` Esben Nielsen
2005-06-01 15:32                                                                                                         ` Andrea Arcangeli
2005-06-01 17:30                                                                                                         ` pi_test predicted results Daniel Walker
2005-06-01 17:58                                                                                                           ` Esben Nielsen
2005-06-01 15:33                                                                                                       ` RT patch acceptance Karim Yaghmour
2005-06-01 15:38                                                                                                         ` Andrea Arcangeli
2005-06-01 17:53                                                                                                           ` Thomas Gleixner
2005-06-01 19:22                                                                                                             ` Andrea Arcangeli
2005-06-01 19:34                                                                                                               ` Esben Nielsen
2005-06-01 19:59                                                                                                                 ` Andrea Arcangeli
2005-06-01 20:17                                                                                                                   ` Bill Huey
2005-06-01 20:32                                                                                                                     ` Andrea Arcangeli
2005-06-01 20:46                                                                                                                       ` Bill Huey
2005-06-01 21:07                                                                                                                         ` Andrea Arcangeli
2005-06-01 21:29                                                                                                                           ` Nicolas Pitre
2005-06-01 21:39                                                                                                                             ` Andrea Arcangeli
2005-06-01 22:29                                                                                                                               ` Nicolas Pitre
2005-06-01 23:44                                                                                                                                 ` Zwane Mwaikambo
2005-06-01 21:42                                                                                                                           ` Bill Huey
2005-06-01 21:59                                                                                                                             ` Bill Huey
2005-06-01 22:32                                                                                                                               ` Andrea Arcangeli
2005-06-01 23:02                                                                                                                                 ` Bill Huey
2005-06-01 23:19                                                                                                                                   ` Bill Huey
2005-06-02  8:50                                                                                                                                     ` Esben Nielsen
2005-06-01 23:59                                                                                                                                   ` Bill Huey
2005-06-02  0:03                                                                                                                                     ` Bill Huey
2005-06-01 19:39                                                                                                               ` Ingo Molnar
2005-06-01 20:44                                                                                                                 ` Andrea Arcangeli
2005-06-01 20:56                                                                                                               ` Thomas Gleixner
2005-06-01 21:09                                                                                                                 ` Andrea Arcangeli
2005-06-01 14:46                                                                                                   ` Andrea Arcangeli
2005-06-01 14:56                                                                                                     ` Chris Friesen
2005-06-01 14:58                                                                                                     ` Paulo Marques
2005-06-01 15:17                                                                                                       ` Andrea Arcangeli
2005-06-01 15:33                                                                                                         ` john cooper
2005-06-01 15:43                                                                                                           ` Andrea Arcangeli
2005-06-01 15:19                                                                                                       ` Karim Yaghmour
2005-06-01 14:59                                                                                                     ` Esben Nielsen
2005-06-01 15:47                                                                                                       ` NZG
2005-06-01 15:58                                                                                                         ` Andrea Arcangeli
2005-06-01 17:20                                                                                                           ` Bill Davidsen
2005-06-01 18:19                                                                                                         ` Esben Nielsen
2005-06-01 15:09                                                                                                   ` Karim Yaghmour
2005-05-31  9:14                                                                         ` Esben Nielsen
2005-05-30 23:32                                                                     ` James Bruce
2005-05-31  1:22                                                                       ` Nick Piggin
2005-05-31  2:06                                                                       ` Lee Revell
2005-05-31  2:26                                                                         ` Lee Revell
2005-05-31  7:34                                                                         ` James Bruce
2005-05-31 19:10                                                                       ` Elladan [this message]
2005-05-30 22:10                                                                 ` Bill Huey
2005-05-30 17:53                                                               ` Karim Yaghmour
2005-05-30 19:24                                                                 ` Esben Nielsen
2005-05-30 19:44                                                                   ` Karim Yaghmour
2005-05-30 21:44                                                                     ` Rich Walker
2005-05-30 22:45                                                                     ` Bill Huey
2005-05-30 23:03                                                                       ` Karim Yaghmour
2005-05-30 23:12                                                                         ` Bill Huey
2005-05-30 23:31                                                                           ` Karim Yaghmour
2005-05-31  0:00                                                                           ` Karim Yaghmour
2005-05-30 11:25                                                             ` Nick Piggin
2005-05-29  1:55                                                           ` Zwane Mwaikambo
2005-05-29  2:48                                                             ` Lee Revell
2005-05-29  2:58                                                               ` Zwane Mwaikambo
2005-05-29  4:08                                                                 ` Valdis.Kletnieks
2005-05-29 15:00                                                                   ` Zwane Mwaikambo
2005-05-29 17:50                                                                     ` Valdis.Kletnieks
2005-05-29 19:52                                                                       ` Zwane Mwaikambo
2005-05-29 21:16                                                                         ` Valdis.Kletnieks
2005-05-30 13:01                                                                         ` Bill Huey
2005-05-31 14:30                                                             ` Zwane Mwaikambo
2005-05-28  6:55                                                       ` Christoph Hellwig
2005-05-28 10:22                                                         ` Bill Huey
2005-05-28 10:24                                                           ` Christoph Hellwig
2005-05-28 10:36                                                             ` Bill Huey
2005-05-28 10:34                                                           ` Bill Huey
2005-05-28 10:50                                                             ` Bill Huey
2005-05-28 10:48                                                               ` Christoph Hellwig
2005-05-28 11:01                                                                 ` Bill Huey
2005-05-28 11:32                                                                   ` Bill Huey
2005-05-27  9:03                                               ` Thomas Gleixner
2005-05-27  9:14                                                 ` Ingo Molnar
2005-05-27  9:22                                                   ` Thomas Gleixner
2005-06-14  0:48                                                   ` Daniel Walker
2005-06-14  3:29                                                     ` Valdis.Kletnieks
2005-06-14  3:38                                                       ` Daniel Walker
2005-05-26 20:38                                           ` john cooper
2005-05-26 21:00                                             ` Sven-Thorsten Dietrich
2005-05-26 21:23                                               ` john cooper
2005-05-26 20:52                                           ` Bill Huey
2005-05-26 21:09                                             ` Sven-Thorsten Dietrich
2005-05-26 21:14                                             ` Sven-Thorsten Dietrich
     [not found]                                           ` <20050526200424.GA27162@elte.hu>
     [not found]                                             ` <20050527123529.GD86087@muc.de>
2005-05-27 12:48                                               ` Ingo Molnar
2005-05-27 12:56                                                 ` Andi Kleen
2005-05-27 13:13                                                   ` Ingo Molnar
2005-05-27 13:31                                                     ` Andi Kleen
2005-05-27 13:44                                                       ` Ingo Molnar
2005-05-27 13:50                                                       ` Ingo Molnar
2005-05-27 13:53                                                       ` Ingo Molnar
2005-05-28 19:55                                                         ` Andi Kleen
2005-05-28 20:57                                                           ` Lee Revell
2005-05-27 13:56                                                       ` Takashi Iwai
2005-05-27 14:27                                                         ` Duncan Sands
2005-05-27 16:35                                                           ` Andrea Arcangeli
2005-05-30  9:53                                                         ` Andi Kleen
2005-05-30 10:33                                                           ` Ingo Molnar
2005-05-30 10:56                                                             ` Andi Kleen
2005-05-30 12:10                                                               ` what is the -RT tree Ingo Molnar
2005-05-30 12:40                                                                 ` Andi Kleen
2005-05-30 13:30                                                                   ` Bill Huey
2005-05-31 16:18                                                                   ` Sven-Thorsten Dietrich
2005-05-30 11:15                                                             ` RT patch acceptance Thomas Gleixner
2005-05-27 13:34                                                   ` Ingo Molnar
2005-05-25  3:48                                 ` john cooper
2005-05-25 11:35                                   ` Esben Nielsen
2005-05-25 13:14                                     ` john cooper
2005-05-25  6:09                               ` Ingo Molnar
2005-05-25  1:05                           ` Bill Huey
2005-05-25  2:37                   ` Karim Yaghmour
2005-05-25  2:36                     ` Sven Dietrich
2005-05-25  2:56                       ` Karim Yaghmour
2005-05-25  3:05                       ` Chris Friesen
2005-05-25 19:24                       ` Tim Bird
2005-05-25  2:38                     ` Lee Revell
2005-05-25  2:58                       ` Karim Yaghmour
2005-05-25  2:51                         ` Sven Dietrich
2005-05-25  6:15                     ` Bill Huey
2005-05-25 13:25                       ` Karim Yaghmour
2005-05-25  8:16                     ` Ingo Molnar
2005-05-25 13:27                       ` Karim Yaghmour
2005-05-25 17:32                         ` Sven-Thorsten Dietrich
2005-05-25 18:16                           ` Chris Friesen
2005-05-27  2:29                           ` Steven Rostedt
2005-05-25 16:01                     ` Jonathan Corbet
2005-05-24 11:22           ` K.R. Foley
2005-05-24 15:21             ` Nick Piggin
2005-05-24 15:41               ` K.R. Foley
2005-05-24 17:31                 ` Daniel Walker
2005-05-24 15:56               ` Ingo Molnar
2005-05-24 23:21               ` Bill Huey
2005-05-24 15:44         ` Daniel Walker
2005-05-24 15:47           ` Ingo Molnar
2005-05-24 16:27           ` john cooper
2005-05-24 18:01         ` Daniel Walker
2005-05-24  9:38   ` Esben Nielsen
2005-05-24 13:29     ` Karim Yaghmour
2005-05-24 13:58       ` Esben Nielsen
2005-05-24 14:42         ` Karim Yaghmour
2005-05-24 15:23           ` Ingo Molnar
2005-05-24 16:17             ` Karim Yaghmour
2005-05-24 21:23               ` Sven Dietrich
2005-05-24 22:31                 ` Karim Yaghmour
2005-05-24 22:54                   ` Sven Dietrich
2005-05-24 23:41                   ` Lee Revell
     [not found]                     ` <1116979434.2912.63.camel@mindpipe>
2005-05-25  0:48                       ` Karim Yaghmour
2005-05-24 23:46                   ` Lee Revell
2005-05-25  3:38                     ` Gene Heskett
2005-05-24 23:49                   ` Bill Huey
2005-05-25 14:26                     ` Philippe Gerum
2005-05-24 14:47         ` Gene Heskett
2005-05-24 17:22         ` Philippe Gerum
2005-05-25  0:21           ` Lee Revell
2005-05-25  2:04     ` Sven Dietrich
2005-05-25 20:58 ` Tom Vier
2005-05-25 21:05   ` Esben Nielsen
2005-05-25 21:10     ` NZG
2005-05-25 21:25     ` Tom Vier
2005-05-25 21:43       ` Esben Nielsen
2005-05-25 21:45       ` Bill Huey
2005-05-25 21:43   ` Bill Huey
  -- strict thread matches above, loose matches on Subject: below --
2005-05-25  6:55 kus Kusche Klaus
2005-05-25  7:06 ` Nick Piggin
2005-05-25  7:58 kus Kusche Klaus
2005-06-01 17:56 ` Paul G. Allen
2005-05-25 18:10 YhLu
2005-05-25 18:51 ` Andi Kleen
2005-05-30  8:25 kus Kusche Klaus
2005-05-30  9:49 ` Nick Piggin
2005-05-30 10:16 kus Kusche Klaus
2005-05-30 10:43 ` Nick Piggin
2005-05-30 11:34   ` Esben Nielsen
2005-05-30 11:58     ` Nick Piggin
2005-05-30 18:26       ` Steven Rostedt
2005-05-30 18:37       ` Esben Nielsen
2005-05-30 18:56         ` Karim Yaghmour
2005-05-30 18:58           ` Karim Yaghmour
2005-05-30 19:33           ` Esben Nielsen
2005-05-30 19:46             ` Karim Yaghmour
2005-05-30 20:00             ` Karim Yaghmour
2005-05-31  8:53               ` Esben Nielsen
2005-05-31 22:01               ` Bill Huey
2005-06-01  0:38                 ` Karim Yaghmour
2005-05-30 22:34           ` Bill Huey
2005-05-30 22:49             ` Karim Yaghmour
2005-05-30 22:49               ` Bill Huey
2005-05-30 23:15                 ` Karim Yaghmour
2005-05-31  0:07                   ` Thomas Gleixner
2005-05-31 16:29                   ` Steven Rostedt
2005-05-31 19:52                     ` Lee Revell
2005-05-31 20:16                       ` Steven Rostedt
2005-05-31 21:36                       ` Bill Huey
2005-05-31 19:55                     ` Lee Revell
2005-05-31 20:01                     ` Andi Kleen
2005-05-31 20:10                       ` Lee Revell
2005-05-31 20:14                       ` David Lang
2005-06-01  2:06                         ` Andrea Arcangeli
2005-06-01  2:13                           ` David Lang
2005-05-30 16:33 ` Zwane Mwaikambo
2005-05-31  4:39 Saksena, Manas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20050531191039.GV1680@eskimo.com \
    --to=elladan@eskimo.com \
    --cc=ak@muc.de \
    --cc=akpm@osdl.org \
    --cc=bhuey@lnxw.com \
    --cc=bruce@andrew.cmu.edu \
    --cc=dwalker@mvista.com \
    --cc=hch@infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=nickpiggin@yahoo.com.au \
    --cc=sdietrich@mvista.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox