From: "Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>
To: Ming Lei <tom.leiming@gmail.com>
Cc: Jan Kara <jack@suse.cz>, Mikulas Patocka <mpatocka@redhat.com>,
Vlastimil Babka <vbabka@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Matthew Wilcox <willy@infradead.org>,
Michal Hocko <mhocko@suse.com>,
stable@vger.kernel.org, regressions@lists.linux.dev,
Alasdair Kergon <agk@redhat.com>,
Mike Snitzer <snitzer@kernel.org>,
dm-devel@lists.linux.dev, linux-mm@kvack.org
Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5
Date: Wed, 1 Nov 2023 03:14:22 +0100 [thread overview]
Message-ID: <ZUG0gcRhUlFm57qN@mail-itl> (raw)
In-Reply-To: <CACVXFVOEWDyzasS7DWDvLOhC3Hr6qOn5ks3HLX+fbRYCxYv26w@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 12717 bytes --]
On Wed, Nov 01, 2023 at 09:27:24AM +0800, Ming Lei wrote:
> On Tue, Oct 31, 2023 at 11:42 PM Marek Marczykowski-Górecki
> <marmarek@invisiblethingslab.com> wrote:
> >
> > On Tue, Oct 31, 2023 at 03:01:36PM +0100, Jan Kara wrote:
> > > On Tue 31-10-23 04:48:44, Marek Marczykowski-Górecki wrote:
> > > > Then tried:
> > > > - PAGE_ALLOC_COSTLY_ORDER=4, order=4 - cannot reproduce,
> > > > - PAGE_ALLOC_COSTLY_ORDER=4, order=5 - cannot reproduce,
> > > > - PAGE_ALLOC_COSTLY_ORDER=4, order=6 - freeze rather quickly
> > > >
> > > > I've retried the PAGE_ALLOC_COSTLY_ORDER=4,order=5 case several times
> > > > and I can't reproduce the issue there. I'm confused...
> > >
> > > And this kind of confirms that allocations > PAGE_ALLOC_COSTLY_ORDER
> > > causing hangs is most likely just a coincidence. Rather something either in
> > > the block layer or in the storage driver has problems with handling bios
> > > with sufficiently high order pages attached. This is going to be a bit
> > > painful to debug I'm afraid. How long does it take for you trigger the
> > > hang? I'm asking to get rough estimate how heavy tracing we can afford so
> > > that we don't overwhelm the system...
> >
> > Sometimes it freezes just after logging in, but in worst case it takes
> > me about 10min of more or less `tar xz` + `dd`.
>
> blk-mq debugfs is usually helpful for hang issue in block layer or
> underlying drivers:
>
> (cd /sys/kernel/debug/block && find . -type f -exec grep -aH . {} \;)
>
> BTW, you can just collect logs of the exact disks if you know what
> are behind dm-crypt,
> which can be figured out by `lsblk`, and it has to be collected after
> the hang is triggered.
dm-crypt lives on the nvme disk, this is what I collected when it
hanged:
nvme0n1/hctx5/type:default
nvme0n1/hctx5/dispatch_busy:0
nvme0n1/hctx5/active:0
nvme0n1/hctx5/run:273
nvme0n1/hctx5/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx5/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx5/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx5/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx5/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx5/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx5/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx5/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx5/tags:nr_tags=1023
nvme0n1/hctx5/tags:nr_reserved_tags=0
nvme0n1/hctx5/tags:active_queues=0
nvme0n1/hctx5/tags:bitmap_tags:
nvme0n1/hctx5/tags:depth=1023
nvme0n1/hctx5/tags:busy=0
nvme0n1/hctx5/tags:cleared=7
nvme0n1/hctx5/tags:bits_per_word=64
nvme0n1/hctx5/tags:map_nr=16
nvme0n1/hctx5/tags:alloc_hint={633, 450, 354, 913, 651, 645}
nvme0n1/hctx5/tags:wake_batch=8
nvme0n1/hctx5/tags:wake_index=0
nvme0n1/hctx5/tags:ws_active=0
nvme0n1/hctx5/tags:ws={
nvme0n1/hctx5/tags: {.wait=inactive},
nvme0n1/hctx5/tags: {.wait=inactive},
nvme0n1/hctx5/tags: {.wait=inactive},
nvme0n1/hctx5/tags: {.wait=inactive},
nvme0n1/hctx5/tags: {.wait=inactive},
nvme0n1/hctx5/tags: {.wait=inactive},
nvme0n1/hctx5/tags: {.wait=inactive},
nvme0n1/hctx5/tags: {.wait=inactive},
nvme0n1/hctx5/tags:}
nvme0n1/hctx5/tags:round_robin=0
nvme0n1/hctx5/tags:min_shallow_depth=4294967295
nvme0n1/hctx5/ctx_map:00000000: 00
nvme0n1/hctx5/flags:alloc_policy=FIFO SHOULD_MERGE
nvme0n1/hctx4/cpu4/default_rq_list:000000000d41998f {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=65, .internal_tag=-1}
nvme0n1/hctx4/cpu4/default_rq_list:00000000d0d04ed2 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=70, .internal_tag=-1}
nvme0n1/hctx4/type:default
nvme0n1/hctx4/dispatch_busy:9
nvme0n1/hctx4/active:0
nvme0n1/hctx4/run:20290468
nvme0n1/hctx4/tags_bitmap:00000000: 0000 0000 0000 0000 4240 0000 0000 0000
nvme0n1/hctx4/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx4/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx4/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx4/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx4/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx4/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx4/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx4/tags:nr_tags=1023
nvme0n1/hctx4/tags:nr_reserved_tags=0
nvme0n1/hctx4/tags:active_queues=0
nvme0n1/hctx4/tags:bitmap_tags:
nvme0n1/hctx4/tags:depth=1023
nvme0n1/hctx4/tags:busy=3
nvme0n1/hctx4/tags:cleared=7
nvme0n1/hctx4/tags:bits_per_word=64
nvme0n1/hctx4/tags:map_nr=16
nvme0n1/hctx4/tags:alloc_hint={899, 846, 390, 472, 73, 439}
nvme0n1/hctx4/tags:wake_batch=8
nvme0n1/hctx4/tags:wake_index=0
nvme0n1/hctx4/tags:ws_active=0
nvme0n1/hctx4/tags:ws={
nvme0n1/hctx4/tags: {.wait=inactive},
nvme0n1/hctx4/tags: {.wait=inactive},
nvme0n1/hctx4/tags: {.wait=inactive},
nvme0n1/hctx4/tags: {.wait=inactive},
nvme0n1/hctx4/tags: {.wait=inactive},
nvme0n1/hctx4/tags: {.wait=inactive},
nvme0n1/hctx4/tags: {.wait=inactive},
nvme0n1/hctx4/tags: {.wait=inactive},
nvme0n1/hctx4/tags:}
nvme0n1/hctx4/tags:round_robin=0
nvme0n1/hctx4/tags:min_shallow_depth=4294967295
nvme0n1/hctx4/ctx_map:00000000: 01
nvme0n1/hctx4/dispatch:00000000b335fa89 {.op=WRITE, .cmd_flags=NOMERGE, .rq_flags=DONTPREP|IO_STAT, .state=idle, .tag=78, .internal_tag=-1}
nvme0n1/hctx4/flags:alloc_policy=FIFO SHOULD_MERGE
nvme0n1/hctx4/state:SCHED_RESTART
nvme0n1/hctx3/type:default
nvme0n1/hctx3/dispatch_busy:0
nvme0n1/hctx3/active:0
nvme0n1/hctx3/run:296
nvme0n1/hctx3/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx3/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx3/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx3/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx3/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx3/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx3/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx3/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx3/tags:nr_tags=1023
nvme0n1/hctx3/tags:nr_reserved_tags=0
nvme0n1/hctx3/tags:active_queues=0
nvme0n1/hctx3/tags:bitmap_tags:
nvme0n1/hctx3/tags:depth=1023
nvme0n1/hctx3/tags:busy=0
nvme0n1/hctx3/tags:cleared=23
nvme0n1/hctx3/tags:bits_per_word=64
nvme0n1/hctx3/tags:map_nr=16
nvme0n1/hctx3/tags:alloc_hint={862, 557, 480, 24, 841, 23}
nvme0n1/hctx3/tags:wake_batch=8
nvme0n1/hctx3/tags:wake_index=0
nvme0n1/hctx3/tags:ws_active=0
nvme0n1/hctx3/tags:ws={
nvme0n1/hctx3/tags: {.wait=inactive},
nvme0n1/hctx3/tags: {.wait=inactive},
nvme0n1/hctx3/tags: {.wait=inactive},
nvme0n1/hctx3/tags: {.wait=inactive},
nvme0n1/hctx3/tags: {.wait=inactive},
nvme0n1/hctx3/tags: {.wait=inactive},
nvme0n1/hctx3/tags: {.wait=inactive},
nvme0n1/hctx3/tags: {.wait=inactive},
nvme0n1/hctx3/tags:}
nvme0n1/hctx3/tags:round_robin=0
nvme0n1/hctx3/tags:min_shallow_depth=4294967295
nvme0n1/hctx3/ctx_map:00000000: 00
nvme0n1/hctx3/flags:alloc_policy=FIFO SHOULD_MERGE
nvme0n1/hctx2/type:default
nvme0n1/hctx2/dispatch_busy:0
nvme0n1/hctx2/active:0
nvme0n1/hctx2/run:279
nvme0n1/hctx2/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx2/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx2/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx2/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx2/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx2/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx2/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx2/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx2/tags:nr_tags=1023
nvme0n1/hctx2/tags:nr_reserved_tags=0
nvme0n1/hctx2/tags:active_queues=0
nvme0n1/hctx2/tags:bitmap_tags:
nvme0n1/hctx2/tags:depth=1023
nvme0n1/hctx2/tags:busy=0
nvme0n1/hctx2/tags:cleared=16
nvme0n1/hctx2/tags:bits_per_word=64
nvme0n1/hctx2/tags:map_nr=16
nvme0n1/hctx2/tags:alloc_hint={960, 528, 145, 730, 447, 1002}
nvme0n1/hctx2/tags:wake_batch=8
nvme0n1/hctx2/tags:wake_index=0
nvme0n1/hctx2/tags:ws_active=0
nvme0n1/hctx2/tags:ws={
nvme0n1/hctx2/tags: {.wait=inactive},
nvme0n1/hctx2/tags: {.wait=inactive},
nvme0n1/hctx2/tags: {.wait=inactive},
nvme0n1/hctx2/tags: {.wait=inactive},
nvme0n1/hctx2/tags: {.wait=inactive},
nvme0n1/hctx2/tags: {.wait=inactive},
nvme0n1/hctx2/tags: {.wait=inactive},
nvme0n1/hctx2/tags: {.wait=inactive},
nvme0n1/hctx2/tags:}
nvme0n1/hctx2/tags:round_robin=0
nvme0n1/hctx2/tags:min_shallow_depth=4294967295
nvme0n1/hctx2/ctx_map:00000000: 00
nvme0n1/hctx2/flags:alloc_policy=FIFO SHOULD_MERGE
nvme0n1/hctx1/type:default
nvme0n1/hctx1/dispatch_busy:0
nvme0n1/hctx1/active:0
nvme0n1/hctx1/run:458
nvme0n1/hctx1/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx1/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx1/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx1/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx1/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx1/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx1/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx1/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx1/tags:nr_tags=1023
nvme0n1/hctx1/tags:nr_reserved_tags=0
nvme0n1/hctx1/tags:active_queues=0
nvme0n1/hctx1/tags:bitmap_tags:
nvme0n1/hctx1/tags:depth=1023
nvme0n1/hctx1/tags:busy=0
nvme0n1/hctx1/tags:cleared=31
nvme0n1/hctx1/tags:bits_per_word=64
nvme0n1/hctx1/tags:map_nr=16
nvme0n1/hctx1/tags:alloc_hint={689, 284, 498, 188, 808, 610}
nvme0n1/hctx1/tags:wake_batch=8
nvme0n1/hctx1/tags:wake_index=0
nvme0n1/hctx1/tags:ws_active=0
nvme0n1/hctx1/tags:ws={
nvme0n1/hctx1/tags: {.wait=inactive},
nvme0n1/hctx1/tags: {.wait=inactive},
nvme0n1/hctx1/tags: {.wait=inactive},
nvme0n1/hctx1/tags: {.wait=inactive},
nvme0n1/hctx1/tags: {.wait=inactive},
nvme0n1/hctx1/tags: {.wait=inactive},
nvme0n1/hctx1/tags: {.wait=inactive},
nvme0n1/hctx1/tags: {.wait=inactive},
nvme0n1/hctx1/tags:}
nvme0n1/hctx1/tags:round_robin=0
nvme0n1/hctx1/tags:min_shallow_depth=4294967295
nvme0n1/hctx1/ctx_map:00000000: 00
nvme0n1/hctx1/flags:alloc_policy=FIFO SHOULD_MERGE
nvme0n1/hctx0/type:default
nvme0n1/hctx0/dispatch_busy:0
nvme0n1/hctx0/active:0
nvme0n1/hctx0/run:375
nvme0n1/hctx0/tags_bitmap:00000000: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx0/tags_bitmap:00000010: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx0/tags_bitmap:00000020: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx0/tags_bitmap:00000030: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx0/tags_bitmap:00000040: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx0/tags_bitmap:00000050: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx0/tags_bitmap:00000060: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx0/tags_bitmap:00000070: 0000 0000 0000 0000 0000 0000 0000 0000
nvme0n1/hctx0/tags:nr_tags=1023
nvme0n1/hctx0/tags:nr_reserved_tags=0
nvme0n1/hctx0/tags:active_queues=0
nvme0n1/hctx0/tags:bitmap_tags:
nvme0n1/hctx0/tags:depth=1023
nvme0n1/hctx0/tags:busy=0
nvme0n1/hctx0/tags:cleared=6
nvme0n1/hctx0/tags:bits_per_word=64
nvme0n1/hctx0/tags:map_nr=16
nvme0n1/hctx0/tags:alloc_hint={5, 18, 320, 448, 728, 9}
nvme0n1/hctx0/tags:wake_batch=8
nvme0n1/hctx0/tags:wake_index=0
nvme0n1/hctx0/tags:ws_active=0
nvme0n1/hctx0/tags:ws={
nvme0n1/hctx0/tags: {.wait=inactive},
nvme0n1/hctx0/tags: {.wait=inactive},
nvme0n1/hctx0/tags: {.wait=inactive},
nvme0n1/hctx0/tags: {.wait=inactive},
nvme0n1/hctx0/tags: {.wait=inactive},
nvme0n1/hctx0/tags: {.wait=inactive},
nvme0n1/hctx0/tags: {.wait=inactive},
nvme0n1/hctx0/tags: {.wait=inactive},
nvme0n1/hctx0/tags:}
nvme0n1/hctx0/tags:round_robin=0
nvme0n1/hctx0/tags:min_shallow_depth=4294967295
nvme0n1/hctx0/ctx_map:00000000: 00
nvme0n1/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE
nvme0n1/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|WC|FUA|REGISTERED|NOWAIT
nvme0n1/pm_only:0
This is without any of the patches Mikulas proposed.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
next prev parent reply other threads:[~2023-11-01 2:14 UTC|newest]
Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-21 3:38 Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Marek Marczykowski-Górecki
2023-10-21 7:48 ` Bagas Sanjaya
2023-10-29 6:23 ` Linux regression tracking #update (Thorsten Leemhuis)
2023-10-23 20:59 ` Mikulas Patocka
2023-10-25 3:10 ` Marek Marczykowski-Górecki
2023-10-25 3:22 ` Marek Marczykowski-Górecki
2023-10-25 10:13 ` Mikulas Patocka
2023-10-27 17:32 ` Mikulas Patocka
2023-10-28 9:23 ` Matthew Wilcox
2023-10-28 15:14 ` Mike Snitzer
2023-10-29 11:15 ` Marek Marczykowski-Górecki
2023-10-29 20:02 ` Vlastimil Babka
2023-10-30 7:37 ` Mikulas Patocka
2023-10-30 8:37 ` Vlastimil Babka
2023-10-30 11:22 ` Mikulas Patocka
2023-10-30 11:30 ` Vlastimil Babka
2023-10-30 11:37 ` Mikulas Patocka
2023-10-30 12:25 ` Jan Kara
2023-10-30 13:30 ` Marek Marczykowski-Górecki
2023-10-30 14:08 ` Mikulas Patocka
2023-10-30 15:56 ` Jan Kara
2023-10-30 16:51 ` Marek Marczykowski-Górecki
2023-10-30 17:50 ` Mikulas Patocka
2023-10-31 3:48 ` Marek Marczykowski-Górecki
2023-10-31 14:01 ` Jan Kara
2023-10-31 15:42 ` Marek Marczykowski-Górecki
2023-10-31 17:17 ` Mikulas Patocka
2023-10-31 17:24 ` Mikulas Patocka
2023-11-02 0:38 ` Marek Marczykowski-Górecki
2023-11-02 9:28 ` Mikulas Patocka
2023-11-02 11:45 ` Marek Marczykowski-Górecki
2023-11-02 17:06 ` Mikulas Patocka
2023-11-03 15:01 ` Marek Marczykowski-Górecki
2023-11-03 15:10 ` Keith Busch
2023-11-03 16:15 ` Marek Marczykowski-Górecki
2023-11-03 16:54 ` Keith Busch
2023-11-03 20:30 ` Marek Marczykowski-G'orecki
2023-11-03 22:42 ` Keith Busch
2023-11-04 9:27 ` Mikulas Patocka
2023-11-04 13:59 ` Keith Busch
2023-11-06 7:10 ` Christoph Hellwig
2023-11-06 14:59 ` [PATCH] swiotlb-xen: provide the "max_mapping_size" method Mikulas Patocka
2023-11-06 15:16 ` Keith Busch
2023-11-06 15:30 ` Mike Snitzer
2023-11-06 17:12 ` [PATCH v2] " Mikulas Patocka
2023-11-07 4:18 ` Stefano Stabellini
2023-11-08 7:31 ` Christoph Hellwig
2023-11-06 7:08 ` Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Christoph Hellwig
2023-11-02 12:21 ` Jan Kara
2023-11-01 1:27 ` Ming Lei
2023-11-01 2:14 ` Marek Marczykowski-Górecki [this message]
2023-11-01 2:15 ` Marek Marczykowski-Górecki
2023-11-01 2:35 ` Marek Marczykowski-Górecki
2023-11-01 3:24 ` Ming Lei
2023-11-01 10:15 ` Hannes Reinecke
2023-11-01 10:26 ` Jan Kara
2023-11-01 11:23 ` Ming Lei
2023-11-02 14:02 ` Keith Busch
2023-11-01 12:16 ` Mikulas Patocka
2023-10-30 11:28 ` Jan Kara
2023-10-30 11:49 ` Mikulas Patocka
2023-10-30 12:11 ` Jan Kara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZUG0gcRhUlFm57qN@mail-itl \
--to=marmarek@invisiblethingslab.com \
--cc=agk@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=dm-devel@lists.linux.dev \
--cc=jack@suse.cz \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.com \
--cc=mpatocka@redhat.com \
--cc=regressions@lists.linux.dev \
--cc=snitzer@kernel.org \
--cc=stable@vger.kernel.org \
--cc=tom.leiming@gmail.com \
--cc=vbabka@suse.cz \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox