Re: linux-next: Boot hangs 3 minutes with device mapper on s390

Re: linux-next: Boot hangs 3 minutes with device mapper on s390

[Mike Snitzer]

On Fri, Mar 01 2019 at 12:33pm -0500,
Michael Holzheu <holzheu@linux.ibm.com> wrote:

> Hi Mike,
> 
> On Fedora 29, the following "linux-next" commit introduced a regression on s390:
> 
>   commit 1efa3bb79d3de8ca1b7f6770313a1fc0bebe25c7
>   Author: Mike Snitzer <snitzer@redhat.com>
>   Date:   Fri Feb 22 11:23:01 2019 -0500
> 
>     dm: must allocate dm_noclone for stacked noclone devices
>     
>     Otherwise various lvm2 testsuite tests fail because the lower layers of
>     the stacked noclone device aren't updated to allocate a new 'struct
>     dm_clone' that reflects the upper layer bio that was issued to it.
>     
>     Fixes: 97a89458020b38 ("dm: improve noclone bio support")
>     Reported-by: Mikulas Patocka <mpatocka@redhat.com>
>     Signed-off-by: Mike Snitzer <snitzer@redhat.com>
> 
> With this commit the boot hangs three minutes on a z/VM system with the
> following device mapper setup:
> 
> # dmsetup ls --tree
> mpathe (252:5)
>  ├─ (8:128)
>  └─ (8:144)
> mpathd (252:4)
>  ├─ (8:96)
>  └─ (8:112)
> mpathc (252:3)
>  ├─ (8:64)
>  └─ (8:80)
> mpathb (252:2)
>  ├─ (8:32)
>  └─ (8:48)
> mpatha1 (252:1)
>  └─mpatha (252:0)
>     ├─ (8:16)
>     └─ (8:0)
> 
> On the console we get messages like the following:
> 
>    10.116863 sd 3:0:0:1083719813: sdj Write Protect is off 
>    10.117170 sd 3:0:0:1083719813: sdj Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
>    10.130562 sd 3:0:0:1083719813: sdj Attached SCSI disk 
>  A start job is running for udev Wai   Device Initialization (6s / 3min)
>  A start job is running for udev Wai   Device Initialization (6s / 3min)
>  A start job is running for udev Wai   Device Initialization (7s / 3min)
>  A start job is running for udev Wai   Device Initialization (7s / 3min)
>  A start job is running for udev Wai   Device Initialization (8s / 3min)
>  ...
> 
> After three minutes the boot process continues and the system comes.
> 
> As kernel config we used "performance_defconfig" (make performance_defconfig).

I'm struggling to see why this particular change would cause such a boot
stall -- but then resolve itself.

Can you provide the output from 'dmsetup table'?

Multipath defaults to blk-mq (request-based).  The change you've called
into question is related to bio-based DM.  So There must be some
bio-based DM layer ontop of the multipath devices.  Is mpatha1 a
dm-linear device layered ontop of multipath?

Thanks,
Mike

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: linux-next: Boot hangs 3 minutes with device mapper on s390

[Steffen Maier]

On 03/01/2019 07:30 PM, Mike Snitzer wrote:
> On Fri, Mar 01 2019 at 12:33pm -0500,
> Michael Holzheu <holzheu@linux.ibm.com> wrote:
> 
>> Hi Mike,
>>
>> On Fedora 29, the following "linux-next" commit introduced a regression on s390:
>>
>>    commit 1efa3bb79d3de8ca1b7f6770313a1fc0bebe25c7
>>    Author: Mike Snitzer <snitzer@redhat.com>
>>    Date:   Fri Feb 22 11:23:01 2019 -0500
>>
>>      dm: must allocate dm_noclone for stacked noclone devices
>>      
>>      Otherwise various lvm2 testsuite tests fail because the lower layers of
>>      the stacked noclone device aren't updated to allocate a new 'struct
>>      dm_clone' that reflects the upper layer bio that was issued to it.
>>      
>>      Fixes: 97a89458020b38 ("dm: improve noclone bio support")
>>      Reported-by: Mikulas Patocka <mpatocka@redhat.com>
>>      Signed-off-by: Mike Snitzer <snitzer@redhat.com>
>>
>> With this commit the boot hangs three minutes on a z/VM system with the
>> following device mapper setup:
>>
>> # dmsetup ls --tree
>> mpathe (252:5)
>>   ├─ (8:128)
>>   └─ (8:144)
>> mpathd (252:4)
>>   ├─ (8:96)
>>   └─ (8:112)
>> mpathc (252:3)
>>   ├─ (8:64)
>>   └─ (8:80)
>> mpathb (252:2)
>>   ├─ (8:32)
>>   └─ (8:48)
>> mpatha1 (252:1)
>>   └─mpatha (252:0)
>>      ├─ (8:16)
>>      └─ (8:0)
>>
>> On the console we get messages like the following:
>>
>>     10.116863 sd 3:0:0:1083719813: sdj Write Protect is off
>>     10.117170 sd 3:0:0:1083719813: sdj Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>     10.130562 sd 3:0:0:1083719813: sdj Attached SCSI disk
>>   A start job is running for udev Wai   Device Initialization (6s / 3min)
>>   A start job is running for udev Wai   Device Initialization (6s / 3min)
>>   A start job is running for udev Wai   Device Initialization (7s / 3min)
>>   A start job is running for udev Wai   Device Initialization (7s / 3min)
>>   A start job is running for udev Wai   Device Initialization (8s / 3min)
>>   ...
>>
>> After three minutes the boot process continues and the system comes.
>>
>> As kernel config we used "performance_defconfig" (make performance_defconfig).
> 
> I'm struggling to see why this particular change would cause such a boot
> stall -- but then resolve itself.
> 
> Can you provide the output from 'dmsetup table'?
> 
> Multipath defaults to blk-mq (request-based).  The change you've called
> into question is related to bio-based DM.  So There must be some
> bio-based DM layer ontop of the multipath devices.  Is mpatha1 a
> dm-linear device layered ontop of multipath?

Yes, mpatha1 is just an automatically generated kpartx linear partition 
mapping, so that one is bio-based.


-- 
Mit freundlichen Gruessen / Kind regards
Steffen Maier

Linux on IBM Z Development

https://www.ibm.com/privacy/us/en/
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: linux-next: Boot hangs 3 minutes with device mapper on s390

[Michael Holzheu]

Am Fri, 1 Mar 2019 13:30:50 -0500
schrieb Mike Snitzer <snitzer@redhat.com>:

> On Fri, Mar 01 2019 at 12:33pm -0500,
> Michael Holzheu <holzheu@linux.ibm.com> wrote:
> 
> > Hi Mike,
> > 
> > On Fedora 29, the following "linux-next" commit introduced a regression on s390:
> > 
> >   commit 1efa3bb79d3de8ca1b7f6770313a1fc0bebe25c7
> >   Author: Mike Snitzer <snitzer@redhat.com>
> >   Date:   Fri Feb 22 11:23:01 2019 -0500
> > 
> >     dm: must allocate dm_noclone for stacked noclone devices
> >     
> >     Otherwise various lvm2 testsuite tests fail because the lower layers of
> >     the stacked noclone device aren't updated to allocate a new 'struct
> >     dm_clone' that reflects the upper layer bio that was issued to it.
> >     
> >     Fixes: 97a89458020b38 ("dm: improve noclone bio support")
> >     Reported-by: Mikulas Patocka <mpatocka@redhat.com>
> >     Signed-off-by: Mike Snitzer <snitzer@redhat.com>
> > 
> > With this commit the boot hangs three minutes on a z/VM system with the
> > following device mapper setup:
> > 
> > # dmsetup ls --tree
> > mpathe (252:5)
> >  ├─ (8:128)
> >  └─ (8:144)
> > mpathd (252:4)
> >  ├─ (8:96)
> >  └─ (8:112)
> > mpathc (252:3)
> >  ├─ (8:64)
> >  └─ (8:80)
> > mpathb (252:2)
> >  ├─ (8:32)
> >  └─ (8:48)
> > mpatha1 (252:1)
> >  └─mpatha (252:0)
> >     ├─ (8:16)
> >     └─ (8:0)
> > 
> > On the console we get messages like the following:
> > 
> >    10.116863 sd 3:0:0:1083719813: sdj Write Protect is off 
> >    10.117170 sd 3:0:0:1083719813: sdj Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
> >    10.130562 sd 3:0:0:1083719813: sdj Attached SCSI disk 
> >  A start job is running for udev Wai   Device Initialization (6s / 3min)
> >  A start job is running for udev Wai   Device Initialization (6s / 3min)
> >  A start job is running for udev Wai   Device Initialization (7s / 3min)
> >  A start job is running for udev Wai   Device Initialization (7s / 3min)
> >  A start job is running for udev Wai   Device Initialization (8s / 3min)
> >  ...
> > 
> > After three minutes the boot process continues and the system comes.
> > 
> > As kernel config we used "performance_defconfig" (make performance_defconfig).
> 
> I'm struggling to see why this particular change would cause such a boot
> stall -- but then resolve itself.
> 
> Can you provide the output from 'dmsetup table'?

# dmsetup table
mpathe: 0 41943040 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 2 8:64 1 1 8:128 1 1 
mpathd: 0 41943040 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 2 8:48 1 1 8:112 1 1 
mpathc: 0 41943040 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 2 8:32 1 1 8:96 1 1 
mpathb: 0 41943040 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 2 8:16 1 1 8:80 1 1 
mpatha: 0 209715200 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 2 8:0 1 1 
mpatha1: 0 209713152 linear 252:0 2048

 
> Multipath defaults to blk-mq (request-based).  The change you've called
> into question is related to bio-based DM.  So There must be some
> bio-based DM layer ontop of the multipath devices.  Is mpatha1 a
> dm-linear device layered ontop of multipath?

See above output of "dmsetup table".

Best Regards,
Michael


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: linux-next: Boot hangs 3 minutes with device mapper on s390

[Mike Snitzer]

Hi,

Alexander reported this same boot hang in another thread.  I was able to
reproduce using an x86_64 .config that Alexander provided.

I've pushed this fix out to linux-next:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.1&id=be2c5301817833f692aadb2e5fa209582db01d3b

If you could verify this fix works for you I'd appreciate it.

Thanks,
Mike

Re: linux-next: Boot hangs 3 minutes with device mapper on s390

[Mikulas Patocka]

On Mon, 4 Mar 2019, Mike Snitzer wrote:

> Hi,
> 
> Alexander reported this same boot hang in another thread.  I was able to
> reproduce using an x86_64 .config that Alexander provided.
> 
> I've pushed this fix out to linux-next:
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.1&id=be2c5301817833f692aadb2e5fa209582db01d3b
> 
> If you could verify this fix works for you I'd appreciate it.
> 
> Thanks,
> Mike

So, remove this no-clone optimization and stage it for the next merge 
window. There's something we don't understand, so don't merge it. This 
patch is just papering over the problem.

"Stacking noclone targets creates more complexity than is tolerable" just 
means that no one knows what is hapenning there.

Meanwile, we could get access to the system that reports hangs and test it 
there.

Mikulas

Re: linux-next: Boot hangs 3 minutes with device mapper on s390

[Mike Snitzer]

On Tue, Mar 05 2019 at  4:46am -0500,
Mikulas Patocka <mpatocka@redhat.com> wrote:

> 
> 
> On Mon, 4 Mar 2019, Mike Snitzer wrote:
> 
> > Hi,
> > 
> > Alexander reported this same boot hang in another thread.  I was able to
> > reproduce using an x86_64 .config that Alexander provided.
> > 
> > I've pushed this fix out to linux-next:
> > https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.1&id=be2c5301817833f692aadb2e5fa209582db01d3b
> > 
> > If you could verify this fix works for you I'd appreciate it.
> > 
> > Thanks,
> > Mike
> 
> So, remove this no-clone optimization and stage it for the next merge 
> window. There's something we don't understand, so don't merge it. This 
> patch is just papering over the problem.

I layered changes that extended your initial noclone support and that
seems to have upset you.  Yes the evolution could've been cleaner but
to say this is papering over anything is purely wrong.

Clearly we're reentering dm_noclone_process_bio() and the relaxed
negative check that only concerned itself with whether we were in
make_request_fn lost sight of the potential for losing an 'struct
dm_noclone' that was already attached to the bio.  _THAT_ is cause for
the hang.  Full stop.

I didn't have time to sort out why we rentered dm_noclone_process_bio()
yesterday.  But I can easily do so today.  Just to fully appreciate
_why_ it happened.

To categorize any of this as papering over is just _wrong_ and I don't
understand why you think it OK to accuse me with that.  Rather than dig
in to help you've sat back and attacked me.

> "Stacking noclone targets creates more complexity than is tolerable" just 
> means that no one knows what is hapenning there.

Not constructive.  What it means is: I wrote the code that enables
splitting + stacking no_clone + dm_work_fn rentry in dm_process_bio.
The duality of use in the shared code paths and flag day to support all
of it made this noclone optimization brittle.  I'll grant you that.

Given the mental gymnastics I had to do to reason through what _could_ be
going on in different stacking scenarios just made me uneasy to support
such noclone complexity from the start.  Could revisit allowing stacking
at a later date though.
 
> Meanwile, we could get access to the system that reports hangs and test it 
> there.

What part of "I was able to reproduce using an x86_64 .config that
Alexander provided" don't you understand?

It wasn't even that the code was just superficially broken.  It required
Alexander's .config to tease out the problem.  It took quite a while to
get my kvm guest testbed sorted out and zero in on reproducing.  I did
that work.  I then asked those who reported the problem to confirm it
fixes the issue for them.

Re: linux-next: Boot hangs 3 minutes with device mapper on s390

[Alexander Duyck]

On Mon, Mar 4, 2019 at 8:03 PM Mike Snitzer <snitzer@redhat.com> wrote:
>
> Hi,
>
> Alexander reported this same boot hang in another thread.  I was able to
> reproduce using an x86_64 .config that Alexander provided.
>
> I've pushed this fix out to linux-next:
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.1&id=be2c5301817833f692aadb2e5fa209582db01d3b
>
> If you could verify this fix works for you I'd appreciate it.
>
> Thanks,
> Mike

I've verified I am no longer seeing the issue with this patch applied.

Thanks.

- Alex

bio-based DM's "noclone" changes have been dropped from 5.1 [was: Re: linux-next: Boot hangs 3 minutes with device mapper on s390]

[Mike Snitzer]

On Tue, Mar 05 2019 at  1:02pm -0500,
Alexander Duyck <alexander.duyck@gmail.com> wrote:

> On Mon, Mar 4, 2019 at 8:03 PM Mike Snitzer <snitzer@redhat.com> wrote:
> >
> > Hi,
> >
> > Alexander reported this same boot hang in another thread.  I was able to
> > reproduce using an x86_64 .config that Alexander provided.
> >
> > I've pushed this fix out to linux-next:
> > https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.1&id=be2c5301817833f692aadb2e5fa209582db01d3b
> >
> > If you could verify this fix works for you I'd appreciate it.
> >
> > Thanks,
> > Mike
> 
> I've verified I am no longer seeing the issue with this patch applied.

Thanks for testing.

But so you and others are aware: based on further internal discussion
with Mikulas, I decided to drop all the bio-based DM "noclone" changes
that were developed for 5.1.

Mikulas had a really good point that the 10-13% performance benefit seen
with 512b IO workloads all but disappears to only a 1% improvement as
soon as the bio's payload is increased to >= 4K.  Meaning the risk
associated with the noclone changes really is _not_ worth the reward.

I've rebased the dm-5.1 branch accordingly and linux-next will pick it
up tonight.

Mike

ps. in case anyone cares, the noclone changes have been preserved in
this dm-noclone-support branch:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-noclone-support

Re: bio-based DM's "noclone" changes have been dropped from 5.1 [was: Re: linux-next: Boot hangs 3 minutes with device mapper on s390]

[Michael Holzheu]

On Tue, 5 Mar 2019 17:21:12 -0500
Mike Snitzer <snitzer@redhat.com> wrote:

> On Tue, Mar 05 2019 at  1:02pm -0500,
> Alexander Duyck <alexander.duyck@gmail.com> wrote:
> 
> > On Mon, Mar 4, 2019 at 8:03 PM Mike Snitzer <snitzer@redhat.com> wrote:
> > >
> > > Hi,
> > >
> > > Alexander reported this same boot hang in another thread.  I was able to
> > > reproduce using an x86_64 .config that Alexander provided.
> > >
> > > I've pushed this fix out to linux-next:
> > > https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.1&id=be2c5301817833f692aadb2e5fa209582db01d3b
> > >
> > > If you could verify this fix works for you I'd appreciate it.
> > >
> > > Thanks,
> > > Mike
> > 
> > I've verified I am no longer seeing the issue with this patch applied.
> 
> Thanks for testing.
> 
> But so you and others are aware: based on further internal discussion
> with Mikulas, I decided to drop all the bio-based DM "noclone" changes
> that were developed for 5.1.
> 
> Mikulas had a really good point that the 10-13% performance benefit seen
> with 512b IO workloads all but disappears to only a 1% improvement as
> soon as the bio's payload is increased to >= 4K.  Meaning the risk
> associated with the noclone changes really is _not_ worth the reward.
> 
> I've rebased the dm-5.1 branch accordingly and linux-next will pick it
> up tonight.

I can confirm that the problem disappeared now on s390.

Thanks!
Michael

Re: linux-next: Boot hangs 3 minutes with device mapper on s390

[Mike Snitzer]

On Fri, Mar 01 2019 at  3:19pm -0500,
Mikulas Patocka <mpatocka@redhat.com> wrote:

> 
> 
> On Fri, 1 Mar 2019, Michael Holzheu wrote:
> 
> > Hi Mike,
> > 
> > On Fedora 29, the following "linux-next" commit introduced a regression on s390:
> > 
> >   commit 1efa3bb79d3de8ca1b7f6770313a1fc0bebe25c7
> >   Author: Mike Snitzer <snitzer@redhat.com>
> >   Date:   Fri Feb 22 11:23:01 2019 -0500
> > 
> >     dm: must allocate dm_noclone for stacked noclone devices
> >     
> >     Otherwise various lvm2 testsuite tests fail because the lower layers of
> >     the stacked noclone device aren't updated to allocate a new 'struct
> >     dm_clone' that reflects the upper layer bio that was issued to it.
> >     
> >     Fixes: 97a89458020b38 ("dm: improve noclone bio support")
> >     Reported-by: Mikulas Patocka <mpatocka@redhat.com>
> >     Signed-off-by: Mike Snitzer <snitzer@redhat.com>
> 
> Should we just drop 97a89458020b388b910160c4f4aa5e24318d2460 and 
> 1efa3bb79d3de8ca1b7f6770313a1fc0bebe25c7?

No.

> 97a89458020b388b910160c4f4aa5e24318d2460 claims to be an improvement, but 
> it broke the tests twice.

This report from Michael needs proper review and triage.

It literally makes _zero_ sense that all 5 mpath devices are suffering
from a 3 minute boot stall (rooted in udev action) when I'm pretty
certain they are using request-based multipath for 4 of the 5 devices.
The 5th likely having a dm-linear ontop of request-based multipath.
And I've tested stacking bio-based linear on request-based multipath.

Mike