Re: [PATCH v3 15/15] blk-mq: use hk cpus only when isolcpus=io_queue is enabled

Ming Lei
2024-08-09T14:53:55Z

On Fri, Aug 09, 2024 at 09:22:11AM +0200, Daniel Wagner wrote:
> On Thu, Aug 08, 2024 at 01:26:41PM GMT, Ming Lei wrote:
> > Isolated CPUs are removed from queue mapping in this patchset, when someone
> > submit IOs from the isolated CPU, what is the correct hctx used for handling
> > these IOs?
>
> No, every possible CPU gets a mapping. What this patch series does, is
> to limit/aligns the number of hardware context to the number of
> housekeeping CPUs. There is still a complete ctx-hctc mapping. So

OK, then I guess patch 1~7 aren't supposed to belong to this series,
cause you just want to reduce nr_hw_queues, meantime spread
house-keeping CPUs first for avoiding queues with all isolated cpu mask.

> whenever an user thread on an isolated CPU is issuing an IO a
> housekeeping CPU will also be involved (with the additional overhead,
> which seems to be okay for these users).
>
> Without hardware queue on the isolated CPUs ensures we really never get
> any unexpected IO on those CPUs unless userspace does it own its own.
> It's a safety net.
>
> Just to illustrate it, the non isolcpus configuration (default) map
> for an 8 CPU setup:
>
> queue mapping for /dev/vda
> hctx0: default 0
> hctx1: default 1
> hctx2: default 2
> hctx3: default 3
> hctx4: default 4
> hctx5: default 5
> hctx6: default 6
> hctx7: default 7
>
> and with isolcpus=io_queue,2-3,6-7
>
> queue mapping for /dev/vda
> hctx0: default 0 2
> hctx1: default 1 3
> hctx2: default 4 6
> hctx3: default 5 7

OK, Looks I missed the point in patch 15 in which you added isolated cpu
into mapping manually, just wondering why not take the current two-stage
policy to cover both house-keeping and isolated CPUs in group_cpus_evenly()?

Such as spread house-keeping CPUs first, then isolated CPUs, just like
what we did for present & non-present cpus.

Then the whole patchset can be simplified a lot.

>
> > From current implementation, it depends on implied zero filled
> > tag_set->map[type].mq_map[isolated_cpu], so hctx 0 is used.
> >
> > During CPU offline, in blk_mq_hctx_notify_offline(),
> > blk_mq_hctx_has_online_cpu() returns true even though the last cpu in
> > hctx 0 is offline because isolated cpus join hctx 0 unexpectedly, so IOs in
> > hctx 0 won't be drained.
> >
> > However managed irq core code still shutdowns the hw queue's irq because all
> > CPUs in this hctx are offline now. Then IO hang is triggered, isn't
> > it?
>
> Thanks for the explanation. I was able to reproduce this scenario, that
> is a hardware context with two CPUs which go offline. Initially, I used
> fio for creating the workload but this never hit the hanger. Instead
> some background workload from systemd-journald is pretty reliable to
> trigger the hanger you describe.
>
> Example:
>
> hctx2: default 4 6
>
> CPU 0 stays online, CPU 1-5 are offline. CPU 6 is offlined:
>
> smpboot: CPU 5 is now offline
> blk_mq_hctx_has_online_cpu:3537 hctx3 offline
> blk_mq_hctx_has_online_cpu:3537 hctx2 offline
>
> and there is no forward progress anymore, the cpuhotplug state machine
> is blocked and an IO is hanging:
>
> # grep busy /sys/kernel/debug/block/*/hctx*/tags | grep -v busy=0
> /sys/kernel/debug/block/vda/hctx2/tags:busy=61
>
> and blk_mq_hctx_notify_offline busy loops forever:
>
> task:cpuhp/6 state:D stack:0 pid:439 tgid:439 ppid:2 flags:0x00004000
> Call Trace:
> <TASK>
> __schedule+0x79d/0x15c0
> ? lockdep_hardirqs_on_prepare+0x152/0x210
> ? kvm_sched_clock_read+0xd/0x20
> ? local_clock_noinstr+0x28/0xb0
> ? local_clock+0x11/0x30
> ? lock_release+0x122/0x4a0
> schedule+0x3d/0xb0
> schedule_timeout+0x88/0xf0
> ? __pfx_process_timeout+0x10/0x10d
> msleep+0x28/0x40
> blk_mq_hctx_notify_offline+0x1b5/0x200
> ? cpuhp_thread_fun+0x41/0x1f0
> cpuhp_invoke_callback+0x27e/0x780
> ? __pfx_blk_mq_hctx_notify_offline+0x10/0x10
> ? cpuhp_thread_fun+0x42/0x1f0
> cpuhp_thread_fun+0x178/0x1f0
> smpboot_thread_fn+0x12e/0x1c0
> ? __pfx_smpboot_thread_fn+0x10/0x10
> kthread+0xe8/0x110
> ? __pfx_kthread+0x10/0x10
> ret_from_fork+0x33/0x40
> ? __pfx_kthread+0x10/0x10
> ret_from_fork_asm+0x1a/0x30
> </TASK>
>
> I don't think this is a new problem this code introduces. This problem
> exists for any hardware context which has more than one CPU. As far I
> understand it, the problem is that there is no forward progress possible
> for the IO itself (I assume the corresponding resources for the CPU

When blk_mq_hctx_notify_offline() is running, the current CPU isn't
offline yet, and the hctx is active, same with the managed irq, so it is fine
to wait until all in-flight IOs originated from this hctx completed there.

The reason is why these requests can't be completed? And the forward
progress is provided by blk-mq. And these requests are very likely
allocated & submitted from CPU6.

Can you figure out what is effective mask for irq of hctx2? It is
supposed to be cpu6. And block debugfs for vda should provide helpful
hint.

> going offline have already been shutdown, thus no progress?) and
> blk_mq_hctx_notifiy_offline isn't doing anything in this scenario.

RH has internal cpu hotplug stress test, but not see such report so far.

I will try to setup such kind of setting and see if it can be
reproduced.

>
> Couldn't we do something like:

I usually won't thinking about any solution until root-cause is figured
out, :-)

Thanks,
Ming