Re: [PATCH v9 21/24] x86/resctrl: Allow overflow/limbo handlers to be scheduled on any-but cpu

Tony Luck
2024-03-25T23:14:25Z

> -static inline unsigned int cpumask_any_housekeeping(const struct cpumask *mask)
> +static inline unsigned int
> +cpumask_any_housekeeping(const struct cpumask *mask, int exclude_cpu)
> {
> unsigned int cpu, hk_cpu;
>
> - cpu = cpumask_any(mask);
> - if (!tick_nohz_full_cpu(cpu))
> + if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
> + cpu = cpumask_any(mask);
> + else
> + cpu = cpumask_any_but(mask, exclude_cpu);
> +
> + if (!IS_ENABLED(CONFIG_NO_HZ_FULL))
> + return cpu;
> +
> + /* If the CPU picked isn't marked nohz_full nothing more needs doing. */
> + if (cpu < nr_cpu_ids && !tick_nohz_full_cpu(cpu))
> return cpu;
>
> + /* Try to find a CPU that isn't nohz_full to use in preference */
> hk_cpu = cpumask_nth_andnot(0, mask, tick_nohz_full_mask);
> + if (hk_cpu == exclude_cpu)
> + hk_cpu = cpumask_nth_andnot(1, mask, tick_nohz_full_mask);

OOPS HERE ^^^^^^^^^^^^^^^^^^
> +
> if (hk_cpu < nr_cpu_ids)
> cpu = hk_cpu;

James,

I'm playing around with v6.9-rc1. When I take all the CPUs in a domain
offline, resctrl wants to stop the mbm overflow handler. While doing
so it crashes with a NULL dereference in the code above.

At this point the cpu_mask for the domain still has the bit for
the last CPU standing in this domain. So cpumask_any_housekeeping()
is called with a mask that just has CPU 36 set, but request to find
a CPU other than 36.

-Tony

Heres the OOPs:

[ 97.107355] smpboot: CPU 37 is now offline
[ 97.166136] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 97.173118] #PF: supervisor read access in kernel mode
[ 97.178263] #PF: error_code(0x0000) - not-present page
[ 97.183410] PGD 0
[ 97.185438] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 97.189805] CPU: 36 PID: 235 Comm: cpuhp/36 Tainted: G T 6.9.0-rc1 #356
[ 97.197810] Hardware name: Intel Corporation WilsonCity/WilsonCity, BIOS WLYDCRB1.SYS.0021.P06.2104260458 04/26/2021
[ 97.208322] RIP: 0010:__find_nth_andnot_bit+0x66/0x110
[ 97.213472] Code: d2 eb 20 48 29 c6 48 8d 41 c0 48 39 ca 0f 82 8b 00 00 00 48 01 f0 48 83 c1 40 48 39 d0 0f 83 9e 00 00 00 4d 89 d3 49 83 c2 01 <4b> 8b 3c d8 48 f7 d7 4b 23 3c d9 f3 48 0f b8 c7 48 39 c6 73 c4 4c
[ 97.232214] RSP: 0018:ff462f11872bbd98 EFLAGS: 00010202
[ 97.237440] RAX: 0000000000000000 RBX: 0000000000000024 RCX: 0000000000000080
[ 97.244574] RDX: 0000000000000090 RSI: 0000000000000000 RDI: 0000000000000000
[ 97.251705] RBP: ff43a0b049599018 R08: 0000000000000000 R09: ff43a0b049599018
[ 97.258835] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000025
[ 97.265970] R13: 0000000000000090 R14: 0000000000000000 R15: ffffffffaa068fb0
[ 97.273100] FS: 0000000000000000(0000) GS:ff43a0b81d800000(0000) knlGS:0000000000000000
[ 97.281187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 97.286932] CR2: 0000000000000000 CR3: 0000000a50e3a004 CR4: 0000000000771ef0
[ 97.294065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 97.301197] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 97.308328] PKRU: 55555554
[ 97.311039] Call Trace:
[ 97.313492] <TASK>
[ 97.315600] ? __die+0x1f/0x60
[ 97.318665] ? page_fault_oops+0x176/0x5a0
[ 97.322777] ? exc_page_fault+0x7f/0x260
[ 97.326708] ? asm_exc_page_fault+0x22/0x30
[ 97.330903] ? __pfx_resctrl_arch_offline_cpu+0x10/0x10
[ 97.336140] ? __find_nth_andnot_bit+0x66/0x110
[ 97.340680] ? __cancel_work+0x7d/0xc0
[ 97.344438] cpumask_any_housekeeping+0x55/0x110
[ 97.349060] mbm_setup_overflow_handler+0x40/0x70
[ 97.353774] resctrl_offline_cpu+0x101/0x110
[ 97.358054] resctrl_arch_offline_cpu+0x19/0x260
[ 97.362673] cpuhp_invoke_callback+0x156/0x6b0
[ 97.367127] ? cpuhp_thread_fun+0x5f/0x250
[ 97.371234] cpuhp_thread_fun+0x1ca/0x250
[ 97.375247] ? __pfx_smpboot_thread_fn+0x10/0x10
[ 97.379874] smpboot_thread_fn+0x184/0x220
[ 97.383974] kthread+0xe0/0x110
[ 97.387129] ? __pfx_kthread+0x10/0x10
[ 97.390888] ret_from_fork+0x2d/0x50
[ 97.394467] ? __pfx_kthread+0x10/0x10
[ 97.398220] ret_from_fork_asm+0x1a/0x30
[ 97.402149] </TASK>