Re: [PATCH 0/5] padata: fix liftime issues after ->serial() has completed

[Nicolai Stange <nstange@suse.de>]

Hi Daniel,

Daniel Jordan <daniel.m.jordan@oracle.com> writes:

> On Wed, Oct 19, 2022 at 10:37:03AM +0200, Nicolai Stange wrote:
>> this series is supposed to fix some lifetime issues all related to the fact that
>> once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well
>> with its right to tear down the associated padata_shell or parallel_data
>> instance respectively.
>> 
>> Only the first one, addressed by patch [2/5], has actually been observed, namely
>> on a (downstream) RT kernel under a very specific workload involving LTP's
>> pcrypt_aead01. On non-RT, I've been unable to reproduce.
>
> I haven't been able to hit the issue in 2/5 on RT on a v6.0 kernel in an
> x86 vm.  Were there any other things running on the system besides
> pcrypt_aead01?  More details about your environment and your kernel
> config would be helpful.

Right, the issue is indeed hard to reproduce, unfortunately. It has
originally been reported internally by our QA Maintenance team, which --
for unknown reason -- suddenly started to hit the issue once every while
in their testing environment. I did manage to reproduce it once or twice
myself, but it took me several days running pcrypt_aead01 in a loop each
time. AFAIR, I allocated a single cpu to the VM only and increased the
priority of pcrypt_aead01 a bit, with the intent to make preemption of
the ->serial() worker by DELALG more likely. But really, I cannot tell
if that did in fact contribute to the likelihood of triggering the race
or whether I've just been lucky.

Also, as mentioned in the cover letter, the RT kernel this has been
observed on is a downstream one, based on 5.3.18 (source tree at [1],
config at [2]), but with quite some additional patches on top. A
backport of this patch series here had been subject to testing in the
same environment the issue originally showed up in on a fairly regular
basis and no new crashes have been observed since.

Let me know if I could provide you with any more details.

Thanks,

Nicolai

[1] https://github.com/SUSE/kernel/tree/SLE15-SP3-RT
[2] https://github.com/SUSE/kernel-source/blob/SLE15-SP3-RT/config/x86_64/rt

-- 
SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany
GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
(HRB 36809, AG Nürnberg)