Re: Regression in io_uring, leading to data corruption

[Jens Axboe <axboe@kernel.dk>]

On 11/8/23 3:15 PM, Timothy Pearson wrote:
> 
> 
> ----- Original Message -----
>> From: "Jens Axboe" <axboe@kernel.dk>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>> Cc: "regressions" <regressions@lists.linux.dev>, "Pavel Begunkov" <asml.silence@gmail.com>
>> Sent: Wednesday, November 8, 2023 4:05:50 PM
>> Subject: Re: Regression in io_uring, leading to data corruption
> 
>> On 11/8/23 12:06 PM, Jens Axboe wrote:
>>>> I have been giving some thought to the CPU pinning of the workers, and
>>>> one thing that may have been overlooked is that this could potentially
>>>> force-serialize worker operations.  Did you just have to pin the io
>>>> workers or did the workqueue also need to be pinned for the corruption
>>>> to disappear?
>>>
>>> Not sure I follow, the workers ARE the workqueue. For the pinning, I
>>> just made sure that the workers are on the same CPU. I honestly don't
>>> remember all the details there outside of what I can read back from the
>>> emails I sent, it's been a while. My suspicion back then was that it was
>>> some weird ppc cache aliasing effect with the copy into kernel memory
>>> happening on cpu X, and then we immediately punt it to cpu Y for
>>> processing.
>>
>> You could try something like this, though I still need to verify that we
>> never end up running it on the wrong CPU. But may be worth a shot, for
>> debug purposes.
> 
> Getting a bunch of:
> 
> [   25.268005] work cpu 5, me 23
> [   25.269217] work cpu 5, me 23
> [   25.269804] work cpu 5, me 23

[snip]

> [  153.029120] work cpu 32, me 10
> 
> They happen both for corrupt and non-corrupt runs.

OK, let me actually test this thing and see if I can make it solid
first...

-- 
Jens Axboe