Re: io_uring failure on parisc (32-bit userspace and 64-bit kernel)

Helge Deller
2023-02-13T22:05:26Z

On 2/13/23 22:05, Jens Axboe wrote:
> On 2/13/23 1:59?PM, Helge Deller wrote:
>>> Yep sounds like it. What's the caching architecture of parisc?
>>
>> parisc is Virtually Indexed, Physically Tagged (VIPT).
>
> That's what I assumed, so virtual aliasing is what we're dealing with
> here.
>
>> Thanks for the patch!
>> Sadly it doesn't fix the problem, as the kernel still sees
>> ctx->rings->sq.tail as being 0.
>> Interestingly it worked once (not reproduceable) directly after bootup,
>> which indicates that we at least look at the right address from kernel side.
>>
>> So, still needs more debugging/testing.
>
> It's not like this is untested stuff, so yeah it'll generally be
> correct, it just seems that parisc is a bit odd in that the virtual
> aliasing occurs between the kernel and userspace addresses too. At least
> that's what it seems like.

True.

> But I wonder if what needs flushing is the user side, not the kernel
> side? Either that, or my patch is not flushing the right thing on the
> kernel side.
>
> Is it possible to flush it from the userspace side? Presumable that's
> what we'd need on the sqe side, and then the kernel side for the cqe
> filling. So probably the patch is half-way correct :-)

I hacked up in __io_uring_flush_sq() in liburing/src/queue.c this code
(which I hope is correct):
if (!(ring->flags & IORING_SETUP_SQPOLL))
IO_URING_WRITE_ONCE(*sq->ktail, tail);
else
io_uring_smp_store_release(sq->ktail, tail);
} /* ADDED: */
{ int i; unsigned long p = (unsigned long)sq->ktail & ~(4096-1);
fprintf(stderr, "FLUSH CACHE OF PAGE %lx\n", p);
for (i=0; i < 4096; i += 8)
asm volatile("fdc 0(%0)" : : "r" (p+i));
}

The kernel sometimes sees the tail value now (it fails afterwards, but that's ok for now).
But I'm not sure yet if this is really the effect of the fdc (flush data cache instruction),
or pure luck because the aliasing of the userspace address and kernel address matches in
a sucessful run.
For me it seems as it's the aliasing which makes it work sometimes.

In this regard I wonder why we don't provide the cacheflush syscall on parisc....

Helge