Re: [RFC PATCH v2 0/7] Defer throttle when task exits to user

Jan Kiszka
2025-04-15T05:29:57Z

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

On 14.04.25 14:04, Aaron Lu wrote:
> Hi Florian,
>
> On Mon, Apr 14, 2025 at 10:54:48AM +0200, Florian Bezdeka wrote:
>> Hi Aaron, Hi Valentin,
>>
>> On Wed, 2025-04-09 at 20:07 +0800, Aaron Lu wrote:
>>> This is a continuous work based on Valentin Schneider's posting here:
>>> Subject: [RFC PATCH v3 00/10] sched/fair: Defer CFS throttle to user entry
>>> https://lore.kernel.org/lkml/20240711130004.2157737-1-vschneid@redhat.com/
>>>
>>> Valentin has described the problem very well in the above link. We also
>>> have task hung problem from time to time in our environment due to cfs quota.
>>> It is mostly visible with rwsem: when a reader is throttled, writer comes in
>>> and has to wait, the writer also makes all subsequent readers wait,
>>> causing problems of priority inversion or even whole system hung.
>>
>> for testing purposes I backported this series to 6.14. We're currently
>> hunting for a sporadic bug with PREEMPT_RT enabled. We see RCU stalls
>> and complete system freezes after a couple of days with some container
>> workload deployed. See [1].
>
> I tried to make a setup last week to reproduce the RT/cfs throttle
> deadlock issue Valentin described but haven't succeeded yet...
>

Attached the bits with which we succeeded, sometimes. Setup: Debian 12,
RT kernel, 2-4 cores VM, 1-5 instances of the test, 2 min - 2 h
patience. As we have to succeed with at least 3 race conditions in a
row, that is still not bad... But maybe someone has an idea how to
increase probabilities further.

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

[-- Attachment #2: epoll-stall.c --]
[-- Type: text/x-csrc, Size: 1173 bytes --]

#include <assert.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/epoll.h>
#include <sys/timerfd.h>

int main(int argc, char *argv[])
{
int pipe, timerfd, epoll;
struct epoll_event ev[2];
struct itimerspec it;
int ret;

assert(argc == 2);
pipe = open(argv[1], O_RDONLY);
assert(pipe >= 0);

timerfd = timerfd_create(CLOCK_MONOTONIC, 0);
assert(timerfd >= 0);
it.it_value.tv_sec = 0;
it.it_value.tv_nsec = 1;
it.it_interval.tv_sec = 0;
it.it_interval.tv_nsec = 50000;
ret = timerfd_settime(timerfd, 0, &it, NULL);
assert(ret == 0);

epoll = epoll_create1(0);
assert(epoll >= 0);

ev[0].events = EPOLLIN;
ev[0].data.fd = pipe;
ret = epoll_ctl(epoll, EPOLL_CTL_ADD, pipe, &ev[0]);
assert(ret == 0);

ev[1].events = EPOLLIN;
ev[1].data.fd = timerfd;
ret = epoll_ctl(epoll, EPOLL_CTL_ADD, timerfd, &ev[1]);
assert(ret == 0);

printf("starting loop\n");
while (1) {
struct epoll_event event;
char buffer[8];
size_t size;

ret = epoll_wait(epoll, &event, 1, -1);
assert(ret == 1);
if (event.data.fd == timerfd)
size = 8;
else
size = 1;
ret = read(event.data.fd, buffer, size);
assert(ret == size);
}
}

[-- Attachment #3: epoll-stall-writer.c --]
[-- Type: text/x-csrc, Size: 302 bytes --]

#include <assert.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
int pipe, ret;

assert(argc == 2);
pipe = open(argv[1], O_WRONLY);
assert(pipe >= 0);

printf("starting writer\n");
while (1) {
ret = write(pipe, "x", 1);
assert(ret == 1);
}
}

[-- Attachment #4: run.sh --]
[-- Type: application/x-shellscript, Size: 384 bytes --]