Problems with wait queues

Problems with wait queues

[Nicolas Bougues]

Dear all,

I'm writing a device driver for a PCI board that sends/receives one
frame (80 bytes) of data every 100th/sec. The kernel is 2.4.18 non SMP.

I use the timer task, and every 100th/sec, I transmit any pending
buffer and store the received one, so that it can be handled by the
user mode app.

The driver is accessible using a standard device node.

I implemented a wait queue mechanism in order to deal with blocking
read/writes from user mode. When the dev_write sees the TX queue is
full, it wait_event_interruptible(txq, condition). Idem for RX.

And every 100th/sec, when it gets more data for RX or more room for
TX, it wake_up_interruptible(txq|rxq).

My problem is that I wrote a test app that loops around read() or
write() on my device in user mode, and when running this app, my
loadavg increases, up to 1.00, which is surprising.

But, in the meantime :
- vmstat shows no user mode nor system activity. 
- time of the user mode test app shows 0.00 for both user and
system time.
- a busy task (such as dd if=/dev/zero of=/dev/null bs=1024k
  count=300000) takes the same amount of time to run when the test app
  is running, as it took when the system was idle.

If I insert an usleep(1000*100), which will wait about 1/100th sec,
in the loop around read()/write(), the wait queues aren't used anymore
in the driver (since there is always room for TX or data for RX), and
the loadavg stays at 0.00, which is what is expected.

Does anybody have any idea on what I may have done wrong, and why
would loadavg increase when vmstat show no activity ?

Thanks.
--
Nicolas Bougues

Re: Problems with wait queues

[Andries Brouwer]

On Wed, Jun 26, 2002 at 12:32:43PM +0200, Nicolas Bougues wrote:

> Does anybody have any idea on what I may have done wrong, and why
> would loadavg increase when vmstat show no activity ?

loadavg does not report what you think it reports

Re: Problems with wait queues

[Nicolas Bougues]

On Wed, Jun 26, 2002 at 12:52:41PM +0200, Andries Brouwer wrote:
> On Wed, Jun 26, 2002 at 12:32:43PM +0200, Nicolas Bougues wrote:
> 
> > Does anybody have any idea on what I may have done wrong, and why
> > would loadavg increase when vmstat show no activity ?
> 
> loadavg does not report what you think it reports
> 

As far as I understand, loadavg reports the average number of
processes in the TASK_RUNNING state.

What happens in my driver, I believe, is that :
- on timer interrupt, I do some stuff, and wake_up the waiting process
- then the loadavg is computed (seeing my waiting task as TASK_RUNNING)
- then the scheduler runs the task
- then the task goes immediatly back to sleep

>From this point of view, then my problem is just "cosmetic". Isn't
there a way to do things in a different order, so that I could still
get a meaningful(*) loadavg ?

(*): by meaningful, I mean representing the number of busy processes
at a random point in time.
--
Nicolas Bougues

Re: Problems with wait queues

[Richard B. Johnson]

On Wed, 26 Jun 2002, Nicolas Bougues wrote:

> On Wed, Jun 26, 2002 at 12:52:41PM +0200, Andries Brouwer wrote:
> > On Wed, Jun 26, 2002 at 12:32:43PM +0200, Nicolas Bougues wrote:
> > 
> > > Does anybody have any idea on what I may have done wrong, and why
> > > would loadavg increase when vmstat show no activity ?
> > 
> > loadavg does not report what you think it reports
> > 
> 
> As far as I understand, loadavg reports the average number of
> processes in the TASK_RUNNING state.
> 
> What happens in my driver, I believe, is that :
> - on timer interrupt, I do some stuff, and wake_up the waiting process
> - then the loadavg is computed (seeing my waiting task as TASK_RUNNING)
> - then the scheduler runs the task
> - then the task goes immediatly back to sleep
> 
> >From this point of view, then my problem is just "cosmetic". Isn't
> there a way to do things in a different order, so that I could still
> get a meaningful(*) loadavg ?
> 
> (*): by meaningful, I mean representing the number of busy processes
> at a random point in time.
> --
> Nicolas Bougues
> 

I am sure that you can have things look correct as well as run
properly. However, you didn't show us the code. You need to
do something like:

            interruptible_sleep_on(&semaphore);

while your wake-up occurs with:

            wake_up_interruptible(&semaphore);

Or, you can do (the hard way) :

             while(!something)
             {
                 current->policy |= SCHED_YIELD;
                 schedule();
             }

Both ways (and others) will look fine with `top` and will sleep
properly.


Cheers,
Dick Johnson

Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).

                 Windows-2000/Professional isn't.

Re: Problems with wait queues

[Nicolas Bougues]

Richard,

On Wed, Jun 26, 2002 at 10:17:45AM -0400, Richard B. Johnson wrote:
> 
> I am sure that you can have things look correct as well as run
> properly. However, you didn't show us the code.

It's quite large, and I didn't think I was "syntactically" wrong. But
the code is available, if somebody wants to have a look.

> You need to do something like:
> 
>             interruptible_sleep_on(&semaphore);
> 
> while your wake-up occurs with:
> 
>             wake_up_interruptible(&semaphore);
> 

That's what I do, although I use the wait_event_interruptible macro
instead of interruptible_sleep_on.

>
> Both ways (and others) will look fine with `top` and will sleep
> properly.
> 

If we're talking about %CPU times, right. If we're talking loadavg,
no. As I said in my previous message, I think it's because my
wake_up_interruptible() is *always* triggered during the timer
interrupt, just before the scheduler runs (new data is available every
100th/sec).

I'm not sure, but I think that the loadavg is computed at the
beginning of each scheduler run, thus, my task always looks "running"
at this time, even though it just runs for a few microseconds each
time.

--
Nicolas Bougues

Re: Problems with wait queues

[David Woodhouse]

For the benefit of any newcomers to the list who haven't clocked RBJ yet...

root@chaos.analogic.com said:
>  I am sure that you can have things look correct as well as run
> properly. However, you didn't show us the code. You need to do
> something like:

>             interruptible_sleep_on(&semaphore);

> while your wake-up occurs with:

>             wake_up_interruptible(&semaphore);

Do not ever use sleep_on() and friends. Almost all usage of these 
functions will be buggy.


--
dwmw2

Re: Problems with wait queues

[Richard B. Johnson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 999 bytes --]

On Wed, 26 Jun 2002, David Woodhouse wrote:

> 
> For the benefit of any newcomers to the list who haven't clocked RBJ yet...
> 
> root@chaos.analogic.com said:
> >  I am sure that you can have things look correct as well as run
> > properly. However, you didn't show us the code. You need to do
> > something like:
> 
> >             interruptible_sleep_on(&semaphore);
> 
> > while your wake-up occurs with:
> 
> >             wake_up_interruptible(&semaphore);
> 
> Do not ever use sleep_on() and friends. Almost all usage of these 
> functions will be buggy.
> 

Whatever your message means; perhaps it was a way of saying "Hello"
to an old friend?

Attached is a file showing about 485 usages of 'sleep_on' in the
kernel drivers. If this usage is, as you say, buggy then will you
please inform us unwashed hordes what we should use to replace these?


Cheers,
Dick Johnson

Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).

                 Windows-2000/Professional isn't.

[-- Attachment #2: Type: APPLICATION/octet-stream, Size: 3804 bytes --]

Re: Problems with wait queues

[Oliver.Neukum]

> > Do not ever use sleep_on() and friends. Almost all usage of these
> > functions will be buggy.
> >
>
> Whatever your message means; perhaps it was a way of saying "Hello"
> to an old friend?
>
> Attached is a file showing about 485 usages of 'sleep_on' in the
> kernel drivers. If this usage is, as you say, buggy then will you
> please inform us unwashed hordes what we should use to replace these?

The simplest way is with the wait_event[_interruptible] macros.
In newer versions of the USB drivers, you'll find examples.
You may also set your task's state and put yourself on a waitqueue,
before you do something that will cause you to be woken up.


And yes, most examples you found are buggy, as they can miss
a wakeup.
To use sleep_on safely you must make sure that you cannot be woken up
before you begin to sleep.

	Regards
		Oliver

Re: Problems with wait queues

[David Woodhouse]

On Thu, 27 Jun 2002, Richard B. Johnson wrote:

> Attached is a file showing about 485 usages of 'sleep_on' in the
> kernel drivers. If this usage is, as you say, buggy then will you
> please inform us unwashed hordes what we should use to replace these?

Anything you like -- as long as it doesn't have gaping races which are 
obvious to anyone with half a clue who actually _thinks_ about the code 
rather than assuming that _driver_ authors all got it right :)

We _really_ need to kill sleep_on() in 2.5. 

-- 
dwmw2