filp usage when cpu busy

filp usage when cpu busy

[bc Wong (chimwong)]

Hi,

With 2.6.16 x86_64 on a 4 core machine, I noticed
that the filp usage (according to /proc/slabinfo)
shoots up and keeps on increasing sharply when one
of the CPUs is (1) locked up, or (2) very busy
doing a lot of printk()'s with KERN_EMERG.

In the case of (1), it's permanent until it runs
out of memory eventually. For (2), it's temporary;
filp count comes back down when the printk()'s are
done.

I can't think of any relationship between a busy/
locked-up CPU and filp count. The system is still
functional. New short-lived processes kept being
created, but the overall number of processes is
stable.

Does anyone know why filp count would go up like
that?

Thanks,
bc

Re: filp usage when cpu busy

[Nick Piggin]

On Thursday 01 November 2007 12:56, bc Wong (chimwong) wrote:
> Hi,
>
> With 2.6.16 x86_64 on a 4 core machine, I noticed
> that the filp usage (according to /proc/slabinfo)
> shoots up and keeps on increasing sharply when one
> of the CPUs is (1) locked up, or (2) very busy
> doing a lot of printk()'s with KERN_EMERG.
>
> In the case of (1), it's permanent until it runs
> out of memory eventually. For (2), it's temporary;
> filp count comes back down when the printk()'s are
> done.
>
> I can't think of any relationship between a busy/
> locked-up CPU and filp count. The system is still
> functional. New short-lived processes kept being
> created, but the overall number of processes is
> stable.
>
> Does anyone know why filp count would go up like
> that?

Yeah, it's probably because filp structures are freed by
RCU, and if you have a locked up CPU then it can't go
through a quiescent state so RCU stops freeing your filps.

If you add some cond_resched()s to your code, you should
find that RCU will force a reschedule and things will work
(actually, for 2.6.16, I'm not sure if RCU had the code to
force a reschedule... it's force_quiescent_state() in
kernel/rcupdate.c upstream).

Re: filp usage when cpu busy

[bc Wong]

Nick Piggin wrote:
> On Thursday 01 November 2007 12:56, bc Wong (chimwong) wrote:
>> Hi,
>>
>> With 2.6.16 x86_64 on a 4 core machine, I noticed
>> that the filp usage (according to /proc/slabinfo)
>> shoots up and keeps on increasing sharply when one
>> of the CPUs is (1) locked up, or (2) very busy
>> doing a lot of printk()'s with KERN_EMERG.
>>
>> In the case of (1), it's permanent until it runs
>> out of memory eventually. For (2), it's temporary;
>> filp count comes back down when the printk()'s are
>> done.
>>
>> I can't think of any relationship between a busy/
>> locked-up CPU and filp count. The system is still
>> functional. New short-lived processes kept being
>> created, but the overall number of processes is
>> stable.
>>
>> Does anyone know why filp count would go up like
>> that?
> 
> Yeah, it's probably because filp structures are freed by
> RCU, and if you have a locked up CPU then it can't go
> through a quiescent state so RCU stops freeing your filps.
> 
> If you add some cond_resched()s to your code, you should
> find that RCU will force a reschedule and things will work
> (actually, for 2.6.16, I'm not sure if RCU had the code to
> force a reschedule... it's force_quiescent_state() in
> kernel/rcupdate.c upstream).

Thanks! You're absolutely right. Btw, 2.6.16 does
have force_quiescent_state().

Cheers,
bc