Re: ext3 leaking buffer_heads

[Don Porter <porterde@cs.utexas.edu>]

Thanks for the tip, Eric.

I spent some time tracing through this in a debugger.

I believe the lingering buffer_heads are allocated by 
journal_write_commit_record()->journal_get_descriptor_buffer()->__getblk().  
The reference count ends up at zero, but the buffer head is never freed.

As best I can tell, this is correct as long as the associated page is 
cached, so that the associated buffer_head can be looked up and reused 
later.

I plan to look into this more, but perhaps the issue is just that the 
kernel should be more aggressive about freeing cached journal pages?  
They are highly unlikely to be used again once a committed transaction 
is completely out on disk, as the journal is roughly an append-only log, 
right?

Thanks again,
Don

Eric Sandeen wrote:
> Don Porter wrote:
>   
>> Hi,
>>
>> It appears that the ext3 journal code has a slow leak of buffer_head 
>> structs.  Try this simple script:
>>
>> perl -e 'while(1){ `sync`; }'
>>
>> and monitor the count of allocated buffer_head structs in 
>> /proc/slabinfo, and it seems to increase without bound.  Even after this 
>> script is killed and the machine is left idle for several minutes, the 
>> count of buffer heads doesn't substantially decrease.
>>
>> Looking around at various machines I have access to, the count of 
>> allocated buffer_heads roughly correlates with uptime when using ext3.  
>> This is a slow leak - one would likely have to run this script for a day 
>> or more to drain enough lowmem to cause problems.
>>
>> Other info:  I have only tried this on x86 machines, but I have tried 
>> both 2.6.22.6 and 2.6.28.8, and both have the problem.  I am running 
>> Ubuntu 7.10 on top of these kernels, but the kernels were built directly 
>> from kernel.org tarballs.
>>
>> Any advice or help with this issue is greatly appreciated. 
>>     
>
> Without investigating too far yet, I did try this, and did indeed see
> the buffer_head usage go up while the script runs.
>
> However, if I did:
>
> # echo 3 > /proc/sys/vm/drop_caches
>
> before the script, and noted the total nr. of buffer heads in use, and
> then did it again after the script had been running a while, I got back
> to the same (low) count of buffer heads in use.   So I don't think this
> is a leak as in "the system has lost all accounting of these buffer
> heads" at least... but it'd be interesting to know what the reason for
> the increase is, I'm not sure offhand.
>
> -Eric
>
>
>