ext3 leaking buffer_heads

ext3 leaking buffer_heads

[Don Porter]

Hi,

It appears that the ext3 journal code has a slow leak of buffer_head 
structs.  Try this simple script:

perl -e 'while(1){ `sync`; }'

and monitor the count of allocated buffer_head structs in 
/proc/slabinfo, and it seems to increase without bound.  Even after this 
script is killed and the machine is left idle for several minutes, the 
count of buffer heads doesn't substantially decrease.

Looking around at various machines I have access to, the count of 
allocated buffer_heads roughly correlates with uptime when using ext3.  
This is a slow leak - one would likely have to run this script for a day 
or more to drain enough lowmem to cause problems.

Other info:  I have only tried this on x86 machines, but I have tried 
both 2.6.22.6 and 2.6.28.8, and both have the problem.  I am running 
Ubuntu 7.10 on top of these kernels, but the kernels were built directly 
from kernel.org tarballs.

Any advice or help with this issue is greatly appreciated. 

Thanks,
Don

Re: ext3 leaking buffer_heads

[Eric Sandeen]

Don Porter wrote:
> Hi,
> 
> It appears that the ext3 journal code has a slow leak of buffer_head 
> structs.  Try this simple script:
> 
> perl -e 'while(1){ `sync`; }'
> 
> and monitor the count of allocated buffer_head structs in 
> /proc/slabinfo, and it seems to increase without bound.  Even after this 
> script is killed and the machine is left idle for several minutes, the 
> count of buffer heads doesn't substantially decrease.
> 
> Looking around at various machines I have access to, the count of 
> allocated buffer_heads roughly correlates with uptime when using ext3.  
> This is a slow leak - one would likely have to run this script for a day 
> or more to drain enough lowmem to cause problems.
> 
> Other info:  I have only tried this on x86 machines, but I have tried 
> both 2.6.22.6 and 2.6.28.8, and both have the problem.  I am running 
> Ubuntu 7.10 on top of these kernels, but the kernels were built directly 
> from kernel.org tarballs.
> 
> Any advice or help with this issue is greatly appreciated. 

Without investigating too far yet, I did try this, and did indeed see
the buffer_head usage go up while the script runs.

However, if I did:

# echo 3 > /proc/sys/vm/drop_caches

before the script, and noted the total nr. of buffer heads in use, and
then did it again after the script had been running a while, I got back
to the same (low) count of buffer heads in use.   So I don't think this
is a leak as in "the system has lost all accounting of these buffer
heads" at least... but it'd be interesting to know what the reason for
the increase is, I'm not sure offhand.

-Eric

Re: ext3 leaking buffer_heads

[Don Porter]

Thanks for the tip, Eric.

I spent some time tracing through this in a debugger.

I believe the lingering buffer_heads are allocated by 
journal_write_commit_record()->journal_get_descriptor_buffer()->__getblk().  
The reference count ends up at zero, but the buffer head is never freed.

As best I can tell, this is correct as long as the associated page is 
cached, so that the associated buffer_head can be looked up and reused 
later.

I plan to look into this more, but perhaps the issue is just that the 
kernel should be more aggressive about freeing cached journal pages?  
They are highly unlikely to be used again once a committed transaction 
is completely out on disk, as the journal is roughly an append-only log, 
right?

Thanks again,
Don

Eric Sandeen wrote:
> Don Porter wrote:
>   
>> Hi,
>>
>> It appears that the ext3 journal code has a slow leak of buffer_head 
>> structs.  Try this simple script:
>>
>> perl -e 'while(1){ `sync`; }'
>>
>> and monitor the count of allocated buffer_head structs in 
>> /proc/slabinfo, and it seems to increase without bound.  Even after this 
>> script is killed and the machine is left idle for several minutes, the 
>> count of buffer heads doesn't substantially decrease.
>>
>> Looking around at various machines I have access to, the count of 
>> allocated buffer_heads roughly correlates with uptime when using ext3.  
>> This is a slow leak - one would likely have to run this script for a day 
>> or more to drain enough lowmem to cause problems.
>>
>> Other info:  I have only tried this on x86 machines, but I have tried 
>> both 2.6.22.6 and 2.6.28.8, and both have the problem.  I am running 
>> Ubuntu 7.10 on top of these kernels, but the kernels were built directly 
>> from kernel.org tarballs.
>>
>> Any advice or help with this issue is greatly appreciated. 
>>     
>
> Without investigating too far yet, I did try this, and did indeed see
> the buffer_head usage go up while the script runs.
>
> However, if I did:
>
> # echo 3 > /proc/sys/vm/drop_caches
>
> before the script, and noted the total nr. of buffer heads in use, and
> then did it again after the script had been running a while, I got back
> to the same (low) count of buffer heads in use.   So I don't think this
> is a leak as in "the system has lost all accounting of these buffer
> heads" at least... but it'd be interesting to know what the reason for
> the increase is, I'm not sure offhand.
>
> -Eric
>
>
>   

Re: ext3 leaking buffer_heads

[Theodore Tso]

On Tue, Mar 24, 2009 at 05:43:10PM -0500, Don Porter wrote:
> Thanks for the tip, Eric.
>
> I spent some time tracing through this in a debugger.
>
> I believe the lingering buffer_heads are allocated by  
> journal_write_commit_record()->journal_get_descriptor_buffer()->__getblk(). 
>  The reference count ends up at zero, but the buffer head is never freed.
>
> As best I can tell, this is correct as long as the associated page is  
> cached, so that the associated buffer_head can be looked up and reused  
> later.
>
> I plan to look into this more, but perhaps the issue is just that the  
> kernel should be more aggressive about freeing cached journal pages?   
> They are highly unlikely to be used again once a committed transaction  
> is completely out on disk, as the journal is roughly an append-only log,  
> right?

The journal is a fixed sized, circular log, so the buffer_head will be
reused eventually.  In the case of memory pressure the unreferenced
buffer_head will be freed, but since we never reference the contents
of the journal, we could be more aggressive about freeing the
buffer_head just to avoid pushing out more valuable memory contents
when we start getting put under memory pressure.

					- Ted

Re: ext3 leaking buffer_heads

[Don Porter]

Theodore Tso wrote:
> The journal is a fixed sized, circular log, so the buffer_head will be
> reused eventually.  In the case of memory pressure the unreferenced
> buffer_head will be freed, but since we never reference the contents
> of the journal, we could be more aggressive about freeing the
> buffer_head just to avoid pushing out more valuable memory contents
> when we start getting put under memory pressure.
>
> 					- Ted
>   

Thanks Ted.  This makes sense.

I am working on a research project that may require more aggressive 
reclaiming of these buffer_heads.  If I were to implement something, 
would this be an optimization the maintainers would be interested in?

Thanks again for the help,

Don