negative dentry_stat.nr_unused causes aggressive dcache pruning

negative dentry_stat.nr_unused causes aggressive dcache pruning

[Steve Lord]

I have seen this stat go negative (just from booting up a multi cpu box),
and looking at the code, it is manipulated without locking in a number
of places. I have only seen this in real life on a 2.4 kernel, but 2.6
also looks vulnerable.

In 2.4 this can cause shrink_dcache_memory to attempt to push the whole
dcache out by calling prune_dcache with a negative parameter, prune_dcache
just keeps going until count hits zero or the dentry_unused list
empties.

In 2.6 kernels I do not see a real use of the variable which would cause harm,
although is someone was to start relying on more than -1 coming back from
shrink_dcache_memory there might be problems.

The problem code is dput which is the only place it is manipulated
without the dcache lock.

nr_dentry is not mp safe either, but no one depends on that.

Steve

Re: negative dentry_stat.nr_unused causes aggressive dcache pruning

[Greg Banks]

On Wed, Dec 08, 2004 at 04:16:52PM -0600, Steve Lord wrote:
> 
> I have seen this stat go negative (just from booting up a multi cpu box),
> and looking at the code, it is manipulated without locking in a number
> of places. I have only seen this in real life on a 2.4 kernel, but 2.6
> also looks vulnerable.

On early 2.6.x, a heavy streaming NFS load was a great way to trigger
this.  I haven't seen it happen since

http://linus.bkbits.net:8080/linux-2.5/cset@40b8cf606MV-gl6VpDyWKzzW1jaIJw

Greg.
-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.

Re: negative dentry_stat.nr_unused causes aggressive dcache pruning

[Andrew Morton]

Steve Lord <lord@xfs.org> wrote:
>
> I have seen this stat go negative (just from booting up a multi cpu box),
>  and looking at the code, it is manipulated without locking in a number
>  of places. I have only seen this in real life on a 2.4 kernel, but 2.6
>  also looks vulnerable.

In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered
by dcache_lock.  I just double-checked and all seems well.

Re: negative dentry_stat.nr_unused causes aggressive dcache pruning

[Steve Lord]

Andrew Morton wrote:
> Steve Lord <lord@xfs.org> wrote:
> 
>>I have seen this stat go negative (just from booting up a multi cpu box),
>> and looking at the code, it is manipulated without locking in a number
>> of places. I have only seen this in real life on a 2.4 kernel, but 2.6
>> also looks vulnerable.
> 
> 
> In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered
> by dcache_lock.  I just double-checked and all seems well.
> 

Looked again, you are right, looks like a 2.4 only problem and even there I
am darned if I can see how it leaks now, still digging as to where it is coming
from. I have a box which goes negative shortly after bootup, this seems to put
it in a feedback loop and  it just keeps heading south from there pushing on the
dcache. This is running on a fedora legacy kernel for redhat 9.0 which I added
kdb to. Starting to wonder if the compiler screwed up somewhere. I have an
external fiberchannel driver loaded, but it does not have any dentry related
code in it.

Steve

Re: negative dentry_stat.nr_unused causes aggressive dcache pruning

[Steve Lord]

Andrew Morton wrote:
> Steve Lord <lord@xfs.org> wrote:
> 
>>I have seen this stat go negative (just from booting up a multi cpu box),
>> and looking at the code, it is manipulated without locking in a number
>> of places. I have only seen this in real life on a 2.4 kernel, but 2.6
>> also looks vulnerable.
> 
> 
> In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered
> by dcache_lock.  I just double-checked and all seems well.
> 

I still do not know exactly how the count gets negative, but I tracked it
down to a user space app from emulex called HBAanywhere. The only thing I
can see this doing which might be related is attempting to open a lot of
non-existant /proc entries:

	/proc/scsi//120
	/proc/scsi//121
	etc...

Yes there is a // in there.

I ran with a BUG call if we manipulate nr_unused without the dcache lock
and it never tripped. All very wierd.

Steve

Re: negative dentry_stat.nr_unused causes aggressive dcache pruning

[Andrew Morton]

Steve Lord <lord@xfs.org> wrote:
>
> Andrew Morton wrote:
> > Steve Lord <lord@xfs.org> wrote:
> > 
> >>I have seen this stat go negative (just from booting up a multi cpu box),
> >> and looking at the code, it is manipulated without locking in a number
> >> of places. I have only seen this in real life on a 2.4 kernel, but 2.6
> >> also looks vulnerable.
> > 
> > 
> > In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered
> > by dcache_lock.  I just double-checked and all seems well.
> > 
> 
> I still do not know exactly how the count gets negative, but I tracked it
> down to a user space app from emulex called HBAanywhere. The only thing I
> can see this doing which might be related is attempting to open a lot of
> non-existant /proc entries:
> 
> 	/proc/scsi//120
> 	/proc/scsi//121
> 	etc...
> 
> Yes there is a // in there.
> 
> I ran with a BUG call if we manipulate nr_unused without the dcache lock
> and it never tripped. All very wierd.
> 

Is that 2.4 or 2.6?

I'd be expecting a systematic counting bug.  After all, nr_unused would
normally be in the thousands and it'd take a lot of races to get that down
to zero.

Re: negative dentry_stat.nr_unused causes aggressive dcache pruning

[Steve Lord]

Andrew Morton wrote:
> Steve Lord <lord@xfs.org> wrote:

>>
>>I still do not know exactly how the count gets negative, but I tracked it
>>down to a user space app from emulex called HBAanywhere. The only thing I
>>can see this doing which might be related is attempting to open a lot of
>>non-existant /proc entries:
>>
>>	/proc/scsi//120
>>	/proc/scsi//121
>>	etc...
>>
>>Yes there is a // in there.
>>
>>I ran with a BUG call if we manipulate nr_unused without the dcache lock
>>and it never tripped. All very wierd.
>>
> 
> 
> Is that 2.4 or 2.6?
> 
> I'd be expecting a systematic counting bug.  After all, nr_unused would
> normally be in the thousands and it'd take a lot of races to get that down
> to zero.
> 

  2.4.20-31.9 (aka Yea Olde Kernel), it is a redhat 9.0 update from the
  fedora legacy project.

Something is pushing down hard on the dcache, I can watch the numbers
drop after boot up. They start out a few hundred but within a few
minutes they go negative.

I have a fiber channel driver from out the tree loaded (emulex lpfcdd)
but it does not appear to mess with dcache entries itself. Nothing else
from outside the tree. I have not had a chance to try 2.6 yet, not
even sure this code would run on 2.6.

I don't have the source of this app, so all I can do is strace it and
watch that. Actually I don't even need the app, so I can make the problem
go away, but it seems very odd that a user space program can do this.

If I start up the program and watch /proc/sys/fs/dentry-state every second
or so I see a sequence something like this:
slord@k4> cat /proc/sys/fs/dentry-state
368     22      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
364     18      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
363     16      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
361     14      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
361     14      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
359     13      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
359     13      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
359     13      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
359     13      45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
348     5       45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
332     -16     45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
332     -16     45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
332     -16     45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
332     -16     45      0       0       0
slord@k4> cat /proc/sys/fs/dentry-state
332     -16     45      0       0       0

Its not like I am low on memory though:
slord@k4> free
              total       used       free     shared    buffers     cached
Mem:       1030108     244932     785176          0      15012     147808
-/+ buffers/cache:      82112     947996
Swap:      1574360         76    1574284

All I can see the program doing is attempting to open these non-existent
/proc entries and doing an ioctl FIBMAP on a /dev/ device bound to the fiber
channel driver. Of course, FIBMAP is not FIBMAP in this driver.

It looks like a steaming pile in there, and I think I am just going to
assume the driver is guilty until proven innocent.


Steve