race leading to held mutexes, inode_cache corruption

race leading to held mutexes, inode_cache corruption

[Sapan Bhatia]

Hi,

We've been trying to investigate a file-system corruption issue in our
kernel (http://svn.planet-lab.org/browser/linux-2.6/trunk) that
manifests itself both with ext3 and ext2. It appears to be happening
to due a contamination of the inode cache (we spent some time
monitoring our systems to arrive at this hypothesis), and can be
reproduced on a vanilla kernel as well.

The race that leads to this issue involves a process being terminated
when it is waiting for a mutex in __mutex_lock_common. eg. when it is
sent a SIGKILL, and the mutex is unlocked, causing the process to be
woken up and sent to exit while now holding the lock.

The way it contaminates the inode_cache slab is that inode->i_mutex is
only initialized once, and assumes that inodes coming back into the
cache are initialized. It seems that in our case such poisoned inodes
were leaking out of pipe.c.

This (www.cs.princeton.edu/~sapanb/mut.c) is the module we used to
test the condition, as follows. Writing to the char device locks a
mutex and reading from it unlocks it.
# echo 1 > /dev/mut
# cat /etc/passwd > /dev/mut &
[2] 6232
# kill -9 6232
# cat /dev/mut
[2]-  Killed                  cat /etc/passwd > /dev/mut
# echo 1 > /dev/mut
(goes to sleep)

I suppose that one could also construct an attack to proactively
corrupt inode_cache, but I haven't tried that as yet.

Our base kernel is 2.6.22.19.

Thank you,
Sapan

Re: race leading to held mutexes, inode_cache corruption

[Andrew Morton]

On Tue, 1 Apr 2008 21:15:52 -0400 "Sapan Bhatia" <sapan.bhatia@gmail.com> wrote:

> Hi,
> 
> We've been trying to investigate a file-system corruption issue in our
> kernel (http://svn.planet-lab.org/browser/linux-2.6/trunk) that
> manifests itself both with ext3 and ext2. It appears to be happening
> to due a contamination of the inode cache (we spent some time
> monitoring our systems to arrive at this hypothesis), and can be
> reproduced on a vanilla kernel as well.
> 
> The race that leads to this issue involves a process being terminated
> when it is waiting for a mutex in __mutex_lock_common. eg. when it is
> sent a SIGKILL, and the mutex is unlocked, causing the process to be
> woken up and sent to exit while now holding the lock.
> 
> The way it contaminates the inode_cache slab is that inode->i_mutex is
> only initialized once, and assumes that inodes coming back into the
> cache are initialized. It seems that in our case such poisoned inodes
> were leaking out of pipe.c.
> 
> This (www.cs.princeton.edu/~sapanb/mut.c) is the module we used to
> test the condition, as follows. Writing to the char device locks a
> mutex and reading from it unlocks it.
> # echo 1 > /dev/mut
> # cat /etc/passwd > /dev/mut &
> [2] 6232
> # kill -9 6232
> # cat /dev/mut
> [2]-  Killed                  cat /etc/passwd > /dev/mut
> # echo 1 > /dev/mut
> (goes to sleep)
> 
> I suppose that one could also construct an attack to proactively
> corrupt inode_cache, but I haven't tried that as yet.
> 
> Our base kernel is 2.6.22.19.

This is ... confusing.

Are you saying that some caller of mutex_lock_interruptible() is getting a
return value of -EINTR from mutex_lock_interruptible(), but this task in
fact _did_ acquire the mutex?

That's the only way in which I can interpret your second paragraph, but as
far as I can tell the code cannot do that.

Can you provide more detail?

Re: race leading to held mutexes, inode_cache corruption

[Andrew Morton]

On Wed, 2 Apr 2008 00:13:04 -0400 "Sapan Bhatia" <sapan.bhatia@gmail.com> wrote:

> >
> >
> > That's the only way in which I can interpret your second paragraph, but as
> > far as I can tell the code cannot do that.
> >
> > Can you provide more detail?
> >
> 
> On running the example again, it seems that attributing the problem to a
> generic locking bug was a misdiagnosis. I apologize for the misinformation.
> The error is more likely a path with a dangling mutex_lock somewhere, or
> something else. I'll investigate further and try to provide a more detailed
> description of the problem when I have something concrete.
> 

OK, thanks.

Recent kernels have this:

config DEBUG_LOCK_ALLOC
	bool "Lock debugging: detect incorrect freeing of live locks"
	depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
	select DEBUG_SPINLOCK
	select DEBUG_MUTEXES
	select LOCKDEP
	help
	 This feature will check whether any held lock (spinlock, rwlock,
	 mutex or rwsem) is incorrectly freed by the kernel, via any of the
	 memory-freeing routines (kfree(), kmem_cache_free(), free_pages(),
	 vfree(), etc.), whether a live lock is incorrectly reinitialized via
	 spin_lock_init()/mutex_init()/etc., or whether there is any lock
	 held during task exit.

which seems rather relevant, no? ;)