Re: 3.9-rc1 NULL pointer crash at find_pid_ns

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Thu, Mar 07, 2013 at 01:14:10PM -0500, Sasha Levin wrote:
> On 03/07/2013 01:05 PM, ebiederm@xmission.com wrote:
> > Sasha Levin <sasha.levin@oracle.com> writes:
> > 
> >> On 03/07/2013 12:46 PM, Eric Dumazet wrote:
> >>> On Thu, 2013-03-07 at 12:36 -0500, Sasha Levin wrote:
> >>>
> >>>> Looks like the hlist change is probably the issue, though it specifically
> >>>> uses:
> >>>>
> >>>> 	#define hlist_entry_safe(ptr, type, member) \
> >>>>         	(ptr) ? hlist_entry(ptr, type, member) : NULL
> >>>>
> >>>> I'm still looking at the code in question and it's assembly, but I can't
> >>>> figure out what's going wrong. I was also trying to see what's so special
> >>>> about this loop in find_pid_ns as opposed to the rest of the kernel code
> >>>> that uses hlist_for_each_entry_rcu() but couldn't find out why.
> >>>>
> >>>> Is it somehow possible that if we rcu_dereference_raw() the same thing twice
> >>>> inside the same rcu_read_lock() section we'll get different results? That's
> >>>> really the only reason for this crash that comes to mind at the moment, very
> >>>> unlikely - but that's all I have right now.
> >>>>
> >>>
> >>> Yep
> >>>
> >>> #define hlist_entry_safe(ptr, type, member) \
> >>> 	(ptr) ? hlist_entry(ptr, type, member) : NULL
> >>>
> >>> Is not safe, as ptr can be evaluated twice, and thats not good at all...
> >>
> >> ptr is being evaluated twice, but in this case this is an
> >> rcu_dereference_raw() value within the same rcu_read_lock() section.
> >>
> >> Is it still problematic?
> > 
> > Definitely.
> > 
> > Head in this instance the expression: &pid_hash[pid_hashfn(nr, ns)]
> > 
> > And the crash clearly shows that when hilst_entry is being evaluated the
> > HEAD is NULL.
> 
> Okay, I'm even more confused now.
> 
> The expression in question is:
> 
> 	hlist_entry_safe(rcu_dereference_bh(hlist_first_rcu(head)))
> 
> You're saying that "rcu_dereference_bh(hlist_first_rcu(head))" can change between
> the two evaluations we do. That would mean that 'head' has changed in between, right?
> 
> In that case, the list itself has changed - which means that RCU has changed the
> list underneath us.
> 
> hlist_first_rcu() doesn't have any side-effects, it doesn't modify the list whatsoever,
> so the only thing that can change is 'head'. Why is it allowed to change if the list
> is protected by RCU?

RCU does not prevent the list from changing.  Instead, it prevents anything
that was in the list from being freed during a given RCU read-side critical
section.  Here is how it is supposed to happen:

	head---->A

Task 1 picks up the pointer from head to A, and sees that it is non-NULL.

Task 2 removes A from the list, so that the pointer from head is now NULL:

	head     A
	  |
	  |
	  V
        NULL

Now task 1 refetches from head, and is fatally disappointed to get a
NULL pointer.

Now, had task 1 avoided the refetch, it would be still working with
a pointer to A.  Since A won't be freed until the end of an RCU grace
period, all would have been well.  Again, one way to handle this is
as follows:

#define hlist_entry_safe(ptr, type, member) \
	({ typeof(ptr) ____ptr = (ptr); \
	   ____ptr ? hlist_entry(____ptr, type, member) : NULL; \
	})

This way, "ptr" is executed exactly once, and the check and the
hlist_entry() are both using the same value.

							Thanx, Paul