linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Jason Gunthorpe <jgg@ziepe.ca>
To: Jerome Glisse <jglisse@redhat.com>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
	linux-rdma@vger.kernel.org, Leon Romanovsky <leonro@mellanox.com>,
	Doug Ledford <dledford@redhat.com>,
	Artemy Kovalyov <artemyko@mellanox.com>,
	Moni Shoua <monis@mellanox.com>,
	Mike Marciniszyn <mike.marciniszyn@intel.com>,
	Kaike Wan <kaike.wan@intel.com>,
	Dennis Dalessandro <dennis.dalessandro@intel.com>
Subject: Re: [PATCH v4 0/1] Use HMM for ODP v4
Date: Wed, 22 May 2019 19:39:06 -0300	[thread overview]
Message-ID: <20190522223906.GA15389@ziepe.ca> (raw)
In-Reply-To: <20190522220419.GB20179@redhat.com>

On Wed, May 22, 2019 at 06:04:20PM -0400, Jerome Glisse wrote:
> On Wed, May 22, 2019 at 05:12:47PM -0300, Jason Gunthorpe wrote:
> > On Wed, May 22, 2019 at 01:48:52PM -0400, Jerome Glisse wrote:
> > 
> > >  static void put_per_mm(struct ib_umem_odp *umem_odp)
> > >  {
> > >  	struct ib_ucontext_per_mm *per_mm = umem_odp->per_mm;
> > > @@ -325,9 +283,10 @@ static void put_per_mm(struct ib_umem_odp *umem_odp)
> > >  	up_write(&per_mm->umem_rwsem);
> > >  
> > >  	WARN_ON(!RB_EMPTY_ROOT(&per_mm->umem_tree.rb_root));
> > > -	mmu_notifier_unregister_no_release(&per_mm->mn, per_mm->mm);
> > > +	hmm_mirror_unregister(&per_mm->mirror);
> > >  	put_pid(per_mm->tgid);
> > > -	mmu_notifier_call_srcu(&per_mm->rcu, free_per_mm);
> > > +
> > > +	kfree(per_mm);
> > 
> > Notice that mmu_notifier only uses SRCU to fence in-progress ops
> > callbacks, so I think hmm internally has the bug that this ODP
> > approach prevents.
> > 
> > hmm should follow the same pattern ODP has and 'kfree_srcu' the hmm
> > struct, use container_of in the mmu_notifier callbacks, and use the
> > otherwise vestigal kref_get_unless_zero() to bail:
> > 
> > From 0cb536dc0150ba964a1d655151d7b7a84d0f915a Mon Sep 17 00:00:00 2001
> > From: Jason Gunthorpe <jgg@mellanox.com>
> > Date: Wed, 22 May 2019 16:52:52 -0300
> > Subject: [PATCH] hmm: Fix use after free with struct hmm in the mmu notifiers
> > 
> > mmu_notifier_unregister_no_release() is not a fence and the mmu_notifier
> > system will continue to reference hmm->mn until the srcu grace period
> > expires.
> > 
> >          CPU0                                     CPU1
> >                                                __mmu_notifier_invalidate_range_start()
> >                                                  srcu_read_lock
> >                                                  hlist_for_each ()
> >                                                    // mn == hmm->mn
> > hmm_mirror_unregister()
> >   hmm_put()
> >     hmm_free()
> >       mmu_notifier_unregister_no_release()
> >          hlist_del_init_rcu(hmm-mn->list)
> > 			                           mn->ops->invalidate_range_start(mn, range);
> > 					             mm_get_hmm()
> >       mm->hmm = NULL;
> >       kfree(hmm)
> >                                                      mutex_lock(&hmm->lock);
> > 
> > Use SRCU to kfree the hmm memory so that the notifiers can rely on hmm
> > existing. Get the now-safe hmm struct through container_of and directly
> > check kref_get_unless_zero to lock it against free.
> 
> It is already badly handled with BUG_ON()

You can't crash the kernel because userspace forced a race, and no it
isn't handled today because there is no RCU locking in mm_get_hmm nor
is there a kfree_rcu for the struct hmm to make the
kref_get_unless_zero work without use-after-free.

> i just need to convert those to return and to use
> mmu_notifier_call_srcu() to free hmm struct.

Isn't that what this patch does?

> The way race is avoided is because mm->hmm will either be NULL or
> point to another hmm struct before an existing hmm is free. 

There is no locking on mm->hmm so it is useless to prevent races.

> Also if range_start/range_end use kref_get_unless_zero() but right
> now this is BUG_ON if it turn out to be NULL, it should just return
> on NULL.

Still needs rcu.

Also the container_of is necessary to avoid some race where you could
be doing:

                  CPU0                                     CPU1                         CPU2
                                                       hlist_for_each ()
       mmu_notifier_unregister_no_release(hmm1)             
       spin_lock(&mm->page_table_lock);                                
       mm->hmm = NULL
       spin_unlock(&mm->page_table_lock);                                                                                      
                                                      				 hmm2 = hmm_get_or_create()
                                                        mn == hmm1->mn
                                                        mn->ops->invalidate_range_start(mn, range)
							  mm_get_mm() == hmm2
                                                      hist_for_each con't
                                                        mn == hmm2->mn
                                                        mn->ops->invalidate_range_start(mn, range)
							  mm_get_mm() == hmm2

Now we called the same notifier twice on hmm2. Ooops.

There is no reason to risk this confusion just to avoid container_of.

So we agree this patch is necessary? Can you test it an ack it please?

Jason


  reply	other threads:[~2019-05-22 22:39 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20190411181314.19465-1-jglisse@redhat.com>
     [not found] ` <20190506195657.GA30261@ziepe.ca>
     [not found]   ` <20190521205321.GC3331@redhat.com>
     [not found]     ` <20190522005225.GA30819@ziepe.ca>
     [not found]       ` <20190522174852.GA23038@redhat.com>
2019-05-22 19:22         ` [PATCH v4 0/1] Use HMM for ODP v4 Jason Gunthorpe
2019-05-22 21:49           ` Jerome Glisse
2019-05-22 22:43             ` Jason Gunthorpe
2019-05-22 20:12         ` Jason Gunthorpe
2019-05-22 21:12           ` Ralph Campbell
2019-05-22 22:06             ` Jerome Glisse
2019-05-22 22:04           ` Jerome Glisse
2019-05-22 22:39             ` Jason Gunthorpe [this message]
2019-05-22 22:42               ` Jerome Glisse
2019-05-22 22:52                 ` Jason Gunthorpe
     [not found]         ` <20190522235737.GD15389@ziepe.ca>
     [not found]           ` <20190523150432.GA5104@redhat.com>
     [not found]             ` <20190523154149.GB12159@ziepe.ca>
     [not found]               ` <20190523155207.GC5104@redhat.com>
     [not found]                 ` <20190523163429.GC12159@ziepe.ca>
     [not found]                   ` <20190523173302.GD5104@redhat.com>
     [not found]                     ` <20190523175546.GE12159@ziepe.ca>
     [not found]                       ` <20190523182458.GA3571@redhat.com>
     [not found]                         ` <20190523191038.GG12159@ziepe.ca>
2019-05-24  6:40                           ` Christoph Hellwig
2019-05-24 12:44                             ` RFC: Run a dedicated hmm.git for 5.3 Jason Gunthorpe
2019-05-24 16:27                               ` Daniel Vetter
2019-05-24 16:53                                 ` Jason Gunthorpe
2019-05-24 16:59                                   ` Daniel Vetter
2019-05-25 22:52                               ` Andrew Morton
2019-05-27 19:12                                 ` Jason Gunthorpe
2019-06-06 15:25                                   ` Jason Gunthorpe
2019-06-06 19:53                                     ` Stephen Rothwell

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190522223906.GA15389@ziepe.ca \
    --to=jgg@ziepe.ca \
    --cc=artemyko@mellanox.com \
    --cc=dennis.dalessandro@intel.com \
    --cc=dledford@redhat.com \
    --cc=jglisse@redhat.com \
    --cc=kaike.wan@intel.com \
    --cc=leonro@mellanox.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mike.marciniszyn@intel.com \
    --cc=monis@mellanox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).