Re: [PATCH] mm/x86: Patch out arch_flush_lazy_mmu_mode() when running on bare metal

Re: [PATCH] mm/x86: Patch out arch_flush_lazy_mmu_mode() when running on bare metal

[Boris Ostrovsky]

----- jwboyer@redhat.com wrote:

> On Wed, Mar 13, 2013 at 09:25:44AM -0400, Boris Ostrovsky wrote:
> > On 03/01/2013 07:14 AM, Josh Boyer wrote:
> > >On Thu, Feb 28, 2013 at 04:52:20PM -0800, H. Peter Anvin wrote:
> > >>On 02/28/2013 04:42 PM, Josh Boyer wrote:
> > >>>On Fri, Mar 01, 2013 at 01:36:29AM +0100, Borislav Petkov wrote:
> > >>>>On Thu, Feb 28, 2013 at 04:15:45PM -0800, H. Peter Anvin wrote:
> > >>>>>>I'll try to get someone to test this tomorrow.
> > >>>>Btw, you'd need to apply that other patch too
> > >>>>
> > >>>>http://marc.info/?l=xen-devel&m=136206183814547&w=2
> > >>>>
> > >>>>so that arch_flush_lazy_mmu_mode() has at least one caller on
> x86_64.
> > >>>Yeah, we already have that applied.  It stops crashes in xen
> > >>>environments so we pulled it in as a bugfix.  Thanks though!
> > >>>
> > >>Who are "we"?
> > >Sorry, Fedora.  That patch has a link to a bug in it.  We applied
> the
> > >patch for that bug.  I'll apply Boris' patch on top and get the
> same
> > >people to test it.
> > 
> > Josh, have you had a chance to test this?
> 
> I've tested it on bare metal for a while now.  No problems noticed at
> all.  I've not heard back from Krishna who was testing it in the Xen
> environment.  Krishna?


Any updates?

Thanks.
-boris

Re: [PATCH] mm/x86: Patch out arch_flush_lazy_mmu_mode() when running on bare metal

[Josh Boyer]

On Wed, Mar 20, 2013 at 06:53:55AM -0700, Boris Ostrovsky wrote:
> 
> ----- jwboyer@redhat.com wrote:
> 
> > On Wed, Mar 13, 2013 at 09:25:44AM -0400, Boris Ostrovsky wrote:
> > > On 03/01/2013 07:14 AM, Josh Boyer wrote:
> > > >On Thu, Feb 28, 2013 at 04:52:20PM -0800, H. Peter Anvin wrote:
> > > >>On 02/28/2013 04:42 PM, Josh Boyer wrote:
> > > >>>On Fri, Mar 01, 2013 at 01:36:29AM +0100, Borislav Petkov wrote:
> > > >>>>On Thu, Feb 28, 2013 at 04:15:45PM -0800, H. Peter Anvin wrote:
> > > >>>>>>I'll try to get someone to test this tomorrow.
> > > >>>>Btw, you'd need to apply that other patch too
> > > >>>>
> > > >>>>http://marc.info/?l=xen-devel&m=136206183814547&w=2
> > > >>>>
> > > >>>>so that arch_flush_lazy_mmu_mode() has at least one caller on
> > x86_64.
> > > >>>Yeah, we already have that applied.  It stops crashes in xen
> > > >>>environments so we pulled it in as a bugfix.  Thanks though!
> > > >>>
> > > >>Who are "we"?
> > > >Sorry, Fedora.  That patch has a link to a bug in it.  We applied
> > the
> > > >patch for that bug.  I'll apply Boris' patch on top and get the
> > same
> > > >people to test it.
> > > 
> > > Josh, have you had a chance to test this?
> > 
> > I've tested it on bare metal for a while now.  No problems noticed at
> > all.  I've not heard back from Krishna who was testing it in the Xen
> > environment.  Krishna?
> 
> 
> Any updates?

No.  I've still not heard from Krishna.

At this point I've tested it on bare metal quite a bit, and Konrad has
tested it on both bare metal and Xen.  That should already cover the
case Krishna was going to test anyway.  I suggest we move forward and
take the patch.

josh

Re: [PATCH] mm/x86: Patch out arch_flush_lazy_mmu_mode() when running on bare metal

[Konrad Rzeszutek Wilk]

On Wed, Mar 20, 2013 at 08:08:45PM -0400, Josh Boyer wrote:
> On Wed, Mar 20, 2013 at 06:53:55AM -0700, Boris Ostrovsky wrote:
> > 
> > ----- jwboyer@redhat.com wrote:
> > 
> > > On Wed, Mar 13, 2013 at 09:25:44AM -0400, Boris Ostrovsky wrote:
> > > > On 03/01/2013 07:14 AM, Josh Boyer wrote:
> > > > >On Thu, Feb 28, 2013 at 04:52:20PM -0800, H. Peter Anvin wrote:
> > > > >>On 02/28/2013 04:42 PM, Josh Boyer wrote:
> > > > >>>On Fri, Mar 01, 2013 at 01:36:29AM +0100, Borislav Petkov wrote:
> > > > >>>>On Thu, Feb 28, 2013 at 04:15:45PM -0800, H. Peter Anvin wrote:
> > > > >>>>>>I'll try to get someone to test this tomorrow.
> > > > >>>>Btw, you'd need to apply that other patch too
> > > > >>>>
> > > > >>>>http://marc.info/?l=xen-devel&m=136206183814547&w=2
> > > > >>>>
> > > > >>>>so that arch_flush_lazy_mmu_mode() has at least one caller on
> > > x86_64.
> > > > >>>Yeah, we already have that applied.  It stops crashes in xen
> > > > >>>environments so we pulled it in as a bugfix.  Thanks though!
> > > > >>>
> > > > >>Who are "we"?
> > > > >Sorry, Fedora.  That patch has a link to a bug in it.  We applied
> > > the
> > > > >patch for that bug.  I'll apply Boris' patch on top and get the
> > > same
> > > > >people to test it.
> > > > 
> > > > Josh, have you had a chance to test this?
> > > 
> > > I've tested it on bare metal for a while now.  No problems noticed at
> > > all.  I've not heard back from Krishna who was testing it in the Xen
> > > environment.  Krishna?
> > 
> > 
> > Any updates?
> 
> No.  I've still not heard from Krishna.
> 
> At this point I've tested it on bare metal quite a bit, and Konrad has
> tested it on both bare metal and Xen.  That should already cover the
> case Krishna was going to test anyway.  I suggest we move forward and
> take the patch.

Peter?

Would you like me or Boris to clean up the two patches with the
appropiate Acks and send them to you? 
> 
> josh

Re: [PATCH] mm/x86: Patch out arch_flush_lazy_mmu_mode() when running on bare metal

[H. Peter Anvin]

Sure.

Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

>On Wed, Mar 20, 2013 at 08:08:45PM -0400, Josh Boyer wrote:
>> On Wed, Mar 20, 2013 at 06:53:55AM -0700, Boris Ostrovsky wrote:
>> > 
>> > ----- jwboyer@redhat.com wrote:
>> > 
>> > > On Wed, Mar 13, 2013 at 09:25:44AM -0400, Boris Ostrovsky wrote:
>> > > > On 03/01/2013 07:14 AM, Josh Boyer wrote:
>> > > > >On Thu, Feb 28, 2013 at 04:52:20PM -0800, H. Peter Anvin
>wrote:
>> > > > >>On 02/28/2013 04:42 PM, Josh Boyer wrote:
>> > > > >>>On Fri, Mar 01, 2013 at 01:36:29AM +0100, Borislav Petkov
>wrote:
>> > > > >>>>On Thu, Feb 28, 2013 at 04:15:45PM -0800, H. Peter Anvin
>wrote:
>> > > > >>>>>>I'll try to get someone to test this tomorrow.
>> > > > >>>>Btw, you'd need to apply that other patch too
>> > > > >>>>
>> > > > >>>>http://marc.info/?l=xen-devel&m=136206183814547&w=2
>> > > > >>>>
>> > > > >>>>so that arch_flush_lazy_mmu_mode() has at least one caller
>on
>> > > x86_64.
>> > > > >>>Yeah, we already have that applied.  It stops crashes in xen
>> > > > >>>environments so we pulled it in as a bugfix.  Thanks though!
>> > > > >>>
>> > > > >>Who are "we"?
>> > > > >Sorry, Fedora.  That patch has a link to a bug in it.  We
>applied
>> > > the
>> > > > >patch for that bug.  I'll apply Boris' patch on top and get
>the
>> > > same
>> > > > >people to test it.
>> > > > 
>> > > > Josh, have you had a chance to test this?
>> > > 
>> > > I've tested it on bare metal for a while now.  No problems
>noticed at
>> > > all.  I've not heard back from Krishna who was testing it in the
>Xen
>> > > environment.  Krishna?
>> > 
>> > 
>> > Any updates?
>> 
>> No.  I've still not heard from Krishna.
>> 
>> At this point I've tested it on bare metal quite a bit, and Konrad
>has
>> tested it on both bare metal and Xen.  That should already cover the
>> case Krishna was going to test anyway.  I suggest we move forward and
>> take the patch.
>
>Peter?
>
>Would you like me or Boris to clean up the two patches with the
>appropiate Acks and send them to you? 
>> 
>> josh

-- 
Sent from my mobile phone. Please excuse brevity and lack of formatting.

[PATCH 1/2] x86: mm: Fix vmalloc_fault oops during lazy MMU updates.

[Konrad Rzeszutek Wilk]

From: Samu Kallio <samu.kallio@aberdeencloud.com>

In paravirtualized x86_64 kernels, vmalloc_fault may cause an oops
when lazy MMU updates are enabled, because set_pgd effects are being
deferred.

One instance of this problem is during process mm cleanup with memory
cgroups enabled. The chain of events is as follows:

- zap_pte_range enables lazy MMU updates
- zap_pte_range eventually calls mem_cgroup_charge_statistics,
  which accesses the vmalloc'd mem_cgroup per-cpu stat area
- vmalloc_fault is triggered which tries to sync the corresponding
  PGD entry with set_pgd, but the update is deferred
- vmalloc_fault oopses due to a mismatch in the PUD entries

The OOPs usually looks as so:

------------[ cut here ]------------
kernel BUG at arch/x86/mm/fault.c:396!
invalid opcode: 0000 [#1] SMP
.. snip ..
CPU 1
Pid: 10866, comm: httpd Not tainted 3.6.10-4.fc18.x86_64 #1
RIP: e030:[<ffffffff816271bf>]  [<ffffffff816271bf>] vmalloc_fault+0x11f/0x208
.. snip ..
Call Trace:
 [<ffffffff81627759>] do_page_fault+0x399/0x4b0
 [<ffffffff81004f4c>] ? xen_mc_extend_args+0xec/0x110
 [<ffffffff81624065>] page_fault+0x25/0x30
 [<ffffffff81184d03>] ? mem_cgroup_charge_statistics.isra.13+0x13/0x50
 [<ffffffff81186f78>] __mem_cgroup_uncharge_common+0xd8/0x350
 [<ffffffff8118aac7>] mem_cgroup_uncharge_page+0x57/0x60
 [<ffffffff8115fbc0>] page_remove_rmap+0xe0/0x150
 [<ffffffff8115311a>] ? vm_normal_page+0x1a/0x80
 [<ffffffff81153e61>] unmap_single_vma+0x531/0x870
 [<ffffffff81154962>] unmap_vmas+0x52/0xa0
 [<ffffffff81007442>] ? pte_mfn_to_pfn+0x72/0x100
 [<ffffffff8115c8f8>] exit_mmap+0x98/0x170
 [<ffffffff810050d9>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff81059ce3>] mmput+0x83/0xf0
 [<ffffffff810624c4>] exit_mm+0x104/0x130
 [<ffffffff8106264a>] do_exit+0x15a/0x8c0
 [<ffffffff810630ff>] do_group_exit+0x3f/0xa0
 [<ffffffff81063177>] sys_exit_group+0x17/0x20
 [<ffffffff8162bae9>] system_call_fastpath+0x16/0x1b

Calling arch_flush_lazy_mmu_mode immediately after set_pgd makes the
changes visible to the consistency checks.

CC: stable@vger.kernel.org
RedHat-Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=914737
Tested-by: Josh Boyer <jwboyer@redhat.com>
Reported-and-Tested-by: Krishna Raman <kraman@redhat.com>
Signed-off-by: Samu Kallio <samu.kallio@aberdeencloud.com>
Tested-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/mm/fault.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 2b97525..0e88336 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -378,10 +378,12 @@ static noinline __kprobes int vmalloc_fault(unsigned long address)
 	if (pgd_none(*pgd_ref))
 		return -1;
 
-	if (pgd_none(*pgd))
+	if (pgd_none(*pgd)) {
 		set_pgd(pgd, *pgd_ref);
-	else
+		arch_flush_lazy_mmu_mode();
+	} else {
 		BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
+	}
 
 	/*
 	 * Below here mismatches are bugs because these lower tables
-- 
1.8.0.2

[tip:x86/urgent] x86, mm, paravirt: Fix vmalloc_fault oops during lazy MMU updates

[tip-bot for Samu Kallio]

Commit-ID:  1160c2779b826c6f5c08e5cc542de58fd1f667d5
Gitweb:     http://git.kernel.org/tip/1160c2779b826c6f5c08e5cc542de58fd1f667d5
Author:     Samu Kallio <samu.kallio@aberdeencloud.com>
AuthorDate: Sat, 23 Mar 2013 09:36:35 -0400
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Wed, 10 Apr 2013 11:25:07 -0700

x86, mm, paravirt: Fix vmalloc_fault oops during lazy MMU updates

In paravirtualized x86_64 kernels, vmalloc_fault may cause an oops
when lazy MMU updates are enabled, because set_pgd effects are being
deferred.

One instance of this problem is during process mm cleanup with memory
cgroups enabled. The chain of events is as follows:

- zap_pte_range enables lazy MMU updates
- zap_pte_range eventually calls mem_cgroup_charge_statistics,
  which accesses the vmalloc'd mem_cgroup per-cpu stat area
- vmalloc_fault is triggered which tries to sync the corresponding
  PGD entry with set_pgd, but the update is deferred
- vmalloc_fault oopses due to a mismatch in the PUD entries

The OOPs usually looks as so:

------------[ cut here ]------------
kernel BUG at arch/x86/mm/fault.c:396!
invalid opcode: 0000 [#1] SMP
.. snip ..
CPU 1
Pid: 10866, comm: httpd Not tainted 3.6.10-4.fc18.x86_64 #1
RIP: e030:[<ffffffff816271bf>]  [<ffffffff816271bf>] vmalloc_fault+0x11f/0x208
.. snip ..
Call Trace:
 [<ffffffff81627759>] do_page_fault+0x399/0x4b0
 [<ffffffff81004f4c>] ? xen_mc_extend_args+0xec/0x110
 [<ffffffff81624065>] page_fault+0x25/0x30
 [<ffffffff81184d03>] ? mem_cgroup_charge_statistics.isra.13+0x13/0x50
 [<ffffffff81186f78>] __mem_cgroup_uncharge_common+0xd8/0x350
 [<ffffffff8118aac7>] mem_cgroup_uncharge_page+0x57/0x60
 [<ffffffff8115fbc0>] page_remove_rmap+0xe0/0x150
 [<ffffffff8115311a>] ? vm_normal_page+0x1a/0x80
 [<ffffffff81153e61>] unmap_single_vma+0x531/0x870
 [<ffffffff81154962>] unmap_vmas+0x52/0xa0
 [<ffffffff81007442>] ? pte_mfn_to_pfn+0x72/0x100
 [<ffffffff8115c8f8>] exit_mmap+0x98/0x170
 [<ffffffff810050d9>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff81059ce3>] mmput+0x83/0xf0
 [<ffffffff810624c4>] exit_mm+0x104/0x130
 [<ffffffff8106264a>] do_exit+0x15a/0x8c0
 [<ffffffff810630ff>] do_group_exit+0x3f/0xa0
 [<ffffffff81063177>] sys_exit_group+0x17/0x20
 [<ffffffff8162bae9>] system_call_fastpath+0x16/0x1b

Calling arch_flush_lazy_mmu_mode immediately after set_pgd makes the
changes visible to the consistency checks.

Cc: <stable@vger.kernel.org>
RedHat-Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=914737
Tested-by: Josh Boyer <jwboyer@redhat.com>
Reported-and-Tested-by: Krishna Raman <kraman@redhat.com>
Signed-off-by: Samu Kallio <samu.kallio@aberdeencloud.com>
Link: http://lkml.kernel.org/r/1364045796-10720-1-git-send-email-konrad.wilk@oracle.com
Tested-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
---
 arch/x86/mm/fault.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 2b97525..0e88336 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -378,10 +378,12 @@ static noinline __kprobes int vmalloc_fault(unsigned long address)
 	if (pgd_none(*pgd_ref))
 		return -1;
 
-	if (pgd_none(*pgd))
+	if (pgd_none(*pgd)) {
 		set_pgd(pgd, *pgd_ref);
-	else
+		arch_flush_lazy_mmu_mode();
+	} else {
 		BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
+	}
 
 	/*
 	 * Below here mismatches are bugs because these lower tables

Re: [PATCH] mm/x86: Patch out arch_flush_lazy_mmu_mode() when running on bare metal

[Boris Ostrovsky]

On 03/01/2013 07:14 AM, Josh Boyer wrote:
> On Thu, Feb 28, 2013 at 04:52:20PM -0800, H. Peter Anvin wrote:
>> On 02/28/2013 04:42 PM, Josh Boyer wrote:
>>> On Fri, Mar 01, 2013 at 01:36:29AM +0100, Borislav Petkov wrote:
>>>> On Thu, Feb 28, 2013 at 04:15:45PM -0800, H. Peter Anvin wrote:
>>>>>> I'll try to get someone to test this tomorrow.
>>>> Btw, you'd need to apply that other patch too
>>>>
>>>> http://marc.info/?l=xen-devel&m=136206183814547&w=2
>>>>
>>>> so that arch_flush_lazy_mmu_mode() has at least one caller on x86_64.
>>> Yeah, we already have that applied.  It stops crashes in xen
>>> environments so we pulled it in as a bugfix.  Thanks though!
>>>
>> Who are "we"?
> Sorry, Fedora.  That patch has a link to a bug in it.  We applied the
> patch for that bug.  I'll apply Boris' patch on top and get the same
> people to test it.

Josh, have you had a chance to test this?

-boris

Re: [PATCH] mm/x86: Patch out arch_flush_lazy_mmu_mode() when running on bare metal

[Josh Boyer]

On Wed, Mar 13, 2013 at 09:25:44AM -0400, Boris Ostrovsky wrote:
> On 03/01/2013 07:14 AM, Josh Boyer wrote:
> >On Thu, Feb 28, 2013 at 04:52:20PM -0800, H. Peter Anvin wrote:
> >>On 02/28/2013 04:42 PM, Josh Boyer wrote:
> >>>On Fri, Mar 01, 2013 at 01:36:29AM +0100, Borislav Petkov wrote:
> >>>>On Thu, Feb 28, 2013 at 04:15:45PM -0800, H. Peter Anvin wrote:
> >>>>>>I'll try to get someone to test this tomorrow.
> >>>>Btw, you'd need to apply that other patch too
> >>>>
> >>>>http://marc.info/?l=xen-devel&m=136206183814547&w=2
> >>>>
> >>>>so that arch_flush_lazy_mmu_mode() has at least one caller on x86_64.
> >>>Yeah, we already have that applied.  It stops crashes in xen
> >>>environments so we pulled it in as a bugfix.  Thanks though!
> >>>
> >>Who are "we"?
> >Sorry, Fedora.  That patch has a link to a bug in it.  We applied the
> >patch for that bug.  I'll apply Boris' patch on top and get the same
> >people to test it.
> 
> Josh, have you had a chance to test this?

I've tested it on bare metal for a while now.  No problems noticed at
all.  I've not heard back from Krishna who was testing it in the Xen
environment.  Krishna?

josh

Re: [PATCH] mm/x86: Patch out arch_flush_lazy_mmu_mode() when running on bare metal

[Konrad Rzeszutek Wilk]

On Wed, Mar 13, 2013 at 09:44:43AM -0400, Josh Boyer wrote:
> On Wed, Mar 13, 2013 at 09:25:44AM -0400, Boris Ostrovsky wrote:
> > On 03/01/2013 07:14 AM, Josh Boyer wrote:
> > >On Thu, Feb 28, 2013 at 04:52:20PM -0800, H. Peter Anvin wrote:
> > >>On 02/28/2013 04:42 PM, Josh Boyer wrote:
> > >>>On Fri, Mar 01, 2013 at 01:36:29AM +0100, Borislav Petkov wrote:
> > >>>>On Thu, Feb 28, 2013 at 04:15:45PM -0800, H. Peter Anvin wrote:
> > >>>>>>I'll try to get someone to test this tomorrow.
> > >>>>Btw, you'd need to apply that other patch too
> > >>>>
> > >>>>http://marc.info/?l=xen-devel&m=136206183814547&w=2
> > >>>>
> > >>>>so that arch_flush_lazy_mmu_mode() has at least one caller on x86_64.
> > >>>Yeah, we already have that applied.  It stops crashes in xen
> > >>>environments so we pulled it in as a bugfix.  Thanks though!
> > >>>
> > >>Who are "we"?
> > >Sorry, Fedora.  That patch has a link to a bug in it.  We applied the
> > >patch for that bug.  I'll apply Boris' patch on top and get the same
> > >people to test it.
> > 
> > Josh, have you had a chance to test this?
> 
> I've tested it on bare metal for a while now.  No problems noticed at
> all.  I've not heard back from Krishna who was testing it in the Xen
> environment.  Krishna?

Roping in Stephan from Canonical as they are seeing it on a regular basis
and is affecting their distro.

Stephan - there are two patches for it - the paravirt of lazy_mmu mode
and then the other one.

This thread should provide the right URLs (I hope). Could you (if it is possible)
try them out and see if they fix the issue?

> 
> josh