Re: stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)

Re: stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)

[Guillaume Tucker]

On 03/01/18 03:11, kernelci.org bot wrote:
> stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)
> 
> Full Boot Summary: https://kernelci.org/boot/all/job/stable/branch/linux-4.14.y/kernel/v4.14.11/
> Full Build Summary: https://kernelci.org/build/stable/branch/linux-4.14.y/kernel/v4.14.11/
> 
> Tree: stable
> Branch: linux-4.14.y
> Git Describe: v4.14.11
> Git Commit: 0d59679df5b53755c00ea0292df696f97bfc950d
> Git URL: http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> Tested: 59 unique boards, 23 SoC families, 16 builds out of 185
> 
> Boot Regressions Detected:
> 
> x86:
> 
>      x86_64_defconfig:
>          qemu:
>              lab-mhart: new failure (last pass: v4.14.10)
> 
> Conflicting Boot Failure Detected: (These likely are not failures as other labs are reporting PASS. Needs review.)
> 
> x86:
> 
>      x86_64_defconfig:
>          qemu:
>              lab-mhart: FAIL
>              lab-collabora: PASS

Well, it turns out this is not exactly a conflict as there's a
subtle difference between the qemu devices in lab-mhart and in
lab-collabora.  The ones in lab-collabora are configured to use
KVM, and it looks like the ones in lab-mhart aren't.

So this job with KVM enabled passes in lab-collabora:

   https://lava.collabora.co.uk/scheduler/job/1032358

but it fails if I tell LAVA (qemu) to disable KVM:

   https://lava.collabora.co.uk/scheduler/job/1032359

with the same panic as in lab-mhart.  It seems like it's failing
to return from an interrupt:

   http://lava.streamtester.net/scheduler/job/87308

   [    2.678828]  ? native_iret+0x7/0x7
   [    2.679208] WARNING: can't dereference iret registers at 00000000ffc66068 for ip page_fault+0x11/0x60

This triggered an automated bisection on kernelci.org, please see
the results below.

I may run another bisection with this config enabled earlier in
the history to track down the actual change in the code that
introduced the issue, let me know if it's worth doing.

Hope this helps!

Best wishes,
Guillaume

--8<-----------------------------------------------------------------------8<--

Bisection result for stable/linux-4.14.y (v4.14.11) on qemu

Good known revision:

     b8ce823 Linux 4.14.10

Bad known revision:

     0d59679 Linux 4.14.11

Extra parameters:

     Tree:      stable
     URL:       http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
     Branch:    linux-4.14.y
     Target:    qemu
     Lab:       lab-mhart
     Defconfig: x86_64_defconfig
     Plan:      boot


Breaking commit found:

-------------------------------------------------------------------------------
commit 3dfd9fd8d897214b1a880c7fd8ed36b88faa1c02
Author: Dave Hansen <dave.hansen@linux.intel.com>
Date:   Mon Dec 4 15:08:03 2017 +0100

     x86/mm/pti: Add Kconfig
     
     commit 385ce0ea4c078517fa51c261882c4e72fba53005 upstream.
     
     Finally allow CONFIG_PAGE_TABLE_ISOLATION to be enabled.
     
     PARAVIRT generally requires that the kernel not manage its own page tables.
     It also means that the hypervisor and kernel must agree wholeheartedly
     about what format the page tables are in and what they contain.
     PAGE_TABLE_ISOLATION, unfortunately, changes the rules and they
     can not be used together.
     
     I've seen conflicting feedback from maintainers lately about whether they
     want the Kconfig magic to go first or last in a patch series.  It's going
     last here because the partially-applied series leads to kernels that can
     not boot in a bunch of cases.  I did a run through the entire series with
     CONFIG_PAGE_TABLE_ISOLATION=y to look for build errors, though.
     
     [ tglx: Removed SMP and !PARAVIRT dependencies as they not longer exist ]
     
     Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
     Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
     Cc: Andy Lutomirski <luto@kernel.org>
     Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
     Cc: Borislav Petkov <bp@alien8.de>
     Cc: Brian Gerst <brgerst@gmail.com>
     Cc: David Laight <David.Laight@aculab.com>
     Cc: Denys Vlasenko <dvlasenk@redhat.com>
     Cc: Eduardo Valentin <eduval@amazon.com>
     Cc: Greg KH <gregkh@linuxfoundation.org>
     Cc: H. Peter Anvin <hpa@zytor.com>
     Cc: Josh Poimboeuf <jpoimboe@redhat.com>
     Cc: Juergen Gross <jgross@suse.com>
     Cc: Linus Torvalds <torvalds@linux-foundation.org>
     Cc: Peter Zijlstra <peterz@infradead.org>
     Cc: Will Deacon <will.deacon@arm.com>
     Cc: aliguori@amazon.com
     Cc: daniel.gruss@iaik.tugraz.at
     Cc: hughd@google.com
     Cc: keescook@google.com
     Cc: linux-mm@kvack.org
     Signed-off-by: Ingo Molnar <mingo@kernel.org>
     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

diff --git a/security/Kconfig b/security/Kconfig
index e8e4494..6614b93 100644
--- a/security/Kconfig
+++ b/security/Kconfig
@@ -54,6 +54,17 @@ config SECURITY_NETWORK
  	  implement socket and networking access controls.
  	  If you are unsure how to answer this question, answer N.
  
+config PAGE_TABLE_ISOLATION
+	bool "Remove the kernel mapping in user mode"
+	depends on X86_64 && !UML
+	default y
+	help
+	  This feature reduces the number of hardware side channels by
+	  ensuring that the majority of kernel addresses are not mapped
+	  into userspace.
+
+	  See Documentation/x86/pagetable-isolation.txt for more details.
+
  config SECURITY_INFINIBAND
  	bool "Infiniband Security Hooks"
  	depends on SECURITY && INFINIBAND
-------------------------------------------------------------------------------


Git bisection log:

-------------------------------------------------------------------------------
git bisect start
# good: [b8ce8232fcc37fe7a97db79ea0a5f32098c25e72] Linux 4.14.10
git bisect good b8ce8232fcc37fe7a97db79ea0a5f32098c25e72
# bad: [0d59679df5b53755c00ea0292df696f97bfc950d] Linux 4.14.11
git bisect bad 0d59679df5b53755c00ea0292df696f97bfc950d
# bad: [621b5ae0f9f4f9ef91bf441afc086ecf5e752d51] s390/qeth: apply takeover changes when mode is toggled
git bisect bad 621b5ae0f9f4f9ef91bf441afc086ecf5e752d51
# bad: [66f833dbed02d39c44440b6b35ac088655c32edb] ring-buffer: Mask out the info bits when returning buffer page length
git bisect bad 66f833dbed02d39c44440b6b35ac088655c32edb
# good: [d230c1917f57c3beee2e0204a4c8c58999758b95] x86/mm/pti: Map ESPFIX into user space
git bisect good d230c1917f57c3beee2e0204a4c8c58999758b95
# good: [36a72ab52c8d969a7a302082f52731c1be0e9ada] x86/mm: Optimize RESTORE_CR3
git bisect good 36a72ab52c8d969a7a302082f52731c1be0e9ada
# bad: [3dfd9fd8d897214b1a880c7fd8ed36b88faa1c02] x86/mm/pti: Add Kconfig
git bisect bad 3dfd9fd8d897214b1a880c7fd8ed36b88faa1c02
# good: [ef4b38472d6b1bf587554dfc7d5ab7abc835c1a5] x86/mm: Clarify the whole ASID/kernel PCID/user PCID naming
git bisect good ef4b38472d6b1bf587554dfc7d5ab7abc835c1a5
# good: [33d9d7836f0fa02777667d72bc815c12fbe61cac] x86/dumpstack: Indicate in Oops whether PTI is configured and enabled
git bisect good 33d9d7836f0fa02777667d72bc815c12fbe61cac
# first bad commit: [3dfd9fd8d897214b1a880c7fd8ed36b88faa1c02] x86/mm/pti: Add Kconfig
-------------------------------------------------------------------------------


> ---
> For more info write to <info@kernelci.org>
> 
> 
> 
> _______________________________________________
> Kernel-build-reports mailing list
> Kernel-build-reports@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/kernel-build-reports

Re: stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)

[Thomas Gleixner]

On Wed, 3 Jan 2018, Guillaume Tucker wrote:

> On 03/01/18 03:11, kernelci.org bot wrote:
> > stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict
> > (v4.14.11)
> > 
> > Full Boot Summary:
> > https://kernelci.org/boot/all/job/stable/branch/linux-4.14.y/kernel/v4.14.11/
> > Full Build Summary:
> > https://kernelci.org/build/stable/branch/linux-4.14.y/kernel/v4.14.11/
> > 
> > Tree: stable
> > Branch: linux-4.14.y
> > Git Describe: v4.14.11
> > Git Commit: 0d59679df5b53755c00ea0292df696f97bfc950d
> > Git URL:
> > http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> > Tested: 59 unique boards, 23 SoC families, 16 builds out of 185
> > 
> > Boot Regressions Detected:
> > 
> > x86:
> > 
> >      x86_64_defconfig:
> >          qemu:
> >              lab-mhart: new failure (last pass: v4.14.10)
> > 
> > Conflicting Boot Failure Detected: (These likely are not failures as other
> > labs are reporting PASS. Needs review.)
> > 
> > x86:
> > 
> >      x86_64_defconfig:
> >          qemu:
> >              lab-mhart: FAIL
> >              lab-collabora: PASS
> 
> Well, it turns out this is not exactly a conflict as there's a
> subtle difference between the qemu devices in lab-mhart and in
> lab-collabora.  The ones in lab-collabora are configured to use
> KVM, and it looks like the ones in lab-mhart aren't.
> 
> So this job with KVM enabled passes in lab-collabora:
> 
>   https://lava.collabora.co.uk/scheduler/job/1032358
> 
> but it fails if I tell LAVA (qemu) to disable KVM:
> 
>   https://lava.collabora.co.uk/scheduler/job/1032359
> 
> with the same panic as in lab-mhart.  It seems like it's failing
> to return from an interrupt:
> 
>   http://lava.streamtester.net/scheduler/job/87308
> 
>   [    2.678828]  ? native_iret+0x7/0x7
>   [    2.679208] WARNING: can't dereference iret registers at 00000000ffc66068
> for ip page_fault+0x11/0x60
> 
> This triggered an automated bisection on kernelci.org, please see
> the results below.
> 
> I may run another bisection with this config enabled earlier in
> the history to track down the actual change in the code that
> introduced the issue, let me know if it's worth doing.

No, because before that commit not all pieces are in place.

Can you please try the failing kernel with pti=off on the command line?

I'll start a test instance here as well.

Thanks,

	tglx

Re: stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)

[Guillaume Tucker]

On 03/01/18 09:48, Thomas Gleixner wrote:
> On Wed, 3 Jan 2018, Guillaume Tucker wrote:
> 
>> On 03/01/18 03:11, kernelci.org bot wrote:
>>> stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict
>>> (v4.14.11)
>>>
>>> Full Boot Summary:
>>> https://kernelci.org/boot/all/job/stable/branch/linux-4.14.y/kernel/v4.14.11/
>>> Full Build Summary:
>>> https://kernelci.org/build/stable/branch/linux-4.14.y/kernel/v4.14.11/
>>>
>>> Tree: stable
>>> Branch: linux-4.14.y
>>> Git Describe: v4.14.11
>>> Git Commit: 0d59679df5b53755c00ea0292df696f97bfc950d
>>> Git URL:
>>> http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
>>> Tested: 59 unique boards, 23 SoC families, 16 builds out of 185
>>>
>>> Boot Regressions Detected:
>>>
>>> x86:
>>>
>>>       x86_64_defconfig:
>>>           qemu:
>>>               lab-mhart: new failure (last pass: v4.14.10)
>>>
>>> Conflicting Boot Failure Detected: (These likely are not failures as other
>>> labs are reporting PASS. Needs review.)
>>>
>>> x86:
>>>
>>>       x86_64_defconfig:
>>>           qemu:
>>>               lab-mhart: FAIL
>>>               lab-collabora: PASS
>>
>> Well, it turns out this is not exactly a conflict as there's a
>> subtle difference between the qemu devices in lab-mhart and in
>> lab-collabora.  The ones in lab-collabora are configured to use
>> KVM, and it looks like the ones in lab-mhart aren't.
>>
>> So this job with KVM enabled passes in lab-collabora:
>>
>>    https://lava.collabora.co.uk/scheduler/job/1032358
>>
>> but it fails if I tell LAVA (qemu) to disable KVM:
>>
>>    https://lava.collabora.co.uk/scheduler/job/1032359
>>
>> with the same panic as in lab-mhart.  It seems like it's failing
>> to return from an interrupt:
>>
>>    http://lava.streamtester.net/scheduler/job/87308
>>
>>    [    2.678828]  ? native_iret+0x7/0x7
>>    [    2.679208] WARNING: can't dereference iret registers at 00000000ffc66068
>> for ip page_fault+0x11/0x60
>>
>> This triggered an automated bisection on kernelci.org, please see
>> the results below.
>>
>> I may run another bisection with this config enabled earlier in
>> the history to track down the actual change in the code that
>> introduced the issue, let me know if it's worth doing.
> 
> No, because before that commit not all pieces are in place.
> 
> Can you please try the failing kernel with pti=off on the command line?

It does boot with pti=off (and KVM disabled):

   https://lava.collabora.co.uk/scheduler/job/1032387

> I'll start a test instance here as well.
> 
> Thanks,
> 
> 	tglx

Re: stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)

[Thomas Gleixner]

On Wed, 3 Jan 2018, Guillaume Tucker wrote:
> On 03/01/18 09:48, Thomas Gleixner wrote:
> > > Well, it turns out this is not exactly a conflict as there's a
> > > subtle difference between the qemu devices in lab-mhart and in
> > > lab-collabora.  The ones in lab-collabora are configured to use
> > > KVM, and it looks like the ones in lab-mhart aren't.
> > > 
> > > So this job with KVM enabled passes in lab-collabora:
> > > 
> > >    https://lava.collabora.co.uk/scheduler/job/1032358
> > > 
> > > but it fails if I tell LAVA (qemu) to disable KVM:
> > > 
> > >    https://lava.collabora.co.uk/scheduler/job/1032359
> > > 
> > > with the same panic as in lab-mhart.  It seems like it's failing
> > > to return from an interrupt:
> > > 
> > >    http://lava.streamtester.net/scheduler/job/87308
> > > 
> > >    [    2.678828]  ? native_iret+0x7/0x7
> > >    [    2.679208] WARNING: can't dereference iret registers at
> > > 00000000ffc66068
> > > for ip page_fault+0x11/0x60
> > > 
> > > This triggered an automated bisection on kernelci.org, please see
> > > the results below.
> > > 
> > > I may run another bisection with this config enabled earlier in
> > > the history to track down the actual change in the code that
> > > introduced the issue, let me know if it's worth doing.
> > 
> > No, because before that commit not all pieces are in place.
> > 
> > Can you please try the failing kernel with pti=off on the command line?
> 
> It does boot with pti=off (and KVM disabled):
> 
>   https://lava.collabora.co.uk/scheduler/job/1032387

So it's a qemu issue. Added qemu folks on Cc.

> > I'll start a test instance here as well.
 
Thanks,

	tglx

Re: stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)

[Paolo Bonzini]

On 03/01/2018 11:36, Thomas Gleixner wrote:
> On Wed, 3 Jan 2018, Guillaume Tucker wrote:
>> On 03/01/18 09:48, Thomas Gleixner wrote:
>>>> Well, it turns out this is not exactly a conflict as there's a
>>>> subtle difference between the qemu devices in lab-mhart and in
>>>> lab-collabora.  The ones in lab-collabora are configured to use
>>>> KVM, and it looks like the ones in lab-mhart aren't.
>>>>
>>>> So this job with KVM enabled passes in lab-collabora:
>>>>
>>>>    https://lava.collabora.co.uk/scheduler/job/1032358
>>>>
>>>> but it fails if I tell LAVA (qemu) to disable KVM:
>>>>
>>>>    https://lava.collabora.co.uk/scheduler/job/1032359
>>>>
>>>> with the same panic as in lab-mhart.  It seems like it's failing
>>>> to return from an interrupt:
>>>>
>>>>    http://lava.streamtester.net/scheduler/job/87308
>>>>
>>>>    [    2.678828]  ? native_iret+0x7/0x7
>>>>    [    2.679208] WARNING: can't dereference iret registers at
>>>> 00000000ffc66068
>>>> for ip page_fault+0x11/0x60
>>>>
>>>> This triggered an automated bisection on kernelci.org, please see
>>>> the results below.
>>>>
>>>> I may run another bisection with this config enabled earlier in
>>>> the history to track down the actual change in the code that
>>>> introduced the issue, let me know if it's worth doing.
>>>
>>> No, because before that commit not all pieces are in place.
>>>
>>> Can you please try the failing kernel with pti=off on the command line?
>>
>> It does boot with pti=off (and KVM disabled):
>>
>>   https://lava.collabora.co.uk/scheduler/job/1032387
> 
> So it's a qemu issue. Added qemu folks on Cc.

Reproduced, thanks.  I will look into it.

Paolo

Re: stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)

[Thomas Gleixner]

On Wed, 3 Jan 2018, Paolo Bonzini wrote:
> On 03/01/2018 11:36, Thomas Gleixner wrote:
> > On Wed, 3 Jan 2018, Guillaume Tucker wrote:
> >> On 03/01/18 09:48, Thomas Gleixner wrote:
> >>>> Well, it turns out this is not exactly a conflict as there's a
> >>>> subtle difference between the qemu devices in lab-mhart and in
> >>>> lab-collabora.  The ones in lab-collabora are configured to use
> >>>> KVM, and it looks like the ones in lab-mhart aren't.
> >>>>
> >>>> So this job with KVM enabled passes in lab-collabora:
> >>>>
> >>>>    https://lava.collabora.co.uk/scheduler/job/1032358
> >>>>
> >>>> but it fails if I tell LAVA (qemu) to disable KVM:
> >>>>
> >>>>    https://lava.collabora.co.uk/scheduler/job/1032359
> >>>>
> >>>> with the same panic as in lab-mhart.  It seems like it's failing
> >>>> to return from an interrupt:
> >>>>
> >>>>    http://lava.streamtester.net/scheduler/job/87308
> >>>>
> >>>>    [    2.678828]  ? native_iret+0x7/0x7
> >>>>    [    2.679208] WARNING: can't dereference iret registers at
> >>>> 00000000ffc66068
> >>>> for ip page_fault+0x11/0x60
> >>>>
> >>>> This triggered an automated bisection on kernelci.org, please see
> >>>> the results below.
> >>>>
> >>>> I may run another bisection with this config enabled earlier in
> >>>> the history to track down the actual change in the code that
> >>>> introduced the issue, let me know if it's worth doing.
> >>>
> >>> No, because before that commit not all pieces are in place.
> >>>
> >>> Can you please try the failing kernel with pti=off on the command line?
> >>
> >> It does boot with pti=off (and KVM disabled):
> >>
> >>   https://lava.collabora.co.uk/scheduler/job/1032387
> > 
> > So it's a qemu issue. Added qemu folks on Cc.
> 
> Reproduced, thanks.  I will look into it.

I just noticed that the qemu instance emulates an AMD CPU.

We discovered an AMD related issue which fits in the problem you are seing
today.

Can you try the patch below please?

Thanks,

	tglx

8<------------------

--- a/arch/x86/entry/entry_64_compat.S
+++ b/arch/x86/entry/entry_64_compat.S
@@ -190,8 +190,13 @@ ENTRY(entry_SYSCALL_compat)
 	/* Interrupts are off on entry. */
 	swapgs
 
-	/* Stash user ESP and switch to the kernel stack. */
+	/* Stash user ESP */
 	movl	%esp, %r8d
+
+	/* Use %rsp as scratch reg. User ESP is stashed in r8 */
+	SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
+	
+	/* Switch to the kernel stack */
 	movq	PER_CPU_VAR(cpu_current_top_of_stack), %rsp
 
 	/* Construct struct pt_regs on stack */
@@ -220,12 +225,6 @@ GLOBAL(entry_SYSCALL_compat_after_hwfram
 	pushq   $0			/* pt_regs->r15 = 0 */
 
 	/*
-	 * We just saved %rdi so it is safe to clobber.  It is not
-	 * preserved during the C calls inside TRACE_IRQS_OFF anyway.
-	 */
-	SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi
-
-	/*
 	 * User mode is traced as though IRQs are on, and SYSENTER
 	 * turned them off.
 	 */