[PATCH] x86/speculation/l1tf: fix off-by-one error when warning that system has too much RAM

[PATCH] x86/speculation/l1tf: fix off-by-one error when warning that system has too much RAM

[Vlastimil Babka]

Two users have reported [1] that they have an "extremely unlikely" system
with more than MAX_PA/2 memory and L1TF mitigation is not effective. In fact
it's a CPU with 36bits phys limit (64GB) and 32GB memory, but due to holes
in the e820 map, the main region is almost 500MB over the 32GB limit:

[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000081effffff] usable

Suggestions to use 'mem=32G' to prefer L1TF mitigation while losing the 500MB
revealed, that there's an off-by-one error in the check in
l1tf_select_mitigation(). l1tf_pfn_limit() returns the last usable pfn
(inclusive), but it's more common and hopefully less error-prone to return the
first pfn that's over limit, so this patch changes that and updates the other
callers.

[1] https://bugzilla.suse.com/show_bug.cgi?id=1105536

Reported-by: xxxxxx xxxxxx <xxxxxx@xxxxxx.xxx>
Reported-by: Christopher Snowhill <kode54@gmail.com>
Fixes: 17dbca119312 ("x86/speculation/l1tf: Add sysfs reporting for l1tf")
Cc: stable@vger.kernel.org
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 arch/x86/include/asm/processor.h | 2 +-
 arch/x86/mm/init.c               | 2 +-
 arch/x86/mm/mmap.c               | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index a0a52274cb4a..c24297268ebc 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -183,7 +183,7 @@ extern void cpu_detect(struct cpuinfo_x86 *c);
 
 static inline unsigned long long l1tf_pfn_limit(void)
 {
-	return BIT_ULL(boot_cpu_data.x86_phys_bits - 1 - PAGE_SHIFT) - 1;
+	return BIT_ULL(boot_cpu_data.x86_phys_bits - 1 - PAGE_SHIFT);
 }
 
 extern void early_cpu_init(void);
diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
index 02de3d6065c4..63a6f9fcaf20 100644
--- a/arch/x86/mm/init.c
+++ b/arch/x86/mm/init.c
@@ -923,7 +923,7 @@ unsigned long max_swapfile_size(void)
 
 	if (boot_cpu_has_bug(X86_BUG_L1TF)) {
 		/* Limit the swap file size to MAX_PA/2 for L1TF workaround */
-		unsigned long long l1tf_limit = l1tf_pfn_limit() + 1;
+		unsigned long long l1tf_limit = l1tf_pfn_limit();
 		/*
 		 * We encode swap offsets also with 3 bits below those for pfn
 		 * which makes the usable limit higher.
diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
index f40ab8185d94..1e95d57760cf 100644
--- a/arch/x86/mm/mmap.c
+++ b/arch/x86/mm/mmap.c
@@ -257,7 +257,7 @@ bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
 	/* If it's real memory always allow */
 	if (pfn_valid(pfn))
 		return true;
-	if (pfn > l1tf_pfn_limit() && !capable(CAP_SYS_ADMIN))
+	if (pfn >= l1tf_pfn_limit() && !capable(CAP_SYS_ADMIN))
 		return false;
 	return true;
 }
-- 
2.18.0

Re: [PATCH] x86/speculation/l1tf: fix off-by-one error when warning that system has too much RAM

[Michal Hocko]

On Thu 23-08-18 15:44:18, Vlastimil Babka wrote:
> Two users have reported [1] that they have an "extremely unlikely" system
> with more than MAX_PA/2 memory and L1TF mitigation is not effective. In fact
> it's a CPU with 36bits phys limit (64GB) and 32GB memory, but due to holes
> in the e820 map, the main region is almost 500MB over the 32GB limit:
> 
> [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000081effffff] usable
> 
> Suggestions to use 'mem=32G' to prefer L1TF mitigation while losing the 500MB
> revealed, that there's an off-by-one error in the check in
> l1tf_select_mitigation(). l1tf_pfn_limit() returns the last usable pfn
> (inclusive), but it's more common and hopefully less error-prone to return the
> first pfn that's over limit, so this patch changes that and updates the other
> callers.
> 
> [1] https://bugzilla.suse.com/show_bug.cgi?id=1105536
> 
> Reported-by: xxxxxx xxxxxx <xxxxxx@xxxxxx.xxx>
> Reported-by: Christopher Snowhill <kode54@gmail.com>
> Fixes: 17dbca119312 ("x86/speculation/l1tf: Add sysfs reporting for l1tf")
> Cc: stable@vger.kernel.org
> Signed-off-by: Vlastimil Babka <vbabka@suse.cz>

Yes this should be less error prone.
Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  arch/x86/include/asm/processor.h | 2 +-
>  arch/x86/mm/init.c               | 2 +-
>  arch/x86/mm/mmap.c               | 2 +-
>  3 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index a0a52274cb4a..c24297268ebc 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -183,7 +183,7 @@ extern void cpu_detect(struct cpuinfo_x86 *c);
>  
>  static inline unsigned long long l1tf_pfn_limit(void)
>  {
> -	return BIT_ULL(boot_cpu_data.x86_phys_bits - 1 - PAGE_SHIFT) - 1;
> +	return BIT_ULL(boot_cpu_data.x86_phys_bits - 1 - PAGE_SHIFT);
>  }
>  
>  extern void early_cpu_init(void);
> diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
> index 02de3d6065c4..63a6f9fcaf20 100644
> --- a/arch/x86/mm/init.c
> +++ b/arch/x86/mm/init.c
> @@ -923,7 +923,7 @@ unsigned long max_swapfile_size(void)
>  
>  	if (boot_cpu_has_bug(X86_BUG_L1TF)) {
>  		/* Limit the swap file size to MAX_PA/2 for L1TF workaround */
> -		unsigned long long l1tf_limit = l1tf_pfn_limit() + 1;
> +		unsigned long long l1tf_limit = l1tf_pfn_limit();
>  		/*
>  		 * We encode swap offsets also with 3 bits below those for pfn
>  		 * which makes the usable limit higher.
> diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
> index f40ab8185d94..1e95d57760cf 100644
> --- a/arch/x86/mm/mmap.c
> +++ b/arch/x86/mm/mmap.c
> @@ -257,7 +257,7 @@ bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
>  	/* If it's real memory always allow */
>  	if (pfn_valid(pfn))
>  		return true;
> -	if (pfn > l1tf_pfn_limit() && !capable(CAP_SYS_ADMIN))
> +	if (pfn >= l1tf_pfn_limit() && !capable(CAP_SYS_ADMIN))
>  		return false;
>  	return true;
>  }
> -- 
> 2.18.0

-- 
Michal Hocko
SUSE Labs

[PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Vlastimil Babka]

Two users have reported [1] that they have an "extremely unlikely" system
with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
make the warning more helpful by suggesting the proper mem=X kernel boot param,
a rough calculation of how much RAM can be lost (not precise if there's holes
between MAX_PA/2 and max_pfn in the e820 map) and a link to the L1TF document
to help decide if the mitigation is worth the unusable RAM.

[1] https://bugzilla.suse.com/show_bug.cgi?id=1105536

Suggested-by: Michal Hocko <mhocko@suse.com>
Cc: stable@vger.kernel.org
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 arch/x86/kernel/cpu/bugs.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index cb4a16292aa7..4b820bb6c504 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -702,6 +702,11 @@ static void __init l1tf_select_mitigation(void)
 	half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;
 	if (e820__mapped_any(half_pa, ULLONG_MAX - half_pa, E820_TYPE_RAM)) {
 		pr_warn("System has more than MAX_PA/2 memory. L1TF mitigation not effective.\n");
+		pr_info("You may make it effective by booting the kernel with mem=%llu parameter.\n",
+				half_pa);
+		pr_info("However, doing so will make up to %llu bytes of RAM unusable.\n",
+				((u64) max_pfn << PAGE_SHIFT) - half_pa);
+		pr_info("Reading Documentation/admin-guide/l1tf.rst might help you decide.");
 		return;
 	}
 
-- 
2.18.0

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Andi Kleen]

On Thu, Aug 23, 2018 at 04:28:12PM +0200, Vlastimil Babka wrote:
> Two users have reported [1] that they have an "extremely unlikely" system
> with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
> make the warning more helpful by suggesting the proper mem=X kernel boot param,
> a rough calculation of how much RAM can be lost (not precise if there's holes
> between MAX_PA/2 and max_pfn in the e820 map) and a link to the L1TF document
> to help decide if the mitigation is worth the unusable RAM.

I'm not sure anyone would really do that. After all you probably prefer
your memory. And if it's really a non ECC client part they are are
already used to to live very dangerously because their undetected RAM bit error
rate will be significant. L1TF is probably one of your smaller problems
in this case...


-Andi

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Michal Hocko]

On Thu 23-08-18 08:46:48, Andi Kleen wrote:
> On Thu, Aug 23, 2018 at 04:28:12PM +0200, Vlastimil Babka wrote:
> > Two users have reported [1] that they have an "extremely unlikely" system
> > with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
> > make the warning more helpful by suggesting the proper mem=X kernel boot param,
> > a rough calculation of how much RAM can be lost (not precise if there's holes
> > between MAX_PA/2 and max_pfn in the e820 map) and a link to the L1TF document
> > to help decide if the mitigation is worth the unusable RAM.
> 
> I'm not sure anyone would really do that. After all you probably prefer
> your memory. And if it's really a non ECC client part they are are
> already used to to live very dangerously because their undetected RAM bit error
> rate will be significant. L1TF is probably one of your smaller problems
> in this case...

There are people who care about L1TF mitigations. I am not going to
question their motivation. In any case a hint how to make the mitigation
active again sounds more useful than something that sounds as scary as
"you are vulnerable".

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Andi Kleen]

> There are people who care about L1TF mitigations. I am not going to
> question their motivation. In any case a hint how to make the mitigation
> active again sounds more useful than something that sounds as scary as
> "you are vulnerable".

FWIW an early version of these patches automatically limited the available
memory, but Linus pointed out that people likely prefer their memory.
I still think his reasoning was right, and likely applies to your
message too.

-Andi

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Michal Hocko]

On Thu 23-08-18 12:38:33, Andi Kleen wrote:
> > There are people who care about L1TF mitigations. I am not going to
> > question their motivation. In any case a hint how to make the mitigation
> > active again sounds more useful than something that sounds as scary as
> > "you are vulnerable".
> 
> FWIW an early version of these patches automatically limited the available
> memory, but Linus pointed out that people likely prefer their memory.

Nobody is questioning that. The point is to give them a hint on how to
make the mitigation active again without going to call for help. The
message does tell them how to _enable_ it and point them to the
documentation on how to _decide_.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Andi Kleen]

On Thu, Aug 23, 2018 at 10:05:20PM +0200, Michal Hocko wrote:
> On Thu 23-08-18 12:38:33, Andi Kleen wrote:
> > > There are people who care about L1TF mitigations. I am not going to
> > > question their motivation. In any case a hint how to make the mitigation
> > > active again sounds more useful than something that sounds as scary as
> > > "you are vulnerable".
> > 
> > FWIW an early version of these patches automatically limited the available
> > memory, but Linus pointed out that people likely prefer their memory.
> 
> Nobody is questioning that. The point is to give them a hint on how to
> make the mitigation active again without going to call for help. The
> message does tell them how to _enable_ it and point them to the
> documentation on how to _decide_.

On the message I guess there are two cases:

- either it's very little memory that is lost like in the 32GB + memory
hole case. In this case maybe it's better if we just limit automatically
if the overlap is small enough (<2GB perhaps?)

- Or it's a lot of memory then people are unlikely to want to lose their
memory and I don't think we really need the message either.

Also I checked the bug again and it looks like the reporter has an IvyBridge.
There is actually a better solution for those (anything Nehalem and newer)
because they internally have at least 44 bits in the cache, which
is good enough for the mitigation. Just need a quirk to override
the bit width in this case (will submit a patch)

-Andi

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 2835 bytes --]

Hi Vlastimil,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on tip/auto-latest]
[also build test ERROR on next-20180822]
[cannot apply to v4.18]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Vlastimil-Babka/x86-speculation-l1tf-suggest-what-to-do-on-systems-with-too-much-RAM/20180824-013859
config: i386-randconfig-x077-201833 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-16) 7.3.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   In file included from include/linux/kernel.h:14:0,
                    from arch/x86/include/asm/percpu.h:45,
                    from arch/x86/include/asm/current.h:6,
                    from include/linux/sched.h:12,
                    from include/linux/utsname.h:6,
                    from arch/x86/kernel/cpu/bugs.c:12:
   arch/x86/kernel/cpu/bugs.c: In function 'l1tf_select_mitigation':
>> arch/x86/kernel/cpu/bugs.c:709:12: error: 'max_pfn' undeclared (first use in this function); did you mean 'pgd_pfn'?
        ((u64) max_pfn << PAGE_SHIFT) - half_pa);
               ^
   include/linux/printk.h:311:34: note: in definition of macro 'pr_info'
     printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
                                     ^~~~~~~~~~~
   arch/x86/kernel/cpu/bugs.c:709:12: note: each undeclared identifier is reported only once for each function it appears in
        ((u64) max_pfn << PAGE_SHIFT) - half_pa);
               ^
   include/linux/printk.h:311:34: note: in definition of macro 'pr_info'
     printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
                                     ^~~~~~~~~~~

vim +709 arch/x86/kernel/cpu/bugs.c

   697	
   698		/*
   699		 * This is extremely unlikely to happen because almost all
   700		 * systems have far more MAX_PA/2 than RAM can be fit into
   701		 * DIMM slots.
   702		 */
   703		half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;
   704		if (e820__mapped_any(half_pa, ULLONG_MAX - half_pa, E820_TYPE_RAM)) {
   705			pr_warn("System has more than MAX_PA/2 memory. L1TF mitigation not effective.\n");
   706			pr_info("You may make it effective by booting the kernel with mem=%llu parameter.\n",
   707					half_pa);
   708			pr_info("However, doing so will make up to %llu bytes of RAM unusable.\n",
 > 709					((u64) max_pfn << PAGE_SHIFT) - half_pa);
   710			pr_info("Reading Documentation/admin-guide/l1tf.rst might help you decide.");
   711			return;
   712		}
   713	
   714		setup_force_cpu_cap(X86_FEATURE_L1TF_PTEINV);
   715	}
   716	

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 32575 bytes --]

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 2804 bytes --]

Hi Vlastimil,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on tip/auto-latest]
[also build test ERROR on next-20180822]
[cannot apply to v4.18]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Vlastimil-Babka/x86-speculation-l1tf-suggest-what-to-do-on-systems-with-too-much-RAM/20180824-013859
config: i386-randconfig-a1-201833 (attached as .config)
compiler: gcc-4.9 (Debian 4.9.4-2) 4.9.4
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   In file included from include/linux/kernel.h:14:0,
                    from arch/x86/include/asm/percpu.h:45,
                    from arch/x86/include/asm/current.h:6,
                    from include/linux/sched.h:12,
                    from include/linux/utsname.h:6,
                    from arch/x86//kernel/cpu/bugs.c:12:
   arch/x86//kernel/cpu/bugs.c: In function 'l1tf_select_mitigation':
>> arch/x86//kernel/cpu/bugs.c:709:12: error: 'max_pfn' undeclared (first use in this function)
        ((u64) max_pfn << PAGE_SHIFT) - half_pa);
               ^
   include/linux/printk.h:311:34: note: in definition of macro 'pr_info'
     printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
                                     ^
   arch/x86//kernel/cpu/bugs.c:709:12: note: each undeclared identifier is reported only once for each function it appears in
        ((u64) max_pfn << PAGE_SHIFT) - half_pa);
               ^
   include/linux/printk.h:311:34: note: in definition of macro 'pr_info'
     printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
                                     ^

vim +/max_pfn +709 arch/x86//kernel/cpu/bugs.c

   697	
   698		/*
   699		 * This is extremely unlikely to happen because almost all
   700		 * systems have far more MAX_PA/2 than RAM can be fit into
   701		 * DIMM slots.
   702		 */
   703		half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;
   704		if (e820__mapped_any(half_pa, ULLONG_MAX - half_pa, E820_TYPE_RAM)) {
   705			pr_warn("System has more than MAX_PA/2 memory. L1TF mitigation not effective.\n");
   706			pr_info("You may make it effective by booting the kernel with mem=%llu parameter.\n",
   707					half_pa);
   708			pr_info("However, doing so will make up to %llu bytes of RAM unusable.\n",
 > 709					((u64) max_pfn << PAGE_SHIFT) - half_pa);
   710			pr_info("Reading Documentation/admin-guide/l1tf.rst might help you decide.");
   711			return;
   712		}
   713	
   714		setup_force_cpu_cap(X86_FEATURE_L1TF_PTEINV);
   715	}
   716	

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 23873 bytes --]

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Michal Hocko]

On Thu 23-08-18 16:28:12, Vlastimil Babka wrote:
> Two users have reported [1] that they have an "extremely unlikely" system
> with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
> make the warning more helpful by suggesting the proper mem=X kernel boot param,
> a rough calculation of how much RAM can be lost (not precise if there's holes
> between MAX_PA/2 and max_pfn in the e820 map) and a link to the L1TF document
> to help decide if the mitigation is worth the unusable RAM.
> 
> [1] https://bugzilla.suse.com/show_bug.cgi?id=1105536
> 
> Suggested-by: Michal Hocko <mhocko@suse.com>
> Cc: stable@vger.kernel.org
> Signed-off-by: Vlastimil Babka <vbabka@suse.cz>

I wouldn't bother with max_pfn-half_pa part but other than that this is
much more useful than the original message.

Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  arch/x86/kernel/cpu/bugs.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index cb4a16292aa7..4b820bb6c504 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -702,6 +702,11 @@ static void __init l1tf_select_mitigation(void)
>  	half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;
>  	if (e820__mapped_any(half_pa, ULLONG_MAX - half_pa, E820_TYPE_RAM)) {
>  		pr_warn("System has more than MAX_PA/2 memory. L1TF mitigation not effective.\n");
> +		pr_info("You may make it effective by booting the kernel with mem=%llu parameter.\n",
> +				half_pa);
> +		pr_info("However, doing so will make up to %llu bytes of RAM unusable.\n",
> +				((u64) max_pfn << PAGE_SHIFT) - half_pa);
> +		pr_info("Reading Documentation/admin-guide/l1tf.rst might help you decide.");
>  		return;
>  	}
>  
> -- 
> 2.18.0

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Vlastimil Babka]

On 08/23/2018 09:27 PM, Michal Hocko wrote:
> On Thu 23-08-18 16:28:12, Vlastimil Babka wrote:
>> Two users have reported [1] that they have an "extremely unlikely" system
>> with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
>> make the warning more helpful by suggesting the proper mem=X kernel boot param,
>> a rough calculation of how much RAM can be lost (not precise if there's holes
>> between MAX_PA/2 and max_pfn in the e820 map) and a link to the L1TF document
>> to help decide if the mitigation is worth the unusable RAM.
>>
>> [1] https://bugzilla.suse.com/show_bug.cgi?id=1105536
>>
>> Suggested-by: Michal Hocko <mhocko@suse.com>
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
> 
> I wouldn't bother with max_pfn-half_pa part but other than that this is
> much more useful than the original message.

Right, and it causes build failures on some configs.

> Acked-by: Michal Hocko <mhocko@suse.com>

Thanks! Here's a v2:

----8<----
>From 977c5db27fe35a84807850b947bc5678c4d467b3 Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka@suse.cz>
Date: Thu, 23 Aug 2018 16:21:29 +0200
Subject: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too
 much RAM

Two users have reported [1] that they have an "extremely unlikely" system
with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
make the warning more helpful by suggesting the proper mem=X kernel boot param
to make it effective and a link to the L1TF document to help decide if the
mitigation is worth the unusable RAM.

[1] https://bugzilla.suse.com/show_bug.cgi?id=1105536

Suggested-by: Michal Hocko <mhocko@suse.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 arch/x86/kernel/cpu/bugs.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index cb4a16292aa7..5c32b5006738 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -702,6 +702,10 @@ static void __init l1tf_select_mitigation(void)
 	half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;
 	if (e820__mapped_any(half_pa, ULLONG_MAX - half_pa, E820_TYPE_RAM)) {
 		pr_warn("System has more than MAX_PA/2 memory. L1TF mitigation not effective.\n");
+		pr_info("You may make it effective by booting the kernel with mem=%llu parameter.\n",
+				half_pa);
+		pr_info("However, doing so will make a part of your RAM unusable.\n");
+		pr_info("Reading Documentation/admin-guide/l1tf.rst might help you decide.\n");
 		return;
 	}
 
-- 
2.18.0

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Vlastimil Babka]

On 08/24/2018 09:32 AM, Vlastimil Babka wrote:
> On 08/23/2018 09:27 PM, Michal Hocko wrote:
>> On Thu 23-08-18 16:28:12, Vlastimil Babka wrote:
>>> Two users have reported [1] that they have an "extremely unlikely" system
>>> with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
>>> make the warning more helpful by suggesting the proper mem=X kernel boot param,
>>> a rough calculation of how much RAM can be lost (not precise if there's holes
>>> between MAX_PA/2 and max_pfn in the e820 map) and a link to the L1TF document
>>> to help decide if the mitigation is worth the unusable RAM.
>>>
>>> [1] https://bugzilla.suse.com/show_bug.cgi?id=1105536
>>>
>>> Suggested-by: Michal Hocko <mhocko@suse.com>
>>> Cc: stable@vger.kernel.org
>>> Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
>>
>> I wouldn't bother with max_pfn-half_pa part but other than that this is
>> much more useful than the original message.
> 
> Right, and it causes build failures on some configs.
> 
>> Acked-by: Michal Hocko <mhocko@suse.com>
> 
> Thanks! Here's a v2:

Just realized that kvm printk's refer to the online version at
https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html
which should be easier for the users of distro kernels, should I change
that?

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Vlastimil Babka]

On 08/24/2018 12:36 PM, Vlastimil Babka wrote:
> On 08/24/2018 09:32 AM, Vlastimil Babka wrote:
>> On 08/23/2018 09:27 PM, Michal Hocko wrote:
>>> On Thu 23-08-18 16:28:12, Vlastimil Babka wrote:
>>>> Two users have reported [1] that they have an "extremely unlikely" system
>>>> with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
>>>> make the warning more helpful by suggesting the proper mem=X kernel boot param,
>>>> a rough calculation of how much RAM can be lost (not precise if there's holes
>>>> between MAX_PA/2 and max_pfn in the e820 map) and a link to the L1TF document
>>>> to help decide if the mitigation is worth the unusable RAM.
>>>>
>>>> [1] https://bugzilla.suse.com/show_bug.cgi?id=1105536
>>>>
>>>> Suggested-by: Michal Hocko <mhocko@suse.com>
>>>> Cc: stable@vger.kernel.org
>>>> Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
>>>
>>> I wouldn't bother with max_pfn-half_pa part but other than that this is
>>> much more useful than the original message.
>>
>> Right, and it causes build failures on some configs.
>>
>>> Acked-by: Michal Hocko <mhocko@suse.com>
>>
>> Thanks! Here's a v2:
> 
> Just realized that kvm printk's refer to the online version at
> https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html
> which should be easier for the users of distro kernels, should I change
> that?

FWIW, if it's not possible to amend anymore...
----8<----
>From a650e6a4b989c6e029ac1ab4e0a68553e074ba7a Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka@suse.cz>
Date: Fri, 24 Aug 2018 14:05:45 +0200
Subject: [PATCH] x86/speculation/l1tf: replace Documentation path with
 kernel.org URL

The URL might be easier to reach for users of distro kernels without unpacked
source, and be more up-to-date. It's also used in KVM warnings related to L1TF.

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 arch/x86/kernel/cpu/bugs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 5c32b5006738..4c2313d0b9ca 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -705,7 +705,7 @@ static void __init l1tf_select_mitigation(void)
 		pr_info("You may make it effective by booting the kernel with mem=%llu parameter.\n",
 				half_pa);
 		pr_info("However, doing so will make a part of your RAM unusable.\n");
-		pr_info("Reading Documentation/admin-guide/l1tf.rst might help you decide.\n");
+		pr_info("Reading https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html might help you decide.\n");
 		return;
 	}
 
-- 
2.18.0


 

Re: [PATCH] x86/speculation/l1tf: suggest what to do on systems with too much RAM

[Michal Hocko]

On Fri 24-08-18 14:10:54, Vlastimil Babka wrote:
> On 08/24/2018 12:36 PM, Vlastimil Babka wrote:
> > On 08/24/2018 09:32 AM, Vlastimil Babka wrote:
> >> On 08/23/2018 09:27 PM, Michal Hocko wrote:
> >>> On Thu 23-08-18 16:28:12, Vlastimil Babka wrote:
> >>>> Two users have reported [1] that they have an "extremely unlikely" system
> >>>> with more than MAX_PA/2 memory and L1TF mitigation is not effective. Let's
> >>>> make the warning more helpful by suggesting the proper mem=X kernel boot param,
> >>>> a rough calculation of how much RAM can be lost (not precise if there's holes
> >>>> between MAX_PA/2 and max_pfn in the e820 map) and a link to the L1TF document
> >>>> to help decide if the mitigation is worth the unusable RAM.
> >>>>
> >>>> [1] https://bugzilla.suse.com/show_bug.cgi?id=1105536
> >>>>
> >>>> Suggested-by: Michal Hocko <mhocko@suse.com>
> >>>> Cc: stable@vger.kernel.org
> >>>> Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
> >>>
> >>> I wouldn't bother with max_pfn-half_pa part but other than that this is
> >>> much more useful than the original message.
> >>
> >> Right, and it causes build failures on some configs.
> >>
> >>> Acked-by: Michal Hocko <mhocko@suse.com>
> >>
> >> Thanks! Here's a v2:
> > 
> > Just realized that kvm printk's refer to the online version at
> > https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html
> > which should be easier for the users of distro kernels, should I change
> > that?
> 
> FWIW, if it's not possible to amend anymore...
> ----8<----
> >From a650e6a4b989c6e029ac1ab4e0a68553e074ba7a Mon Sep 17 00:00:00 2001
> From: Vlastimil Babka <vbabka@suse.cz>
> Date: Fri, 24 Aug 2018 14:05:45 +0200
> Subject: [PATCH] x86/speculation/l1tf: replace Documentation path with
>  kernel.org URL
> 
> The URL might be easier to reach for users of distro kernels without unpacked
> source, and be more up-to-date. It's also used in KVM warnings related to L1TF.

Yes, URL is much more convenient than a reference to the kernel source
documentation. If the link below is meant to be long lived then sure.

> Signed-off-by: Vlastimil Babka <vbabka@suse.cz>

Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  arch/x86/kernel/cpu/bugs.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 5c32b5006738..4c2313d0b9ca 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -705,7 +705,7 @@ static void __init l1tf_select_mitigation(void)
>  		pr_info("You may make it effective by booting the kernel with mem=%llu parameter.\n",
>  				half_pa);
>  		pr_info("However, doing so will make a part of your RAM unusable.\n");
> -		pr_info("Reading Documentation/admin-guide/l1tf.rst might help you decide.\n");
> +		pr_info("Reading https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html might help you decide.\n");
>  		return;
>  	}
>  
> -- 
> 2.18.0
> 
> 
>  

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] x86/speculation/l1tf: fix off-by-one error when warning that system has too much RAM

[Andi Kleen]

On Thu, Aug 23, 2018 at 03:44:18PM +0200, Vlastimil Babka wrote:
> Two users have reported [1] that they have an "extremely unlikely" system
> with more than MAX_PA/2 memory and L1TF mitigation is not effective. In fact
> it's a CPU with 36bits phys limit (64GB) and 32GB memory, but due to holes
> in the e820 map, the main region is almost 500MB over the 32GB limit:

Ah I see it's a client part with very large DIMMs and someone being
very brave and using that much memory without ECC.

> 
> [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000081effffff] usable
> 
> Suggestions to use 'mem=32G' to prefer L1TF mitigation while losing the 500MB
> revealed, that there's an off-by-one error in the check in
> l1tf_select_mitigation(). l1tf_pfn_limit() returns the last usable pfn
> (inclusive), but it's more common and hopefully less error-prone to return the
> first pfn that's over limit, so this patch changes that and updates the other
> callers.

I can see the off by one, but does it really cause the user's problem?

They will be still over the limit in any case, with or without off-by-one.

So the description has nothing to do with the fix. Or do I miss something?

-Andi

Re: [PATCH] x86/speculation/l1tf: fix off-by-one error when warning that system has too much RAM

[Vlastimil Babka]

On 8/23/18 5:44 PM, Andi Kleen wrote:
> On Thu, Aug 23, 2018 at 03:44:18PM +0200, Vlastimil Babka wrote:
>> Two users have reported [1] that they have an "extremely unlikely" system
>> with more than MAX_PA/2 memory and L1TF mitigation is not effective. In fact
>> it's a CPU with 36bits phys limit (64GB) and 32GB memory, but due to holes
>> in the e820 map, the main region is almost 500MB over the 32GB limit:
> 
> Ah I see it's a client part with very large DIMMs and someone being
> very brave and using that much memory without ECC.

Are you trying to mock the users and diminsh their report because of that?

>>
>> [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000081effffff] usable
>>
>> Suggestions to use 'mem=32G' to prefer L1TF mitigation while losing the 500MB
>> revealed, that there's an off-by-one error in the check in
>> l1tf_select_mitigation(). l1tf_pfn_limit() returns the last usable pfn
>> (inclusive), but it's more common and hopefully less error-prone to return the
>> first pfn that's over limit, so this patch changes that and updates the other
>> callers.
> 
> I can see the off by one, but does it really cause the user's problem?
> 
> They will be still over the limit in any case, with or without off-by-one.
> 
> So the description has nothing to do with the fix. Or do I miss something?

The off-by-one happens when 'mem=32G' is used to limit the amount
memory. It's easier to do "mem=32G" with fixed code than "mem=33554428k"
to workaround the error.

> 
> -Andi
> 

Re: [PATCH] x86/speculation/l1tf: fix off-by-one error when warning that system has too much RAM

[Andre Tomt]

On 23. aug. 2018 17:44, Andi Kleen wrote:
> On Thu, Aug 23, 2018 at 03:44:18PM +0200, Vlastimil Babka wrote:
>> Two users have reported [1] that they have an "extremely unlikely" system
>> with more than MAX_PA/2 memory and L1TF mitigation is not effective. In fact
>> it's a CPU with 36bits phys limit (64GB) and 32GB memory, but due to holes
>> in the e820 map, the main region is almost 500MB over the 32GB limit:
> 
> Ah I see it's a client part with very large DIMMs and someone being
> very brave and using that much memory without ECC.

FYI; It is also happening on Xeon E3v2 (Ivy Bridge generation) w/ 32GB 
of ECC RAM here, a low-end server part that officially supports up to 32GB.

> [    0.000000] microcode: microcode updated early to revision 0x20, date = 2018-04-10
> [    0.029728] L1TF: System has more than MAX_PA/2 memory. L1TF mitigation not effective.
> [    1.063155] microcode: sig=0x306a9, pf=0x2, revision=0x20


> processor	: 7
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 58
> model name	: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
> stepping	: 9
> microcode	: 0x20
> cpu MHz		: 3500.044
> cache size	: 8192 KB
> physical id	: 0
> siblings	: 8
> core id		: 3
> cpu cores	: 4
> apicid		: 7
> initial apicid	: 7
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 13
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts flush_l1d
> bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
> bogomips	: 6602.15
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 36 bits physical, 48 bits virtual
> power management:

Re: [PATCH] x86/speculation/l1tf: fix off-by-one error when warning that system has too much RAM

[Andi Kleen]

On Fri, Aug 24, 2018 at 04:22:57AM +0200, Andre Tomt wrote:
> On 23. aug. 2018 17:44, Andi Kleen wrote:
> > On Thu, Aug 23, 2018 at 03:44:18PM +0200, Vlastimil Babka wrote:
> > > Two users have reported [1] that they have an "extremely unlikely" system
> > > with more than MAX_PA/2 memory and L1TF mitigation is not effective. In fact
> > > it's a CPU with 36bits phys limit (64GB) and 32GB memory, but due to holes
> > > in the e820 map, the main region is almost 500MB over the 32GB limit:
> > 
> > Ah I see it's a client part with very large DIMMs and someone being
> > very brave and using that much memory without ECC.
> 
> FYI; It is also happening on Xeon E3v2 (Ivy Bridge generation) w/ 32GB of
> ECC RAM here, a low-end server part that officially supports up to 32GB.

Good point.

-andi

Re: [PATCH] x86/speculation/l1tf: fix off-by-one error when warning that system has too much RAM

[Christopher Snowhill]

[-- Attachment #1: Type: text/plain, Size: 2746 bytes --]



On 08/23/2018 07:22 PM, Andre Tomt wrote:
> On 23. aug. 2018 17:44, Andi Kleen wrote:
>> On Thu, Aug 23, 2018 at 03:44:18PM +0200, Vlastimil Babka wrote:
>>> Two users have reported [1] that they have an "extremely unlikely"
>>> system
>>> with more than MAX_PA/2 memory and L1TF mitigation is not effective.
>>> In fact
>>> it's a CPU with 36bits phys limit (64GB) and 32GB memory, but due to
>>> holes
>>> in the e820 map, the main region is almost 500MB over the 32GB limit:
>>
>> Ah I see it's a client part with very large DIMMs and someone being
>> very brave and using that much memory without ECC.
>
> FYI; It is also happening on Xeon E3v2 (Ivy Bridge generation) w/ 32GB
> of ECC RAM here, a low-end server part that officially supports up to
> 32GB.
>

Indeed, I must be "very brave" to not have chucked this CPU and
motherboard and RAM in the bin, and bought a new board, Xeon CPU, and
ECC RAM. Maybe I'll consider that in the future, when I again have
$1000+ to buy new kit. Which will probably be never, at this rate.

>> [    0.000000] microcode: microcode updated early to revision 0x20,
>> date = 2018-04-10
>> [    0.029728] L1TF: System has more than MAX_PA/2 memory. L1TF
>> mitigation not effective.
>> [    1.063155] microcode: sig=0x306a9, pf=0x2, revision=0x20
>
>
>> processor    : 7
>> vendor_id    : GenuineIntel
>> cpu family    : 6
>> model        : 58
>> model name    : Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
>> stepping    : 9
>> microcode    : 0x20
>> cpu MHz        : 3500.044
>> cache size    : 8192 KB
>> physical id    : 0
>> siblings    : 8
>> core id        : 3
>> cpu cores    : 4
>> apicid        : 7
>> initial apicid    : 7
>> fpu        : yes
>> fpu_exception    : yes
>> cpuid level    : 13
>> wp        : yes
>> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>> syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
>> xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor
>> ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic
>> popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm
>> cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority
>> ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts flush_l1d
>> bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
>> bogomips    : 6602.15
>> clflush size    : 64
>> cache_alignment    : 64
>> address sizes    : 36 bits physical, 48 bits virtual
>> power management:
>
>


[-- Attachment #2: pEpkey.asc --]
[-- Type: application/pgp-keys, Size: 2505 bytes --]