CPU vulnerabilities in public clouds

CPU vulnerabilities in public clouds

[Stefan Hajnoczi]

Hi Vitaly,
I just watched your FOSDEM talk on CPU vulnerabilities in public clouds:
https://mirror.cyberbits.eu/fosdem/2020/H.1309/vai_pubic_clouds_and_vulnerable_cpus.webm

If I understand correctly the situation for cloud users is:
1. The cloud provider takes care of hypervisor and CPU microcode fixes
but the instance may still be vulnerable to inter-process or guest
kernel attacks.
2. /sys/devices/system/cpu/vulnerabilities lists vulnerabilities that
the guest kernel knows about.  This might be outdated if new
vulnerabilities have been discovered since the kernel was installed.
False negatives are possible where your slides show the guest kernel
thinks there is no mitigation but you suspect the cloud provider has a
fix in place.
3. Cloud users still need to learn about every vulnerability to
understand whether inter-process or guest kernel attacks are possible.

Overall this seems to leave cloud users in a bad situation.  They
still need to become experts in each vulnerability and don't have
reliable information on their protection status.

Users with deep pockets will pay someone to do the work for them. For
many users the answer will probably be to apply guest OS updates and
hope for the best? :(

It would be nice if /sys/devices/system/cpu/vulnerabilities was at
least accurate...  Do you have any thoughts on improving the situation
for users?

Stefan

Re: CPU vulnerabilities in public clouds

[Vitaly Kuznetsov]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> Hi Vitaly,
> I just watched your FOSDEM talk on CPU vulnerabilities in public clouds:
> https://mirror.cyberbits.eu/fosdem/2020/H.1309/vai_pubic_clouds_and_vulnerable_cpus.webm
>
> If I understand correctly the situation for cloud users is:
> 1. The cloud provider takes care of hypervisor and CPU microcode fixes
> but the instance may still be vulnerable to inter-process or guest
> kernel attacks.

Correct.

> 2. /sys/devices/system/cpu/vulnerabilities lists vulnerabilities that
> the guest kernel knows about.  This might be outdated if new
> vulnerabilities have been discovered since the kernel was installed.

We don't know what we don't know, yes.

> False negatives are possible where your slides show the guest kernel
> thinks there is no mitigation but you suspect the cloud provider has a
> fix in place.

Well, I'm assuming that cloud providers are not crazy :-)

In particular, when you don't see STIBP/IBPB features on your instance
*now* there are two options:
1) Microcode wasn't updated since 2017
2) Features are deliberately hidden from you.

Why are these features hidden? Well, performace. When guest kernel sees
them it will start using them! And it doesn't come for free.

The situation with MDS/TAA is somewhat unique. To flush these internal
CPU buffers, an existing 'verw' instruction was re-purposed so if the
guest sees 'md_clear' flag it knows that the flush is happening,
however, if it doesn't see the flag (e.g. it was hidden by the
hypervisor) it cannot tell for sure if microcode update was deployed or
not. Linux made a choice to still try by default (MDS_MITIGATION_VMWERV/
TAA_MITIGATION_UCODE_NEEDED).

> 3. Cloud users still need to learn about every vulnerability to
> understand whether inter-process or guest kernel attacks are possible.
>
> Overall this seems to leave cloud users in a bad situation.  They
> still need to become experts in each vulnerability and don't have
> reliable information on their protection status.

The situation is not any better outside of cloud space. Linux (upstream
or vendors) tries to make reasonable choices for its defaults but they
may or may not work well for everyone. E.g. for mitigating Spectre_v2 we
now default to 'conditional' STIBP/IBPB meaning it will be enabled for
seccomp'ed tasks and tasks which explicitly ask for it with
prctl(). This is a good default but not a universal solution for
everyone.

>
> Users with deep pockets will pay someone to do the work for them. For
> many users the answer will probably be to apply guest OS updates and
> hope for the best? :(

Yes, it is, of course, possible that the user is in danger, however, to
mount an attack an intruder needs to have local access so it's mostly
multi-tenant environments which are at risk (or, if you're running an
untrusted code in your environment. Just don't).

>
> It would be nice if /sys/devices/system/cpu/vulnerabilities was at
> least accurate...  Do you have any thoughts on improving the situation
> for users?

The biggest ambiguity I see now is with Spectre_v2, I was sending an RFC
last week:
https://lore.kernel.org/lkml/20200121160257.302999-1-vkuznets@redhat.com/
but it seems I'm not the only one who noticed that.

Also, for KVM we probably should adjust our defaults for
L1TF/ITLB_MULTIHIT when running as a nested (L1) hypervisor.

There is also a big question with SMT (and I need to revive my
'NO_NONARCH_CORESHARING' work).

-- 
Vitaly

Re: CPU vulnerabilities in public clouds

[Christophe de Dinechin]

> On 5 Feb 2020, at 17:06, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
> Hi Vitaly,
> I just watched your FOSDEM talk on CPU vulnerabilities in public clouds:
> https://mirror.cyberbits.eu/fosdem/2020/H.1309/vai_pubic_clouds_and_vulnerable_cpus.webm
> 
> If I understand correctly the situation for cloud users is:
> 1. The cloud provider takes care of hypervisor and CPU microcode fixes
> but the instance may still be vulnerable to inter-process or guest
> kernel attacks.
> 2. /sys/devices/system/cpu/vulnerabilities lists vulnerabilities that
> the guest kernel knows about.  This might be outdated if new
> vulnerabilities have been discovered since the kernel was installed.
> False negatives are possible where your slides show the guest kernel
> thinks there is no mitigation but you suspect the cloud provider has a
> fix in place.
> 3. Cloud users still need to learn about every vulnerability to
> understand whether inter-process or guest kernel attacks are possible.
> 
> Overall this seems to leave cloud users in a bad situation.  They
> still need to become experts in each vulnerability and don't have
> reliable information on their protection status.
> 
> Users with deep pockets will pay someone to do the work for them. For
> many users the answer will probably be to apply guest OS updates and
> hope for the best? :(
> 
> It would be nice if /sys/devices/system/cpu/vulnerabilities was at
> least accurate...  Do you have any thoughts on improving the situation
> for users?

I understand your concern, and it’s a great point.

However, /sys is about the local system, so I’m not overly shocked
that it does not know about what is outside the system :-)

What could be nice, though, is if /sys/…/vulnerabilities exposed
a list of CVEs that have been taken into account at the time
the kernel was built.

# cat /sys/devices/system/cpu/vulnerabilities/CVE_list 
2017-5715
2017-5753
2017-5754
2018-3615
2018-3620
2018-3646
2018-12207
2018-12130
2018-12126
2018-12127
2019-11091
2018-3639
2019-11135

That way, you would know at least what you are measuring against.
The implementation is quite easy, see experiment here:

https://github.com/c3d/linux/commits/cpu-bugs-cve-list

Do you think that would have any value?


Thanks
Christophe

> 
> Stefan
> 

Re: CPU vulnerabilities in public clouds

[Vitaly Kuznetsov]

Christophe de Dinechin <dinechin@redhat.com> writes:

>> On 5 Feb 2020, at 17:06, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> 
>> Hi Vitaly,
>> I just watched your FOSDEM talk on CPU vulnerabilities in public clouds:
>> https://mirror.cyberbits.eu/fosdem/2020/H.1309/vai_pubic_clouds_and_vulnerable_cpus.webm
>> 
>> If I understand correctly the situation for cloud users is:
>> 1. The cloud provider takes care of hypervisor and CPU microcode fixes
>> but the instance may still be vulnerable to inter-process or guest
>> kernel attacks.
>> 2. /sys/devices/system/cpu/vulnerabilities lists vulnerabilities that
>> the guest kernel knows about.  This might be outdated if new
>> vulnerabilities have been discovered since the kernel was installed.
>> False negatives are possible where your slides show the guest kernel
>> thinks there is no mitigation but you suspect the cloud provider has a
>> fix in place.
>> 3. Cloud users still need to learn about every vulnerability to
>> understand whether inter-process or guest kernel attacks are possible.
>> 
>> Overall this seems to leave cloud users in a bad situation.  They
>> still need to become experts in each vulnerability and don't have
>> reliable information on their protection status.
>> 
>> Users with deep pockets will pay someone to do the work for them. For
>> many users the answer will probably be to apply guest OS updates and
>> hope for the best? :(
>> 
>> It would be nice if /sys/devices/system/cpu/vulnerabilities was at
>> least accurate...  Do you have any thoughts on improving the situation
>> for users?
>
> I understand your concern, and it’s a great point.
>
> However, /sys is about the local system, so I’m not overly shocked
> that it does not know about what is outside the system :-)
>
> What could be nice, though, is if /sys/…/vulnerabilities exposed
> a list of CVEs that have been taken into account at the time
> the kernel was built.
>
> # cat /sys/devices/system/cpu/vulnerabilities/CVE_list 
> 2017-5715
> 2017-5753
> 2017-5754
> 2018-3615
> 2018-3620
> 2018-3646
> 2018-12207
> 2018-12130
> 2018-12126
> 2018-12127
> 2019-11091
> 2018-3639
> 2019-11135
>
> That way, you would know at least what you are measuring against.
> The implementation is quite easy, see experiment here:
>
> https://github.com/c3d/linux/commits/cpu-bugs-cve-list
>
> Do you think that would have any value?
>

In a way, yes. The list basically tells you 'your kernel *knows* about
these CVEs as it was released after their release dates' but there is
still some uncertainty for the user: OK, a CVE exists but is my CPU
actually vulnerable? It may happen that when CVE releases we think that
some CPUs (or a particular arch) are not vulnerable but we change our
mind later. Also, all vulnerabilities are different as the severity
depends a lot on the workload. The actions one need to undertake differ
(update microcode, disable SMT, pin tasks to certain cores/threads,
...). The bottom line is I don't see an easy path for users. They either
trust the source of their bits (e.g. distro), the infrastructure
(hardware/virtual/cloud) and the default settings as being 'secure
enough' or they have to study the details...

-- 
Vitaly