[Qemu-devel] [RFC] sanitize memory on system reset

[Qemu-devel] [RFC] sanitize memory on system reset

[Peter Lieven]

Hi,

I was thinking if it would be a good idea to zeroize all memory resources on system reset and
madvise dontneed them afterwards. This would avoid system reset attacks in case the attacker
has only access to the console of a vServer but not on the physical host and it would shrink
RSS size of the vServer siginificantly.

BR,
Peter

-- 

Mit freundlichen Grüßen

Peter Lieven

...........................................................

   KAMP Netzwerkdienste GmbH
   Vestische Str. 89-91 | 46117 Oberhausen
   Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40
   pl@kamp.de | http://www.kamp.de

   Geschäftsführer: Heiner Lante | Michael Lante
   Amtsgericht Duisburg | HRB Nr. 12154
   USt-Id-Nr.: DE 120607556

...........................................................

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Stefan Hajnoczi]

On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
> I was thinking if it would be a good idea to zeroize all memory resources on system reset and
> madvise dontneed them afterwards. This would avoid system reset attacks in case the attacker
> has only access to the console of a vServer but not on the physical host and it would shrink
> RSS size of the vServer siginificantly.

I wonder if you'll hit weird OS installers or PXE clients that rely on
stashing stuff in memory across reset.

Stefan

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Peter Lieven]

On 13.06.2013 10:40, Stefan Hajnoczi wrote:
> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>> I was thinking if it would be a good idea to zeroize all memory resources on system reset and
>> madvise dontneed them afterwards. This would avoid system reset attacks in case the attacker
>> has only access to the console of a vServer but not on the physical host and it would shrink
>> RSS size of the vServer siginificantly.
> I wonder if you'll hit weird OS installers or PXE clients that rely on
> stashing stuff in memory across reset.
Mhh, that indeed would be weird.

What do you think of the idea in general? You concerns could be addresses by adding
a switch for this which defaults to off.

Peter

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Stefan Hajnoczi]

On Thu, Jun 13, 2013 at 10:46:39AM +0200, Peter Lieven wrote:
> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
> >On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
> >>I was thinking if it would be a good idea to zeroize all memory resources on system reset and
> >>madvise dontneed them afterwards. This would avoid system reset attacks in case the attacker
> >>has only access to the console of a vServer but not on the physical host and it would shrink
> >>RSS size of the vServer siginificantly.
> >I wonder if you'll hit weird OS installers or PXE clients that rely on
> >stashing stuff in memory across reset.
> Mhh, that indeed would be weird.
> 
> What do you think of the idea in general? You concerns could be addresses by adding
> a switch for this which defaults to off.

Any time we deviate from how real hardware behaves we end up in trouble
later on.  Something will depend on this behavior.

It is nice to reduce the RSS footprint on reboot in some cases, your
idea makes sense.  I think it should be disabled by default for
compatibility.

Stefan

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Peter Lieven]

On 13.06.2013 10:40, Stefan Hajnoczi wrote:
> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>> I was thinking if it would be a good idea to zeroize all memory resources on system reset and
>> madvise dontneed them afterwards. This would avoid system reset attacks in case the attacker
>> has only access to the console of a vServer but not on the physical host and it would shrink
>> RSS size of the vServer siginificantly.
> I wonder if you'll hit weird OS installers or PXE clients that rely on
> stashing stuff in memory across reset.
One point:
Wouldn't a memory test which some systems do at startup break these as well?

Peter

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Markus Armbruster]

Peter Lieven <pl@kamp.de> writes:

> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>> I was thinking if it would be a good idea to zeroize all memory
>>> resources on system reset and
>>> madvise dontneed them afterwards. This would avoid system reset
>>> attacks in case the attacker
>>> has only access to the console of a vServer but not on the physical
>>> host and it would shrink
>>> RSS size of the vServer siginificantly.
>> I wonder if you'll hit weird OS installers or PXE clients that rely on
>> stashing stuff in memory across reset.
> One point:
> Wouldn't a memory test which some systems do at startup break these as well?

Systems that distinguish between warm and cold boot (such as PCs)
generally run POST only on cold boot.

I'm not saying triggering warm reboot and expecting memory contents to
survive is a good idea, but it has been done.

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Anthony Liguori]

Markus Armbruster <armbru@redhat.com> writes:

> Peter Lieven <pl@kamp.de> writes:
>
>> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>>> I was thinking if it would be a good idea to zeroize all memory
>>>> resources on system reset and
>>>> madvise dontneed them afterwards. This would avoid system reset
>>>> attacks in case the attacker
>>>> has only access to the console of a vServer but not on the physical
>>>> host and it would shrink
>>>> RSS size of the vServer siginificantly.
>>> I wonder if you'll hit weird OS installers or PXE clients that rely on
>>> stashing stuff in memory across reset.
>> One point:
>> Wouldn't a memory test which some systems do at startup break these as well?
>
> Systems that distinguish between warm and cold boot (such as PCs)
> generally run POST only on cold boot.
>
> I'm not saying triggering warm reboot and expecting memory contents to
> survive is a good idea, but it has been done.

Doesn't kexec do a warm reboot stashing the new kernel somewhere in
memory?

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Paolo Bonzini]

Il 13/06/2013 07:56, Anthony Liguori ha scritto:
> Markus Armbruster <armbru@redhat.com> writes:
> 
>> Peter Lieven <pl@kamp.de> writes:
>>
>>> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>>>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>>>> I was thinking if it would be a good idea to zeroize all memory
>>>>> resources on system reset and
>>>>> madvise dontneed them afterwards. This would avoid system reset
>>>>> attacks in case the attacker
>>>>> has only access to the console of a vServer but not on the physical
>>>>> host and it would shrink
>>>>> RSS size of the vServer siginificantly.
>>>> I wonder if you'll hit weird OS installers or PXE clients that rely on
>>>> stashing stuff in memory across reset.
>>> One point:
>>> Wouldn't a memory test which some systems do at startup break these as well?
>>
>> Systems that distinguish between warm and cold boot (such as PCs)
>> generally run POST only on cold boot.
>>
>> I'm not saying triggering warm reboot and expecting memory contents to
>> survive is a good idea, but it has been done.
> 
> Doesn't kexec do a warm reboot stashing the new kernel somewhere in
> memory?

No, it undoes most of the hardware setup and jumps to the new kernel.
It never goes through the BIOS.

Paolo

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Christian Borntraeger]

On 13/06/13 13:56, Anthony Liguori wrote:
> Markus Armbruster <armbru@redhat.com> writes:
> 
>> Peter Lieven <pl@kamp.de> writes:
>>
>>> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>>>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>>>> I was thinking if it would be a good idea to zeroize all memory
>>>>> resources on system reset and
>>>>> madvise dontneed them afterwards. This would avoid system reset
>>>>> attacks in case the attacker
>>>>> has only access to the console of a vServer but not on the physical
>>>>> host and it would shrink
>>>>> RSS size of the vServer siginificantly.
>>>> I wonder if you'll hit weird OS installers or PXE clients that rely on
>>>> stashing stuff in memory across reset.
>>> One point:
>>> Wouldn't a memory test which some systems do at startup break these as well?
>>
>> Systems that distinguish between warm and cold boot (such as PCs)
>> generally run POST only on cold boot.
>>
>> I'm not saying triggering warm reboot and expecting memory contents to
>> survive is a good idea, but it has been done.
> 
> Doesn't kexec do a warm reboot stashing the new kernel somewhere in
> memory?

It does something like that on s390.
There is a diagnose instruction to the machine, that resets all
subsystems and cpus in a defined state, but lets the memory untouched.
So we want to be able to define the components of the system which we are 
going to reset and have two cases:
1. reset everything and clear the memory
2. just reset the cpus and devices, but leave the memory untouched

For case 2 we basically want to avoid memory clearing AND bios reloading

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Alexander Graf]

Am 14.06.2013 um 08:56 schrieb Christian Borntraeger <borntraeger@de.ibm.com>:

> On 13/06/13 13:56, Anthony Liguori wrote:
>> Markus Armbruster <armbru@redhat.com> writes:
>> 
>>> Peter Lieven <pl@kamp.de> writes:
>>> 
>>>> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>>>>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>>>>> I was thinking if it would be a good idea to zeroize all memory
>>>>>> resources on system reset and
>>>>>> madvise dontneed them afterwards. This would avoid system reset
>>>>>> attacks in case the attacker
>>>>>> has only access to the console of a vServer but not on the physical
>>>>>> host and it would shrink
>>>>>> RSS size of the vServer siginificantly.
>>>>> I wonder if you'll hit weird OS installers or PXE clients that rely on
>>>>> stashing stuff in memory across reset.
>>>> One point:
>>>> Wouldn't a memory test which some systems do at startup break these as well?
>>> 
>>> Systems that distinguish between warm and cold boot (such as PCs)
>>> generally run POST only on cold boot.
>>> 
>>> I'm not saying triggering warm reboot and expecting memory contents to
>>> survive is a good idea, but it has been done.
>> 
>> Doesn't kexec do a warm reboot stashing the new kernel somewhere in
>> memory?
> 
> It does something like that on s390.
> There is a diagnose instruction to the machine, that resets all
> subsystems and cpus in a defined state, but lets the memory untouched.
> So we want to be able to define the components of the system which we are 
> going to reset and have two cases:
> 1. reset everything and clear the memory
> 2. just reset the cpus and devices, but leave the memory untouched
> 
> For case 2 we basically want to avoid memory clearing AND bios reloading

Legacy 286 protected mode to real mode switching also happens through the CPU reset PIN, so there certainly is a need to distinguish.

Alex

> 
> 
> 

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Paolo Bonzini]

Il 14/06/2013 05:44, Alexander Graf ha scritto:
> Legacy 286 protected mode to real mode switching also happens through
> the CPU reset PIN, so there certainly is a need to distinguish.

That's a separate thing because devices aren't reset at all---not just
memory.

I have pending patches for that, they've been rotting in my tree for a
while. :)

Paolo

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Peter Lieven]

Am 14.06.2013 15:43, schrieb Paolo Bonzini:
> Il 14/06/2013 05:44, Alexander Graf ha scritto:
>> Legacy 286 protected mode to real mode switching also happens through
>> the CPU reset PIN, so there certainly is a need to distinguish.
> That's a separate thing because devices aren't reset at all---not just
> memory.
>
> I have pending patches for that, they've been rotting in my tree for a
> while. :)
Which strategy would you follow? Wait until your patches are in or
start with a memory sanitize patch that can be enabled via commandline

Peter

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[H. Peter Anvin]

Only on a real 286.  At least since 486 the legacy switch has been INIT, not RESET.

Alexander Graf <agraf@suse.de> wrote:

>
>
>Am 14.06.2013 um 08:56 schrieb Christian Borntraeger
><borntraeger@de.ibm.com>:
>
>> On 13/06/13 13:56, Anthony Liguori wrote:
>>> Markus Armbruster <armbru@redhat.com> writes:
>>> 
>>>> Peter Lieven <pl@kamp.de> writes:
>>>> 
>>>>> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>>>>>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>>>>>> I was thinking if it would be a good idea to zeroize all memory
>>>>>>> resources on system reset and
>>>>>>> madvise dontneed them afterwards. This would avoid system reset
>>>>>>> attacks in case the attacker
>>>>>>> has only access to the console of a vServer but not on the
>physical
>>>>>>> host and it would shrink
>>>>>>> RSS size of the vServer siginificantly.
>>>>>> I wonder if you'll hit weird OS installers or PXE clients that
>rely on
>>>>>> stashing stuff in memory across reset.
>>>>> One point:
>>>>> Wouldn't a memory test which some systems do at startup break
>these as well?
>>>> 
>>>> Systems that distinguish between warm and cold boot (such as PCs)
>>>> generally run POST only on cold boot.
>>>> 
>>>> I'm not saying triggering warm reboot and expecting memory contents
>to
>>>> survive is a good idea, but it has been done.
>>> 
>>> Doesn't kexec do a warm reboot stashing the new kernel somewhere in
>>> memory?
>> 
>> It does something like that on s390.
>> There is a diagnose instruction to the machine, that resets all
>> subsystems and cpus in a defined state, but lets the memory
>untouched.
>> So we want to be able to define the components of the system which we
>are 
>> going to reset and have two cases:
>> 1. reset everything and clear the memory
>> 2. just reset the cpus and devices, but leave the memory untouched
>> 
>> For case 2 we basically want to avoid memory clearing AND bios
>reloading
>
>Legacy 286 protected mode to real mode switching also happens through
>the CPU reset PIN, so there certainly is a need to distinguish.
>
>Alex
>
>> 
>> 
>> 

-- 
Sent from my mobile phone. Please excuse brevity and lack of formatting.

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Peter Lieven]

On 13.06.2013 12:55, Markus Armbruster wrote:
> Peter Lieven <pl@kamp.de> writes:
>
>> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>>> I was thinking if it would be a good idea to zeroize all memory
>>>> resources on system reset and
>>>> madvise dontneed them afterwards. This would avoid system reset
>>>> attacks in case the attacker
>>>> has only access to the console of a vServer but not on the physical
>>>> host and it would shrink
>>>> RSS size of the vServer siginificantly.
>>> I wonder if you'll hit weird OS installers or PXE clients that rely on
>>> stashing stuff in memory across reset.
>> One point:
>> Wouldn't a memory test which some systems do at startup break these as well?
> Systems that distinguish between warm and cold boot (such as PCs)
> generally run POST only on cold boot.
>
> I'm not saying triggering warm reboot and expecting memory contents to
> survive is a good idea, but it has been done.
so you would vote for not touching it or at least enable it only through
a cmdline paramter?

Peter

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Markus Armbruster]

Peter Lieven <pl@kamp.de> writes:

> On 13.06.2013 12:55, Markus Armbruster wrote:
>> Peter Lieven <pl@kamp.de> writes:
>>
>>> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>>>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>>>> I was thinking if it would be a good idea to zeroize all memory
>>>>> resources on system reset and
>>>>> madvise dontneed them afterwards. This would avoid system reset
>>>>> attacks in case the attacker
>>>>> has only access to the console of a vServer but not on the physical
>>>>> host and it would shrink
>>>>> RSS size of the vServer siginificantly.
>>>> I wonder if you'll hit weird OS installers or PXE clients that rely on
>>>> stashing stuff in memory across reset.
>>> One point:
>>> Wouldn't a memory test which some systems do at startup break these as well?
>> Systems that distinguish between warm and cold boot (such as PCs)
>> generally run POST only on cold boot.
>>
>> I'm not saying triggering warm reboot and expecting memory contents to
>> survive is a good idea, but it has been done.
> so you would vote for not touching it or at least enable it only through
> a cmdline paramter?

If you can demonstrate practical advantages of clearing memory, we can
talk about how to best do it, and whether it really needs to be
optional.

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Peter Lieven]

Am 13.06.2013 um 17:51 schrieb Markus Armbruster <armbru@redhat.com>:

> Peter Lieven <pl@kamp.de> writes:
> 
>> On 13.06.2013 12:55, Markus Armbruster wrote:
>>> Peter Lieven <pl@kamp.de> writes:
>>> 
>>>> On 13.06.2013 10:40, Stefan Hajnoczi wrote:
>>>>> On Thu, Jun 13, 2013 at 08:09:09AM +0200, Peter Lieven wrote:
>>>>>> I was thinking if it would be a good idea to zeroize all memory
>>>>>> resources on system reset and
>>>>>> madvise dontneed them afterwards. This would avoid system reset
>>>>>> attacks in case the attacker
>>>>>> has only access to the console of a vServer but not on the physical
>>>>>> host and it would shrink
>>>>>> RSS size of the vServer siginificantly.
>>>>> I wonder if you'll hit weird OS installers or PXE clients that rely on
>>>>> stashing stuff in memory across reset.
>>>> One point:
>>>> Wouldn't a memory test which some systems do at startup break these as well?
>>> Systems that distinguish between warm and cold boot (such as PCs)
>>> generally run POST only on cold boot.
>>> 
>>> I'm not saying triggering warm reboot and expecting memory contents to
>>> survive is a good idea, but it has been done.
>> so you would vote for not touching it or at least enable it only through
>> a cmdline paramter?
> 
> If you can demonstrate practical advantages of clearing memory, we can
> talk about how to best do it, and whether it really needs to be
> optional.

i was thinking of 2 things:

a) after some runtime the RSS size of a vServer grows up to its physical memory size. on a reset
we could zero out and free the memory.

b) one could think of attack scenario where an attacker has access to a vServer management
console but not to the physical vServer node or the vServer itself. in this case the attacker could hard reset the
vServer and boot a small rescue system and recover data from the vServer such as keys or other
sensitive data. if the memory is zeroed out this can be avoided. a common suggestion to mitigate
such attacks is to enable memory power on self-test because it overwrites the memory.

Peter

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Andreas Färber]

Hi,

Am 13.06.2013 08:09, schrieb Peter Lieven:
> I was thinking if it would be a good idea to zeroize all memory
> resources on system reset and
> madvise dontneed them afterwards.

The current way of not zeroing memory has led to discovery of some
firmware bugs that we wouldn't have found if QEMU defaulted to zeroing.

> This would avoid system reset attacks
> in case the attacker
> has only access to the console of a vServer but not on the physical host
> and it would shrink
> RSS size of the vServer siginificantly.

Apart from the guest issue Stefan brought up (so far by definition we do
a hard reset, so guests cannot assume soft reset semantics, but we
should keep our options open), would not zeroing while marking pages as
unused be an option? E.g., -reset-memory=DEADBEEF or some other
command-line-specifiable pattern, absence would mean current behavior.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [RFC] sanitize memory on system reset

[Peter Lieven]

On 13.06.2013 11:22, Andreas Färber wrote:
> Hi,
>
> Am 13.06.2013 08:09, schrieb Peter Lieven:
>> I was thinking if it would be a good idea to zeroize all memory
>> resources on system reset and
>> madvise dontneed them afterwards.
> The current way of not zeroing memory has led to discovery of some
> firmware bugs that we wouldn't have found if QEMU defaulted to zeroing.
The memory is zero at the start due to the use of mmap. Maybe
we need to add an option to add an initialization value anyway
because I am unsure if PERTURB works with mmap?!
>
>> This would avoid system reset attacks
>> in case the attacker
>> has only access to the console of a vServer but not on the physical host
>> and it would shrink
>> RSS size of the vServer siginificantly.
> Apart from the guest issue Stefan brought up (so far by definition we do
> a hard reset, so guests cannot assume soft reset semantics, but we
> should keep our options open), would not zeroing while marking pages as
> unused be an option? E.g., -reset-memory=DEADBEEF or some other
> command-line-specifiable pattern, absence would mean current behavior.
This would overwrite all contents with 0xdeadbeaf avoiding information
leak, but it would unnecessarily keep the memory alocated. So what
about an option -mem-sanitize with an optional parameter to write
a initialization value.

option missing -> no change
-mem-sanitize -> zeroize and madv_dontneed
-mem-sanitze=deadbeaf -> fill memory with 0xdeadbeaf

Where is the right postion to add this hook. qemu_system_reset() ?

Peter
>
> Regards,
> Andreas
>


-- 

Mit freundlichen Grüßen

Peter Lieven

...........................................................

   KAMP Netzwerkdienste GmbH
   Vestische Str. 89-91 | 46117 Oberhausen
   Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40
   pl@kamp.de | http://www.kamp.de

   Geschäftsführer: Heiner Lante | Michael Lante
   Amtsgericht Duisburg | HRB Nr. 12154
   USt-Id-Nr.: DE 120607556

...........................................................