Memory image save/restore

Memory image save/restore

[Chris Lalancette]

Hello all,

    I have been trying to implement some sort of save/restore kernel 
memory image for the linux kernel (x86 only right now), without much 
success.  Let me explain the situation:

I have a hardware device that I can generate interrupts with.  I also 
have a machine with 512M of memory, and I am passing the kernel the 
command line mem=256M.  My idea is to generate an interrupt with the 
hardware device, and then inside of the interrupt handler make a copy of 
the entire contents of RAM into the unused upper 256M of memory; later 
on, with another interrupt, I would like to restore that previously 
saved memory image.  This way we can go "back in time", similar to what 
software suspend is doing, but without as many constraints (i.e. we have 
a hardware interrupt to work with, we reserved the same amount of 
physical memory to use, etc.).  Before I went much further, I figured I 
would ask if anyone on the list has tried this, and if there are any 
reasons why this is not possible.

Thank you,
Chris Lalancette

P.S.  If you respond to the list, please CC me; I am not subscribed

Re: Memory image save/restore

[Paulo Marques]

Chris Lalancette wrote:

> Hello all,
> 
>    I have been trying to implement some sort of save/restore kernel 
> memory image for the linux kernel (x86 only right now), without much 
> success.  Let me explain the situation:
> 
> I have a hardware device that I can generate interrupts with.  I also 
> have a machine with 512M of memory, and I am passing the kernel the 
> command line mem=256M.  My idea is to generate an interrupt with the 
> hardware device, and then inside of the interrupt handler make a copy of 
> the entire contents of RAM into the unused upper 256M of memory; later 
> on, with another interrupt, I would like to restore that previously 
> saved memory image.  This way we can go "back in time", similar to what 
> software suspend is doing, but without as many constraints (i.e. we have 
> a hardware interrupt to work with, we reserved the same amount of 
> physical memory to use, etc.).  Before I went much further, I figured I 
> would ask if anyone on the list has tried this, and if there are any 
> reasons why this is not possible.

You're assuming that the state of the memory is the *state* of the entire system.

This fails because there is a lot of state information in hardware registers, 
external peripheral devices, etc., etc.

-- 
Paulo Marques - www.grupopie.com
"In a world without walls and fences who needs windows and gates?"

Re: Memory image save/restore

[Chris Lalancette]

Paulo,
    Thanks for responding.  You are right in that the state of memory 
isn't the state of the entire machine; however, for instance, the 
swsusp2 project can save the contents of memory out to disk, go to 
sleep, and then resume to it later.  Basically, what I am trying to do 
is something like swsusp2, with less restrictions; I know I have another 
region the size of physical memory to work with, I know I am executing 
from an interrupt handler, and at the end I don't want to go to sleep, 
just continue on.  Then I might want to use that memory image later.

Thanks,
Chris Lalancette

pmarques@grupopie.com wrote:

> Chris Lalancette wrote:
>
>> Hello all,
>>
>>    I have been trying to implement some sort of save/restore kernel 
>> memory image for the linux kernel (x86 only right now), without much 
>> success.  Let me explain the situation:
>>
>> I have a hardware device that I can generate interrupts with.  I also 
>> have a machine with 512M of memory, and I am passing the kernel the 
>> command line mem=256M.  My idea is to generate an interrupt with the 
>> hardware device, and then inside of the interrupt handler make a copy 
>> of the entire contents of RAM into the unused upper 256M of memory; 
>> later on, with another interrupt, I would like to restore that 
>> previously saved memory image.  This way we can go "back in time", 
>> similar to what software suspend is doing, but without as many 
>> constraints (i.e. we have a hardware interrupt to work with, we 
>> reserved the same amount of physical memory to use, etc.).  Before I 
>> went much further, I figured I would ask if anyone on the list has 
>> tried this, and if there are any reasons why this is not possible.
>
>
> You're assuming that the state of the memory is the *state* of the 
> entire system.
>
> This fails because there is a lot of state information in hardware 
> registers, external peripheral devices, etc., etc.
>

Re: Memory image save/restore

[Paulo Marques]

Chris Lalancette wrote:

> Paulo,
>    Thanks for responding.  You are right in that the state of memory 
> isn't the state of the entire machine; however, for instance, the 
> swsusp2 project can save the contents of memory out to disk, go to 
> sleep, and then resume to it later.  Basically, what I am trying to do 
> is something like swsusp2, with less restrictions; I know I have another 
> region the size of physical memory to work with, I know I am executing 
> from an interrupt handler, and at the end I don't want to go to sleep, 
> just continue on.  Then I might want to use that memory image later.


That is not *all* the swsusp2 project does. Probably the biggest challenge in 
the software suspend project is to get all the devices to acknowledge that they 
are being "suspended" and to have them resume properly.

Just as an example, imagine that the usb host controller is currently 
dispatching an urb to a device. The kernel in-memory structures reflect this 
occurrence. The kernel is ready for the interrupt the controller will generate 
to update these structures.

You now interrupt, and save the memory contents. You let everything proceed, and 
after a while try to recover the previous memory contents.

Now the kernel is expecting the interrupt from the controller, regarding an urb 
that the controller has dispatched a long time ago. The state of the controller 
and the kernel in-memory structures is out of sync.

This is just a small example. I could think of this sort of examples for disk 
reads/writes and IDE/SCSI controllers, accelerated graphics adapters, etc., etc. 
Even much simpler examples, like keyboard state (a scancode was just received to 
press a key, and the corresponding release is lost), internal timer/clock, etc. 
will cause you problems just the same.

However, I'm not an expert in this area. Maybe someone like Nigel Cunningham or 
Pavel Machek can explain better the problems that arise from trying to save the 
complete *state* of the system. I just wrote this to try to save these guys the 
trouble, since they can occupy their time better in developing the damn thing :)

-- 
Paulo Marques - www.grupopie.com
"In a world without walls and fences who needs windows and gates?"

Re: Memory image save/restore

[Adam Kropelin]

On Tue, Apr 13, 2004 at 02:22:47PM -0400, Chris Lalancette wrote:
> Paulo,
>     Thanks for responding.  You are right in that the state of memory 
> isn't the state of the entire machine; however, for instance, the 
> swsusp2 project can save the contents of memory out to disk, go to 
> sleep, and then resume to it later.

You're forgetting an important step. swsusp does not just haul off and
save the contents of memory. It has to quiesce the entire system first
by getting all hardware devices into a known state. THAT is the tricky
bit, with deadlock lurking around every corner.

> Basically, what I am trying to do 
> is something like swsusp2, with less restrictions; I know I have another 
> region the size of physical memory to work with, I know I am executing 
> from an interrupt handler, and at the end I don't want to go to sleep, 
> just continue on.

That's relatively easy.

> Then I might want to use that memory image later.

That's hard. REALLY, REALLY hard. Impossibly hard.

Make no mistake about it: You need to save and restore HARDWARE state as
well as software state (i.e. RAM image). The only practical way to do
that is to quiesce the system before you snapshot it. Your case is even
harder because you essentially want to revert to a previous point in time.
That means reverting fun things like your disk images (since the layout
of the data in your filesystem no longer matches what's in your outdated
RAM image) and how are you going to suck all those tx'ed network packets
back off the wire, anyway?

Your "simplifications" are really complexifications.

--Adam


> pmarques@grupopie.com wrote:
> 
> > Chris Lalancette wrote:
> >
> >> Hello all,
> >>
> >>    I have been trying to implement some sort of save/restore kernel 
> >> memory image for the linux kernel (x86 only right now), without much 
> >> success.  Let me explain the situation:
> >>
> >> I have a hardware device that I can generate interrupts with.  I also 
> >> have a machine with 512M of memory, and I am passing the kernel the 
> >> command line mem=256M.  My idea is to generate an interrupt with the 
> >> hardware device, and then inside of the interrupt handler make a copy 
> >> of the entire contents of RAM into the unused upper 256M of memory; 
> >> later on, with another interrupt, I would like to restore that 
> >> previously saved memory image.  This way we can go "back in time", 
> >> similar to what software suspend is doing, but without as many 
> >> constraints (i.e. we have a hardware interrupt to work with, we 
> >> reserved the same amount of physical memory to use, etc.).  Before I 
> >> went much further, I figured I would ask if anyone on the list has 
> >> tried this, and if there are any reasons why this is not possible.
> >
> >
> > You're assuming that the state of the memory is the *state* of the 
> > entire system.
> >
> > This fails because there is a lot of state information in hardware 
> > registers, external peripheral devices, etc., etc.
> >

Re: Memory image save/restore

[Pavel Machek]

Hi!

> >     Thanks for responding.  You are right in that the state of memory 
> > isn't the state of the entire machine; however, for instance, the 
> > swsusp2 project can save the contents of memory out to disk, go to 
> > sleep, and then resume to it later.
> 
> You're forgetting an important step. swsusp does not just haul off and
> save the contents of memory. It has to quiesce the entire system first
> by getting all hardware devices into a known state. THAT is the tricky
> bit, with deadlock lurking around every corner.
> 
> > Basically, what I am trying to do 
> > is something like swsusp2, with less restrictions; I know I have another 
> > region the size of physical memory to work with, I know I am executing 
> > from an interrupt handler, and at the end I don't want to go to sleep, 
> > just continue on.
> 
> That's relatively easy.
> 
> > Then I might want to use that memory image later.
> 
> That's hard. REALLY, REALLY hard. Impossibly hard.

Its not impossible. You basically may not write to disk after you save
snapshot... Or you can keep the logs to be able to revert disk state.
Its possible but pretty hard.

Okay, make it impossible if that machine is connected to internet.

> RAM image) and how are you going to suck all those tx'ed network packets
> back off the wire, anyway?


Heh... It would actually make for good april-fools rfc :-).


You could make machine where any action can be undone within five seconds...
...but you'd need mdelay(5000) in send packet path, and that would kind-of suck.

				Pavel

-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: Memory image save/restore

[Pavel Machek]

Hi!

>    I have been trying to implement some sort of save/restore kernel 
> memory image for the linux kernel (x86 only right now), without much 
> success.  Let me explain the situation:
> 
> I have a hardware device that I can generate interrupts with.  I also 
> have a machine with 512M of memory, and I am passing the kernel the 
> command line mem=256M.  My idea is to generate an interrupt with the 
> hardware device, and then inside of the interrupt handler make a copy 
> of the entire contents of RAM into the unused upper 256M of memory; 
> later on, with another interrupt, I would like to restore that 
> previously saved memory image.  This way we can go "back in time", 
> similar to what software suspend is doing, but without as many 
> constraints (i.e. we have a hardware interrupt to work with, we 
> reserved the same amount of physical memory to use, etc.).  Before I 
> went much further, I figured I would ask if anyone on the list has 
> tried this, and if there are any reasons why this is not possible.

As swsusp shows, it definitely is possible...

...if you can live with system unresponsive for few seconds while it
does snapshotting.

Creating "bastardized swsusp" would actually be good way to start.
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms