[PATCH] powerpc: add Book E support to 64-bit hibernation

[PATCH] powerpc: add Book E support to 64-bit hibernation

[Wang Dongsheng]

Update the 64-bit hibernation code to support Book E CPUs.
Some registers and instructions are not defined for Book3e
(SDR reg, tlbia instruction).

SDR: Storage Description Register. Book3S and Book3E have different
address translation mode, we do not need HTABORG & HTABSIZE to
translate virtual address to real address.

More registers are saved in BookE-64bit.(TCR, SPRGx)

Signed-off-by: Wang Dongsheng <dongsheng.wang@freescale.com>
---
 arch/powerpc/kernel/swsusp_asm64.S |   64 ++++++++++++++++++++++++++++++++++-
 1 files changed, 62 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/kernel/swsusp_asm64.S b/arch/powerpc/kernel/swsusp_asm64.S
index 86ac1d9..608e4ceb 100644
--- a/arch/powerpc/kernel/swsusp_asm64.S
+++ b/arch/powerpc/kernel/swsusp_asm64.S
@@ -46,10 +46,29 @@
 #define SL_r29		0xe8
 #define SL_r30		0xf0
 #define SL_r31		0xf8
-#define SL_SIZE		SL_r31+8
+#define SL_SPRG0	0x100
+#define SL_SPRG1	0x108
+#define SL_SPRG2	0x110
+#define SL_SPRG3	0x118
+#define SL_SPRG4	0x120
+#define SL_SPRG5	0x128
+#define SL_SPRG6	0x130
+#define SL_SPRG7	0x138
+#define SL_TCR		0x140
+#define SL_SIZE		SL_TCR+8
 
 /* these macros rely on the save area being
  * pointed to by r11 */
+
+#define SAVE_SPR(register)		\
+	mfspr	r0,SPRN_##register	;\
+	std	r0,SL_##register(r11)
+#define RESTORE_SPR(register)		\
+	ld	r0,SL_##register(r11)	;\
+	mtspr	SPRN_##register,r0
+#define RESTORE_SPRG(n)			\
+	ld	r0,SL_SPRG##n(r11)	;\
+	mtsprg	n,r0
 #define SAVE_SPECIAL(special)		\
 	mf##special	r0		;\
 	std	r0, SL_##special(r11)
@@ -103,8 +122,21 @@ _GLOBAL(swsusp_arch_suspend)
 	SAVE_REGISTER(r30)
 	SAVE_REGISTER(r31)
 	SAVE_SPECIAL(MSR)
-	SAVE_SPECIAL(SDR1)
 	SAVE_SPECIAL(XER)
+#ifdef CONFIG_PPC_BOOK3S_64
+	SAVE_SPECIAL(SDR1)
+#else
+	SAVE_SPR(TCR)
+	/* Save SPRGs */
+	SAVE_SPR(SPRG0)
+	SAVE_SPR(SPRG1)
+	SAVE_SPR(SPRG2)
+	SAVE_SPR(SPRG3)
+	SAVE_SPR(SPRG4)
+	SAVE_SPR(SPRG5)
+	SAVE_SPR(SPRG6)
+	SAVE_SPR(SPRG7)
+#endif
 
 	/* we push the stack up 128 bytes but don't store the
 	 * stack pointer on the stack like a real stackframe */
@@ -151,6 +183,7 @@ copy_page_loop:
 	bne+	copyloop
 nothing_to_copy:
 
+#ifdef CONFIG_PPC_BOOK3S_64
 	/* flush caches */
 	lis	r3, 0x10
 	mtctr	r3
@@ -167,6 +200,7 @@ nothing_to_copy:
 	sync
 
 	tlbia
+#endif
 
 	ld	r11,swsusp_save_area_ptr@toc(r2)
 
@@ -208,16 +242,42 @@ nothing_to_copy:
 	RESTORE_REGISTER(r29)
 	RESTORE_REGISTER(r30)
 	RESTORE_REGISTER(r31)
+
+#ifdef CONFIG_PPC_BOOK3S_64
 	/* can't use RESTORE_SPECIAL(MSR) */
 	ld	r0, SL_MSR(r11)
 	mtmsrd	r0, 0
 	RESTORE_SPECIAL(SDR1)
+#else
+	/* Save SPRGs */
+	RESTORE_SPRG(0)
+	RESTORE_SPRG(1)
+	RESTORE_SPRG(2)
+	RESTORE_SPRG(3)
+	RESTORE_SPRG(4)
+	RESTORE_SPRG(5)
+	RESTORE_SPRG(6)
+	RESTORE_SPRG(7)
+
+	RESTORE_SPECIAL(MSR)
+
+	/* Restore TCR and clear any pending bits in TSR. */
+	RESTORE_SPR(TCR)
+	lis	r0, (TSR_ENW | TSR_WIS | TSR_DIS | TSR_FIS)@h
+	mtspr	SPRN_TSR,r0
+
+	/* Kick decrementer */
+	li	r0,1
+	mtdec	r0
+#endif
 	RESTORE_SPECIAL(XER)
 
 	sync
 
 	addi	r1,r1,-128
+#ifdef CONFIG_PPC_BOOK3S_64
 	bl	slb_flush_and_rebolt
+#endif
 	bl	do_after_copyback
 	addi	r1,r1,128
 
-- 
1.7.5.1

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Johannes Berg]

On Thu, 2013-03-14 at 11:36 +0800, Wang Dongsheng wrote:

> +#ifdef CONFIG_PPC_BOOK3S_64
>  	/* can't use RESTORE_SPECIAL(MSR) */
>  	ld	r0, SL_MSR(r11)
>  	mtmsrd	r0, 0

Unfortunately, I forgot the reason for this comment, and didn't put a
better one (almost 6 years ago!!)

> +	RESTORE_SPECIAL(MSR)

but maybe you also can't do this on BookE?

johannes

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Scott Wood]

On 03/14/2013 03:37:53 AM, Johannes Berg wrote:
> On Thu, 2013-03-14 at 11:36 +0800, Wang Dongsheng wrote:
>=20
> > +#ifdef CONFIG_PPC_BOOK3S_64
> >  	/* can't use RESTORE_SPECIAL(MSR) */
> >  	ld	r0, SL_MSR(r11)
> >  	mtmsrd	r0, 0
>=20
> Unfortunately, I forgot the reason for this comment, and didn't put a
> better one (almost 6 years ago!!)
>=20
> > +	RESTORE_SPECIAL(MSR)
>=20
> but maybe you also can't do this on BookE?

If it's because book3s needs mtmsrd instead of mtmsr, that doesn't =20
apply to booke.

-Scott=

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Johannes Berg]

On Thu, 2013-03-14 at 11:52 -0500, Scott Wood wrote:
> > > +#ifdef CONFIG_PPC_BOOK3S_64
> > >  	/* can't use RESTORE_SPECIAL(MSR) */
> > >  	ld	r0, SL_MSR(r11)
> > >  	mtmsrd	r0, 0
> > 
> > Unfortunately, I forgot the reason for this comment, and didn't put a
> > better one (almost 6 years ago!!)

> If it's because book3s needs mtmsrd instead of mtmsr, that doesn't  
> apply to booke.

Indeed, looking at the code again now that seems pretty obvious.

Looking at the patch again, I'd be a little concerned about the lack of
cache flushing, seems a bit odd but I'm sure you know what you're doing
(and I don't know book3e at all, and hardly remember book3s -- or even
that name...) :-)

johannes

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Scott Wood]

On 03/15/2013 10:22:06 AM, Johannes Berg wrote:
> On Thu, 2013-03-14 at 11:52 -0500, Scott Wood wrote:
> > > > +#ifdef CONFIG_PPC_BOOK3S_64
> > > >  	/* can't use RESTORE_SPECIAL(MSR) */
> > > >  	ld	r0, SL_MSR(r11)
> > > >  	mtmsrd	r0, 0
> > >
> > > Unfortunately, I forgot the reason for this comment, and didn't =20
> put a
> > > better one (almost 6 years ago!!)
>=20
> > If it's because book3s needs mtmsrd instead of mtmsr, that doesn't
> > apply to booke.
>=20
> Indeed, looking at the code again now that seems pretty obvious.
>=20
> Looking at the patch again, I'd be a little concerned about the lack =20
> of
> cache flushing, seems a bit odd but I'm sure you know what you're =20
> doing
> (and I don't know book3e at all, and hardly remember book3s -- or even
> that name...) :-)

Could you elaborate on why book3s flushes the way it does?  What's =20
special about the first 32 MiB?  If it's to cover kernel code, why =20
would that be changing from what's already there?

-Scott=

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Johannes Berg]

On Mon, 2013-03-18 at 17:12 -0500, Scott Wood wrote:

> Could you elaborate on why book3s flushes the way it does?  What's  
> special about the first 32 MiB?  If it's to cover kernel code, why  
> would that be changing from what's already there?

I was going to say I have no idea, but looking at it again ... this is
in the *resume* code, not the suspend code as I'd assumed, and on resume
I guess I felt it was safer to not assume it didn't change, since it
could be a slightly different kernel that loaded and restored the
hibernation image? It should be the same one, so I guess it should be
exactly the same code, but I guess I wanted to make sure there wasn't
anything weird there. As for why it'd be 32 MiB? No idea. Although that
really ought to flush all your possible caches anyway, I guess.

johannes

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Scott Wood]

On 03/19/2013 03:55:34 PM, Johannes Berg wrote:
> On Mon, 2013-03-18 at 17:12 -0500, Scott Wood wrote:
>=20
> > Could you elaborate on why book3s flushes the way it does?  What's
> > special about the first 32 MiB?  If it's to cover kernel code, why
> > would that be changing from what's already there?
>=20
> I was going to say I have no idea, but looking at it again ... this is
> in the *resume* code, not the suspend code as I'd assumed, and on =20
> resume
> I guess I felt it was safer to not assume it didn't change, since it
> could be a slightly different kernel that loaded and restored the
> hibernation image?

Wouldn't that be doomed for other reasons?

I wonder about kernel modules, though flushing 32 MiB wouldn't be =20
adequate there.

> It should be the same one, so I guess it should be
> exactly the same code, but I guess I wanted to make sure there wasn't
> anything weird there. As for why it'd be 32 MiB? No idea. Although =20
> that
> really ought to flush all your possible caches anyway, I guess.

It's not a displacement flush (i.e. you don't do a separate load pass =20
first) -- it just flushes lines if they happen to be present, and =20
leaves alone anything outside that range.  Given that you just finished =20
copying a bunch of data, most likely what's in the cache is the last =20
bit of data you copied.

-Scott=

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Johannes Berg]

On Tue, 2013-03-19 at 16:10 -0500, Scott Wood wrote:

> > I was going to say I have no idea, but looking at it again ... this is
> > in the *resume* code, not the suspend code as I'd assumed, and on  
> > resume
> > I guess I felt it was safer to not assume it didn't change, since it
> > could be a slightly different kernel that loaded and restored the
> > hibernation image?
> 
> Wouldn't that be doomed for other reasons?

Most likely, yeah.

> I wonder about kernel modules, though flushing 32 MiB wouldn't be  
> adequate there.

Good question, but would they be running? You have to have everything
built in that you need to load the image? Or maybe not, with the
userspace image restoration that became possible at some point...

> > It should be the same one, so I guess it should be
> > exactly the same code, but I guess I wanted to make sure there wasn't
> > anything weird there. As for why it'd be 32 MiB? No idea. Although  
> > that
> > really ought to flush all your possible caches anyway, I guess.
> 
> It's not a displacement flush (i.e. you don't do a separate load pass  
> first) -- it just flushes lines if they happen to be present, and  
> leaves alone anything outside that range.  Given that you just finished  
> copying a bunch of data, most likely what's in the cache is the last  
> bit of data you copied.

Oops, good point.

Maybe there's a way to completely flush the (i)cache? :-)

johannes

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Scott Wood]

On 03/19/2013 04:22:07 PM, Johannes Berg wrote:
> On Tue, 2013-03-19 at 16:10 -0500, Scott Wood wrote:
>=20
> > > I was going to say I have no idea, but looking at it again ... =20
> this is
> > > in the *resume* code, not the suspend code as I'd assumed, and on
> > > resume
> > > I guess I felt it was safer to not assume it didn't change, since =20
> it
> > > could be a slightly different kernel that loaded and restored the
> > > hibernation image?
> >
> > Wouldn't that be doomed for other reasons?
>=20
> Most likely, yeah.
>=20
> > I wonder about kernel modules, though flushing 32 MiB wouldn't be
> > adequate there.
>=20
> Good question, but would they be running? You have to have everything
> built in that you need to load the image? Or maybe not, with the
> userspace image restoration that became possible at some point...

Is that all that's being restored in this step, or would we be loading =20
all modules that were loaded before suspend (as they're normally not =20
swappable)?  I'm not too familiar with what gets saved where.

> > It's not a displacement flush (i.e. you don't do a separate load =20
> pass
> > first) -- it just flushes lines if they happen to be present, and
> > leaves alone anything outside that range.  Given that you just =20
> finished
> > copying a bunch of data, most likely what's in the cache is the last
> > bit of data you copied.
>=20
> Oops, good point.
>=20
> Maybe there's a way to completely flush the (i)cache? :-)

There is, but it's platform-dependent, and not pleasant on our chips =20
(need a displacement flush for L1, and sometimes errata are involved =20
for L2).

-Scott=

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Johannes Berg]

On Tue, 2013-03-19 at 17:16 -0500, Scott Wood wrote:

> > > I wonder about kernel modules, though flushing 32 MiB wouldn't be
> > > adequate there.
> > 
> > Good question, but would they be running? You have to have everything
> > built in that you need to load the image? Or maybe not, with the
> > userspace image restoration that became possible at some point...
> 
> Is that all that's being restored in this step, or would we be loading  
> all modules that were loaded before suspend (as they're normally not  
> swappable)?  I'm not too familiar with what gets saved where.

Yes, they would be restored since full memory is restored, but the
kernel doing the restore hasn't typically had a chance to load modules.
Although with uswsusp there probably are ways it can already have.

> > Maybe there's a way to completely flush the (i)cache? :-)
> 
> There is, but it's platform-dependent, and not pleasant on our chips  
> (need a displacement flush for L1, and sometimes errata are involved  
> for L2).

:-)

I'll be the last one to tell you what to do, and I really don't remember
for sure why I did the flushing this way. It may very well be wrong for
what I thought I was doing, but it doesn't matter because I didn't need
to be doing it, I can't say any more now, too long ago. Sorry. May you
should ask Rafael what he thinks about flushing.

johannes

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Scott Wood]

On 03/22/2013 05:58:37 AM, Johannes Berg wrote:
> On Tue, 2013-03-19 at 17:16 -0500, Scott Wood wrote:
>=20
> > > > I wonder about kernel modules, though flushing 32 MiB wouldn't =20
> be
> > > > adequate there.
> > >
> > > Good question, but would they be running? You have to have =20
> everything
> > > built in that you need to load the image? Or maybe not, with the
> > > userspace image restoration that became possible at some point...
> >
> > Is that all that's being restored in this step, or would we be =20
> loading
> > all modules that were loaded before suspend (as they're normally not
> > swappable)?  I'm not too familiar with what gets saved where.
>=20
> Yes, they would be restored since full memory is restored, but the
> kernel doing the restore hasn't typically had a chance to load =20
> modules.
> Although with uswsusp there probably are ways it can already have.

If we're loading executable code that wasn't there before[1], we need =20
to flush -- but there's no reason why modules would only be in the =20
lower 32 MiB.

-Scott

[1] within the resume process, not before suspend=

RE: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Wang Dongsheng-B40534]

Hi scott & Johannes,

Thanks for reviewing.

@scott, About this patch, could you please help ack this patch?

> -----Original Message-----
> From: Wood Scott-B07421
> Sent: Saturday, March 23, 2013 6:15 AM
> To: Johannes Berg
> Cc: Wang Dongsheng-B40534; linuxppc-dev@lists.ozlabs.org
> Subject: Re: [PATCH] powerpc: add Book E support to 64-bit hibernation
>=20
> On 03/22/2013 05:58:37 AM, Johannes Berg wrote:
> > On Tue, 2013-03-19 at 17:16 -0500, Scott Wood wrote:
> >
> > > > > I wonder about kernel modules, though flushing 32 MiB wouldn't
> > be
> > > > > adequate there.
> > > >
> > > > Good question, but would they be running? You have to have
> > everything
> > > > built in that you need to load the image? Or maybe not, with the
> > > > userspace image restoration that became possible at some point...
> > >
> > > Is that all that's being restored in this step, or would we be
> > loading
> > > all modules that were loaded before suspend (as they're normally not
> > > swappable)?  I'm not too familiar with what gets saved where.
> >
> > Yes, they would be restored since full memory is restored, but the
> > kernel doing the restore hasn't typically had a chance to load
> > modules.
> > Although with uswsusp there probably are ways it can already have.
>=20
> If we're loading executable code that wasn't there before[1], we need to
> flush -- but there's no reason why modules would only be in the lower 32
> MiB.
>=20
> -Scott
>=20
> [1] within the resume process, not before suspend

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Scott Wood]

On 04/02/2013 12:28:40 AM, Wang Dongsheng-B40534 wrote:
> Hi scott & Johannes,
>=20
> Thanks for reviewing.
>=20
> @scott, About this patch, could you please help ack this patch?

Please investigate the issue of whether we are loading kernel module =20
code in this step, and whether cache flushing is needed as a result.

Is there any chance we could even be loading user code here, e.g. if =20
it's mlocked and thus couldn't be swapped out?

-Scott=

RE: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Wang Dongsheng-B40534]

> -----Original Message-----
> From: Wood Scott-B07421
> Sent: Wednesday, April 03, 2013 8:35 AM
> To: Wang Dongsheng-B40534
> Cc: Wood Scott-B07421; Johannes Berg; linuxppc-dev@lists.ozlabs.org
> Subject: Re: [PATCH] powerpc: add Book E support to 64-bit hibernation
>=20
> On 04/02/2013 12:28:40 AM, Wang Dongsheng-B40534 wrote:
> > Hi scott & Johannes,
> >
> > Thanks for reviewing.
> >
> > @scott, About this patch, could you please help ack this patch?
>=20
> Please investigate the issue of whether we are loading kernel module
> code in this step, and whether cache flushing is needed as a result.
>=20
Sorry, I am not very clear what you mean.
When the kernel boot end, modprobe some xx.ko?

> Is there any chance we could even be loading user code here, e.g. if
> it's mlocked and thus couldn't be swapped out?

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Scott Wood]

On 04/03/2013 12:36:41 AM, Wang Dongsheng-B40534 wrote:
>=20
>=20
> > -----Original Message-----
> > From: Wood Scott-B07421
> > Sent: Wednesday, April 03, 2013 8:35 AM
> > To: Wang Dongsheng-B40534
> > Cc: Wood Scott-B07421; Johannes Berg; linuxppc-dev@lists.ozlabs.org
> > Subject: Re: [PATCH] powerpc: add Book E support to 64-bit =20
> hibernation
> >
> > On 04/02/2013 12:28:40 AM, Wang Dongsheng-B40534 wrote:
> > > Hi scott & Johannes,
> > >
> > > Thanks for reviewing.
> > >
> > > @scott, About this patch, could you please help ack this patch?
> >
> > Please investigate the issue of whether we are loading kernel module
> > code in this step, and whether cache flushing is needed as a result.
> >
> Sorry, I am not very clear what you mean.
> When the kernel boot end, modprobe some xx.ko?

Suppose, before the kernel was suspended, modules had been loaded.  At =20
what point do those modules get restored, and when does the cache get =20
flushed?

-Scott=

RE: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Wang Dongsheng-B40534]

> -----Original Message-----
> From: Wood Scott-B07421
> Sent: Thursday, April 04, 2013 4:16 AM
> To: Wang Dongsheng-B40534
> Cc: Wood Scott-B07421; Johannes Berg; linuxppc-dev@lists.ozlabs.org
> Subject: Re: [PATCH] powerpc: add Book E support to 64-bit hibernation
>=20
> On 04/03/2013 12:36:41 AM, Wang Dongsheng-B40534 wrote:
> >
> >
> > > -----Original Message-----
> > > From: Wood Scott-B07421
> > > Sent: Wednesday, April 03, 2013 8:35 AM
> > > To: Wang Dongsheng-B40534
> > > Cc: Wood Scott-B07421; Johannes Berg; linuxppc-dev@lists.ozlabs.org
> > > Subject: Re: [PATCH] powerpc: add Book E support to 64-bit
> > hibernation
> > >
> > > On 04/02/2013 12:28:40 AM, Wang Dongsheng-B40534 wrote:
> > > > Hi scott & Johannes,
> > > >
> > > > Thanks for reviewing.
> > > >
> > > > @scott, About this patch, could you please help ack this patch?
> > >
> > > Please investigate the issue of whether we are loading kernel module
> > > code in this step, and whether cache flushing is needed as a result.
> > >
> > Sorry, I am not very clear what you mean.
> > When the kernel boot end, modprobe some xx.ko?
>=20
> Suppose, before the kernel was suspended, modules had been loaded.  At
> what point do those modules get restored, and when does the cache get
> flushed?
>=20
Before the kernel was suspended, modules had been loaded, the modules is
already in memory. And /lib/modules/* is belong to vfs.
When suspend to disk, all used pages will be saved.(Include VFS, Loaded mod=
ules)
When restore, the kernel will not modprobe again.
The non-bootcpu will restore all pages.(Include VFS, Loaded modules)

So, It does not need to flush.

Re: [PATCH] powerpc: add Book E support to 64-bit hibernation

[Scott Wood]

On 04/06/2013 10:01:45 PM, Wang Dongsheng-B40534 wrote:
>=20
>=20
> > -----Original Message-----
> > From: Wood Scott-B07421
> > Sent: Thursday, April 04, 2013 4:16 AM
> > To: Wang Dongsheng-B40534
> > Cc: Wood Scott-B07421; Johannes Berg; linuxppc-dev@lists.ozlabs.org
> > Subject: Re: [PATCH] powerpc: add Book E support to 64-bit =20
> hibernation
> >
> > On 04/03/2013 12:36:41 AM, Wang Dongsheng-B40534 wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Wood Scott-B07421
> > > > Sent: Wednesday, April 03, 2013 8:35 AM
> > > > To: Wang Dongsheng-B40534
> > > > Cc: Wood Scott-B07421; Johannes Berg; =20
> linuxppc-dev@lists.ozlabs.org
> > > > Subject: Re: [PATCH] powerpc: add Book E support to 64-bit
> > > hibernation
> > > >
> > > > On 04/02/2013 12:28:40 AM, Wang Dongsheng-B40534 wrote:
> > > > > Hi scott & Johannes,
> > > > >
> > > > > Thanks for reviewing.
> > > > >
> > > > > @scott, About this patch, could you please help ack this =20
> patch?
> > > >
> > > > Please investigate the issue of whether we are loading kernel =20
> module
> > > > code in this step, and whether cache flushing is needed as a =20
> result.
> > > >
> > > Sorry, I am not very clear what you mean.
> > > When the kernel boot end, modprobe some xx.ko?
> >
> > Suppose, before the kernel was suspended, modules had been loaded.  =20
> At
> > what point do those modules get restored, and when does the cache =20
> get
> > flushed?
> >
> Before the kernel was suspended, modules had been loaded, the modules =20
> is
> already in memory.

They *were* in memory, until the hardware was powered down.

> And /lib/modules/* is belong to vfs.

Huh?  I'm talking about modules that have been loaded, not where in the =20
filesystem they were loaded from.  Loading a module is not like =20
mmap()ing a file.

> When suspend to disk, all used pages will be saved.(Include VFS, =20
> Loaded modules)
> When restore, the kernel will not modprobe again.

Of course it won't modprobe again.  Still, at some point during the =20
resume process, the code has to be loaded from disk into RAM.  What I =20
don't know is if this is where that happens.

> The non-bootcpu will restore all pages.(Include VFS, Loaded modules)

I don't know what "non-bootcpu" has to do with anything.  What matters =20
is what piece of code does the restoring, and if the cache flush =20
properly happens then.

-Scott=