Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks

Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks

[Jon Hunter]

Hi Ard,

On 28/12/2021 14:39, Geert Uytterhoeven wrote:

...

>> As i don't have access to this hardware, I am going to have to rely on
>> someone who does to debug this further. The only alternative is
>> marking CONFIG_VMAP_STACK broken on MACH_EXYNOS but that would be
>> unfortunate.
> 
> Wish I had seen this thread before...
> 
> I've just bisected a resume after s2ram failure on R-Car Gen2 to the same
> commit a1c510d0adc604bb ("ARM: implement support for vmap'ed stacks")
> in arm/for-next.
> 
> Expected output:
> 
>      PM: suspend entry (deep)
>      Filesystems sync: 0.000 seconds
>      Freezing user space processes ... (elapsed 0.010 seconds) done.
>      OOM killer disabled.
>      Freezing remaining freezable tasks ... (elapsed 0.009 seconds) done.
>      Disabling non-boot CPUs ...
> 
> [system suspended, this is also where it hangs on failure]
> 
>      Enabling non-boot CPUs ...
>      CPU1 is up
>      sh-eth ee700000.ethernet eth0: Link is Down
>      Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY
> driver (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=193)
>      OOM killer enabled.
>      Restarting tasks ... done.
>      PM: suspend exit
> 
> Both wake-on-LAN and wake-up by gpio-keys fail.
> Nothing interesting in the kernel log, cfr. above.
> 
> Disabling CONFIG_VMAP_STACK fixes the issue for me.
> 
> Just like arch/arm/mach-exynos/ (and others), arch/arm/mach-shmobile/
> has several *.S files related to secondary CPU bringup.


This is also breaking suspend on our 32-bit Tegra platforms. Reverting 
this change on top of -next fixes the problem.

Cheers
Jon

-- 
nvpublic

Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks

[Ard Biesheuvel]

On Wed, 5 Jan 2022 at 12:08, Jon Hunter <jonathanh@nvidia.com> wrote:
>
> Hi Ard,
>
> On 28/12/2021 14:39, Geert Uytterhoeven wrote:
>
> ...
>
> >> As i don't have access to this hardware, I am going to have to rely on
> >> someone who does to debug this further. The only alternative is
> >> marking CONFIG_VMAP_STACK broken on MACH_EXYNOS but that would be
> >> unfortunate.
> >
> > Wish I had seen this thread before...
> >
> > I've just bisected a resume after s2ram failure on R-Car Gen2 to the same
> > commit a1c510d0adc604bb ("ARM: implement support for vmap'ed stacks")
> > in arm/for-next.
> >
> > Expected output:
> >
> >      PM: suspend entry (deep)
> >      Filesystems sync: 0.000 seconds
> >      Freezing user space processes ... (elapsed 0.010 seconds) done.
> >      OOM killer disabled.
> >      Freezing remaining freezable tasks ... (elapsed 0.009 seconds) done.
> >      Disabling non-boot CPUs ...
> >
> > [system suspended, this is also where it hangs on failure]
> >
> >      Enabling non-boot CPUs ...
> >      CPU1 is up
> >      sh-eth ee700000.ethernet eth0: Link is Down
> >      Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY
> > driver (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=193)
> >      OOM killer enabled.
> >      Restarting tasks ... done.
> >      PM: suspend exit
> >
> > Both wake-on-LAN and wake-up by gpio-keys fail.
> > Nothing interesting in the kernel log, cfr. above.
> >
> > Disabling CONFIG_VMAP_STACK fixes the issue for me.
> >
> > Just like arch/arm/mach-exynos/ (and others), arch/arm/mach-shmobile/
> > has several *.S files related to secondary CPU bringup.
>
>
> This is also breaking suspend on our 32-bit Tegra platforms. Reverting
> this change on top of -next fixes the problem.
>

Thanks for the report.

It would be helpful if you could provide some more context:
- does it happen on a LPAE build too?
- does it only happen on SMP capable systems?
- does it reproduce on such systems when using only a single CPU?
(i.e., pass 'nosmp' on the kernel command line)
- when passing 'no_console_suspend' on the kernel command line, are
any useful diagnostics produced?
- is there any way you could tell whether the crash/hang (assuming
that is what you are observing) occurs on the suspend path or on
resume?
- any other observations that could narrow this down?

Thanks,
Ard.

Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks

[Jon Hunter]

On 05/01/2022 11:12, Ard Biesheuvel wrote:

...

> Thanks for the report.
> 
> It would be helpful if you could provide some more context:
> - does it happen on a LPAE build too?
> - does it only happen on SMP capable systems?

These are all SMP systems.

> - does it reproduce on such systems when using only a single CPU?
> (i.e., pass 'nosmp' on the kernel command line)

I would need to try this.

> - when passing 'no_console_suspend' on the kernel command line, are
> any useful diagnostics produced?
> - is there any way you could tell whether the crash/hang (assuming
> that is what you are observing) occurs on the suspend path or on
> resume?
> - any other observations that could narrow this down?

I can run the above and let you know what I find.

Cheers
Jon

-- 
nvpublic

Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks

[Russell King (Oracle)]

On Wed, Jan 05, 2022 at 11:33:48AM +0000, Jon Hunter wrote:
> On 05/01/2022 11:12, Ard Biesheuvel wrote:
> > Thanks for the report.
> > 
> > It would be helpful if you could provide some more context:
> > - does it happen on a LPAE build too?
> > - does it only happen on SMP capable systems?
> 
> These are all SMP systems.
> 
> > - does it reproduce on such systems when using only a single CPU?
> > (i.e., pass 'nosmp' on the kernel command line)
> 
> I would need to try this.

Please note that I want an answer on the vmap stack patches by the
end of today (UK time - so about five hours after this email has
been sent) as we have only tonight and tomorrow's linux-next before
the probable opening of the merge window.

The options are:

1. The problem gets fixed today and I merge the fix today so it can
   get tested in linux-next over the next few days by the various
   build farms and test setups.
2. We postpone the merging of this until the very end of the merge
   window to give more time to sort out this mess - but what it
   means is keeping it in linux-next and keeping various platforms
   broken during that period. However, this is really not fair for
   other people, and some would say this isn't even an option.
3. We drop the entire series for this merge window, meaning it gets
   dropped from linux-next, and have another go for the neext merge
   window.

Sorry for being so demanding, but we're far too close to the merge
window to be trying to debug a feature that is clearly causing a
regression for several platforms.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks

[Jon Hunter]

On 05/01/2022 11:12, Ard Biesheuvel wrote:

...

> Thanks for the report.
> 
> It would be helpful if you could provide some more context:
> - does it happen on a LPAE build too?

Enabling CONFIG_ARM_LPAE does work.

> - does it only happen on SMP capable systems?
> - does it reproduce on such systems when using only a single CPU?
> (i.e., pass 'nosmp' on the kernel command line)

Adding 'nosmp' does not help.

> - when passing 'no_console_suspend' on the kernel command line, are
> any useful diagnostics produced?

Adding 'no_console_suspend' does not produce any interesting logs.

> - is there any way you could tell whether the crash/hang (assuming
> that is what you are observing) occurs on the suspend path or on
> resume?

That is not clear. I see it entering suspend, but not clear if it is 
failing on entering suspend or resuming.

Cheers
Jon

-- 
nvpublic

Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks

[Ard Biesheuvel]

On Wed, 5 Jan 2022 at 17:50, Jon Hunter <jonathanh@nvidia.com> wrote:
>
>
> On 05/01/2022 11:12, Ard Biesheuvel wrote:
>
> ...
>
> > Thanks for the report.
> >
> > It would be helpful if you could provide some more context:
> > - does it happen on a LPAE build too?
>
> Enabling CONFIG_ARM_LPAE does work.
>
> > - does it only happen on SMP capable systems?
> > - does it reproduce on such systems when using only a single CPU?
> > (i.e., pass 'nosmp' on the kernel command line)
>
> Adding 'nosmp' does not help.
>
> > - when passing 'no_console_suspend' on the kernel command line, are
> > any useful diagnostics produced?
>
> Adding 'no_console_suspend' does not produce any interesting logs.
>
> > - is there any way you could tell whether the crash/hang (assuming
> > that is what you are observing) occurs on the suspend path or on
> > resume?
>
> That is not clear. I see it entering suspend, but not clear if it is
> failing on entering suspend or resuming.
>

Thanks a lot for providing this info.

The fact that enabling LPAE makes the issue go away is a fairly strong
hint that one of the CPUs comes up running in an address space that
lacks the stack's vmapping in its copy of the swapper_pg_dir region -
LPAE builds map swapper_pg_dir directly so there it can never go out
of sync.

Given that vmappings are global, and therefore cached in the TLB
across context switches, it is not unlikely that the missing vmapping
of the stack is in a task that runs before suspend, but does not cause
any issues until after the CPU is reset completely (which takes cached
TLB entries down with it)

So in summary, this gives me something to chew on, and hopefully, I
will be able to provide a proper fix shortly.