* [PATCH RFC] kexec: Freeze processes before kexec
@ 2022-11-16 19:56 ` Joel Fernandes (Google)
0 siblings, 0 replies; 10+ messages in thread
From: Joel Fernandes (Google) @ 2022-11-16 19:56 UTC (permalink / raw)
To: linux-kernel
Cc: Joel Fernandes (Google), rostedt, ribalda, zwisler, robdclark,
Eric Biederman, kexec
During kexec, it is possible for userspace to race with
device_shutdown() causing accesses to GPU after pm_runtime suspend has
already happened. Fix by freezing userspace before device_shutdown().
Such freezing is already being done if kernel supports KEXEC_JUMP and
kexec_image->preserve_context is true. However, doing it if either of
these are not true prevents crashes/races.
This fixes the following crash during kexec reboot on an ARM64 device
with adreno GPU:
[ 292.534314] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 292.534323] Hardware name: Google Lazor (rev3 - 8) with LTE (DT)
[ 292.534326] Call trace:
[ 292.534328] dump_backtrace+0x0/0x1d4
[ 292.534337] show_stack+0x20/0x2c
[ 292.534342] dump_stack_lvl+0x60/0x78
[ 292.534347] dump_stack+0x18/0x38
[ 292.534352] panic+0x148/0x3b0
[ 292.534357] nmi_panic+0x80/0x94
[ 292.534364] arm64_serror_panic+0x70/0x7c
[ 292.534369] do_serror+0x0/0x7c
[ 292.534372] do_serror+0x54/0x7c
[ 292.534377] el1h_64_error_handler+0x34/0x4c
[ 292.534381] el1h_64_error+0x7c/0x80
[ 292.534386] el1_interrupt+0x20/0x58
[ 292.534389] el1h_64_irq_handler+0x18/0x24
[ 292.534395] el1h_64_irq+0x7c/0x80
[ 292.534399] local_daif_inherit+0x10/0x18
[ 292.534405] el1h_64_sync_handler+0x48/0xb4
[ 292.534410] el1h_64_sync+0x7c/0x80
[ 292.534414] a6xx_gmu_set_oob+0xbc/0x1fc
[ 292.534422] a6xx_get_timestamp+0x40/0xb4
[ 292.534426] adreno_get_param+0x12c/0x1e0
[ 292.534433] msm_ioctl_get_param+0x64/0x70
[ 292.534440] drm_ioctl_kernel+0xe8/0x158
[ 292.534448] drm_ioctl+0x208/0x320
[ 292.534453] __arm64_sys_ioctl+0x98/0xd0
[ 292.534461] invoke_syscall+0x4c/0x118
[ 292.534467] el0_svc_common+0x98/0x104
[ 292.534473] do_el0_svc+0x30/0x80
[ 292.534478] el0_svc+0x20/0x50
[ 292.534481] el0t_64_sync_handler+0x78/0x108
[ 292.534485] el0t_64_sync+0x1a4/0x1a8
[ 292.534632] Kernel Offset: 0x1a5f800000 from 0xffffffc008000000
[ 292.534635] PHYS_OFFSET: 0x80000000
[ 292.534638] CPU features: 0x40018541,a3300e42
[ 292.534644] Memory Limit: none
Cc: rostedt@goodmis.org
Cc: ribalda@google.com
Cc: zwisler@google.com
Cc: robdclark@gmail.com
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
kernel/kexec_core.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index ca2743f9c634..2614a7f1f8a6 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1175,6 +1175,12 @@ int kernel_kexec(void)
} else
#endif
{
+ error = freeze_processes();
+ if (error) {
+ error = -EBUSY;
+ goto Unlock;
+ }
+
kexec_in_progress = true;
kernel_restart_prepare("kexec reboot");
migrate_to_reboot_cpu();
--
2.38.1.584.g0f3c55d4c2-goog
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH RFC] kexec: Freeze processes before kexec
@ 2022-11-16 19:56 ` Joel Fernandes (Google)
0 siblings, 0 replies; 10+ messages in thread
From: Joel Fernandes (Google) @ 2022-11-16 19:56 UTC (permalink / raw)
To: linux-kernel
Cc: Joel Fernandes (Google), rostedt, ribalda, zwisler, robdclark,
Eric Biederman, kexec
During kexec, it is possible for userspace to race with
device_shutdown() causing accesses to GPU after pm_runtime suspend has
already happened. Fix by freezing userspace before device_shutdown().
Such freezing is already being done if kernel supports KEXEC_JUMP and
kexec_image->preserve_context is true. However, doing it if either of
these are not true prevents crashes/races.
This fixes the following crash during kexec reboot on an ARM64 device
with adreno GPU:
[ 292.534314] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 292.534323] Hardware name: Google Lazor (rev3 - 8) with LTE (DT)
[ 292.534326] Call trace:
[ 292.534328] dump_backtrace+0x0/0x1d4
[ 292.534337] show_stack+0x20/0x2c
[ 292.534342] dump_stack_lvl+0x60/0x78
[ 292.534347] dump_stack+0x18/0x38
[ 292.534352] panic+0x148/0x3b0
[ 292.534357] nmi_panic+0x80/0x94
[ 292.534364] arm64_serror_panic+0x70/0x7c
[ 292.534369] do_serror+0x0/0x7c
[ 292.534372] do_serror+0x54/0x7c
[ 292.534377] el1h_64_error_handler+0x34/0x4c
[ 292.534381] el1h_64_error+0x7c/0x80
[ 292.534386] el1_interrupt+0x20/0x58
[ 292.534389] el1h_64_irq_handler+0x18/0x24
[ 292.534395] el1h_64_irq+0x7c/0x80
[ 292.534399] local_daif_inherit+0x10/0x18
[ 292.534405] el1h_64_sync_handler+0x48/0xb4
[ 292.534410] el1h_64_sync+0x7c/0x80
[ 292.534414] a6xx_gmu_set_oob+0xbc/0x1fc
[ 292.534422] a6xx_get_timestamp+0x40/0xb4
[ 292.534426] adreno_get_param+0x12c/0x1e0
[ 292.534433] msm_ioctl_get_param+0x64/0x70
[ 292.534440] drm_ioctl_kernel+0xe8/0x158
[ 292.534448] drm_ioctl+0x208/0x320
[ 292.534453] __arm64_sys_ioctl+0x98/0xd0
[ 292.534461] invoke_syscall+0x4c/0x118
[ 292.534467] el0_svc_common+0x98/0x104
[ 292.534473] do_el0_svc+0x30/0x80
[ 292.534478] el0_svc+0x20/0x50
[ 292.534481] el0t_64_sync_handler+0x78/0x108
[ 292.534485] el0t_64_sync+0x1a4/0x1a8
[ 292.534632] Kernel Offset: 0x1a5f800000 from 0xffffffc008000000
[ 292.534635] PHYS_OFFSET: 0x80000000
[ 292.534638] CPU features: 0x40018541,a3300e42
[ 292.534644] Memory Limit: none
Cc: rostedt@goodmis.org
Cc: ribalda@google.com
Cc: zwisler@google.com
Cc: robdclark@gmail.com
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
kernel/kexec_core.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index ca2743f9c634..2614a7f1f8a6 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1175,6 +1175,12 @@ int kernel_kexec(void)
} else
#endif
{
+ error = freeze_processes();
+ if (error) {
+ error = -EBUSY;
+ goto Unlock;
+ }
+
kexec_in_progress = true;
kernel_restart_prepare("kexec reboot");
migrate_to_reboot_cpu();
--
2.38.1.584.g0f3c55d4c2-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH RFC] kexec: Freeze processes before kexec
2022-11-16 19:56 ` Joel Fernandes (Google)
@ 2022-11-16 20:16 ` Steven Rostedt
-1 siblings, 0 replies; 10+ messages in thread
From: Steven Rostedt @ 2022-11-16 20:16 UTC (permalink / raw)
To: Joel Fernandes (Google)
Cc: linux-kernel, ribalda, zwisler, robdclark, Eric Biederman, kexec
On Wed, 16 Nov 2022 19:56:24 +0000
"Joel Fernandes (Google)" <joel@joelfernandes.org> wrote:
> --- a/kernel/kexec_core.c
> +++ b/kernel/kexec_core.c
> @@ -1175,6 +1175,12 @@ int kernel_kexec(void)
> } else
> #endif
> {
> + error = freeze_processes();
> + if (error) {
> + error = -EBUSY;
> + goto Unlock;
> + }
If this is the path of a kernel panic, do we really want to check the
return error of freeze_processes()? We are panicing, there's not much more
we can do.
-- Steve
> +
> kexec_in_progress = true;
> kernel_restart_prepare("kexec reboot");
> migrate_to_reboot_cpu();
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH RFC] kexec: Freeze processes before kexec
@ 2022-11-16 20:16 ` Steven Rostedt
0 siblings, 0 replies; 10+ messages in thread
From: Steven Rostedt @ 2022-11-16 20:16 UTC (permalink / raw)
To: Joel Fernandes (Google)
Cc: linux-kernel, ribalda, zwisler, robdclark, Eric Biederman, kexec
On Wed, 16 Nov 2022 19:56:24 +0000
"Joel Fernandes (Google)" <joel@joelfernandes.org> wrote:
> --- a/kernel/kexec_core.c
> +++ b/kernel/kexec_core.c
> @@ -1175,6 +1175,12 @@ int kernel_kexec(void)
> } else
> #endif
> {
> + error = freeze_processes();
> + if (error) {
> + error = -EBUSY;
> + goto Unlock;
> + }
If this is the path of a kernel panic, do we really want to check the
return error of freeze_processes()? We are panicing, there's not much more
we can do.
-- Steve
> +
> kexec_in_progress = true;
> kernel_restart_prepare("kexec reboot");
> migrate_to_reboot_cpu();
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH RFC] kexec: Freeze processes before kexec
2022-11-16 20:16 ` Steven Rostedt
@ 2022-11-16 21:07 ` Joel Fernandes
-1 siblings, 0 replies; 10+ messages in thread
From: Joel Fernandes @ 2022-11-16 21:07 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-kernel, ribalda, zwisler, robdclark, Eric Biederman, kexec
Hey Steve,
On Wed, Nov 16, 2022 at 8:15 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Wed, 16 Nov 2022 19:56:24 +0000
> "Joel Fernandes (Google)" <joel@joelfernandes.org> wrote:
>
> > --- a/kernel/kexec_core.c
> > +++ b/kernel/kexec_core.c
> > @@ -1175,6 +1175,12 @@ int kernel_kexec(void)
> > } else
> > #endif
> > {
> > + error = freeze_processes();
> > + if (error) {
> > + error = -EBUSY;
> > + goto Unlock;
> > + }
>
> If this is the path of a kernel panic, do we really want to check the
> return error of freeze_processes()? We are panicing, there's not much more
> we can do.
I am OK with not checking the return of freeze_processes() and trying
to shut down anyway. Will re-spin after any other feedback.
Thanks,
- Joel
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH RFC] kexec: Freeze processes before kexec
@ 2022-11-16 21:07 ` Joel Fernandes
0 siblings, 0 replies; 10+ messages in thread
From: Joel Fernandes @ 2022-11-16 21:07 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-kernel, ribalda, zwisler, robdclark, Eric Biederman, kexec
Hey Steve,
On Wed, Nov 16, 2022 at 8:15 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Wed, 16 Nov 2022 19:56:24 +0000
> "Joel Fernandes (Google)" <joel@joelfernandes.org> wrote:
>
> > --- a/kernel/kexec_core.c
> > +++ b/kernel/kexec_core.c
> > @@ -1175,6 +1175,12 @@ int kernel_kexec(void)
> > } else
> > #endif
> > {
> > + error = freeze_processes();
> > + if (error) {
> > + error = -EBUSY;
> > + goto Unlock;
> > + }
>
> If this is the path of a kernel panic, do we really want to check the
> return error of freeze_processes()? We are panicing, there's not much more
> we can do.
I am OK with not checking the return of freeze_processes() and trying
to shut down anyway. Will re-spin after any other feedback.
Thanks,
- Joel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH RFC] kexec: Freeze processes before kexec
2022-11-16 20:16 ` Steven Rostedt
@ 2022-11-17 15:46 ` Philipp Rudo
-1 siblings, 0 replies; 10+ messages in thread
From: Philipp Rudo @ 2022-11-17 15:46 UTC (permalink / raw)
To: Steven Rostedt
Cc: Joel Fernandes (Google), linux-kernel, ribalda, zwisler,
robdclark, Eric Biederman, kexec
Hi Steve,
On Wed, 16 Nov 2022 15:16:10 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:
> On Wed, 16 Nov 2022 19:56:24 +0000
> "Joel Fernandes (Google)" <joel@joelfernandes.org> wrote:
>
> > --- a/kernel/kexec_core.c
> > +++ b/kernel/kexec_core.c
> > @@ -1175,6 +1175,12 @@ int kernel_kexec(void)
> > } else
> > #endif
> > {
> > + error = freeze_processes();
> > + if (error) {
> > + error = -EBUSY;
> > + goto Unlock;
> > + }
>
> If this is the path of a kernel panic, do we really want to check the
> return error of freeze_processes()? We are panicing, there's not much more
> we can do.
kernel_kexec isn't called during panic. We don't need to worry about it
here.
Having that said I think this is a problem in the device driver _not_
in kexec. In my opinion it's the job of the driver to prevent such
races during shutdown.
Thanks
Philipp
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH RFC] kexec: Freeze processes before kexec
@ 2022-11-17 15:46 ` Philipp Rudo
0 siblings, 0 replies; 10+ messages in thread
From: Philipp Rudo @ 2022-11-17 15:46 UTC (permalink / raw)
To: Steven Rostedt
Cc: Joel Fernandes (Google), linux-kernel, ribalda, zwisler,
robdclark, Eric Biederman, kexec
Hi Steve,
On Wed, 16 Nov 2022 15:16:10 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:
> On Wed, 16 Nov 2022 19:56:24 +0000
> "Joel Fernandes (Google)" <joel@joelfernandes.org> wrote:
>
> > --- a/kernel/kexec_core.c
> > +++ b/kernel/kexec_core.c
> > @@ -1175,6 +1175,12 @@ int kernel_kexec(void)
> > } else
> > #endif
> > {
> > + error = freeze_processes();
> > + if (error) {
> > + error = -EBUSY;
> > + goto Unlock;
> > + }
>
> If this is the path of a kernel panic, do we really want to check the
> return error of freeze_processes()? We are panicing, there's not much more
> we can do.
kernel_kexec isn't called during panic. We don't need to worry about it
here.
Having that said I think this is a problem in the device driver _not_
in kexec. In my opinion it's the job of the driver to prevent such
races during shutdown.
Thanks
Philipp
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH RFC] kexec: Freeze processes before kexec
2022-11-17 15:46 ` Philipp Rudo
@ 2022-11-17 16:07 ` Joel Fernandes
-1 siblings, 0 replies; 10+ messages in thread
From: Joel Fernandes @ 2022-11-17 16:07 UTC (permalink / raw)
To: Philipp Rudo
Cc: Steven Rostedt, linux-kernel, ribalda, zwisler, robdclark,
Eric Biederman, kexec
On Thu, Nov 17, 2022 at 3:46 PM Philipp Rudo <prudo@redhat.com> wrote:
> On Wed, 16 Nov 2022 15:16:10 -0500
> Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > On Wed, 16 Nov 2022 19:56:24 +0000
> > "Joel Fernandes (Google)" <joel@joelfernandes.org> wrote:
> >
> > > --- a/kernel/kexec_core.c
> > > +++ b/kernel/kexec_core.c
> > > @@ -1175,6 +1175,12 @@ int kernel_kexec(void)
> > > } else
> > > #endif
> > > {
> > > + error = freeze_processes();
> > > + if (error) {
> > > + error = -EBUSY;
> > > + goto Unlock;
> > > + }
> >
> > If this is the path of a kernel panic, do we really want to check the
> > return error of freeze_processes()? We are panicing, there's not much more
> > we can do.
>
> kernel_kexec isn't called during panic. We don't need to worry about it
> here.
Indeed, sorry for my hasty comment and for misleading Steve. This is
seen to happen when doing manual kexec from the reboot(2) system call.
During a regular panic, machine_shutdown() is not called so this would
not happen. However, for testing, the crash is a hurdle.
> Having that said I think this is a problem in the device driver _not_
> in kexec. In my opinion it's the job of the driver to prevent such
> races during shutdown.
Thanks a lot for your input. Rob, what do you think?
thanks,
- Joel
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH RFC] kexec: Freeze processes before kexec
@ 2022-11-17 16:07 ` Joel Fernandes
0 siblings, 0 replies; 10+ messages in thread
From: Joel Fernandes @ 2022-11-17 16:07 UTC (permalink / raw)
To: Philipp Rudo
Cc: Steven Rostedt, linux-kernel, ribalda, zwisler, robdclark,
Eric Biederman, kexec
On Thu, Nov 17, 2022 at 3:46 PM Philipp Rudo <prudo@redhat.com> wrote:
> On Wed, 16 Nov 2022 15:16:10 -0500
> Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > On Wed, 16 Nov 2022 19:56:24 +0000
> > "Joel Fernandes (Google)" <joel@joelfernandes.org> wrote:
> >
> > > --- a/kernel/kexec_core.c
> > > +++ b/kernel/kexec_core.c
> > > @@ -1175,6 +1175,12 @@ int kernel_kexec(void)
> > > } else
> > > #endif
> > > {
> > > + error = freeze_processes();
> > > + if (error) {
> > > + error = -EBUSY;
> > > + goto Unlock;
> > > + }
> >
> > If this is the path of a kernel panic, do we really want to check the
> > return error of freeze_processes()? We are panicing, there's not much more
> > we can do.
>
> kernel_kexec isn't called during panic. We don't need to worry about it
> here.
Indeed, sorry for my hasty comment and for misleading Steve. This is
seen to happen when doing manual kexec from the reboot(2) system call.
During a regular panic, machine_shutdown() is not called so this would
not happen. However, for testing, the crash is a hurdle.
> Having that said I think this is a problem in the device driver _not_
> in kexec. In my opinion it's the job of the driver to prevent such
> races during shutdown.
Thanks a lot for your input. Rob, what do you think?
thanks,
- Joel
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-11-17 16:11 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-11-16 19:56 [PATCH RFC] kexec: Freeze processes before kexec Joel Fernandes (Google)
2022-11-16 19:56 ` Joel Fernandes (Google)
2022-11-16 20:16 ` Steven Rostedt
2022-11-16 20:16 ` Steven Rostedt
2022-11-16 21:07 ` Joel Fernandes
2022-11-16 21:07 ` Joel Fernandes
2022-11-17 15:46 ` Philipp Rudo
2022-11-17 15:46 ` Philipp Rudo
2022-11-17 16:07 ` Joel Fernandes
2022-11-17 16:07 ` Joel Fernandes
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.