Invalid pstore_blk use?

Invalid pstore_blk use?

[Florian Fainelli]

Hi Kees, WeiXiong,

I am trying to make use of pstore_blk which is BTW exactly what I had 
been looking for to store panic/console logs onto an eMMC partition.

Using the 5.10 kernel plus:

7e2e92e9861b Revert "mark pstore-blk as broken"
01c28bc8f389 pstore/blk: Use the normal block device I/O path
2a7507999638 pstore/blk: remove {un,}register_pstore_blk
fef0b337cd25 pstore/zone: cap the maximum device size

or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with 
no changes and using:

mount -t pstore pstore /sys/fs/pstore
modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes

upon triggering a crash with:

echo c > /proc/sysrq-trigger

and rebooting and remounting the pstore filesystem and loading 
pstore_blk, I only have:

# ls /sys/fs/pstore/
console-pstore_blk-0

which contains the entire console log up to, but excluding the crash. 
The kernel does show that pstore_blk was used for all 3 types of kmsg, 
pmsg and console:

[   28.649514] pstore_zone: capping size to 128MiB
[   28.712894] pstore_zone: registered pstore_blk as backend for 
kmsg(Oops) pmsg console
[   28.721145] pstore: Using crash dump compression: deflate
[   28.906253] printk: console [pstore_blk-1] enabled
[   28.911229] pstore: Registered pstore_blk as persistent store backend
[   28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9 
(134217728) (no dedicated panic_write!)

there is no automatic reboot upon panic, so I just tend to reboot after 
2-3 seconds manually. The kernel is configured with the default 
CONFIG_PSTORE_* options.

Is the observed behavior a limitation of the best_effort mode? If so, do 
we have any plans to implementing a non-best effort mode for eMMC devices?

Thanks!
-- 
Florian

Re: Invalid pstore_blk use?

[Florian Fainelli]

Hi Kees, WeiXiong,

On 7/14/22 20:49, Florian Fainelli wrote:
> Hi Kees, WeiXiong,
> 
> I am trying to make use of pstore_blk which is BTW exactly what I had 
> been looking for to store panic/console logs onto an eMMC partition.
> 
> Using the 5.10 kernel plus:
> 
> 7e2e92e9861b Revert "mark pstore-blk as broken"
> 01c28bc8f389 pstore/blk: Use the normal block device I/O path
> 2a7507999638 pstore/blk: remove {un,}register_pstore_blk
> fef0b337cd25 pstore/zone: cap the maximum device size
> 
> or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with 
> no changes and using:
> 
> mount -t pstore pstore /sys/fs/pstore
> modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
> 
> upon triggering a crash with:
> 
> echo c > /proc/sysrq-trigger
> 
> and rebooting and remounting the pstore filesystem and loading 
> pstore_blk, I only have:
> 
> # ls /sys/fs/pstore/
> console-pstore_blk-0
> 
> which contains the entire console log up to, but excluding the crash. 
> The kernel does show that pstore_blk was used for all 3 types of kmsg, 
> pmsg and console:
> 
> [   28.649514] pstore_zone: capping size to 128MiB
> [   28.712894] pstore_zone: registered pstore_blk as backend for 
> kmsg(Oops) pmsg console
> [   28.721145] pstore: Using crash dump compression: deflate
> [   28.906253] printk: console [pstore_blk-1] enabled
> [   28.911229] pstore: Registered pstore_blk as persistent store backend
> [   28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9 
> (134217728) (no dedicated panic_write!)
> 
> there is no automatic reboot upon panic, so I just tend to reboot after 
> 2-3 seconds manually. The kernel is configured with the default 
> CONFIG_PSTORE_* options.
> 
> Is the observed behavior a limitation of the best_effort mode? If so, do 
> we have any plans to implementing a non-best effort mode for eMMC devices?

Any feedback on my email? I did try to get kernel panics to be dumped 
out to a dedicated /dev/mtdblock* partition for which there ought to be 
support for mtd->panic_write, but it still did not work any better. Is 
there something obvious that I am missing which prevents kernel panics 
from being logged?

Thank you!
-- 
Florian

Re: Invalid pstore_blk use?

[Kees Cook]

On Tue, Aug 09, 2022 at 11:35:08AM -0700, Florian Fainelli wrote:
> Hi Kees, WeiXiong,
> 
> On 7/14/22 20:49, Florian Fainelli wrote:
> > Hi Kees, WeiXiong,
> > 
> > I am trying to make use of pstore_blk which is BTW exactly what I had
> > been looking for to store panic/console logs onto an eMMC partition.
> > 
> > Using the 5.10 kernel plus:
> > 
> > 7e2e92e9861b Revert "mark pstore-blk as broken"
> > 01c28bc8f389 pstore/blk: Use the normal block device I/O path
> > 2a7507999638 pstore/blk: remove {un,}register_pstore_blk
> > fef0b337cd25 pstore/zone: cap the maximum device size
> > 
> > or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with
> > no changes and using:
> > 
> > mount -t pstore pstore /sys/fs/pstore
> > modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
> > 
> > upon triggering a crash with:
> > 
> > echo c > /proc/sysrq-trigger
> > 
> > and rebooting and remounting the pstore filesystem and loading
> > pstore_blk, I only have:
> > 
> > # ls /sys/fs/pstore/
> > console-pstore_blk-0
> > 
> > which contains the entire console log up to, but excluding the crash.
> > The kernel does show that pstore_blk was used for all 3 types of kmsg,
> > pmsg and console:
> > 
> > [   28.649514] pstore_zone: capping size to 128MiB
> > [   28.712894] pstore_zone: registered pstore_blk as backend for
> > kmsg(Oops) pmsg console
> > [   28.721145] pstore: Using crash dump compression: deflate
> > [   28.906253] printk: console [pstore_blk-1] enabled
> > [   28.911229] pstore: Registered pstore_blk as persistent store backend
> > [   28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9
> > (134217728) (no dedicated panic_write!)
> > 
> > there is no automatic reboot upon panic, so I just tend to reboot after
> > 2-3 seconds manually. The kernel is configured with the default
> > CONFIG_PSTORE_* options.
> > 
> > Is the observed behavior a limitation of the best_effort mode? If so, do
> > we have any plans to implementing a non-best effort mode for eMMC
> > devices?
> 
> Any feedback on my email? I did try to get kernel panics to be dumped out to

Hi! Sorry I lost this email originally. :)

> a dedicated /dev/mtdblock* partition for which there ought to be support for
> mtd->panic_write, but it still did not work any better. Is there something

With the mtdblock driver, do you still see:

    pstore_blk: attached pstore_blk:/dev/... (no dedicated panic_write!)
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^

> obvious that I am missing which prevents kernel panics from being logged?

Unfortunately it really depends on how the drivers are built. If the
block layer is shut down during a panic, pstore_blk won't catch the
panic. :(

-- 
Kees Cook

Re: Invalid pstore_blk use?

[Kamal Dasu]

Kees,

I am in the process of implementing  mmcpstore  backend for mmc  based
on the mtdpstore driver

This is what is registered with register_pstore_device(&cxt->dev);
cxt->dev.flags = PSTORE_FLAGS_DMESG;
cxt->dev.zone.read = mmcpstore_read;
cxt->dev.zone.write = mmcpstore_write;
cxt->dev.zone.erase = mmcpstore_erase;
cxt->dev.zone.panic_write = mmcpstore_panic_write;

# dmesg | grep pstor
[    0.000000] Kernel command line: pstore_blk.blkdev=/dev/mmcblk1p8
crash_kexec_post_notifiers printk.always_kmsg_dump
[    1.993986] pstore_zone: registered pstore_blk as backend for
kmsg(Oops,panic_write) pmsg
[    2.002582] pstore: Using crash dump compression: deflate
[    2.008133] pstore: Registered pstore_blk as persistent store backend
[    2.020907] mmcpstore: /dev/mmcblk1p8 size 131072 start sector
34468 registered as psblk backend
[   17.868753] psz_kmsg_recover_meta: pstore_zone: no valid data in
kmsg dump zone 0
[   18.298933] psz_recover_zone: pstore_zone: no valid data in zone pmsg
[   18.305398] psz_recovery: pstore_zone: recover end!

The driver is successfully registered and the read path works when
/sys/fs/pstor is mounted , however mmc_pstore_panic_write is not
called.
Need help in understanding what could be missing. I am using the
latest upstream kernel for testing.

Kamal

On Tue, Aug 9, 2022 at 7:06 PM Kees Cook <keescook@chromium.org> wrote:
>
> On Tue, Aug 09, 2022 at 11:35:08AM -0700, Florian Fainelli wrote:
> > Hi Kees, WeiXiong,
> >
> > On 7/14/22 20:49, Florian Fainelli wrote:
> > > Hi Kees, WeiXiong,
> > >
> > > I am trying to make use of pstore_blk which is BTW exactly what I had
> > > been looking for to store panic/console logs onto an eMMC partition.
> > >
> > > Using the 5.10 kernel plus:
> > >
> > > 7e2e92e9861b Revert "mark pstore-blk as broken"
> > > 01c28bc8f389 pstore/blk: Use the normal block device I/O path
> > > 2a7507999638 pstore/blk: remove {un,}register_pstore_blk
> > > fef0b337cd25 pstore/zone: cap the maximum device size
> > >
> > > or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with
> > > no changes and using:
> > >
> > > mount -t pstore pstore /sys/fs/pstore
> > > modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
> > >
> > > upon triggering a crash with:
> > >
> > > echo c > /proc/sysrq-trigger
> > >
> > > and rebooting and remounting the pstore filesystem and loading
> > > pstore_blk, I only have:
> > >
> > > # ls /sys/fs/pstore/
> > > console-pstore_blk-0
> > >
> > > which contains the entire console log up to, but excluding the crash.
> > > The kernel does show that pstore_blk was used for all 3 types of kmsg,
> > > pmsg and console:
> > >
> > > [   28.649514] pstore_zone: capping size to 128MiB
> > > [   28.712894] pstore_zone: registered pstore_blk as backend for
> > > kmsg(Oops) pmsg console
> > > [   28.721145] pstore: Using crash dump compression: deflate
> > > [   28.906253] printk: console [pstore_blk-1] enabled
> > > [   28.911229] pstore: Registered pstore_blk as persistent store backend
> > > [   28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9
> > > (134217728) (no dedicated panic_write!)
> > >
> > > there is no automatic reboot upon panic, so I just tend to reboot after
> > > 2-3 seconds manually. The kernel is configured with the default
> > > CONFIG_PSTORE_* options.
> > >
> > > Is the observed behavior a limitation of the best_effort mode? If so, do
> > > we have any plans to implementing a non-best effort mode for eMMC
> > > devices?
> >
> > Any feedback on my email? I did try to get kernel panics to be dumped out to
>
> Hi! Sorry I lost this email originally. :)
>
> > a dedicated /dev/mtdblock* partition for which there ought to be support for
> > mtd->panic_write, but it still did not work any better. Is there something
>
> With the mtdblock driver, do you still see:
>
>     pstore_blk: attached pstore_blk:/dev/... (no dedicated panic_write!)
>                                               ^^^^^^^^^^^^^^^^^^^^^^^^^
>
> > obvious that I am missing which prevents kernel panics from being logged?
>
> Unfortunately it really depends on how the drivers are built. If the
> block layer is shut down during a panic, pstore_blk won't catch the
> panic. :(
>
> --
> Kees Cook

Re: Invalid pstore_blk use?

[Kamal Dasu]

+ Adrain Hunter
+ Ulf Hansoon



On Fri, Nov 18, 2022 at 3:41 PM Kamal Dasu <kdasu.kdev@gmail.com> wrote:
>
> Kees,
>
> I am in the process of implementing  mmcpstore  backend for mmc  based
> on the mtdpstore driver
>
> This is what is registered with register_pstore_device(&cxt->dev);
> cxt->dev.flags = PSTORE_FLAGS_DMESG;
> cxt->dev.zone.read = mmcpstore_read;
> cxt->dev.zone.write = mmcpstore_write;
> cxt->dev.zone.erase = mmcpstore_erase;
> cxt->dev.zone.panic_write = mmcpstore_panic_write;
>
> # dmesg | grep pstor
> [    0.000000] Kernel command line: pstore_blk.blkdev=/dev/mmcblk1p8
> crash_kexec_post_notifiers printk.always_kmsg_dump
> [    1.993986] pstore_zone: registered pstore_blk as backend for
> kmsg(Oops,panic_write) pmsg
> [    2.002582] pstore: Using crash dump compression: deflate
> [    2.008133] pstore: Registered pstore_blk as persistent store backend
> [    2.020907] mmcpstore: /dev/mmcblk1p8 size 131072 start sector
> 34468 registered as psblk backend
> [   17.868753] psz_kmsg_recover_meta: pstore_zone: no valid data in
> kmsg dump zone 0
> [   18.298933] psz_recover_zone: pstore_zone: no valid data in zone pmsg
> [   18.305398] psz_recovery: pstore_zone: recover end!
>
> The driver is successfully registered and the read path works when
> /sys/fs/pstor is mounted , however mmc_pstore_panic_write is not
> called.
> Need help in understanding what could be missing. I am using the
> latest upstream kernel for testing.
>
> Kamal
>
> On Tue, Aug 9, 2022 at 7:06 PM Kees Cook <keescook@chromium.org> wrote:
> >
> > On Tue, Aug 09, 2022 at 11:35:08AM -0700, Florian Fainelli wrote:
> > > Hi Kees, WeiXiong,
> > >
> > > On 7/14/22 20:49, Florian Fainelli wrote:
> > > > Hi Kees, WeiXiong,
> > > >
> > > > I am trying to make use of pstore_blk which is BTW exactly what I had
> > > > been looking for to store panic/console logs onto an eMMC partition.
> > > >
> > > > Using the 5.10 kernel plus:
> > > >
> > > > 7e2e92e9861b Revert "mark pstore-blk as broken"
> > > > 01c28bc8f389 pstore/blk: Use the normal block device I/O path
> > > > 2a7507999638 pstore/blk: remove {un,}register_pstore_blk
> > > > fef0b337cd25 pstore/zone: cap the maximum device size
> > > >
> > > > or the android13-5.15 (at Merge 5.15.40 into android13-5.15) kernel with
> > > > no changes and using:
> > > >
> > > > mount -t pstore pstore /sys/fs/pstore
> > > > modprobe pstore_blk blkdev=/dev/mmcblk1p9 best_effort=yes
> > > >
> > > > upon triggering a crash with:
> > > >
> > > > echo c > /proc/sysrq-trigger
> > > >
> > > > and rebooting and remounting the pstore filesystem and loading
> > > > pstore_blk, I only have:
> > > >
> > > > # ls /sys/fs/pstore/
> > > > console-pstore_blk-0
> > > >
> > > > which contains the entire console log up to, but excluding the crash.
> > > > The kernel does show that pstore_blk was used for all 3 types of kmsg,
> > > > pmsg and console:
> > > >
> > > > [   28.649514] pstore_zone: capping size to 128MiB
> > > > [   28.712894] pstore_zone: registered pstore_blk as backend for
> > > > kmsg(Oops) pmsg console
> > > > [   28.721145] pstore: Using crash dump compression: deflate
> > > > [   28.906253] printk: console [pstore_blk-1] enabled
> > > > [   28.911229] pstore: Registered pstore_blk as persistent store backend
> > > > [   28.917735] pstore_blk: attached pstore_blk:/dev/mmcblk1p9
> > > > (134217728) (no dedicated panic_write!)
> > > >
> > > > there is no automatic reboot upon panic, so I just tend to reboot after
> > > > 2-3 seconds manually. The kernel is configured with the default
> > > > CONFIG_PSTORE_* options.
> > > >
> > > > Is the observed behavior a limitation of the best_effort mode? If so, do
> > > > we have any plans to implementing a non-best effort mode for eMMC
> > > > devices?
> > >
> > > Any feedback on my email? I did try to get kernel panics to be dumped out to
> >
> > Hi! Sorry I lost this email originally. :)
> >
> > > a dedicated /dev/mtdblock* partition for which there ought to be support for
> > > mtd->panic_write, but it still did not work any better. Is there something
> >
> > With the mtdblock driver, do you still see:
> >
> >     pstore_blk: attached pstore_blk:/dev/... (no dedicated panic_write!)
> >                                               ^^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > > obvious that I am missing which prevents kernel panics from being logged?
> >
> > Unfortunately it really depends on how the drivers are built. If the
> > block layer is shut down during a panic, pstore_blk won't catch the
> > panic. :(
> >
> > --
> > Kees Cook

Re: Invalid pstore_blk use?

[Kees Cook]

On Fri, Nov 18, 2022 at 03:41:44PM -0500, Kamal Dasu wrote:
> Kees,
> 
> I am in the process of implementing  mmcpstore  backend for mmc  based
> on the mtdpstore driver
> 
> This is what is registered with register_pstore_device(&cxt->dev);
> cxt->dev.flags = PSTORE_FLAGS_DMESG;
> cxt->dev.zone.read = mmcpstore_read;
> cxt->dev.zone.write = mmcpstore_write;
> cxt->dev.zone.erase = mmcpstore_erase;
> cxt->dev.zone.panic_write = mmcpstore_panic_write;
> 
> # dmesg | grep pstor
> [    0.000000] Kernel command line: pstore_blk.blkdev=/dev/mmcblk1p8
> crash_kexec_post_notifiers printk.always_kmsg_dump
> [    1.993986] pstore_zone: registered pstore_blk as backend for
> kmsg(Oops,panic_write) pmsg
> [    2.002582] pstore: Using crash dump compression: deflate
> [    2.008133] pstore: Registered pstore_blk as persistent store backend
> [    2.020907] mmcpstore: /dev/mmcblk1p8 size 131072 start sector
> 34468 registered as psblk backend
> [   17.868753] psz_kmsg_recover_meta: pstore_zone: no valid data in
> kmsg dump zone 0
> [   18.298933] psz_recover_zone: pstore_zone: no valid data in zone pmsg
> [   18.305398] psz_recovery: pstore_zone: recover end!
> 
> The driver is successfully registered and the read path works when
> /sys/fs/pstor is mounted , however mmc_pstore_panic_write is not
> called.
> Need help in understanding what could be missing. I am using the
> latest upstream kernel for testing.

Hi!

Can you send an RFC patch? I'd expect this to work as you've currently
described it, but without code I'd just be guessing. :)

-Kees

-- 
Kees Cook

Re: Invalid pstore_blk use?

[Kamal Dasu]

Kees,

Thanks for your reply. I will send the RFC patch.

Kamal

On Wed, Dec 7, 2022 at 5:13 PM Kees Cook <keescook@chromium.org> wrote:
>
> On Fri, Nov 18, 2022 at 03:41:44PM -0500, Kamal Dasu wrote:
> > Kees,
> >
> > I am in the process of implementing  mmcpstore  backend for mmc  based
> > on the mtdpstore driver
> >
> > This is what is registered with register_pstore_device(&cxt->dev);
> > cxt->dev.flags = PSTORE_FLAGS_DMESG;
> > cxt->dev.zone.read = mmcpstore_read;
> > cxt->dev.zone.write = mmcpstore_write;
> > cxt->dev.zone.erase = mmcpstore_erase;
> > cxt->dev.zone.panic_write = mmcpstore_panic_write;
> >
> > # dmesg | grep pstor
> > [    0.000000] Kernel command line: pstore_blk.blkdev=/dev/mmcblk1p8
> > crash_kexec_post_notifiers printk.always_kmsg_dump
> > [    1.993986] pstore_zone: registered pstore_blk as backend for
> > kmsg(Oops,panic_write) pmsg
> > [    2.002582] pstore: Using crash dump compression: deflate
> > [    2.008133] pstore: Registered pstore_blk as persistent store backend
> > [    2.020907] mmcpstore: /dev/mmcblk1p8 size 131072 start sector
> > 34468 registered as psblk backend
> > [   17.868753] psz_kmsg_recover_meta: pstore_zone: no valid data in
> > kmsg dump zone 0
> > [   18.298933] psz_recover_zone: pstore_zone: no valid data in zone pmsg
> > [   18.305398] psz_recovery: pstore_zone: recover end!
> >
> > The driver is successfully registered and the read path works when
> > /sys/fs/pstor is mounted , however mmc_pstore_panic_write is not
> > called.
> > Need help in understanding what could be missing. I am using the
> > latest upstream kernel for testing.
>
> Hi!
>
> Can you send an RFC patch? I'd expect this to work as you've currently
> described it, but without code I'd just be guessing. :)
>
> -Kees
>
> --
> Kees Cook