* RFC design of device coredump collection on panic in Pstore
@ 2023-05-08 15:51 Mukesh Ojha
2023-05-15 6:52 ` Pavan Kondeti
2023-05-16 20:55 ` Kees Cook
0 siblings, 2 replies; 4+ messages in thread
From: Mukesh Ojha @ 2023-05-08 15:51 UTC (permalink / raw)
To: keescook, gregkh, tony.luck, gpiccoli, johannes, rafael
Cc: quic_mojha, linux-hardening, linux-kernel, quic_shashim,
quic_pkondeti
Hi All,
We are exploring a way where device driver(registered/interested and optional could be
callback) can dump their data(consist of list of buffers or hardware registers) for post
mortam debugging of a crash via dumping the content to pstore region. Some of the example
data could be like clock dump/regulators/ etc.
These stuff should be already part of entire RAM dump but in some cases it's however not
feasible to capture the entire content of RAM, so was thinking if pstore region can be
used to capture above information either in the form of elf or binary and how pstore
can get this in human readable format can be discussed further.
Also, existing devcoredump driver does not cover panic usecases so we thought of something
like below RFC design where,
1. Device_coredump allocates some configurable contigous memory that can be controlled
via CONFIG or bootargs and later registers for panic notifiers.
2. Notifier gets added.
3. Pstore adds device_coredump as its front-end via dumper registration similar
to kmsg being dump today.
4. Successful registration of dumper.
5. A device driver(A-Z) can register their buffer to be dumped as part of panic.
6. buffer gets added to the dump list.
7. Panic occurs.
8. iterate over registered drivers and copy their dump list to its own memory and if
it crosses device core dump memory log an error stop iterating.
9. Similar to devcore_dump() inline with kmsg_dump()
10.Copy the content to pstore region and this could be elf or raw binary that can be
discussed.
Device coredump memory(1) size could be passed from pstore and should be same as size
of devcoredump frontend size given in DT or some other way.
Let me know your concern and view on this.
pstore device_coredump deviceA..Z panic
┼ │
│ │ panic_notifier(1) │
│ ├───────────────────┬─────────────────►│
│ │ notifier added │ (2) │
│ │◄──────────────────┼──────────────────┤
pstore can give │ (3) │ │ │
its region for │ dumper_registration │ │ │
dump ├─────────────────────►│ │ │
│ (4) │ │ │
│◄─────────────────────┤ │ │
│ register_dumper │ │ │
│ │ │ │
│ │ │ │
│ │ │ init │
│ │ ───┼── │
│ │ (5) │ │
│ │devcoredump_register(dev,buf)); │
│ │◄──────────────────┐ │
│ │ (6) │ │
│ ├──────────────────►│ │
│ │device/buf gets add│d │
│ │ to the list │ │panic
│ │ │ ──┼───
│ │ │ │
│ │(7) panic_notifier │ call │
│ │◄──────────────────┼──────────────────┤
│ │ prepare coredump │for the deviceA-Z │
│ ├──────────────────►├────┐ │
│ │ (8) │ │ │
│ │ │ │ │
│ │ │ │ │
│ │ │◄───┘ │
dumper will │ │ (9) │dev_coredump()
write all the device dump to │◄──────────────────┼──────────────────┤
pstore ┌◄───┬─────────────────┤ │ │
region │ │ (10) │ │ │
│ │ │ │ │
│ │ │ │ │
│◄───┘ │ │ │
│ │ │ │
│ │
Regards,
Mukesh
--
2.7.4
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: RFC design of device coredump collection on panic in Pstore 2023-05-08 15:51 RFC design of device coredump collection on panic in Pstore Mukesh Ojha @ 2023-05-15 6:52 ` Pavan Kondeti 2023-05-16 20:55 ` Kees Cook 1 sibling, 0 replies; 4+ messages in thread From: Pavan Kondeti @ 2023-05-15 6:52 UTC (permalink / raw) To: Mukesh Ojha Cc: keescook, gregkh, tony.luck, gpiccoli, johannes, rafael, linux-hardening, linux-kernel, quic_shashim, quic_pkondeti On Mon, May 08, 2023 at 09:21:00PM +0530, Mukesh Ojha wrote: > Hi All, > > We are exploring a way where device driver(registered/interested and optional could be > callback) can dump their data(consist of list of buffers or hardware registers) for post > mortam debugging of a crash via dumping the content to pstore region. Some of the example > data could be like clock dump/regulators/ etc. > > These stuff should be already part of entire RAM dump but in some cases it's however not > feasible to capture the entire content of RAM, so was thinking if pstore region can be > used to capture above information either in the form of elf or binary and how pstore > can get this in human readable format can be discussed further. > > Also, existing devcoredump driver does not cover panic usecases so we thought of something > like below RFC design where, > > 1. Device_coredump allocates some configurable contigous memory that can be controlled > via CONFIG or bootargs and later registers for panic notifiers. > 2. Notifier gets added. > 3. Pstore adds device_coredump as its front-end via dumper registration similar > to kmsg being dump today. > 4. Successful registration of dumper. > 5. A device driver(A-Z) can register their buffer to be dumped as part of panic. > 6. buffer gets added to the dump list. > 7. Panic occurs. > 8. iterate over registered drivers and copy their dump list to its own memory and if > it crosses device core dump memory log an error stop iterating. > 9. Similar to devcore_dump() inline with kmsg_dump() > 10.Copy the content to pstore region and this could be elf or raw binary that can be > discussed. > > > Device coredump memory(1) size could be passed from pstore and should be same as size > of devcoredump frontend size given in DT or some other way. > > Let me know your concern and view on this. > > > pstore device_coredump deviceA..Z panic > ┼ │ > │ │ panic_notifier(1) │ > │ ├───────────────────┬─────────────────►│ > │ │ notifier added │ (2) │ > │ │◄──────────────────┼──────────────────┤ > pstore can give │ (3) │ │ │ > its region for │ dumper_registration │ │ │ > dump ├─────────────────────►│ │ │ > │ (4) │ │ │ > │◄─────────────────────┤ │ │ > │ register_dumper │ │ │ > │ │ │ │ > │ │ │ │ > │ │ │ init │ > │ │ ───┼── │ > │ │ (5) │ │ > │ │devcoredump_register(dev,buf)); │ > │ │◄──────────────────┐ │ > │ │ (6) │ │ > │ ├──────────────────►│ │ > │ │device/buf gets add│d │ > │ │ to the list │ │panic > │ │ │ ──┼─── > │ │ │ │ > │ │(7) panic_notifier │ call │ > │ │◄──────────────────┼──────────────────┤ > │ │ prepare coredump │for the deviceA-Z │ > │ ├──────────────────►├────┐ │ > │ │ (8) │ │ │ > │ │ │ │ │ > │ │ │ │ │ > │ │ │◄───┘ │ > dumper will │ │ (9) │dev_coredump() > write all the device dump to │◄──────────────────┼──────────────────┤ > pstore ┌◄───┬─────────────────┤ │ │ > region │ │ (10) │ │ │ > │ │ │ │ │ > │ │ │ │ │ > │◄───┘ │ │ │ > │ │ │ │ > │ │ > > I really like the idea of providing an interface for device drivers to dump specific data for offline use. Currently in MSM/QCOM downstream kernel, we do register for panic notifiers and dump relavant data for later parsing/analysis. In some cases, this message gets printed to console so that it would also available in pstore / extracted dmesg. However having separate record/buffer for individual drivers would help. I believe dev_coredump() is just an analogy here as it is meant for taking a dump of the device when the driver thinks its device/firmware is hung/crashed etc, AFAICT. IIUC, What we are looking for here a different interface/semantics i.e querying devices to dump any information when the *system* is crashed/paniced. May be something like dev_panicdump() and a separate method in device_driver. Also, are there any use cases for this dev_panicdump() outside the pstore()? For ex: it would still be helpful to gather all the relavant data of a device and put it a separate buffer. The list of such buffers can be parsed offline via standard ramdump analysis. Thanks, Pavan ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: RFC design of device coredump collection on panic in Pstore 2023-05-08 15:51 RFC design of device coredump collection on panic in Pstore Mukesh Ojha 2023-05-15 6:52 ` Pavan Kondeti @ 2023-05-16 20:55 ` Kees Cook 2025-01-21 13:12 ` Eugen Hristev 1 sibling, 1 reply; 4+ messages in thread From: Kees Cook @ 2023-05-16 20:55 UTC (permalink / raw) To: Mukesh Ojha Cc: gregkh, tony.luck, gpiccoli, johannes, rafael, linux-hardening, linux-kernel, quic_shashim, quic_pkondeti On Mon, May 08, 2023 at 09:21:00PM +0530, Mukesh Ojha wrote: > 1. Device_coredump allocates some configurable contigous memory that can be controlled > via CONFIG or bootargs and later registers for panic notifiers. > 2. Notifier gets added. > 3. Pstore adds device_coredump as its front-end via dumper registration similar > to kmsg being dump today. > 4. Successful registration of dumper. > 5. A device driver(A-Z) can register their buffer to be dumped as part of panic. > 6. buffer gets added to the dump list. > 7. Panic occurs. > 8. iterate over registered drivers and copy their dump list to its own memory and if > it crosses device core dump memory log an error stop iterating. > 9. Similar to devcore_dump() inline with kmsg_dump() > 10.Copy the content to pstore region and this could be elf or raw binary that can be > discussed. Yeah, I think having something like the panic dump handlers for driver-specific data sounds like a fine idea (as long as we have actual users of such an interface). I think it would be easy to add another front-end to pstore (via the enums, etc), and expose it via a new template filename in the pstorefs. -- Kees Cook ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: RFC design of device coredump collection on panic in Pstore 2023-05-16 20:55 ` Kees Cook @ 2025-01-21 13:12 ` Eugen Hristev 0 siblings, 0 replies; 4+ messages in thread From: Eugen Hristev @ 2025-01-21 13:12 UTC (permalink / raw) To: Kees Cook, Mukesh Ojha Cc: gregkh, tony.luck, gpiccoli, johannes, rafael, linux-hardening, linux-kernel, quic_shashim, quic_pkondeti On 5/16/23 23:55, Kees Cook wrote: > On Mon, May 08, 2023 at 09:21:00PM +0530, Mukesh Ojha wrote: >> 1. Device_coredump allocates some configurable contigous memory that can be controlled >> via CONFIG or bootargs and later registers for panic notifiers. >> 2. Notifier gets added. >> 3. Pstore adds device_coredump as its front-end via dumper registration similar >> to kmsg being dump today. >> 4. Successful registration of dumper. >> 5. A device driver(A-Z) can register their buffer to be dumped as part of panic. >> 6. buffer gets added to the dump list. >> 7. Panic occurs. >> 8. iterate over registered drivers and copy their dump list to its own memory and if >> it crosses device core dump memory log an error stop iterating. >> 9. Similar to devcore_dump() inline with kmsg_dump() >> 10.Copy the content to pstore region and this could be elf or raw binary that can be >> discussed. > > Yeah, I think having something like the panic dump handlers for > driver-specific data sounds like a fine idea (as long as we have actual > users of such an interface). > > I think it would be easy to add another front-end to pstore (via the > enums, etc), and expose it via a new template filename in the pstorefs. > Hello Kees, Currently I am working to get a similar design on Pstore as described by Mukesh, with some differences. I want to be able to register memory areas into Pstore, not by doing a copy, just by registering via a new API (as a frontend, area pointer and size). Further, these memory areas would be registered to a new type of zone inside pstore/zone called directly mapped zone, and then registered to any kind of backend that would support it. The backend would then decide what to do with the memory area, have it ready in a list for a panic case, hangup, or hardware fault; or something else. This implementation would avoid memory copy operations, memory reservations, and dependency to a panic handler being run. It would work in multiple scenarios, not just when the kernel is being able to run a panic handler. Further, the data can be extracted either by JTAG, firmware, remote processor or any other solution, including dedicated storage, shared memory. One could then use it to construct a coredump or specific dump information. Having different memory areas being registered has the advantage of being flexible and small, ranging from nothing to a full kernel memory range. What do you think about this approach ? Do you see it working like this ? Thank you for your suggestions, Eugen ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-01-21 13:12 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-05-08 15:51 RFC design of device coredump collection on panic in Pstore Mukesh Ojha 2023-05-15 6:52 ` Pavan Kondeti 2023-05-16 20:55 ` Kees Cook 2025-01-21 13:12 ` Eugen Hristev
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox