[Linux-ia64] Dump driver module

[Linux-ia64] Dump driver module

[Bruno Vidal]

	Hi all.
I've already wrote a dump modules driver for linux-parisc.
This module goal is to create a memory image on a swap area, and
at reboot time to save it to disk with all kernel modules, in
order to analyze it after by "support" people with tools like
gdb/p4. The problem while dumping is that the dump modules
cannot trust anymore the system, so dumping means: no interruption,
no disk driver, no buffer, nothing. The solution is to use low level
call. For parisc I use the IODC calls, for ia64 I think I'll
use the EFI calls. My questions:
-do you think it is a "good" and realistic solution.
-because I think "yes", does exist somewhere an example of
reading/writing with EFI call on disk using BLOCK IO. I've already
looked in elilo, but it seems that it use FS access.

	Thanks.

-- 
	Vidal Bruno, (770-4271)
         SSD-HA Team, HP-UX & LINUX Support
	bruno_vidal@admin.france.hp.com

Re: [Linux-ia64] Dump driver module

[Stephane Eranian]

Salut Bruno,

On Tue, May 13, 2003 at 09:02:48AM +0200, Bruno Vidal wrote:
> I've already wrote a dump modules driver for linux-parisc.
> This module goal is to create a memory image on a swap area, and
> at reboot time to save it to disk with all kernel modules, in
> order to analyze it after by "support" people with tools like
> gdb/p4. The problem while dumping is that the dump modules
> cannot trust anymore the system, so dumping means: no interruption,
> no disk driver, no buffer, nothing. The solution is to use low level
> call. For parisc I use the IODC calls, for ia64 I think I'll
> use the EFI calls. My questions:
> -do you think it is a "good" and realistic solution.

I don't think it is because, I think you are missing an important
feature of EFI. The last step taken by elilo before jumping to the
first instruction of the kernel is to switch EFI into runtime mode
(compared to boot time mode intially). There is no way back. In
runtime mode, almost none of the EFI services you get at boot time
remain available. The code/data memory areas are reclaimed by the kernel. 
The parts of EFI that remain are tagged 'runtime' and mostly consist
of a subset of what you have at bootime: reset, gettimeofday, runtime
drivers (fpswa). None of the I/O support is left.


the kernel, EFI is 
> -because I think "yes", does exist somewhere an example of
> reading/writing with EFI call on disk using BLOCK IO. I've already
> looked in elilo, but it seems that it use FS access.

Not quite. If you looked at the ext2 emulation, it uses the BlockIO 
protocol.

-- 
-Stephane

Re: [Linux-ia64] Dump driver module

[David Mosberger]

>>>>> On Tue, 13 May 2003 09:02:48 +0200, Bruno Vidal <bruno_vidal@hp.com> said:

  Bruno> I've already wrote a dump modules driver for linux-parisc.
  Bruno> This module goal is to create a memory image on a swap area, and
  Bruno> at reboot time to save it to disk with all kernel modules, in
  Bruno> order to analyze it after by "support" people with tools like
  Bruno> gdb/p4. The problem while dumping is that the dump modules
  Bruno> cannot trust anymore the system, so dumping means: no interruption,
  Bruno> no disk driver, no buffer, nothing. The solution is to use low level
  Bruno> call. For parisc I use the IODC calls, for ia64 I think I'll
  Bruno> use the EFI calls. My questions:
  Bruno> -do you think it is a "good" and realistic solution.
  Bruno> -because I think "yes", does exist somewhere an example of
  Bruno> reading/writing with EFI call on disk using BLOCK IO. I've already
  Bruno> looked in elilo, but it seems that it use FS access.

Once the kernel is booted, you can access only the EFI runtime
services (see struct efi in include/linux/efi.h).  I'm afraid there is
no support for writing disk blocks.  Given this limitation, I suspect
you'd have to include your own (simple) disk driver(s).

	--david

Re: [Linux-ia64] Dump driver module

[Bruno Vidal]

David Mosberger wrote:
>>>>>>On Tue, 13 May 2003 09:02:48 +0200, Bruno Vidal <bruno_vidal@hp.com> said:
>>>>>
> 
>   Bruno> I've already wrote a dump modules driver for linux-parisc.
>   Bruno> This module goal is to create a memory image on a swap area, and
>   Bruno> at reboot time to save it to disk with all kernel modules, in
>   Bruno> order to analyze it after by "support" people with tools like
>   Bruno> gdb/p4. The problem while dumping is that the dump modules
>   Bruno> cannot trust anymore the system, so dumping means: no interruption,
>   Bruno> no disk driver, no buffer, nothing. The solution is to use low level
>   Bruno> call. For parisc I use the IODC calls, for ia64 I think I'll
>   Bruno> use the EFI calls. My questions:
>   Bruno> -do you think it is a "good" and realistic solution.
>   Bruno> -because I think "yes", does exist somewhere an example of
>   Bruno> reading/writing with EFI call on disk using BLOCK IO. I've already
>   Bruno> looked in elilo, but it seems that it use FS access.
> 
> Once the kernel is booted, you can access only the EFI runtime
> services (see struct efi in include/linux/efi.h).  I'm afraid there is
> no support for writing disk blocks.  Given this limitation, I suspect
> you'd have to include your own (simple) disk driver(s).
> 
> 	--david
> 
> .
> 

I've already seen this EFI runtime service, and there is nearly nothing in it
(nothing usefull for IO), so what I can understand is I have to write (rewrite)
a driver for each kind of scsi/ide interface....it is not really realistic :-(
There is no other way to access EFI or firmware ? Or reset back EFI to have access
to BLOCK IO interface ? For example in parisc, the first thing to do, is to a
do a PDC_IO_RESET, it freeze the entire system and it is possible to have access
to all firmware calls. Does exist something similar with EFI.

	Thanks.

-- 
	Vidal Bruno, (770-4271)
         SSD-HA Team, HP-UX & LINUX Support
	bruno_vidal@admin.france.hp.com

Re: [Linux-ia64] Dump driver module

[David Mosberger]

>>>>> On Tue, 13 May 2003 09:38:05 +0200, Bruno Vidal <bruno_vidal@hp.com> said:

  Bruno> There is no other way to access EFI or firmware ?

The EFI standard certainly won't help you here.  Whether or not
vendors have their own backdoors to a low-level disk driver, I don't
know.

  Bruno> Or reset back EFI to have access to BLOCK IO interface ?

You can warm-boot via EFI.  But I'm not sure there is any guarantee
that all kernel memory will be preserved.

	--david

RE: [Linux-ia64] Dump driver module

[Howell, David P]

Have you looked at the LKCD project? It supports ia64 and does about
what you are doing on the dump side, and has dump analysis tools.

If you've already considered this just delete this.

Thanks,
Dave Howell


These are my opinions and not official opinions of Intel Corp.

David Howell
Intel Corporation
Telco Server Development
Server Products Division
Voice: (803) 461-6112  Fax: (803) 461-6292

Intel Corporation
Columbia Design Center, CBA-2
250 Berryhill Road, Suite 100
Columbia, SC 29210

david.p.howell@intel.com


-----Original Message-----
From: Bruno Vidal [mailto:bruno_vidal@hp.com] 
Sent: Tuesday, May 13, 2003 3:03 AM
To: linux-ia64@linuxia64.org
Subject: [Linux-ia64] Dump driver module

	Hi all.
I've already wrote a dump modules driver for linux-parisc.
This module goal is to create a memory image on a swap area, and
at reboot time to save it to disk with all kernel modules, in
order to analyze it after by "support" people with tools like
gdb/p4. The problem while dumping is that the dump modules
cannot trust anymore the system, so dumping means: no interruption,
no disk driver, no buffer, nothing. The solution is to use low level
call. For parisc I use the IODC calls, for ia64 I think I'll
use the EFI calls. My questions:
-do you think it is a "good" and realistic solution.
-because I think "yes", does exist somewhere an example of
reading/writing with EFI call on disk using BLOCK IO. I've already
looked in elilo, but it seems that it use FS access.

	Thanks.

-- 
	Vidal Bruno, (770-4271)
         SSD-HA Team, HP-UX & LINUX Support
	bruno_vidal@admin.france.hp.com


_______________________________________________
Linux-IA64 mailing list
Linux-IA64@linuxia64.org
http://lists.linuxia64.org/lists/listinfo/linux-ia64

Re: [Linux-ia64] Dump driver module

[Bruno Vidal]

	Hi
Sorry, but LKCD is really not usefull because it use the buffer
to write on the device. So it just hang in case of data page fault for
example (because interuption are masked), LKCD is also not working in case
of buffer corruption, disk driver pb, etc.... so LKCD is not usable in
lot's of case (and it happen really often to have data page fault).

Cheers.

Howell, David P wrote:
> Have you looked at the LKCD project? It supports ia64 and does about
> what you are doing on the dump side, and has dump analysis tools.
> 
> If you've already considered this just delete this.
> 
> Thanks,
> Dave Howell
> 
> 
> These are my opinions and not official opinions of Intel Corp.
> 
> David Howell
> Intel Corporation
> Telco Server Development
> Server Products Division
> Voice: (803) 461-6112  Fax: (803) 461-6292
> 
> Intel Corporation
> Columbia Design Center, CBA-2
> 250 Berryhill Road, Suite 100
> Columbia, SC 29210
> 
> david.p.howell@intel.com
> 
> 
> -----Original Message-----
> From: Bruno Vidal [mailto:bruno_vidal@hp.com] 
> Sent: Tuesday, May 13, 2003 3:03 AM
> To: linux-ia64@linuxia64.org
> Subject: [Linux-ia64] Dump driver module
> 
> 	Hi all.
> I've already wrote a dump modules driver for linux-parisc.
> This module goal is to create a memory image on a swap area, and
> at reboot time to save it to disk with all kernel modules, in
> order to analyze it after by "support" people with tools like
> gdb/p4. The problem while dumping is that the dump modules
> cannot trust anymore the system, so dumping means: no interruption,
> no disk driver, no buffer, nothing. The solution is to use low level
> call. For parisc I use the IODC calls, for ia64 I think I'll
> use the EFI calls. My questions:
> -do you think it is a "good" and realistic solution.
> -because I think "yes", does exist somewhere an example of
> reading/writing with EFI call on disk using BLOCK IO. I've already
> looked in elilo, but it seems that it use FS access.
> 
> 	Thanks.
> 


-- 
	Vidal Bruno, (770-4271)
         SSD-HA Team, HP-UX & LINUX Support
	bruno_vidal@hp.com

Re: [Linux-ia64] Dump driver module

[Keith Owens]

On Tue, 13 May 2003 16:39:30 +0200, 
Bruno Vidal <bruno_vidal@hp.com> wrote:
>Sorry, but LKCD is really not usefull because it use the buffer
>to write on the device. So it just hang in case of data page fault for
>example (because interuption are masked), LKCD is also not working in case
>of buffer corruption, disk driver pb, etc.... so LKCD is not usable in
>lot's of case (and it happen really often to have data page fault).

lkcd should not get data page faults.  Are you sure that is what you
are seeing and not some other problem that hangs lkcd?

Re: [Linux-ia64] Dump driver module

[Martin Pool]

On 13 May 2003, Bruno Vidal <bruno_vidal@hp.com> wrote:
> 	Hi
> Sorry, but LKCD is really not usefull because it use the buffer
> to write on the device. So it just hang in case of data page fault for
> example (because interuption are masked), LKCD is also not working in case
> of buffer corruption, disk driver pb, etc.... so LKCD is not usable in
> lot's of case (and it happen really often to have data page fault).

There is also the netconsole/netdump system.  This is supposed to rely
on only a very minimal network driver.  As you say, writing to disk
when kernel memory may have been corrupted is a risky business, not
only because you might hang but also because you might write over the
wrong region.  netdump doesn't do any disk IO for that reason.  I seem
to recall that Linus liked this property.

Of course it's only useful if the machine is on a network where there
is another machine to catch the data, but I would expect that to be
the case for most ia64 machines.

  http://www.redhat.com/support/wpapers/redhat/netdump/

-- 
Martin 

Re: [Linux-ia64] Dump driver module

[David Mosberger]

>>>>> On Wed, 14 May 2003 12:14:21 +1000, Martin Pool <mbp@samba.org> said:

  Martin> On 13 May 2003, Bruno Vidal <bruno_vidal@hp.com> wrote:
  >> Hi
  >> Sorry, but LKCD is really not usefull because it use the buffer
  >> to write on the device. So it just hang in case of data page fault for
  >> example (because interuption are masked), LKCD is also not working in case
  >> of buffer corruption, disk driver pb, etc.... so LKCD is not usable in
  >> lot's of case (and it happen really often to have data page fault).

  Martin> There is also the netconsole/netdump system.  This is supposed to rely
  Martin> on only a very minimal network driver.  As you say, writing to disk
  Martin> when kernel memory may have been corrupted is a risky business, not
  Martin> only because you might hang but also because you might write over the
  Martin> wrong region.  netdump doesn't do any disk IO for that reason.  I seem
  Martin> to recall that Linus liked this property.

  Martin> Of course it's only useful if the machine is on a network where there
  Martin> is another machine to catch the data, but I would expect that to be
  Martin> the case for most ia64 machines.

  Martin> http://www.redhat.com/support/wpapers/redhat/netdump/

It strikes me that for a really reliable crash-dump, you'd want to
read _all_ the bits needed to do the dumping from ROM.  Assuming you
have a (minimal) disk driver/network driver written in EFI byte code,
all you'd need is the EFI byte-code interpreter which hopefully would
fit in a fixed (and reasonable) amount of space.  Hence you could
reserve some memory for this purpose and on a crash, load the
byte-code interpreter from ROM and get going that way.

	--david

Re: [Linux-ia64] Dump driver module

[Martin Pool]

On 13 May 2003, David Mosberger <davidm@napali.hpl.hp.com> wrote:

> It strikes me that for a really reliable crash-dump, you'd want to
> read _all_ the bits needed to do the dumping from ROM.

But you also need to know which disk to write to, and where.  And
presumably "which disk" actually means knowing the PCI path to the
SCSI host adapter, and then the path to the disk.  "Where" might be
just the index of start block, if you reserved a contiguous partition,
or it might be more complex.  If any of this got screwed up, you might
write your dump over /home instead -- and then rather than a possibly
once-off kernel crash, you've lost lots of data.

So for a really reliable dump, you'd want to check and "lock down"
these variables at startup into a region of memory not easily changed
by the kernel, and accessible by EFI.  NVRAM?  I don't know if there's
anything suitable on IPF.

It would be an interesting project, although since network dumps run a
lower risk of clobbering filesystems they're probably a smarter
design.  It would be cool to do it in EFI, if that's possible.

> Assuming you have a (minimal) disk driver/network driver written in
> EFI byte code, all you'd need is the EFI byte-code interpreter which
> hopefully would fit in a fixed (and reasonable) amount of space.
> Hence you could reserve some memory for this purpose and on a crash,
> load the byte-code interpreter from ROM and get going that way.

-- 
Martin 

Re: [Linux-ia64] Dump driver module

[David Mosberger]

>>>>> On Wed, 14 May 2003 12:32:17 +1000, Martin Pool <mbp@samba.org> said:

  Martin> So for a really reliable dump, you'd want to check and "lock
  Martin> down" these variables at startup into a region of memory not
  Martin> easily changed by the kernel, and accessible by EFI.  NVRAM?
  Martin> I don't know if there's anything suitable on IPF.

EFI provides (persistent) environment variables for this purpose.
Don't know how well they're protected, typically, but that's probably
more of an implementation issue than anything else.

  Martin> It would be an interesting project, although since network
  Martin> dumps run a lower risk of clobbering filesystems they're
  Martin> probably a smarter design.  It would be cool to do it in
  Martin> EFI, if that's possible.

"Smart" is relative. I'm typing this on an Itanium box which doesn't
have anything on the network that could serve as a dump device (it's
own disk would be fine, though).

	--david

Re: [Linux-ia64] Dump driver module

[Grant Grundler]

On Wed, May 14, 2003 at 12:32:17PM +1000, Martin Pool wrote:
> But you also need to know which disk to write to, and where.
...
> If any of this got screwed up, you might
> write your dump over /home instead -- and then rather than a possibly
> once-off kernel crash, you've lost lots of data.

I'll take that risk since I've never seen it nor heard of it
happening with HPUX after ~6 years of working HPUX IO subsystems.
I'm sure HPUX has done things to help mitigate that risk.
(like storing paths in NVRAM or some other hard to reach place.)
All problem reports (including memory corruption problems) came with
crash dumps.

Besides, fsck after the reboot is much more likely to trash data
on disk than the original memory corruption problem.

While it's clearly feasible to dump over gigabit networks,
I'm not sure how pervasive gigabit networks are.
Probably standard now for core commercial/scientific centers.
I fear dumping 4GB of memory over 100BT would be painful.
(I pulled 4GB out of a hat; rx5670 supports 48GB of RAM).

grant

Re: [Linux-ia64] Dump driver module

[Bruno Vidal]

	Hi
Many thanks, it seems that my question was interresting.
I just want to answer to some point:
> 
> But you also need to know which disk to write to, and where.  And
> presumably "which disk" actually means knowing the PCI path to the
> SCSI host adapter, and then the path to the disk.  "Where" might be
> just the index of start block, if you reserved a contiguous partition,
> or it might be more complex.  If any of this got screwed up, you might
> write your dump over /home instead -- and then rather than a possibly
> once-off kernel crash, you've lost lots of data.

HP-UX is already doing that. It keep in memory where to put the dump, and
the best place is swap partition. And it is really easy to verify before writing
that we are at the right place by checking any swap magic number in order to
avoid any overwriting (with linux it is really easy to check that).
And about the PCI path, it is not so hard to find it, and I saw that EFI provide
a function to retrieve the media ID by using the hardware path. Now it is clear
that in case of hardware failure  their is still cases where it avoid to take dumps,
but I never heard/seen HP-UX overwriting data while dumping, even in case of memory
corruption.

Now there is some other solution in order to take a dump. It is another path
I didn't try yet but if you can say what you think about it:
1. dump module reserve some amount of memory at the end of the memory (1Mb)
2. system call panic()
3. dump module start to compress memory, starting by the last address (without
the pages reserved by the module), and put the result in the last reserved pages
4. free the compressed pages, and then compress new pages to put it in freed pages
and so on until no more pages to compress
5. put a flag somewhere in NVRAM (or elsewhere)
6. do a warm boot in order to not reset the memory, and the compress memory
dump pages must be marked as allocated be the new kernel.
7. now we have a brand new running kernel with all drivers and a bunch of
memory pages that contain the dump, so we can push it directly on filesystem

The main problem with this kind of dump, is that it is not possible to takes
early dumps. It means also that firmware is able to reboot without modifying
any memory pages, it mean also that the kernel is able at boot time to flag
a large amount of pages as allocated. But the good effects are: a quick dump
(only memory access), a system up and running fairly fast (with less memory
at boot time but for a short periode of time). It can be a good choice for
critical application that require a low donwtime, and that have often a large
amount of memory (if you already try to dump a superdome with 64Gb by using
PDC call you understand what I mean).

	Cheers.

-- 
	Vidal Bruno, (770-4271)
         SSD-HA Team, HP-UX & LINUX Support
	bruno_vidal@hp.com

Re: [Linux-ia64] Dump driver module

[Matthew Wilcox]

On Tue, May 13, 2003 at 11:33:33PM -0700, Grant Grundler wrote:
> On Wed, May 14, 2003 at 12:32:17PM +1000, Martin Pool wrote:
> > But you also need to know which disk to write to, and where.
> ...
> > If any of this got screwed up, you might
> > write your dump over /home instead -- and then rather than a possibly
> > once-off kernel crash, you've lost lots of data.
> 
> I'll take that risk since I've never seen it nor heard of it
> happening with HPUX after ~6 years of working HPUX IO subsystems.
> I'm sure HPUX has done things to help mitigate that risk.
> (like storing paths in NVRAM or some other hard to reach place.)
> All problem reports (including memory corruption problems) came with
> crash dumps.

But those HPUX systems were on PA which has PDC which can write to
the disc.  The problem on ia64 is that EFI can't after the OS has started.

I seem to remember lamont eulogising the MPE way of doing crashdumps
-- Transfer of Control really does transfer control to a completely
different kernel that does the dump for you.  That seems a much better
way of doing things to me.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [Linux-ia64] Dump driver module

[Grant Grundler]

On Wed, May 14, 2003 at 02:29:00PM +0100, Matthew Wilcox wrote:
> But those HPUX systems were on PA which has PDC which can write to
> the disc.  The problem on ia64 is that EFI can't after the OS has started.

yup - I understood that. The "where to write" information getting
corrupted is the same problem regardless of where the driver code
to do the writes exists.

grant