[PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Mahesh J Salgaonkar]

Hi All,

Please find the version 2 of makedumpfile enhancement patchset that introduces
a data filtering feature which enables makedumpfile to filter out desired kernel
symbol data and it's members from the specified VMCORE file. The data to be
filtered out is poisoned with character 'X' (58 in Hex).

This feature will be very useful for the customers who wants to erase the
customer sensitive data like security keys and other confidential data, in
DUMPFILE before sending it to support team for analysis.

This feature introduces a filter config file where, using filter commands,
user can specify desired kernel data symbols and it's members that need to be
filtered out while creating o/p DUMPFILE. The Syntax for filter commands are
provided in the makedumpfile.conf(8) man page.

The first 4 patches prepares the base work for filtering framework. The
patches 5 and 6 implements the generic filtering framework to erase desired
kernel data. The last two patches implements inclusion of eraseinfo data into
o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the
effective lines from configuration files.

I have tested these patches on x86_64 and s390x architecture against RHEL6 GA
kernel. The feature supports filtering data from ELF as well as
kdump-compressed formatted dump.

Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data
  into kdump compressed as well as ELF formatted dumpfile.

Please review the patchset and let me know your comments.

Thanks,
-Mahesh.
---

Mahesh Salgaonkar (8):
      makedumpfile: Add '--config' option to specify filter config file.
      makedumpfile: Apply relocation while loading module debuginfo.
      makedumpfile: Load the module symbol data from vmcore.
      makedumpfile: Introduce routines to get type name from debuginfo.
      makedumpfile: Read and process filter commands from config file.
      makedumpfile: Read and process 'for' command from config file.
      makedumpfile: Add erased information in compressed kdump file
      makedumpfile: Add erase information in ELF formatted dumpfile


 IMPLEMENTATION      |    6 
 Makefile            |    8 
 diskdump_mod.h      |    2 
 makedumpfile.8      |   20 
 makedumpfile.c      | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++---
 makedumpfile.conf   |  149 +++
 makedumpfile.conf.8 |  419 ++++++++
 makedumpfile.h      |  199 ++++
 8 files changed, 3394 insertions(+), 156 deletions(-)
 create mode 100644 makedumpfile.conf
 create mode 100644 makedumpfile.conf.8

-- 
Signature

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Ken'ichi Ohmichi]

Hi Mahesh,

Thank you for the patches.

But now I am out of the makedumpfile devel and other OSS devels.
The main maintainer of makedumpfile is Tachibana-san (tachibana@mxm.nes.nec.co.jp)
and he will help you.


Thanks
Ken'ichi Ohmichi

--
On Wed, 18 May 2011 01:29:06 +0530
Mahesh J Salgaonkar <mahesh@linux.vnet.ibm.com> wrote:
>
> Hi All,
> 
> Please find the version 2 of makedumpfile enhancement patchset that introduces
> a data filtering feature which enables makedumpfile to filter out desired kernel
> symbol data and it's members from the specified VMCORE file. The data to be
> filtered out is poisoned with character 'X' (58 in Hex).
> 
> This feature will be very useful for the customers who wants to erase the
> customer sensitive data like security keys and other confidential data, in
> DUMPFILE before sending it to support team for analysis.
> 
> This feature introduces a filter config file where, using filter commands,
> user can specify desired kernel data symbols and it's members that need to be
> filtered out while creating o/p DUMPFILE. The Syntax for filter commands are
> provided in the makedumpfile.conf(8) man page.
> 
> The first 4 patches prepares the base work for filtering framework. The
> patches 5 and 6 implements the generic filtering framework to erase desired
> kernel data. The last two patches implements inclusion of eraseinfo data into
> o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the
> effective lines from configuration files.
> 
> I have tested these patches on x86_64 and s390x architecture against RHEL6 GA
> kernel. The feature supports filtering data from ELF as well as
> kdump-compressed formatted dump.
> 
> Change in v2:
> - Changes in patch 5 and inclusion of new patches 7 & 8
> - Re-named filter.conf file to makedumpfile.conf and some cleanup work.
> - Introduced last two patches that implements inclusion of eraseinfo data
>   into kdump compressed as well as ELF formatted dumpfile.
> 
> Please review the patchset and let me know your comments.
> 
> Thanks,
> -Mahesh.
> ---
> 
> Mahesh Salgaonkar (8):
>       makedumpfile: Add '--config' option to specify filter config file.
>       makedumpfile: Apply relocation while loading module debuginfo.
>       makedumpfile: Load the module symbol data from vmcore.
>       makedumpfile: Introduce routines to get type name from debuginfo.
>       makedumpfile: Read and process filter commands from config file.
>       makedumpfile: Read and process 'for' command from config file.
>       makedumpfile: Add erased information in compressed kdump file
>       makedumpfile: Add erase information in ELF formatted dumpfile
> 
> 
>  IMPLEMENTATION      |    6 
>  Makefile            |    8 
>  diskdump_mod.h      |    2 
>  makedumpfile.8      |   20 
>  makedumpfile.c      | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++---
>  makedumpfile.conf   |  149 +++
>  makedumpfile.conf.8 |  419 ++++++++
>  makedumpfile.h      |  199 ++++
>  8 files changed, 3394 insertions(+), 156 deletions(-)
>  create mode 100644 makedumpfile.conf
>  create mode 100644 makedumpfile.conf.8
> 
> -- 
> Signature


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Vivek Goyal]

On Wed, May 18, 2011 at 01:29:06AM +0530, Mahesh J Salgaonkar wrote:
> Hi All,
> 
> Please find the version 2 of makedumpfile enhancement patchset that introduces
> a data filtering feature which enables makedumpfile to filter out desired kernel
> symbol data and it's members from the specified VMCORE file. The data to be
> filtered out is poisoned with character 'X' (58 in Hex).
> 
> This feature will be very useful for the customers who wants to erase the
> customer sensitive data like security keys and other confidential data, in
> DUMPFILE before sending it to support team for analysis.

Mahesh,

Apart from security keys, what are the other examples of data which needs
to be removed?

By erasing the data, you mean you set it to zero? Will it make sense to
poison it with some specific pattern so that if crash or other tool
are looking into it they know it has been posioned and it is not a
case of corruption. This might help if a user erased a symbol which was
required by crash for some analysis etc.

Thanks
Vivek

> 
> This feature introduces a filter config file where, using filter commands,
> user can specify desired kernel data symbols and it's members that need to be
> filtered out while creating o/p DUMPFILE. The Syntax for filter commands are
> provided in the makedumpfile.conf(8) man page.
> 
> The first 4 patches prepares the base work for filtering framework. The
> patches 5 and 6 implements the generic filtering framework to erase desired
> kernel data. The last two patches implements inclusion of eraseinfo data into
> o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the
> effective lines from configuration files.
> 
> I have tested these patches on x86_64 and s390x architecture against RHEL6 GA
> kernel. The feature supports filtering data from ELF as well as
> kdump-compressed formatted dump.
> 
> Change in v2:
> - Changes in patch 5 and inclusion of new patches 7 & 8
> - Re-named filter.conf file to makedumpfile.conf and some cleanup work.
> - Introduced last two patches that implements inclusion of eraseinfo data
>   into kdump compressed as well as ELF formatted dumpfile.
> 
> Please review the patchset and let me know your comments.
> 
> Thanks,
> -Mahesh.
> ---
> 
> Mahesh Salgaonkar (8):
>       makedumpfile: Add '--config' option to specify filter config file.
>       makedumpfile: Apply relocation while loading module debuginfo.
>       makedumpfile: Load the module symbol data from vmcore.
>       makedumpfile: Introduce routines to get type name from debuginfo.
>       makedumpfile: Read and process filter commands from config file.
>       makedumpfile: Read and process 'for' command from config file.
>       makedumpfile: Add erased information in compressed kdump file
>       makedumpfile: Add erase information in ELF formatted dumpfile
> 
> 
>  IMPLEMENTATION      |    6 
>  Makefile            |    8 
>  diskdump_mod.h      |    2 
>  makedumpfile.8      |   20 
>  makedumpfile.c      | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++---
>  makedumpfile.conf   |  149 +++
>  makedumpfile.conf.8 |  419 ++++++++
>  makedumpfile.h      |  199 ++++
>  8 files changed, 3394 insertions(+), 156 deletions(-)
>  create mode 100644 makedumpfile.conf
>  create mode 100644 makedumpfile.conf.8
> 
> -- 
> Signature
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Dave Anderson]

----- Original Message -----
> On Wed, May 18, 2011 at 01:29:06AM +0530, Mahesh J Salgaonkar wrote:
> > Hi All,
> >
> > Please find the version 2 of makedumpfile enhancement patchset that introduces
> > a data filtering feature which enables makedumpfile to filter out desired kernel
> > symbol data and it's members from the specified VMCORE file. The data to be
> > filtered out is poisoned with character 'X' (58 in Hex).
> >
> > This feature will be very useful for the customers who wants to erase the
> > customer sensitive data like security keys and other confidential data, in
> > DUMPFILE before sending it to support team for analysis.
> 
> Mahesh,
> 
> Apart from security keys, what are the other examples of data which needs
> to be removed?
> 
> By erasing the data, you mean you set it to zero? Will it make sense to
> poison it with some specific pattern so that if crash or other tool
> are looking into it they know it has been posioned and it is not a
> case of corruption. This might help if a user erased a symbol which was
> required by crash for some analysis etc.
> 
> Thanks
> Vivek

As I said when the first patch-set was proposed, there is no way
that I am going to put code into the crash utility that will somehow 
"work around" memory that has been poisoned.  The only thing I plan to
do is to check whether the new segments exist in the compressed
kdump or ELF kdump headers -- and if they do, print a warning message 
during invocation informing the user of that fact.  If Mahesh wants to
dump the contents of the sections, he can post a patch to the crash-utility
list.

And then, whatever breaks, breaks -- it is not the crash utility's
problem to handle.  Just looking at the examples in the documentation
made me nervous -- IIRC the kernel "vmlist" pointer, and "vm_struct.addr"
are shown -- both of which would cripple the crash utility.

Dave


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Vivek Goyal]

On Wed, May 25, 2011 at 10:41:55AM +0200, Reinhard Buendgen wrote:
> Hi,
> 
> to answer Vivek questions first: Eventually we want to be able to erase 
> all data that a customer may consider sensitive to her privacy. In 
> addition to encryption key that may be the contents (i.e. payload within) 
> of all kinds of I/O buffers. Consider you are running a kvm based 
> hypervisor and want its dump to be analyized while promising your 
> customers whose guests you run on that hypervisor that none of their data 
> will be externalized. Or consider your system reads a spreadsheat with 
> bank account or health information. You might not want to send fractions 
> of that information sitting in some buffers to a service organization.

So for direct IO, buffer is still in user space and should be filtered
out when we filter out user space pages using mkdumpfile. For kvm, I am
assuming that all the pages belong to qemu process and once we are
filtering out user space pages, any data belonging to guest will go away.

So atleast for above examples it does not sound as if we need symbol
erase infrastructure.

> 
> to answer Daves concern: there is no intention that crash should ever look 
> into the erased structures. In theroy it should not be needed because the 
> contents of structures to be deleted should be irrelevant to kernel 
> debugging.

So what are those kernel structures which we are planning to delete and
are irrelevant to kernel debugging by crash?

I think we are missing something here. If there are only few known
structures we want to get rid of, lets hardcode it in makedumpfile
instead of giving user a generic infrastructure. That way we know
that we are not leaking information at the same time making sure
that analysis tools are working.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Reinhard Buendgen]

[-- Attachment #1.1: Type: text/plain, Size: 3347 bytes --]

Vivek,

I/O is not restricted to disk I/O (it may be network I/O, data sent to 
crtypto cards etc.) and it is not always direct, Device drivers may have 
buffers to which such data is copied. 

So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because 
you would need a new makedumpfile release for every Distribution 
(release).

Allowing to configure makedumpfile allows each distribution and each 
platform to provide appropriate filters.

Mit freundlichen Grüßen/Best Regards/Cordialement 

Reinhard

Dr. Reinhard Bündgen 
RAS & Crypto Architect for Linux on System z 
Virtualization and Systems Management 
 
Mail:buendgen@de.ibm.com
Phone: ++49-(0)7031-16-1130
Fax: ++49-(0)7031-16-3456 
 
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Dirk Wittkopp 
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294





From:   Vivek Goyal <vgoyal@redhat.com>
To:     Reinhard Buendgen/Germany/IBM@IBMDE
Cc:     Dave Anderson <anderson@redhat.com>, Ananth N Mavinakayanahalli 
<ananth@in.ibm.com>, kexec@lists.infradead.org, Mahesh J Salgaonkar 
<mahesh@linux.vnet.ibm.com>, "Ken'ichi Ohmichi" 
<oomichi@mxs.nes.nec.co.jp>, V Srivatsa <vsrivatsa@in.ibm.com>
Date:   25.05.2011 21:53
Subject:        Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement 
to filter out kernel data from vmcore



On Wed, May 25, 2011 at 10:41:55AM +0200, Reinhard Buendgen wrote:
> Hi,
> 
> to answer Vivek questions first: Eventually we want to be able to erase 
> all data that a customer may consider sensitive to her privacy. In 
> addition to encryption key that may be the contents (i.e. payload 
within) 
> of all kinds of I/O buffers. Consider you are running a kvm based 
> hypervisor and want its dump to be analyized while promising your 
> customers whose guests you run on that hypervisor that none of their 
data 
> will be externalized. Or consider your system reads a spreadsheat with 
> bank account or health information. You might not want to send fractions 

> of that information sitting in some buffers to a service organization.

So for direct IO, buffer is still in user space and should be filtered
out when we filter out user space pages using mkdumpfile. For kvm, I am
assuming that all the pages belong to qemu process and once we are
filtering out user space pages, any data belonging to guest will go away.

So atleast for above examples it does not sound as if we need symbol
erase infrastructure.

> 
> to answer Daves concern: there is no intention that crash should ever 
look 
> into the erased structures. In theroy it should not be needed because 
the 
> contents of structures to be deleted should be irrelevant to kernel 
> debugging.

So what are those kernel structures which we are planning to delete and
are irrelevant to kernel debugging by crash?

I think we are missing something here. If there are only few known
structures we want to get rid of, lets hardcode it in makedumpfile
instead of giving user a generic infrastructure. That way we know
that we are not leaking information at the same time making sure
that analysis tools are working.

Thanks
Vivek


[-- Attachment #1.2: Type: text/html, Size: 5200 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Vivek Goyal]

On Thu, May 26, 2011 at 01:15:14PM +0200, Reinhard Buendgen wrote:
> Vivek,
> 
> I/O is not restricted to disk I/O (it may be network I/O, data sent to 
> crtypto cards etc.) and it is not always direct, Device drivers may have 
> buffers to which such data is copied. 
> 
> So it is more than just keys, and it may change over time.
> I do not think hardwiring a filter in makedumpfile is a good idea because 
> you would need a new makedumpfile release for every Distribution 
> (release).

Ok, so we are worried about data being in slub/slab caches or driver's
kmalloced buffers etc.

When do I need access to debuginfo files. I am assuming that makedumpfile
reads them in first kernel sometime and stores relevant info in initramfs.
Otherwise, it is not possible to get to it in second kernel after crash.

> 
> Allowing to configure makedumpfile allows each distribution and each 
> platform to provide appropriate filters.
> 

I was thinking that there are very few symbols and these will not change
frequently. makdumpfile is already dependent on some kenrel data strcutres
and symbols for doing filtering. 

But looks like you are planning to filter out lots of symbols. So in that
case agreed that hardcoding is not a good idea.

Thanks
Vivek

> Mit freundlichen Grüßen/Best Regards/Cordialement 
> 
> Reinhard
> 
> Dr. Reinhard Bündgen 
> RAS & Crypto Architect for Linux on System z 
> Virtualization and Systems Management 
>  
> Mail:buendgen@de.ibm.com
> Phone: ++49-(0)7031-16-1130
> Fax: ++49-(0)7031-16-3456 
>  
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Martin Jetter
> Geschäftsführung: Dirk Wittkopp 
> Sitz der Gesellschaft: Böblingen
> Registergericht: Amtsgericht Stuttgart, HRB 243294
> 
> 
> 
> 
> 
> From:   Vivek Goyal <vgoyal@redhat.com>
> To:     Reinhard Buendgen/Germany/IBM@IBMDE
> Cc:     Dave Anderson <anderson@redhat.com>, Ananth N Mavinakayanahalli 
> <ananth@in.ibm.com>, kexec@lists.infradead.org, Mahesh J Salgaonkar 
> <mahesh@linux.vnet.ibm.com>, "Ken'ichi Ohmichi" 
> <oomichi@mxs.nes.nec.co.jp>, V Srivatsa <vsrivatsa@in.ibm.com>
> Date:   25.05.2011 21:53
> Subject:        Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement 
> to filter out kernel data from vmcore
> 
> 
> 
> On Wed, May 25, 2011 at 10:41:55AM +0200, Reinhard Buendgen wrote:
> > Hi,
> > 
> > to answer Vivek questions first: Eventually we want to be able to erase 
> > all data that a customer may consider sensitive to her privacy. In 
> > addition to encryption key that may be the contents (i.e. payload 
> within) 
> > of all kinds of I/O buffers. Consider you are running a kvm based 
> > hypervisor and want its dump to be analyized while promising your 
> > customers whose guests you run on that hypervisor that none of their 
> data 
> > will be externalized. Or consider your system reads a spreadsheat with 
> > bank account or health information. You might not want to send fractions 
> 
> > of that information sitting in some buffers to a service organization.
> 
> So for direct IO, buffer is still in user space and should be filtered
> out when we filter out user space pages using mkdumpfile. For kvm, I am
> assuming that all the pages belong to qemu process and once we are
> filtering out user space pages, any data belonging to guest will go away.
> 
> So atleast for above examples it does not sound as if we need symbol
> erase infrastructure.
> 
> > 
> > to answer Daves concern: there is no intention that crash should ever 
> look 
> > into the erased structures. In theroy it should not be needed because 
> the 
> > contents of structures to be deleted should be irrelevant to kernel 
> > debugging.
> 
> So what are those kernel structures which we are planning to delete and
> are irrelevant to kernel debugging by crash?
> 
> I think we are missing something here. If there are only few known
> structures we want to get rid of, lets hardcode it in makedumpfile
> instead of giving user a generic infrastructure. That way we know
> that we are not leaking information at the same time making sure
> that analysis tools are working.
> 
> Thanks
> Vivek
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Mahesh J Salgaonkar]

On 2011-05-26 09:39:23 Thu, Vivek Goyal wrote:
> On Thu, May 26, 2011 at 01:15:14PM +0200, Reinhard Buendgen wrote:
> > Vivek,
> > 
> > I/O is not restricted to disk I/O (it may be network I/O, data sent to 
> > crtypto cards etc.) and it is not always direct, Device drivers may have 
> > buffers to which such data is copied. 
> > 
> > So it is more than just keys, and it may change over time.
> > I do not think hardwiring a filter in makedumpfile is a good idea because 
> > you would need a new makedumpfile release for every Distribution 
> > (release).
> 
> Ok, so we are worried about data being in slub/slab caches or driver's
> kmalloced buffers etc.
> 
> When do I need access to debuginfo files. I am assuming that makedumpfile
> reads them in first kernel sometime and stores relevant info in initramfs.
> Otherwise, it is not possible to get to it in second kernel after crash.
> 

The current approach is to re-run the makedumpfile on the crash dump
(generated in the second kernel) when we are back into production kernel
(1st kernel).

> > 
> > Allowing to configure makedumpfile allows each distribution and each 
> > platform to provide appropriate filters.
> > 
> 
> I was thinking that there are very few symbols and these will not change
> frequently. makdumpfile is already dependent on some kenrel data strcutres
> and symbols for doing filtering. 
> 
> But looks like you are planning to filter out lots of symbols. So in that
> case agreed that hardcoding is not a good idea.
> 
> Thanks
> Vivek
> 
> > Mit freundlichen Grüßen/Best Regards/Cordialement 
> > 
> > Reinhard
> > 
> > Dr. Reinhard Bündgen 
> > RAS & Crypto Architect for Linux on System z 
> > Virtualization and Systems Management 
> >  
> > Mail:buendgen@de.ibm.com
> > Phone: ++49-(0)7031-16-1130
> > Fax: ++49-(0)7031-16-3456 
> >  
> > IBM Deutschland Research & Development GmbH
> > Vorsitzender des Aufsichtsrats: Martin Jetter
> > Geschäftsführung: Dirk Wittkopp 
> > Sitz der Gesellschaft: Böblingen
> > Registergericht: Amtsgericht Stuttgart, HRB 243294
> > 
> > 
> > 
> > 
> > 
> > From:   Vivek Goyal <vgoyal@redhat.com>
> > To:     Reinhard Buendgen/Germany/IBM@IBMDE
> > Cc:     Dave Anderson <anderson@redhat.com>, Ananth N Mavinakayanahalli 
> > <ananth@in.ibm.com>, kexec@lists.infradead.org, Mahesh J Salgaonkar 
> > <mahesh@linux.vnet.ibm.com>, "Ken'ichi Ohmichi" 
> > <oomichi@mxs.nes.nec.co.jp>, V Srivatsa <vsrivatsa@in.ibm.com>
> > Date:   25.05.2011 21:53
> > Subject:        Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement 
> > to filter out kernel data from vmcore
> > 
> > 
> > 
> > On Wed, May 25, 2011 at 10:41:55AM +0200, Reinhard Buendgen wrote:
> > > Hi,
> > > 
> > > to answer Vivek questions first: Eventually we want to be able to erase 
> > > all data that a customer may consider sensitive to her privacy. In 
> > > addition to encryption key that may be the contents (i.e. payload 
> > within) 
> > > of all kinds of I/O buffers. Consider you are running a kvm based 
> > > hypervisor and want its dump to be analyized while promising your 
> > > customers whose guests you run on that hypervisor that none of their 
> > data 
> > > will be externalized. Or consider your system reads a spreadsheat with 
> > > bank account or health information. You might not want to send fractions 
> > 
> > > of that information sitting in some buffers to a service organization.
> > 
> > So for direct IO, buffer is still in user space and should be filtered
> > out when we filter out user space pages using mkdumpfile. For kvm, I am
> > assuming that all the pages belong to qemu process and once we are
> > filtering out user space pages, any data belonging to guest will go away.
> > 
> > So atleast for above examples it does not sound as if we need symbol
> > erase infrastructure.
> > 
> > > 
> > > to answer Daves concern: there is no intention that crash should ever 
> > look 
> > > into the erased structures. In theroy it should not be needed because 
> > the 
> > > contents of structures to be deleted should be irrelevant to kernel 
> > > debugging.
> > 
> > So what are those kernel structures which we are planning to delete and
> > are irrelevant to kernel debugging by crash?
> > 
> > I think we are missing something here. If there are only few known
> > structures we want to get rid of, lets hardcode it in makedumpfile
> > instead of giving user a generic infrastructure. That way we know
> > that we are not leaking information at the same time making sure
> > that analysis tools are working.
> > 
> > Thanks
> > Vivek
> > 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

-- 
Mahesh J Salgaonkar

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Ken'ichi Ohmichi]

Hi Mahesh,

(I'm back to makedumpfile devel for merging this patchset,
 because Tachibana-san is busy.)

Sorry for replying an old mail.

On Fri, 27 May 2011 12:26:21 +0530
Mahesh J Salgaonkar <mahesh@linux.vnet.ibm.com> wrote:
> On 2011-05-26 09:39:23 Thu, Vivek Goyal wrote:
> > On Thu, May 26, 2011 at 01:15:14PM +0200, Reinhard Buendgen wrote:
> > > Vivek,
> > > 
> > > I/O is not restricted to disk I/O (it may be network I/O, data sent to 
> > > crtypto cards etc.) and it is not always direct, Device drivers may have 
> > > buffers to which such data is copied. 
> > > 
> > > So it is more than just keys, and it may change over time.
> > > I do not think hardwiring a filter in makedumpfile is a good idea because 
> > > you would need a new makedumpfile release for every Distribution 
> > > (release).
> > 
> > Ok, so we are worried about data being in slub/slab caches or driver's
> > kmalloced buffers etc.
> > 
> > When do I need access to debuginfo files. I am assuming that makedumpfile
> > reads them in first kernel sometime and stores relevant info in initramfs.
> > Otherwise, it is not possible to get to it in second kernel after crash.
> > 
> 
> The current approach is to re-run the makedumpfile on the crash dump
> (generated in the second kernel) when we are back into production kernel
> (1st kernel).

IIUC, there are be 2 dumpfiles on customer site by the above approach.
The one is with privacy/secret data, and another is without.

If correct, I like this approach because a customer can have two options
when the crash utility cannot read a dumpfile without privacy/secret data
on support site.

First option:
  For digging a problem, a customer sends a dumpfile with privacy/secret
  data to support site.

Second option:
  For protecting privacy/secret data, a customer gives up digging a problem.


Thanks
Ken'ichi Ohmichi

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Mahesh J Salgaonkar]

Hi Ken'ichi,

On 2011-07-15 14:20:41 Fri, Ken'ichi Ohmichi wrote:
> 
> Hi Mahesh,
> 
> (I'm back to makedumpfile devel for merging this patchset,
>  because Tachibana-san is busy.)

Good to see that you are back.

> 
> Sorry for replying an old mail.
> 
> On Fri, 27 May 2011 12:26:21 +0530
> Mahesh J Salgaonkar <mahesh@linux.vnet.ibm.com> wrote:
> > On 2011-05-26 09:39:23 Thu, Vivek Goyal wrote:
> > > On Thu, May 26, 2011 at 01:15:14PM +0200, Reinhard Buendgen wrote:
> > > > Vivek,
> > > > 
> > > > I/O is not restricted to disk I/O (it may be network I/O, data sent to 
> > > > crtypto cards etc.) and it is not always direct, Device drivers may have 
> > > > buffers to which such data is copied. 
> > > > 
> > > > So it is more than just keys, and it may change over time.
> > > > I do not think hardwiring a filter in makedumpfile is a good idea because 
> > > > you would need a new makedumpfile release for every Distribution 
> > > > (release).
> > > 
> > > Ok, so we are worried about data being in slub/slab caches or driver's
> > > kmalloced buffers etc.
> > > 
> > > When do I need access to debuginfo files. I am assuming that makedumpfile
> > > reads them in first kernel sometime and stores relevant info in initramfs.
> > > Otherwise, it is not possible to get to it in second kernel after crash.
> > > 
> > 
> > The current approach is to re-run the makedumpfile on the crash dump
> > (generated in the second kernel) when we are back into production kernel
> > (1st kernel).
> 
> IIUC, there are be 2 dumpfiles on customer site by the above approach.
> The one is with privacy/secret data, and another is without.

Correct.

> 
> If correct, I like this approach because a customer can have two options
> when the crash utility cannot read a dumpfile without privacy/secret data
> on support site.

The crash utility would just display a warning message early on during
invocation. Most of the time crash tool will be able to read/analyze the
dump unless someone scrubs out the data on which crash utility is dependant
on. And as I mentioned previously, this is intended just as a security filter
and not to be used as detrimental to crash's analysis of the dump.

> 
> First option:
>   For digging a problem, a customer sends a dumpfile with privacy/secret
>   data to support site.
> 
> Second option:
>   For protecting privacy/secret data, a customer gives up digging a problem.
> 
> 
> Thanks
> Ken'ichi Ohmichi

Thanks,
-Mahesh.

> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

-- 
Mahesh J Salgaonkar

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Ken'ichi Ohmichi]

Hi Mahesh,

Thank you for quick response.

On Fri, 15 Jul 2011 12:40:29 +0530
Mahesh J Salgaonkar <mahesh@linux.vnet.ibm.com> wrote:
> > > 
> > > The current approach is to re-run the makedumpfile on the crash dump
> > > (generated in the second kernel) when we are back into production kernel
> > > (1st kernel).
> > 
> > IIUC, there are be 2 dumpfiles on customer site by the above approach.
> > The one is with privacy/secret data, and another is without.
> 
> Correct.
> 
> > 
> > If correct, I like this approach because a customer can have two options
> > when the crash utility cannot read a dumpfile without privacy/secret data
> > on support site.
> 
> The crash utility would just display a warning message early on during
> invocation. Most of the time crash tool will be able to read/analyze the
> dump unless someone scrubs out the data on which crash utility is dependant
> on. And as I mentioned previously, this is intended just as a security filter
> and not to be used as detrimental to crash's analysis of the dump.

I see, this feature will not be a problem in most cases.
However a customer can create unreadable dumpfile by using this feature,
and current makedumpfile does not have such feature.

Of course unreadable dumpfile is due to wrong makedumpfile.conf,
and I like this feature anyway because a customer has options.


Thanks
Ken'ichi Ohmichi

> > First option:
> >   For digging a problem, a customer sends a dumpfile with privacy/secret
> >   data to support site.
> > 
> > Second option:
> >   For protecting privacy/secret data, a customer gives up digging a problem.


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Mahesh J Salgaonkar]

On 2011-05-24 16:35:42 Tue, Vivek Goyal wrote:
> On Wed, May 18, 2011 at 01:29:06AM +0530, Mahesh J Salgaonkar wrote:
> > Hi All,
> > 
> > Please find the version 2 of makedumpfile enhancement patchset that introduces
> > a data filtering feature which enables makedumpfile to filter out desired kernel
> > symbol data and it's members from the specified VMCORE file. The data to be
> > filtered out is poisoned with character 'X' (58 in Hex).
> > 
> > This feature will be very useful for the customers who wants to erase the
> > customer sensitive data like security keys and other confidential data, in
> > DUMPFILE before sending it to support team for analysis.
> 
> Mahesh,
> 
> Apart from security keys, what are the other examples of data which needs
> to be removed?

The idea is to allow customer to remove any such confidential data that
he thinks should not be disclosed outside. The framework is generic
enough to accommodate any new such fields. The is intended just as a
security filter and not to be used as detrimental to crash's analysis of the
dump.

> 
> By erasing the data, you mean you set it to zero? Will it make sense to
> poison it with some specific pattern so that if crash or other tool
> are looking into it they know it has been posioned and it is not a
> case of corruption. This might help if a user erased a symbol which was
> required by crash for some analysis etc.

The data to be erased is poisoned with character 'X' (58 in Hex).

The last two patches 7/8 and 8/8 introduces eraseinfo section into
filtered compressed kdump and ELF kdump file. The compressed kdump file
now carries additional fields namely offset_eraseinfo and size_eraseinfo in
kdump sub header that can help crash to identify whether filtering is
been done. Similarly, ELF kdump file now contains new ELF note of type
NT_ERASEINFO, that can help crash to identify whether the filtering has been
done or not.

Thanks,
-Mahesh.

> 
> Thanks
> Vivek
> 
> > 
> > This feature introduces a filter config file where, using filter commands,
> > user can specify desired kernel data symbols and it's members that need to be
> > filtered out while creating o/p DUMPFILE. The Syntax for filter commands are
> > provided in the makedumpfile.conf(8) man page.
> > 
> > The first 4 patches prepares the base work for filtering framework. The
> > patches 5 and 6 implements the generic filtering framework to erase desired
> > kernel data. The last two patches implements inclusion of eraseinfo data into
> > o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the
> > effective lines from configuration files.
> > 
> > I have tested these patches on x86_64 and s390x architecture against RHEL6 GA
> > kernel. The feature supports filtering data from ELF as well as
> > kdump-compressed formatted dump.
> > 
> > Change in v2:
> > - Changes in patch 5 and inclusion of new patches 7 & 8
> > - Re-named filter.conf file to makedumpfile.conf and some cleanup work.
> > - Introduced last two patches that implements inclusion of eraseinfo data
> >   into kdump compressed as well as ELF formatted dumpfile.
> > 
> > Please review the patchset and let me know your comments.
> > 
> > Thanks,
> > -Mahesh.
> > ---
> > 
> > Mahesh Salgaonkar (8):
> >       makedumpfile: Add '--config' option to specify filter config file.
> >       makedumpfile: Apply relocation while loading module debuginfo.
> >       makedumpfile: Load the module symbol data from vmcore.
> >       makedumpfile: Introduce routines to get type name from debuginfo.
> >       makedumpfile: Read and process filter commands from config file.
> >       makedumpfile: Read and process 'for' command from config file.
> >       makedumpfile: Add erased information in compressed kdump file
> >       makedumpfile: Add erase information in ELF formatted dumpfile
> > 
> > 
> >  IMPLEMENTATION      |    6 
> >  Makefile            |    8 
> >  diskdump_mod.h      |    2 
> >  makedumpfile.8      |   20 
> >  makedumpfile.c      | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++---
> >  makedumpfile.conf   |  149 +++
> >  makedumpfile.conf.8 |  419 ++++++++
> >  makedumpfile.h      |  199 ++++
> >  8 files changed, 3394 insertions(+), 156 deletions(-)
> >  create mode 100644 makedumpfile.conf
> >  create mode 100644 makedumpfile.conf.8
> > 
> > -- 
> > Signature
> > 
> > _______________________________________________
> > kexec mailing list
> > kexec@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/kexec

-- 
Mahesh J Salgaonkar

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Vivek Goyal]

On Wed, May 25, 2011 at 01:14:07PM +0530, Mahesh J Salgaonkar wrote:
> On 2011-05-24 16:35:42 Tue, Vivek Goyal wrote:
> > On Wed, May 18, 2011 at 01:29:06AM +0530, Mahesh J Salgaonkar wrote:
> > > Hi All,
> > > 
> > > Please find the version 2 of makedumpfile enhancement patchset that introduces
> > > a data filtering feature which enables makedumpfile to filter out desired kernel
> > > symbol data and it's members from the specified VMCORE file. The data to be
> > > filtered out is poisoned with character 'X' (58 in Hex).
> > > 
> > > This feature will be very useful for the customers who wants to erase the
> > > customer sensitive data like security keys and other confidential data, in
> > > DUMPFILE before sending it to support team for analysis.
> > 
> > Mahesh,
> > 
> > Apart from security keys, what are the other examples of data which needs
> > to be removed?
> 
> The idea is to allow customer to remove any such confidential data that
> he thinks should not be disclosed outside. The framework is generic
> enough to accommodate any new such fields.

So what is that info which customer whould like to erase (apart from
security keys?) Do we have some more concrete examples?

> The is intended just as a
> security filter and not to be used as detrimental to crash's analysis of the
> dump.

How would we know in advance that what symbols crash is dependent on
to even open crash dump and we are not erasing one of those symbols. In
other mail Dave mentioned two symbols (vmlist and one more) which crash
is dependent on and we seem to be clearing that.

I am worried about filtering out symbols which make crash non functional.
Dave mentioned that one can remove the pointer which contains the list
of modules etc. I am worried what if module caused the crash and we break
the chain in such a way that reaching to module data structures is not
possible anymore.

If security key is only example, should we just hardcode that in makedumpfile
to wipe that info instead of creating a generic framework for this. If we
give user too much flexibility they might delete way too many symbols to
be safe and that might result in un-analyzable crash dump.

> 
> > 
> > By erasing the data, you mean you set it to zero? Will it make sense to
> > poison it with some specific pattern so that if crash or other tool
> > are looking into it they know it has been posioned and it is not a
> > case of corruption. This might help if a user erased a symbol which was
> > required by crash for some analysis etc.
> 
> The data to be erased is poisoned with character 'X' (58 in Hex).
> 
> The last two patches 7/8 and 8/8 introduces eraseinfo section into
> filtered compressed kdump and ELF kdump file. The compressed kdump file
> now carries additional fields namely offset_eraseinfo and size_eraseinfo in
> kdump sub header that can help crash to identify whether filtering is
> been done. Similarly, ELF kdump file now contains new ELF note of type
> NT_ERASEINFO, that can help crash to identify whether the filtering has been
> done or not.

And how this information is useful to crash? I mean why would crash care
whether filtering has been done or not and if filtering has been done
then do what?

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

[Dave Anderson]

----- Original Message -----
>
> > The data to be erased is poisoned with character 'X' (58 in Hex).
> >
> > The last two patches 7/8 and 8/8 introduces eraseinfo section into
> > filtered compressed kdump and ELF kdump file. The compressed kdump file
> > now carries additional fields namely offset_eraseinfo and size_eraseinfo in
> > kdump sub header that can help crash to identify whether filtering is
> > been done. Similarly, ELF kdump file now contains new ELF note of type
> > NT_ERASEINFO, that can help crash to identify whether the filtering has been
> > done or not.
> 
> And how this information is useful to crash? I mean why would crash care
> whether filtering has been done or not and if filtering has been done
> then do what?

It's not particularly useful to crash per se -- but rather it would be 
useful to a user trying to analyze the dumpfile.  If a prominent warning
message is displayed early on during invocation, then it won't be a big
surprise if/when things are subsequently broken/crippled.

Dave
 
> Thanks
> Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec