deb/rpm package purge

deb/rpm package purge

[Sage Weil]

Should the package purge remove /var/lib/ceph/* (potential mon data, osd 
data) and/or /var/log/ceph/* (logs)?  Right now it does, but mysql, for 
example, leaves /var/lib/mysql where it is (not sure about logs).

sage

Re: deb/rpm package purge

[Greg Farnum]

On Tuesday, March 19, 2013 at 12:48 PM, Sage Weil wrote:
> Should the package purge remove /var/lib/ceph/* (potential mon data, osd 
> data) and/or /var/log/ceph/* (logs)? Right now it does, but mysql, for 
> example, leaves /var/lib/mysql where it is (not sure about logs).
> 


I'm with Mark's ticket on this (http://tracker.ceph.com/issues/4505). Config data in /etc/ceph and logs in var/log/ceph is fine to remove, but storage data isn't. That's essentially user-generated and not something that can be recovered or rebuilt following the purge. Keyrings might not be unreasonable to delete, but I don't think they're necessary and certainly aren't worth putting in the work to separate from the other user data in var/lib/ceph.

-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

Re: deb/rpm package purge

[Mark Nelson]

On 03/19/2013 02:48 PM, Sage Weil wrote:
> Should the package purge remove /var/lib/ceph/* (potential mon data, osd
> data) and/or /var/log/ceph/* (logs)?  Right now it does, but mysql, for
> example, leaves /var/lib/mysql where it is (not sure about logs).

I'm definitely for leaving mon/osd data in place.  Those files are 
created at cluster creation time, not when the packages are installed. 
They may have been created by a totally different installation of Ceph 
than the packaged version.

What's worse is that you can't get the files back simply by reinstalling 
the package.  With the way things currently are, a package purge will 
effectively permanently destroy your cluster.  Purge should get rid of 
configuration files, but I don't think it should destroy user data which 
is what it effectively is doing now.

>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: deb/rpm package purge

[Dan Mick]

Is there a way out if you remove the keyrings?

On 03/19/2013 01:05 PM, Mark Nelson wrote:
>
>
> On 03/19/2013 02:48 PM, Sage Weil wrote:
>> Should the package purge remove /var/lib/ceph/* (potential mon data, osd
>> data) and/or /var/log/ceph/* (logs)?  Right now it does, but mysql, for
>> example, leaves /var/lib/mysql where it is (not sure about logs).
>
> I'm definitely for leaving mon/osd data in place.  Those files are
> created at cluster creation time, not when the packages are installed.
> They may have been created by a totally different installation of Ceph
> than the packaged version.
>
> What's worse is that you can't get the files back simply by reinstalling
> the package.  With the way things currently are, a package purge will
> effectively permanently destroy your cluster.  Purge should get rid of
> configuration files, but I don't think it should destroy user data which
> is what it effectively is doing now.

Re: deb/rpm package purge

[Sage Weil]

On Tue, 19 Mar 2013, Dan Mick wrote:
> Is there a way out if you remove the keyrings?

Yeah; you can use the keyring in the mon data dir to refetch any keyrings 
you deleted from /etc/ceph/.

...

I don't disagree with any of this discussion, BTW; I ask the question 
because I want to understand what the packaging best-practices are before 
we put our foot down for what is "right".

sage

Re: deb/rpm package purge

[Laszlo Boszormenyi (GCS)]

On Tue, 2013-03-19 at 13:51 -0700, Sage Weil wrote:
> On Tue, 19 Mar 2013, Dan Mick wrote:
> > Is there a way out if you remove the keyrings?
> 
> Yeah; you can use the keyring in the mon data dir to refetch any keyrings 
> you deleted from /etc/ceph/.
 But if packages are purged on mon as well...

> I don't disagree with any of this discussion, BTW; I ask the question 
> because I want to understand what the packaging best-practices are before 
> we put our foot down for what is "right".
 Actually there are two methods to uninstall a package from the system.
One is '-r' or '--remove' options of dpkg . As it's name and the
manpage confirms: "Remove  an  installed  package. -r or --remove remove
everything _except conffiles_. This may avoid having to reconfigure the
package if it is reinstalled later.". The second is '-P' or '--purge',
which stands for '[...] removes  everything,  including conffiles.".
Debian policy chapter 10 [1] writes about files and their handling. Its
subchapter 10.7 [2] talks about configuration files. Their handling is
expressed in subchapter 6.8 [3].

 In short, if user asks to purge the package, then the keys have to be
removed as well. If someone thinks about a reinstallation, s/he should
use remove instead. To be safe, I see one option. On purge, a debconf
message can be displayed notifying the user about the keys are going to
be removed as well. Abort of purge or backup of keys can be offered,
based on the choice.

Hope this helps,
Laszlo/GCS
[1] http://www.debian.org/doc/debian-policy/ch-files.html
[2] http://www.debian.org/doc/debian-policy/ch-files.html#s-config-files
[3] http://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails

Re: deb/rpm package purge

[Sage Weil]

On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
> On Tue, 2013-03-19 at 13:51 -0700, Sage Weil wrote:
> > On Tue, 19 Mar 2013, Dan Mick wrote:
> > > Is there a way out if you remove the keyrings?
> > 
> > Yeah; you can use the keyring in the mon data dir to refetch any keyrings 
> > you deleted from /etc/ceph/.
>  But if packages are purged on mon as well...
> 
> > I don't disagree with any of this discussion, BTW; I ask the question 
> > because I want to understand what the packaging best-practices are before 
> > we put our foot down for what is "right".
>  Actually there are two methods to uninstall a package from the system.
> One is '-r' or '--remove' options of dpkg . As it's name and the
> manpage confirms: "Remove  an  installed  package. -r or --remove remove
> everything _except conffiles_. This may avoid having to reconfigure the
> package if it is reinstalled later.". The second is '-P' or '--purge',
> which stands for '[...] removes  everything,  including conffiles.".
> Debian policy chapter 10 [1] writes about files and their handling. Its
> subchapter 10.7 [2] talks about configuration files. Their handling is
> expressed in subchapter 6.8 [3].
> 
>  In short, if user asks to purge the package, then the keys have to be
> removed as well. If someone thinks about a reinstallation, s/he should
> use remove instead. To be safe, I see one option. On purge, a debconf
> message can be displayed notifying the user about the keys are going to
> be removed as well. Abort of purge or backup of keys can be offered,
> based on the choice.

The keys aren't a problem; they are still in the mon database 
(/var/lib/ceph/mon/...).

The real question is whether purge should remove /var/lib/ceph and 
/var/log/ceph...

sage

Re: deb/rpm package purge

[Laszlo Boszormenyi (GCS)]

On Tue, 2013-03-19 at 15:29 -0700, Sage Weil wrote:
> On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
> >  In short, if user asks to purge the package, then the keys have to be
> > removed as well. If someone thinks about a reinstallation, s/he should
> > use remove instead.
> The keys aren't a problem; they are still in the mon database 
> (/var/lib/ceph/mon/...).
> 
> The real question is whether purge should remove /var/lib/ceph and 
> /var/log/ceph...
 Sorry if I was not clear and/or generic enough. Use of 'dpkg --remove'
is to remove binaries of the package, but leave other things, runtime
data (configuration, logs, database settings/users and so on) to be left
untouched.
On the other hand, 'dpkg --purge' is to remove everything the package
has installed and/or generated. This includes debconf answers as well.
With other words, purge is used to make the system totally clean of the
package. As such, if the sysadmin install the package again, all debconf
questions will be asked again and all generated files will be generated
again from scratch.

Laszlo/GCS

Re: deb/rpm package purge

[Sage Weil]

On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
>  Sorry if I was not clear and/or generic enough. Use of 'dpkg --remove'
> is to remove binaries of the package, but leave other things, runtime
> data (configuration, logs, database settings/users and so on) to be left
> untouched.
> On the other hand, 'dpkg --purge' is to remove everything the package
> has installed and/or generated. This includes debconf answers as well.
> With other words, purge is used to make the system totally clean of the
> package. As such, if the sysadmin install the package again, all debconf
> questions will be asked again and all generated files will be generated
> again from scratch.

I understand that part, but the policy isn't very clear about files that 
are not part of the package but are generated as a result of the package 
being installed (i.e., user data).

/var/lib/ceph/ has all of the *user data*, like critical monitor files 
that are necessary for the cluster to function (/var/lib/ceph/mon/*), and 
actual ceph objects (/var/lib/ceph/osd/*, although these are usually mount 
points that we would unmount, not delete).  Doing a purge on the monitor 
nodes would destroy the cluster (and all of its data) in a 
not-easily-recoverable way.

As a point of comparison, mysql removes the config files but not 
/var/lib/mysql.

The question is, is that okay/typical/desireable/recommended/a bad idea?

The less important question is whether /var/log/ceph should be removed; 
I'm assuming yes?

Thanks!
sage

Re: deb/rpm package purge

[Mark Nelson]

On 03/19/2013 05:51 PM, Laszlo Boszormenyi (GCS) wrote:
> On Tue, 2013-03-19 at 15:29 -0700, Sage Weil wrote:
>> On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
>>>   In short, if user asks to purge the package, then the keys have to be
>>> removed as well. If someone thinks about a reinstallation, s/he should
>>> use remove instead.
>> The keys aren't a problem; they are still in the mon database
>> (/var/lib/ceph/mon/...).
>>
>> The real question is whether purge should remove /var/lib/ceph and
>> /var/log/ceph...
>   Sorry if I was not clear and/or generic enough. Use of 'dpkg --remove'
> is to remove binaries of the package, but leave other things, runtime
> data (configuration, logs, database settings/users and so on) to be left
> untouched.
> On the other hand, 'dpkg --purge' is to remove everything the package
> has installed and/or generated. This includes debconf answers as well.
> With other words, purge is used to make the system totally clean of the
> package. As such, if the sysadmin install the package again, all debconf
> questions will be asked again and all generated files will be generated
> again from scratch.
>

Hi Lazlo,

Thanks for the answers!  The question is whether or not the data in 
/var/lib/ceph should be removed when that data was not generated as part 
of the package installation process, but rather as part of the cluster 
configuration after the package was installed.  IE the package creates 
the directory, but the user creates the data that resides in it.  What 
should purge do with the directory now that the user has placed 
(critical) data in it that was not generated as part of the package 
installation process?

> Laszlo/GCS
>

Re: deb/rpm package purge

[Laszlo Boszormenyi (GCS)]

On Tue, 2013-03-19 at 15:59 -0700, Sage Weil wrote:
> On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
> > On the other hand, 'dpkg --purge' is to remove everything the package
> > has installed and/or generated. This includes debconf answers as well.
> > With other words, purge is used to make the system totally clean of the
> > package. As such, if the sysadmin install the package again, all debconf
> > questions will be asked again and all generated files will be generated
> > again from scratch.
> 
> I understand that part, but the policy isn't very clear about files that 
> are not part of the package but are generated as a result of the package 
> being installed (i.e., user data).
 Forgive me, I just learnt English and my wording may not be that clear
for a natural speaker.

> As a point of comparison, mysql removes the config files but not 
> /var/lib/mysql.
 As I remember, MySQL asks if /var/lib/mysql/ should be purged or not; I
may mix with an other (database related) package.

> The question is, is that okay/typical/desireable/recommended/a bad idea?
 I can rephrase my words. Purge removes the (binary) package files, its
configuration and logs (its generated files). To emphasis, user files
are _not_ fall into this category and must remain as-is, _intact_.
Some packages writes a console message that 'your files remain at xxx,
they were not removed' on purge. Others just leave the dpkg warning
"directory not empty so not removed" which means user files may have
left there and that may be the reason the directory is not empty.
 I'm in a rush, but hopefully will be able to note policy parts in the
afternoon (CET).

> The less important question is whether /var/log/ceph should be removed; 
> I'm assuming yes?
 Yes, logs are going to be removed.

Hope I could answer your question now. Please note me if I should clear
more parts of my answer.
Laszlo/GCS

Re: deb/rpm package purge

[Sage Weil]

On Wed, 20 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
> On Tue, 2013-03-19 at 15:59 -0700, Sage Weil wrote:
> > On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
> > > On the other hand, 'dpkg --purge' is to remove everything the package
> > > has installed and/or generated. This includes debconf answers as well.
> > > With other words, purge is used to make the system totally clean of the
> > > package. As such, if the sysadmin install the package again, all debconf
> > > questions will be asked again and all generated files will be generated
> > > again from scratch.
> > 
> > I understand that part, but the policy isn't very clear about files that 
> > are not part of the package but are generated as a result of the package 
> > being installed (i.e., user data).
>  Forgive me, I just learnt English and my wording may not be that clear
> for a natural speaker.
> 
> > As a point of comparison, mysql removes the config files but not 
> > /var/lib/mysql.
>  As I remember, MySQL asks if /var/lib/mysql/ should be purged or not; I
> may mix with an other (database related) package.
> 
> > The question is, is that okay/typical/desireable/recommended/a bad idea?
>  I can rephrase my words. Purge removes the (binary) package files, its
> configuration and logs (its generated files). To emphasis, user files
> are _not_ fall into this category and must remain as-is, _intact_.
> Some packages writes a console message that 'your files remain at xxx,
> they were not removed' on purge. Others just leave the dpkg warning
> "directory not empty so not removed" which means user files may have
> left there and that may be the reason the directory is not empty.
>  I'm in a rush, but hopefully will be able to note policy parts in the
> afternoon (CET).

Thanks, Laszlo, that's exactly what I was after!  Sorry for the confusing 
exchange.  :)

Sounds like in this case, the fix is simply to leave /var/lib/ceph 
untouched.

We'll need to update teuthology ceph.py and nuke to clean up /var/lib/ceph 
(for qa runs), and I think we should add a ceph-deploy 'purgedata' command 
to clean out /var/lib/ceph on a given host.

Thanks!
sage

Re: deb/rpm package purge

[Mark Nelson]

On 03/20/2013 07:48 AM, Sage Weil wrote:
> On Wed, 20 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
>> On Tue, 2013-03-19 at 15:59 -0700, Sage Weil wrote:
>>> On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
>>>> On the other hand, 'dpkg --purge' is to remove everything the package
>>>> has installed and/or generated. This includes debconf answers as well.
>>>> With other words, purge is used to make the system totally clean of the
>>>> package. As such, if the sysadmin install the package again, all debconf
>>>> questions will be asked again and all generated files will be generated
>>>> again from scratch.
>>>
>>> I understand that part, but the policy isn't very clear about files that
>>> are not part of the package but are generated as a result of the package
>>> being installed (i.e., user data).
>>   Forgive me, I just learnt English and my wording may not be that clear
>> for a natural speaker.
>>
>>> As a point of comparison, mysql removes the config files but not
>>> /var/lib/mysql.
>>   As I remember, MySQL asks if /var/lib/mysql/ should be purged or not; I
>> may mix with an other (database related) package.
>>
>>> The question is, is that okay/typical/desireable/recommended/a bad idea?
>>   I can rephrase my words. Purge removes the (binary) package files, its
>> configuration and logs (its generated files). To emphasis, user files
>> are _not_ fall into this category and must remain as-is, _intact_.
>> Some packages writes a console message that 'your files remain at xxx,
>> they were not removed' on purge. Others just leave the dpkg warning
>> "directory not empty so not removed" which means user files may have
>> left there and that may be the reason the directory is not empty.
>>   I'm in a rush, but hopefully will be able to note policy parts in the
>> afternoon (CET).
>
> Thanks, Laszlo, that's exactly what I was after!  Sorry for the confusing
> exchange.  :)
>
> Sounds like in this case, the fix is simply to leave /var/lib/ceph
> untouched.
>
> We'll need to update teuthology ceph.py and nuke to clean up /var/lib/ceph
> (for qa runs), and I think we should add a ceph-deploy 'purgedata' command
> to clean out /var/lib/ceph on a given host.

It's not as important given that it won't outright destroy the cluster, 
but perhaps we should also leave /etc/ceph untouched on purge if a 
ceph.conf file has been placed in it (since that also was not installed 
by the package, but rather by a user?).  I figure we should probably try 
to get it right now.  The message about the directory not being empty 
sounds good.

My thought here is:

- remove anything created by the packages in /var/lib/ceph that has been 
untouched since package installation.
- remove /var/lib/ceph if it has been untouched
- remove /etc/ceph if it has been untouched

Thoughts?

>
> Thanks!
> sage
>

Mark

Re: deb/rpm package purge

[James Page]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 20/03/13 14:48, Mark Nelson wrote:
>> 
>> We'll need to update teuthology ceph.py and nuke to clean up 
>> /var/lib/ceph (for qa runs), and I think we should add a
>> ceph-deploy 'purgedata' command to clean out /var/lib/ceph on a
>> given host.
> 
> It's not as important given that it won't outright destroy the
> cluster, but perhaps we should also leave /etc/ceph untouched on
> purge if a ceph.conf file has been placed in it (since that also
> was not installed by the package, but rather by a user?).  I figure
> we should probably try to get it right now.  The message about the
> directory not being empty sounds good.
> 
> My thought here is:
> 
> - remove anything created by the packages in /var/lib/ceph that has
> been untouched since package installation. - remove /var/lib/ceph
> if it has been untouched - remove /etc/ceph if it has been
> untouched

If those directories are created by dpkg rather than maintainer
scripts (i.e. in ceph.dirs rather than ceph.postinst) you should
should get that all for free; if the directories contain anything dpkg
does not know about it will just not removed them.


- -- 
James Page
Ubuntu Core Developer
Debian Maintainer
james.page@ubuntu.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBCAAGBQJRSfFoAAoJEL/srsug59jDJgsP/0zD4aKFtimFPh02/zdfJ3X0
BEY8Jmnmt3HcCxSPbGleZ+2p/38iLfLz8HdM1ZpOwGDVIv13N45vG0wW9zF/843R
8vKoGHJY7gAt/uY1fqa115m9txeNXAIZoaxwjrd6Zd31pgPvTBmfZhVsFKUnk7E5
9JmUs/K8gjjAZajhkUKgddp2ID70n/WGdHR+iu5cy72TyuvXVBQV1OmyYi9lMIxM
yHGnCM/X7x5DE1g61x532VP2D0gAegA2WWURoqQ6vAM3IZfVGVuvat+HdzZ8Ej8z
LfTk+8n2alTj6s1Xp3KGbb/D231MIi3VBaFMQx5pBlM29lAv8OYKidRQpZc9bZe2
5m5vhDutp4ZOZmqxDvDdayZgb/s8uVuodT2XK7qn4KbBRDEJN5aJKiUzXH0wTVTZ
Lg/A/criFuzRP+ZH/Sh1pfSnLkNrrLMbdTglUv4krM2L6ZPOmEU3fh+UIXkr2u9t
f6lnu4fVBwikDy/4hDztVL76IqB3wjnxYlJ1uHMHOrCugDeLRsGHdbCdFcoZonRW
rUhdrtqzuuSbPHkzs/dEMCm4vF439YdmuL4WGsfzxEu+djcESDtzuAw4D1LRO12V
JbEc9s8+L84oBJUv5dCSDG333jWc/8eihSs1ZG33NKZsWsNheppLR6aeXbg/nGi1
53R63uM1Lv7f1UaTxvH8
=6adU
-----END PGP SIGNATURE-----

Re: deb/rpm package purge

[Laszlo Boszormenyi (GCS)]

On Wed, 2013-03-20 at 05:48 -0700, Sage Weil wrote:
> On Wed, 20 Mar 2013, Laszlo Boszormenyi (GCS) wrote:
> > On Tue, 2013-03-19 at 15:59 -0700, Sage Weil wrote:
> > > As a point of comparison, mysql removes the config files but not 
> > > /var/lib/mysql.
> > > The question is, is that okay/typical/desireable/recommended/a bad idea?
 I should have asked this sooner. Do you know _any_ program that removes
your favorite music collection, your family photos or your business
emails when you do uninstall it?
I suspect that your question was theoretical instead.

On Wed, 2013-03-20 at 09:48 -0500, Mark Nelson wrote:
On 03/20/2013 07:48 AM, Sage Weil wrote:
> It's not as important given that it won't outright destroy the cluster, 
> but perhaps we should also leave /etc/ceph untouched on purge if a 
> ceph.conf file has been placed in it (since that also was not installed 
> by the package, but rather by a user?).  I figure we should probably try 
> to get it right now.  The message about the directory not being empty 
> sounds good.
 Sure, personal user data must be kept. If it's a big amount of data and
left under a non-standard location (ie, not under his/her $HOME) then
s/he should be informed where those files are located on purge.

> My thought here is:
> - remove anything created by the packages in /var/lib/ceph that has been 
> untouched since package installation.
> - remove /var/lib/ceph if it has been untouched
 Please note that you have to store some kind of checksum for the files
then. Probably md5sum is enough.

> - remove /etc/ceph if it has been untouched
 This is an other case. dpkg itself handle package files here, called
conffiles. I should check the method (md5sum and/or sha1 variants) used
for the checksum on these files. On upgrade it's used not to overwrite
local changes by the user. It may worth to read a bit more about it[1]
from Raphaël Hertzog. He is the co-author of Debian Administrator's
handbook[2] BTW.
 On purge dpkg will remove the package conffiles no matter what. It
won't check if those were changed or not. You may not mark the files
under /etc as conffiles, but then you'll lose the mentioned
merge logic on upgrades; dpkg will just overwrite those.
 In short, files under /var/lib/ceph are the only candidates for
in-package checksumming. How many files under there that essential for
the packages?

Laszlo/GCS
[1] http://raphaelhertzog.com/2010/09/21/debian-conffile-configuration-file-managed-by-dpkg/
[2] http://debian-handbook.info/

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: deb/rpm package purge

[Dan Mick]

>>>> As a point of comparison, mysql removes the config files but not
>>>> /var/lib/mysql.
>>>> The question is, is that okay/typical/desireable/recommended/a bad idea?
>   I should have asked this sooner. Do you know _any_ program that removes
> your favorite music collection, your family photos or your business
> emails when you do uninstall it?
> I suspect that your question was theoretical instead.

It's somewhat different in that the data is not owned by one user, but 
there are clear parallels.  The thing to be careful about here, IMO, is 
not only to preserve the data, but the associated files that allow
(reasonably-easy) access to that data.  (It's no good preserving the OSD
filestore if the keys, monmap, or osdmap are gone or hard to recover.)

>> - remove /etc/ceph if it has been untouched
>   This is an other case. dpkg itself handle package files here, called
> conffiles. I should check the method (md5sum and/or sha1 variants) used
> for the checksum on these files. On upgrade it's used not to overwrite
> local changes by the user. It may worth to read a bit more about it[1]
> from Raphaël Hertzog. He is the co-author of Debian Administrator's
> handbook[2] BTW.

Excellent reference; thanks for the pointer.

>   On purge dpkg will remove the package conffiles no matter what. It
> won't check if those were changed or not. You may not mark the files
> under /etc as conffiles, but then you'll lose the mentioned
> merge logic on upgrades; dpkg will just overwrite those.
>   In short, files under /var/lib/ceph are the only candidates for
> in-package checksumming. How many files under there that essential for
> the packages?
>
> Laszlo/GCS
> [1] http://raphaelhertzog.com/2010/09/21/debian-conffile-configuration-file-managed-by-dpkg/
> [2] http://debian-handbook.info/
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html