I would like to propose that we add compression to handle all policy files on disk.

I would like to propose that we add compression to handle all policy files on disk.

[Daniel J Walsh]

We are currently storing three sets of pp files on disk in Fedora as 
well as a policy.21 file
To get an idea of how much space we can save, I did this little 
experiment.    I believe this change is
critical for minimal installs, and if we want to eventually use SELinux 
on certain small platforms.

I think just changing libsemanage to handle compressed policy packages 
and to create its itermediary files as compressed files and changing 
libselinux able to read a compressed policy.21 file.

# tar cvf /tmp/selinux /etc/selinux/targeted /usr/share/selinux/targeted/
# gzip -c /tmp/selinux > /tm/selinux.gz

# du /tmp/selinux*
57380   /tmp/selinux
3772    /tmp/selinux.gz


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: I would like to propose that we add compression to handle all policy files on disk.

[Joshua Brindle]

Daniel J Walsh wrote:
> We are currently storing three sets of pp files on disk in Fedora as 
> well as a policy.21 file
3 policy.21 files (deceptively named policy.kern in the module store...) 
plus 2 fully linked copies (base.linked) which is essentially a copy of 
all the policy packages concatenated (which is much larger than the 
policy.21 file)

As of right now base.linked is unused, we had some plans to eventually 
use it for 1) verification of the policy via external tools (libsemanage 
already supports this but noone is using it) and 2) a while back I 
suggested we could do incremental linking to speed up the process.

Getting rid of it almost halves the store size alone:

[root@poisonivy targeted]# du -sh modules/
38M     modules/
[root@poisonivy targeted]# rm modules/*/base.linked
[root@poisonivy targeted]# du -sh modules/        
21M     modules/

> To get an idea of how much space we can save, I did this little 
> experiment.    I believe this change is
> critical for minimal installs, and if we want to eventually use 
> SELinux on certain small platforms.
>
> I think just changing libsemanage to handle compressed policy packages 
> and to create its itermediary files as compressed files and changing 
> libselinux able to read a compressed policy.21 file.
>
libsepol does all the reading and writing, if we were to support 
compressed policies it would have to be there, not in libsemanage.

> # tar cvf /tmp/selinux /etc/selinux/targeted /usr/share/selinux/targeted/
there are files in here that we probably wouldn't want to compress like 
contexts/*, conf files, etc. (granted they are very small compared to 
policy packages and kernel policy..
> # gzip -c /tmp/selinux > /tm/selinux.gz
>
> # du /tmp/selinux*
> 57380   /tmp/selinux
> 3772    /tmp/selinux.gz

I wouldn't be adverse to someone adding a libsemanage option to not save 
the previous directory on success, we don't actually provide a revert 
interface as of now (though we probably should if we are going to keep 
it around..)

[root@poisonivy targeted]# rm -rf modules/previous/
[root@poisonivy targeted]# du -sh modules/
11M     modules/

(this was after the base.linked getting deleted above, add 9 meg to this 
if we keep that around..)

so there we've cut the module directory by 1/4 potentially, just how 
much space are we needing to save?

The main concern I have with this is that libz hasn't had an awesome 
security record and it would have to be used by the component of the 
toolchain closest to the kernel, load_policy.

Additionally, on Fedora libz is in /usr/lib which means init won't be 
able to use it to decompress the policy at boot time, I really don't 
think we should be pulling a static libz into libselinux and libsepol.


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: I would like to propose that we add compression to handle all policy files on disk.

[Stephen Smalley]

On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> Daniel J Walsh wrote:
> > We are currently storing three sets of pp files on disk in Fedora as 
> > well as a policy.21 file
> 3 policy.21 files (deceptively named policy.kern in the module store...) 
> plus 2 fully linked copies (base.linked) which is essentially a copy of 
> all the policy packages concatenated (which is much larger than the 
> policy.21 file)
> 
> As of right now base.linked is unused, we had some plans to eventually 
> use it for 1) verification of the policy via external tools (libsemanage 
> already supports this but noone is using it) and 2) a while back I 
> suggested we could do incremental linking to speed up the process.
> 
> Getting rid of it almost halves the store size alone:
> 
> [root@poisonivy targeted]# du -sh modules/
> 38M     modules/
> [root@poisonivy targeted]# rm modules/*/base.linked
> [root@poisonivy targeted]# du -sh modules/        
> 21M     modules/
> 
> > To get an idea of how much space we can save, I did this little 
> > experiment.    I believe this change is
> > critical for minimal installs, and if we want to eventually use 
> > SELinux on certain small platforms.
> >
> > I think just changing libsemanage to handle compressed policy packages 
> > and to create its itermediary files as compressed files and changing 
> > libselinux able to read a compressed policy.21 file.
> >
> libsepol does all the reading and writing, if we were to support 
> compressed policies it would have to be there, not in libsemanage.
> 
> > # tar cvf /tmp/selinux /etc/selinux/targeted /usr/share/selinux/targeted/
> there are files in here that we probably wouldn't want to compress like 
> contexts/*, conf files, etc. (granted they are very small compared to 
> policy packages and kernel policy..
> > # gzip -c /tmp/selinux > /tm/selinux.gz
> >
> > # du /tmp/selinux*
> > 57380   /tmp/selinux
> > 3772    /tmp/selinux.gz
> 
> I wouldn't be adverse to someone adding a libsemanage option to not save 
> the previous directory on success, we don't actually provide a revert 
> interface as of now (though we probably should if we are going to keep 
> it around..)
> 
> [root@poisonivy targeted]# rm -rf modules/previous/
> [root@poisonivy targeted]# du -sh modules/
> 11M     modules/
> 
> (this was after the base.linked getting deleted above, add 9 meg to this 
> if we keep that around..)
> 
> so there we've cut the module directory by 1/4 potentially, just how 
> much space are we needing to save?
> 
> The main concern I have with this is that libz hasn't had an awesome 
> security record and it would have to be used by the component of the 
> toolchain closest to the kernel, load_policy.
> 
> Additionally, on Fedora libz is in /usr/lib which means init won't be 
> able to use it to decompress the policy at boot time, I really don't 
> think we should be pulling a static libz into libselinux and libsepol.

Sounds like dropping base.linked and making previous optional would
address the problem more effectively.  Also, do we need to keep
policy.kern after successful installation of policy.N?  If not, we can
have libsemanage unlink it automatically after installation.

-- 
Stephen Smalley
National Security Agency


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: I would like to propose that we add compression to handle all policy files on disk.

[Stephen Smalley]

On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> > Daniel J Walsh wrote:
> > > We are currently storing three sets of pp files on disk in Fedora as 
> > > well as a policy.21 file
> > 3 policy.21 files (deceptively named policy.kern in the module store...) 
> > plus 2 fully linked copies (base.linked) which is essentially a copy of 
> > all the policy packages concatenated (which is much larger than the 
> > policy.21 file)
> > 
> > As of right now base.linked is unused, we had some plans to eventually 
> > use it for 1) verification of the policy via external tools (libsemanage 
> > already supports this but noone is using it) and 2) a while back I 
> > suggested we could do incremental linking to speed up the process.
> > 
> > Getting rid of it almost halves the store size alone:
> > 
> > [root@poisonivy targeted]# du -sh modules/
> > 38M     modules/
> > [root@poisonivy targeted]# rm modules/*/base.linked
> > [root@poisonivy targeted]# du -sh modules/        
> > 21M     modules/
> > 
> > > To get an idea of how much space we can save, I did this little 
> > > experiment.    I believe this change is
> > > critical for minimal installs, and if we want to eventually use 
> > > SELinux on certain small platforms.
> > >
> > > I think just changing libsemanage to handle compressed policy packages 
> > > and to create its itermediary files as compressed files and changing 
> > > libselinux able to read a compressed policy.21 file.
> > >
> > libsepol does all the reading and writing, if we were to support 
> > compressed policies it would have to be there, not in libsemanage.
> > 
> > > # tar cvf /tmp/selinux /etc/selinux/targeted /usr/share/selinux/targeted/
> > there are files in here that we probably wouldn't want to compress like 
> > contexts/*, conf files, etc. (granted they are very small compared to 
> > policy packages and kernel policy..
> > > # gzip -c /tmp/selinux > /tm/selinux.gz
> > >
> > > # du /tmp/selinux*
> > > 57380   /tmp/selinux
> > > 3772    /tmp/selinux.gz
> > 
> > I wouldn't be adverse to someone adding a libsemanage option to not save 
> > the previous directory on success, we don't actually provide a revert 
> > interface as of now (though we probably should if we are going to keep 
> > it around..)
> > 
> > [root@poisonivy targeted]# rm -rf modules/previous/
> > [root@poisonivy targeted]# du -sh modules/
> > 11M     modules/
> > 
> > (this was after the base.linked getting deleted above, add 9 meg to this 
> > if we keep that around..)
> > 
> > so there we've cut the module directory by 1/4 potentially, just how 
> > much space are we needing to save?
> > 
> > The main concern I have with this is that libz hasn't had an awesome 
> > security record and it would have to be used by the component of the 
> > toolchain closest to the kernel, load_policy.
> > 
> > Additionally, on Fedora libz is in /usr/lib which means init won't be 
> > able to use it to decompress the policy at boot time, I really don't 
> > think we should be pulling a static libz into libselinux and libsepol.
> 
> Sounds like dropping base.linked and making previous optional would
> address the problem more effectively.  Also, do we need to keep
> policy.kern after successful installation of policy.N?  If not, we can
> have libsemanage unlink it automatically after installation.

Same question for any other file regenerated by every commit, although
we may not get much of a savings from the others.
file_contexts.template, file_contexts, and netfilter_contexts are the
most obvious ones.

-- 
Stephen Smalley
National Security Agency


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: I would like to propose that we add compression to handle all policy files on disk.

[Joshua Brindle]

Stephen Smalley wrote:
> On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
>> On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
>>>
>>> Additionally, on Fedora libz is in /usr/lib which means init won't be 
>>> able to use it to decompress the policy at boot time, I really don't 
>>> think we should be pulling a static libz into libselinux and libsepol.
>> Sounds like dropping base.linked and making previous optional would
>> address the problem more effectively.  Also, do we need to keep
>> policy.kern after successful installation of policy.N?  If not, we can
>> have libsemanage unlink it automatically after installation.
> 
> Same question for any other file regenerated by every commit, although
> we may not get much of a savings from the others.
> file_contexts.template, file_contexts, and netfilter_contexts are the
> most obvious ones.
> 

We obviously didn't optimize this for space before, it was nice having 
all the files around for development and debugging purposes, and we 
might have been able to do some speed optimizations later by keeping 
caches of everything around but that isn't much of a concern now (I 
haven't had speed issues with semodule since optionals in base take 2)


deleting everything except base.pp, commit_num and modules gave a couple 
more meg:

[root@poisonivy targeted]# du -sh modules/
9.1M    modules/

Note, this is with a huge base.pp (8.1 meg), I would expect embedded 
systems to have a very cut down SELinux policy (for a number of reasons 
other than policy storage size). So this number should go down 
significantly on a true embedded configuration.


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Joshua Brindle]

> From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
> 
> On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:

> > Sounds like dropping base.linked and making previous optional would 
> > address the problem more effectively.  Also, do we need to keep 
> > policy.kern after successful installation of policy.N?  If 
> not, we can 
> > have libsemanage unlink it automatically after installation.
> 
> Same question for any other file regenerated by every commit, 
> although we may not get much of a savings from the others.
> file_contexts.template, file_contexts, and netfilter_contexts 
> are the most obvious ones.
> 

Karl suggested that we can compress the policy packages but not the
kernel policy. As long as this isn't a policy package format change
(eg., the policy packages in /usr/share/selinux are the same they've
always been) and it is only libsemanage manipulating the files in the
store I'm fine with that. The module store is a private resource of
libsemanage so nothing else should be affected in any way by this. 

This will slow down some otherwise cheap operations such as semodule -l,
rather than just opening the files and reading the policy name it'll
have to decompress them first, I'm not sure what the performance cost
will be.. Perhaps this should be configurable as well. 

Matt Anderson also mentioned using libbz2 which is more space efficient
and has a better security history, so embedded installations include
that library?

With bzip2 compression and nothing removed from the store I'm getting
around 670k for the active store (so *2 if previous sticks around). With
all the superfluous files removed the store is around 210k.

What kind of size were we looking for again?


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: I would like to propose that we add compression to handle allpolicy files on disk.

[Daniel J Walsh]

Joshua Brindle wrote:
>> From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
>>
>> On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
>>     
>>> On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
>>>       
>
>   
>>> Sounds like dropping base.linked and making previous optional would 
>>> address the problem more effectively.  Also, do we need to keep 
>>> policy.kern after successful installation of policy.N?  If 
>>>       
>> not, we can 
>>     
>>> have libsemanage unlink it automatically after installation.
>>>       
>> Same question for any other file regenerated by every commit, 
>> although we may not get much of a savings from the others.
>> file_contexts.template, file_contexts, and netfilter_contexts 
>> are the most obvious ones.
>>
>>     
>
> Karl suggested that we can compress the policy packages but not the
> kernel policy. As long as this isn't a policy package format change
> (eg., the policy packages in /usr/share/selinux are the same they've
> always been) and it is only libsemanage manipulating the files in the
> store I'm fine with that. The module store is a private resource of
> libsemanage so nothing else should be affected in any way by this. 
>
> This will slow down some otherwise cheap operations such as semodule -l,
> rather than just opening the files and reading the policy name it'll
> have to decompress them first, I'm not sure what the performance cost
> will be.. Perhaps this should be configurable as well. 
>
> Matt Anderson also mentioned using libbz2 which is more space efficient
> and has a better security history, so embedded installations include
> that library?
>
> With bzip2 compression and nothing removed from the store I'm getting
> around 670k for the active store (so *2 if previous sticks around). With
> all the superfluous files removed the store is around 210k.
>
> What kind of size were we looking for again?
>   
What ever we can get.  This discussion started with, the minimal install 
of Fedora currently is approaching 500 Meg.  A fairly large percentage 
of this is SELinux related.  I would like to make the percentage 
insignificant.  Since most of the files sit around doing nothing.  :^)



--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Karl MacMillan]

On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
> > 
> > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> 
> > > Sounds like dropping base.linked and making previous optional would 
> > > address the problem more effectively.  Also, do we need to keep 
> > > policy.kern after successful installation of policy.N?  If 
> > not, we can 
> > > have libsemanage unlink it automatically after installation.
> > 
> > Same question for any other file regenerated by every commit, 
> > although we may not get much of a savings from the others.
> > file_contexts.template, file_contexts, and netfilter_contexts 
> > are the most obvious ones.
> > 
> 
> Karl suggested that we can compress the policy packages but not the
> kernel policy. As long as this isn't a policy package format change
> (eg., the policy packages in /usr/share/selinux are the same they've
> always been) and it is only libsemanage manipulating the files in the
> store I'm fine with that. The module store is a private resource of
> libsemanage so nothing else should be affected in any way by this. 
> 

Making semodule recognize bzipped files should be pretty simple as well
- why wouldn't we do that to save space in /usr/share/selinux?

> This will slow down some otherwise cheap operations such as semodule -l,
> rather than just opening the files and reading the policy name it'll
> have to decompress them first, I'm not sure what the performance cost
> will be.. Perhaps this should be configurable as well. 
> 

If we make printing the version optional we should be able to do this
with only name mangling on the file names in the module directory. We
should probably make it possible just to get a list of module names
anyway to facilitate scripting.

Karl


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handleallpolicy files on disk.

[Joshua Brindle]

> From: Karl MacMillan [mailto:kmacmillan@mentalrootkit.com] 
> 
> On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > > From: Stephen Smalley [mailto:sds@tycho.nsa.gov]
> > > 
> > > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> > 
> > Karl suggested that we can compress the policy packages but not the 
> > kernel policy. As long as this isn't a policy package format change 
> > (eg., the policy packages in /usr/share/selinux are the 
> same they've 
> > always been) and it is only libsemanage manipulating the 
> files in the 
> > store I'm fine with that. The module store is a private resource of 
> > libsemanage so nothing else should be affected in any way by this.
> > 
> 
> Making semodule recognize bzipped files should be pretty 
> simple as well
> - why wouldn't we do that to save space in /usr/share/selinux?
> 

What about everything else that will read modules (including setools in
the future)? They need to be stored in their native format and I don't
think that libsepol should link against a compression library to make
that happen.

> > This will slow down some otherwise cheap operations such as 
> semodule 
> > -l, rather than just opening the files and reading the policy name 
> > it'll have to decompress them first, I'm not sure what the 
> performance 
> > cost will be.. Perhaps this should be configurable as well.
> > 
> 
> If we make printing the version optional we should be able to 
> do this with only name mangling on the file names in the 
> module directory. We should probably make it possible just to 
> get a list of module names anyway to facilitate scripting.
> 



--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handleallpolicy files on disk.

[Karl MacMillan]

On Thu, 2006-11-09 at 13:50 -0500, Joshua Brindle wrote:
> > From: Karl MacMillan [mailto:kmacmillan@mentalrootkit.com] 
> > 
> > On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > > > From: Stephen Smalley [mailto:sds@tycho.nsa.gov]
> > > > 
> > > > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> > > 
> > > Karl suggested that we can compress the policy packages but not the 
> > > kernel policy. As long as this isn't a policy package format change 
> > > (eg., the policy packages in /usr/share/selinux are the 
> > same they've 
> > > always been) and it is only libsemanage manipulating the 
> > files in the 
> > > store I'm fine with that. The module store is a private resource of 
> > > libsemanage so nothing else should be affected in any way by this.
> > > 
> > 
> > Making semodule recognize bzipped files should be pretty 
> > simple as well
> > - why wouldn't we do that to save space in /usr/share/selinux?
> > 
> 
> What about everything else that will read modules (including setools in
> the future)?

It's a small amount of logic to detect.

>  They need to be stored in their native format and I don't
> think that libsepol should link against a compression library to make
> that happen.

What's wrong with letting libsepol dlopen libbz2.so?

Karl


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression tohandleallpolicy files on disk.

[Chris Stone]

For embedded devices, size is an issue. We have just started looking at
this, because it is an issue we need to solve for our customers. I am an
SELinux newbie, but, I wanted to add my two cents to the discussion. Please
forgive me if my comments are not relevant.

I would think that tools (such as setools) used for policy analysis or
debugging would be host based. In other words, only the binary policy files
are loaded on the target, and any other files needed to support analysis
would be loaded on the host. Tools would connect to the device remotely and
use the local files to interpret the devices policy, just like gdb loads
symbols on a host to debug a binary running on a remote device. This is
because policy analysis would only ever get done in the lab. Once a device
is deployed, policy analysis would no longer be required. Nobody is going to
deploy setools on an embedded device. If a problem occurs in the field, it
would have to be reproduced in the lab, and then a new corrected binary
policy deployed. I know that the tools don't currently work this way, but,
they are going to have to be modified to support it.

As far as storing the binary policy in the embedded devices file system,
that can be solved by using a compressed file system such as sqaushfs,
cramfs, or jffs2/3. It is certainly true that policies customized for
embedded devices will have to be developed so that they are smaller. It
seems to me that current policies are designed to handle the generic PC
environment, which is not the same as an embedded environment at all.

Finally, and this where I am really past my knowledge, but, it may be
possible to use memory mapped files to reduce the amount of the binary
policy that is actually loaded into memory.

I think the bottom line is that it may be too soon to consider compressing
the policy. We need to define the problem better, and consider the solution
space, before we jump to a solution. MV is doing that and we hope to
contribute in this area.

Cheers,
   Chris.

----------------------------------------
Christopher Stone
Senior Software Engineer
MontaVista Software,Inc. 
2929 Patrick Henry Drive
Santa Clara, CA. 95054-1831 
Web: www.mvista.com
613-831-1892
> -----Original Message-----
> From: owner-selinux@tycho.nsa.gov [mailto:owner-selinux@tycho.nsa.gov] On
> Behalf Of Karl MacMillan
> Sent: November 9, 2006 2:11 PM
> To: Joshua Brindle
> Cc: Stephen Smalley; Daniel J Walsh; SE Linux
> Subject: RE: I would like to propose that we add compression
> tohandleallpolicy files on disk.
> 
> On Thu, 2006-11-09 at 13:50 -0500, Joshua Brindle wrote:
> > > From: Karl MacMillan [mailto:kmacmillan@mentalrootkit.com]
> > >
> > > On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > > > > From: Stephen Smalley [mailto:sds@tycho.nsa.gov]
> > > > >
> > > > > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > > > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> > > >
> > > > Karl suggested that we can compress the policy packages but not the
> > > > kernel policy. As long as this isn't a policy package format change
> > > > (eg., the policy packages in /usr/share/selinux are the
> > > same they've
> > > > always been) and it is only libsemanage manipulating the
> > > files in the
> > > > store I'm fine with that. The module store is a private resource of
> > > > libsemanage so nothing else should be affected in any way by this.
> > > >
> > >
> > > Making semodule recognize bzipped files should be pretty
> > > simple as well
> > > - why wouldn't we do that to save space in /usr/share/selinux?
> > >
> >
> > What about everything else that will read modules (including setools in
> > the future)?
> 
> It's a small amount of logic to detect.
> 
> >  They need to be stored in their native format and I don't
> > think that libsepol should link against a compression library to make
> > that happen.
> 
> What's wrong with letting libsepol dlopen libbz2.so?
> 
> Karl
> 
> 
> --
> This message was distributed to subscribers of the selinux mailing list.
> If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov
> with
> the words "unsubscribe selinux" without quotes as the message.


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Stephen Smalley]

On Thu, 2006-11-09 at 13:43 -0500, Karl MacMillan wrote:
> On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > > From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
> > > 
> > > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> > 
> > > > Sounds like dropping base.linked and making previous optional would 
> > > > address the problem more effectively.  Also, do we need to keep 
> > > > policy.kern after successful installation of policy.N?  If 
> > > not, we can 
> > > > have libsemanage unlink it automatically after installation.
> > > 
> > > Same question for any other file regenerated by every commit, 
> > > although we may not get much of a savings from the others.
> > > file_contexts.template, file_contexts, and netfilter_contexts 
> > > are the most obvious ones.
> > > 
> > 
> > Karl suggested that we can compress the policy packages but not the
> > kernel policy. As long as this isn't a policy package format change
> > (eg., the policy packages in /usr/share/selinux are the same they've
> > always been) and it is only libsemanage manipulating the files in the
> > store I'm fine with that. The module store is a private resource of
> > libsemanage so nothing else should be affected in any way by this. 
> > 
> 
> Making semodule recognize bzipped files should be pretty simple as well
> - why wouldn't we do that to save space in /usr/share/selinux?

Why do we need to keep /usr/share/selinux/$SELINUXTYPE/*.pp around
_after_ a successful run of semodule from %post?  Why not just remove
them after installation?  And move enableaudit.pp into a separate -debug
package.

-- 
Stephen Smalley
National Security Agency


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Stephen Smalley]

On Thu, 2006-11-09 at 16:01 -0500, Stephen Smalley wrote:
> On Thu, 2006-11-09 at 13:43 -0500, Karl MacMillan wrote:
> > On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > > > From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
> > > > 
> > > > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> > > 
> > > > > Sounds like dropping base.linked and making previous optional would 
> > > > > address the problem more effectively.  Also, do we need to keep 
> > > > > policy.kern after successful installation of policy.N?  If 
> > > > not, we can 
> > > > > have libsemanage unlink it automatically after installation.
> > > > 
> > > > Same question for any other file regenerated by every commit, 
> > > > although we may not get much of a savings from the others.
> > > > file_contexts.template, file_contexts, and netfilter_contexts 
> > > > are the most obvious ones.
> > > > 
> > > 
> > > Karl suggested that we can compress the policy packages but not the
> > > kernel policy. As long as this isn't a policy package format change
> > > (eg., the policy packages in /usr/share/selinux are the same they've
> > > always been) and it is only libsemanage manipulating the files in the
> > > store I'm fine with that. The module store is a private resource of
> > > libsemanage so nothing else should be affected in any way by this. 
> > > 
> > 
> > Making semodule recognize bzipped files should be pretty simple as well
> > - why wouldn't we do that to save space in /usr/share/selinux?
> 
> Why do we need to keep /usr/share/selinux/$SELINUXTYPE/*.pp around
> _after_ a successful run of semodule from %post?  Why not just remove
> them after installation?  And move enableaudit.pp into a separate -debug
> package.

In fact, semodule could have an option like lpr -r (but something other
than -r since it is already taken for module removal) that indicates
that it should remove any .pp files it installed or upgraded after a
successful commit.

(btw it would be quite nice if semodule had some way to install a list
of .pp files without the current unpleasant workaround via awk)

-- 
Stephen Smalley
National Security Agency


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Karl MacMillan]

On Thu, 2006-11-09 at 16:01 -0500, Stephen Smalley wrote:
> On Thu, 2006-11-09 at 13:43 -0500, Karl MacMillan wrote:
> > On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > > > From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
> > > > 
> > > > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> > > 
> > > > > Sounds like dropping base.linked and making previous optional would 
> > > > > address the problem more effectively.  Also, do we need to keep 
> > > > > policy.kern after successful installation of policy.N?  If 
> > > > not, we can 
> > > > > have libsemanage unlink it automatically after installation.
> > > > 
> > > > Same question for any other file regenerated by every commit, 
> > > > although we may not get much of a savings from the others.
> > > > file_contexts.template, file_contexts, and netfilter_contexts 
> > > > are the most obvious ones.
> > > > 
> > > 
> > > Karl suggested that we can compress the policy packages but not the
> > > kernel policy. As long as this isn't a policy package format change
> > > (eg., the policy packages in /usr/share/selinux are the same they've
> > > always been) and it is only libsemanage manipulating the files in the
> > > store I'm fine with that. The module store is a private resource of
> > > libsemanage so nothing else should be affected in any way by this. 
> > > 
> > 
> > Making semodule recognize bzipped files should be pretty simple as well
> > - why wouldn't we do that to save space in /usr/share/selinux?
> 
> Why do we need to keep /usr/share/selinux/$SELINUXTYPE/*.pp around
> _after_ a successful run of semodule from %post?  Why not just remove
> them after installation?  And move enableaudit.pp into a separate -debug
> package.
> 

Removing files managed by an RPM doesn't seem like a good idea, even in
%post. At the very least, won't this will make rpm verification report
changes?

Karl


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Stephen Smalley]

On Thu, 2006-11-09 at 16:54 -0500, Karl MacMillan wrote:
> On Thu, 2006-11-09 at 16:01 -0500, Stephen Smalley wrote:
> > On Thu, 2006-11-09 at 13:43 -0500, Karl MacMillan wrote:
> > > On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > > > > From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
> > > > > 
> > > > > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > > > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> > > > 
> > > > > > Sounds like dropping base.linked and making previous optional would 
> > > > > > address the problem more effectively.  Also, do we need to keep 
> > > > > > policy.kern after successful installation of policy.N?  If 
> > > > > not, we can 
> > > > > > have libsemanage unlink it automatically after installation.
> > > > > 
> > > > > Same question for any other file regenerated by every commit, 
> > > > > although we may not get much of a savings from the others.
> > > > > file_contexts.template, file_contexts, and netfilter_contexts 
> > > > > are the most obvious ones.
> > > > > 
> > > > 
> > > > Karl suggested that we can compress the policy packages but not the
> > > > kernel policy. As long as this isn't a policy package format change
> > > > (eg., the policy packages in /usr/share/selinux are the same they've
> > > > always been) and it is only libsemanage manipulating the files in the
> > > > store I'm fine with that. The module store is a private resource of
> > > > libsemanage so nothing else should be affected in any way by this. 
> > > > 
> > > 
> > > Making semodule recognize bzipped files should be pretty simple as well
> > > - why wouldn't we do that to save space in /usr/share/selinux?
> > 
> > Why do we need to keep /usr/share/selinux/$SELINUXTYPE/*.pp around
> > _after_ a successful run of semodule from %post?  Why not just remove
> > them after installation?  And move enableaudit.pp into a separate -debug
> > package.
> > 
> 
> Removing files managed by an RPM doesn't seem like a good idea, even in
> %post. At the very least, won't this will make rpm verification report
> changes?

True.  Nonetheless, keeping those files around indefinitely when they
are never used again seems pointless.  Is there a different way to
package them that would let them be transient?

-- 
Stephen Smalley
National Security Agency


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: I would like to propose that we add compression to handle allpolicy files on disk.

[Karl MacMillan]

Stephen Smalley wrote:
> On Thu, 2006-11-09 at 16:54 -0500, Karl MacMillan wrote:
>> On Thu, 2006-11-09 at 16:01 -0500, Stephen Smalley wrote:
>>> On Thu, 2006-11-09 at 13:43 -0500, Karl MacMillan wrote:
>>>> On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
>>>>>> From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
>>>>>>
>>>>>> On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
>>>>>>> On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
>>>>>>> Sounds like dropping base.linked and making previous optional would 
>>>>>>> address the problem more effectively.  Also, do we need to keep 
>>>>>>> policy.kern after successful installation of policy.N?  If 
>>>>>> not, we can 
>>>>>>> have libsemanage unlink it automatically after installation.
>>>>>> Same question for any other file regenerated by every commit, 
>>>>>> although we may not get much of a savings from the others.
>>>>>> file_contexts.template, file_contexts, and netfilter_contexts 
>>>>>> are the most obvious ones.
>>>>>>
>>>>> Karl suggested that we can compress the policy packages but not the
>>>>> kernel policy. As long as this isn't a policy package format change
>>>>> (eg., the policy packages in /usr/share/selinux are the same they've
>>>>> always been) and it is only libsemanage manipulating the files in the
>>>>> store I'm fine with that. The module store is a private resource of
>>>>> libsemanage so nothing else should be affected in any way by this. 
>>>>>
>>>> Making semodule recognize bzipped files should be pretty simple as well
>>>> - why wouldn't we do that to save space in /usr/share/selinux?
>>> Why do we need to keep /usr/share/selinux/$SELINUXTYPE/*.pp around
>>> _after_ a successful run of semodule from %post?  Why not just remove
>>> them after installation?  And move enableaudit.pp into a separate -debug
>>> package.
>>>
>> Removing files managed by an RPM doesn't seem like a good idea, even in
>> %post. At the very least, won't this will make rpm verification report
>> changes?
> 
> True.  Nonetheless, keeping those files around indefinitely when they
> are never used again seems pointless.

It easily lets you go back to the shipped configuration, but probably 
only policy developers chaning base.pp care.

   Is there a different way to
> package them that would let them be transient?
> 

I'm not a packaging expert, but the whole reason we took this route was 
because we saw no other way to let the package manager manage its files 
and have the module store fully managed by SELinux.

Karl

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Joshua Brindle]

> From: Karl MacMillan [mailto:kmacmillan@mentalrootkit.com] 
> 
> Stephen Smalley wrote:
> > On Thu, 2006-11-09 at 16:54 -0500, Karl MacMillan wrote:
> >> On Thu, 2006-11-09 at 16:01 -0500, Stephen Smalley wrote:
> >>> On Thu, 2006-11-09 at 13:43 -0500, Karl MacMillan wrote:
> >>>> On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> >>>>>> From: Stephen Smalley [mailto:sds@tycho.nsa.gov]
> >>>>>>
> >>>>>> On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> >>>>>>> On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> >>>>>>> Sounds like dropping base.linked and making previous optional 
> >>>>>>> would address the problem more effectively.  Also, do 
> we need to 
> >>>>>>> keep policy.kern after successful installation of 
> policy.N?  If
> >>>>>> not, we can
> >>>>>>> have libsemanage unlink it automatically after installation.
> >>>>>> Same question for any other file regenerated by every commit, 
> >>>>>> although we may not get much of a savings from the others.
> >>>>>> file_contexts.template, file_contexts, and 
> netfilter_contexts are 
> >>>>>> the most obvious ones.
> >>>>>>
> >>>>> Karl suggested that we can compress the policy packages but not 
> >>>>> the kernel policy. As long as this isn't a policy 
> package format 
> >>>>> change (eg., the policy packages in /usr/share/selinux are the 
> >>>>> same they've always been) and it is only libsemanage 
> manipulating 
> >>>>> the files in the store I'm fine with that. The module 
> store is a 
> >>>>> private resource of libsemanage so nothing else should 
> be affected in any way by this.
> >>>>>
> >>>> Making semodule recognize bzipped files should be pretty 
> simple as 
> >>>> well
> >>>> - why wouldn't we do that to save space in /usr/share/selinux?
> >>> Why do we need to keep 
> /usr/share/selinux/$SELINUXTYPE/*.pp around 
> >>> _after_ a successful run of semodule from %post?  Why not just 
> >>> remove them after installation?  And move enableaudit.pp into a 
> >>> separate -debug package.
> >>>
> >> Removing files managed by an RPM doesn't seem like a good 
> idea, even 
> >> in %post. At the very least, won't this will make rpm verification 
> >> report changes?
> > 
> > True.  Nonetheless, keeping those files around indefinitely 
> when they 
> > are never used again seems pointless.
> 
> It easily lets you go back to the shipped configuration, but 
> probably only policy developers chaning base.pp care.
> 

I'd rather keep the /usr/share/selinux ones around as well, mainly
because you'll need them if you ever semodule -r a module. That's just
me though, if this use case seems marginal to everyone else that is
fine, we should remove them. 

Granted this is a RH only configuration so really you can get rid of
them regardless of what I think, this is just IMO.


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: I would like to propose that we add compression to handle allpolicy files on disk.

[Daniel J Walsh]

Stephen Smalley wrote:
> On Thu, 2006-11-09 at 13:43 -0500, Karl MacMillan wrote:
>   
>> On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
>>     
>>>> From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
>>>>
>>>> On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
>>>>         
>>>>> On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
>>>>>           
>>>>> Sounds like dropping base.linked and making previous optional would 
>>>>> address the problem more effectively.  Also, do we need to keep 
>>>>> policy.kern after successful installation of policy.N?  If 
>>>>>           
>>>> not, we can 
>>>>         
>>>>> have libsemanage unlink it automatically after installation.
>>>>>           
>>>> Same question for any other file regenerated by every commit, 
>>>> although we may not get much of a savings from the others.
>>>> file_contexts.template, file_contexts, and netfilter_contexts 
>>>> are the most obvious ones.
>>>>
>>>>         
>>> Karl suggested that we can compress the policy packages but not the
>>> kernel policy. As long as this isn't a policy package format change
>>> (eg., the policy packages in /usr/share/selinux are the same they've
>>> always been) and it is only libsemanage manipulating the files in the
>>> store I'm fine with that. The module store is a private resource of
>>> libsemanage so nothing else should be affected in any way by this. 
>>>
>>>       
>> Making semodule recognize bzipped files should be pretty simple as well
>> - why wouldn't we do that to save space in /usr/share/selinux?
>>     
>
> Why do we need to keep /usr/share/selinux/$SELINUXTYPE/*.pp around
> _after_ a successful run of semodule from %post?  Why not just remove
> them after installation?  And move enableaudit.pp into a separate -debug
> package.
>
>   
Because that is not the way RPM works.  :^(  Anything in the payload 
gets left on disk, Removing them in the post would be bad and would 
screw up rpm -V.

I am planning on compressing them, and changing the post install to 
uncompress into a temp dir during install.  That way it will at least be 
less of a problem.

That along with elimination of some of the stuff in modules subdirs 
should save us a lot of space, until we get some consensus on 
compression of the pp files.

Dan

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Joshua Brindle]

> From: Daniel J Walsh [mailto:dwalsh@redhat.com] 
> 

> I am planning on compressing them, and changing the post 
> install to uncompress into a temp dir during install.  That 
> way it will at least be less of a problem.
> 
> That along with elimination of some of the stuff in modules 
> subdirs should save us a lot of space, until we get some 
> consensus on compression of the pp files.
> 

So you are going to compress them *until* we get a consensus on
compressing them? This seems like a bad idea if we are going to try to
have any kind of consistency between selinux distributions (or even
between RH and upstream..)

I think having libsemanage compress files in the store is fine, as
that's a private resource of libsemanage users but having files on the
filesystem that are compressed but has nothing that can read them in the
format they are in on disk is not ideal I don't think..


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: I would like to propose that we add compression to handle allpolicy files on disk.

[Karl MacMillan]

Joshua Brindle wrote:
>> From: Daniel J Walsh [mailto:dwalsh@redhat.com] 
>>
> 
>> I am planning on compressing them, and changing the post 
>> install to uncompress into a temp dir during install.  That 
>> way it will at least be less of a problem.
>>
>> That along with elimination of some of the stuff in modules 
>> subdirs should save us a lot of space, until we get some 
>> consensus on compression of the pp files.
>>
> 
> So you are going to compress them *until* we get a consensus on
> compressing them? This seems like a bad idea if we are going to try to
> have any kind of consistency between selinux distributions (or even
> between RH and upstream..)
> 

Just the files from the rpm.

> I think having libsemanage compress files in the store is fine, as
> that's a private resource of libsemanage users but having files on the
> filesystem that are compressed but has nothing that can read them in the
> format they are in on disk is not ideal I don't think..
> 

If they sit there with a .bz2 extension I think it is not a big deal.

Karl


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

RE: I would like to propose that we add compression to handle allpolicy files on disk.

[Stephen Smalley]

On Thu, 2006-11-09 at 12:00 -0500, Joshua Brindle wrote:
> > From: Stephen Smalley [mailto:sds@tycho.nsa.gov] 
> > 
> > On Thu, 2006-11-09 at 10:13 -0500, Stephen Smalley wrote:
> > > On Thu, 2006-11-09 at 09:34 -0500, Joshua Brindle wrote:
> 
> > > Sounds like dropping base.linked and making previous optional would 
> > > address the problem more effectively.  Also, do we need to keep 
> > > policy.kern after successful installation of policy.N?  If 
> > not, we can 
> > > have libsemanage unlink it automatically after installation.
> > 
> > Same question for any other file regenerated by every commit, 
> > although we may not get much of a savings from the others.
> > file_contexts.template, file_contexts, and netfilter_contexts 
> > are the most obvious ones.
> > 
> 
> Karl suggested that we can compress the policy packages but not the
> kernel policy. As long as this isn't a policy package format change
> (eg., the policy packages in /usr/share/selinux are the same they've
> always been) and it is only libsemanage manipulating the files in the
> store I'm fine with that. The module store is a private resource of
> libsemanage so nothing else should be affected in any way by this. 
> 
> This will slow down some otherwise cheap operations such as semodule -l,
> rather than just opening the files and reading the policy name it'll
> have to decompress them first, I'm not sure what the performance cost
> will be.. Perhaps this should be configurable as well. 
> 
> Matt Anderson also mentioned using libbz2 which is more space efficient
> and has a better security history, so embedded installations include
> that library?
> 
> With bzip2 compression and nothing removed from the store I'm getting
> around 670k for the active store (so *2 if previous sticks around). With
> all the superfluous files removed the store is around 210k.
> 
> What kind of size were we looking for again?

I'm not clear it is justified; let's just start with the simple measures
of eliminating the unnecessary and transient files.

-- 
Stephen Smalley
National Security Agency


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.