* SELinux performance depending on type count
@ 2012-08-07 13:02 Ole Kliemann
2012-08-07 14:12 ` David Quigley
` (4 more replies)
0 siblings, 5 replies; 27+ messages in thread
From: Ole Kliemann @ 2012-08-07 13:02 UTC (permalink / raw)
To: selinux
[-- Attachment #1: Type: text/plain, Size: 636 bytes --]
I read on some locations (Fedora FAQ...) that there is an overall
performance impact of about 7% when running with SELinux.
Does anyone know if this impact is dependent upon the number of
types the policy has? I would assume no: A lot of types only take
up memory and caching should prevent any impact on the runtime
performance.
But if there was a performance problem with a lot of types, at
what number n would it start to hit hard? And how does it
increase (linear, quadratic...)?
And would it be better performance-wise to run a MCS-policy with
say categories c0.cn than to have types c0_t, ... cn_t?
Ole
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-07 13:02 SELinux performance depending on type count Ole Kliemann
@ 2012-08-07 14:12 ` David Quigley
2012-08-07 17:07 ` William Roberts
` (3 subsequent siblings)
4 siblings, 0 replies; 27+ messages in thread
From: David Quigley @ 2012-08-07 14:12 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux
On 08/07/2012 09:02, Ole Kliemann wrote:
> I read on some locations (Fedora FAQ...) that there is an overall
> performance impact of about 7% when running with SELinux.
>
> Does anyone know if this impact is dependent upon the number of
> types the policy has? I would assume no: A lot of types only take
> up memory and caching should prevent any impact on the runtime
> performance.
>
> But if there was a performance problem with a lot of types, at
> what number n would it start to hit hard? And how does it
> increase (linear, quadratic...)?
>
> And would it be better performance-wise to run a MCS-policy with
> say categories c0.cn than to have types c0_t, ... cn_t?
>
> Ole
I don't believe anyone has done recent benchmarks on SELinux overhead.
However in that study the overhead mostly comes from the permission
checks in the various layers of the Linux kernel. There were some issues
associated with access vector cache overhead but those were fixed I
believe by some contributors from Japan. The largest offender was the
checks on read/write since we checked on every single call to read/write
before. That was fixed so we don't do the full computation every time.
We only do it on the first read/write and only recheck on policy change
or label change since it invalidates our earlier check. It would be nice
to see a more recent study on SELinux overhead.
Dave
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-07 13:02 SELinux performance depending on type count Ole Kliemann
2012-08-07 14:12 ` David Quigley
@ 2012-08-07 17:07 ` William Roberts
2012-08-07 17:36 ` Daniel J Walsh
2012-08-08 20:44 ` Ole Kliemann
` (2 subsequent siblings)
4 siblings, 1 reply; 27+ messages in thread
From: William Roberts @ 2012-08-07 17:07 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux
Well as far as caching goes, their are cache misses, so as the amount
of data increase and your cache size stays fixed, they may be an
issue...
On Tue, Aug 7, 2012 at 6:02 AM, Ole Kliemann <ole@plastictree.net> wrote:
> I read on some locations (Fedora FAQ...) that there is an overall
> performance impact of about 7% when running with SELinux.
>
> Does anyone know if this impact is dependent upon the number of
> types the policy has? I would assume no: A lot of types only take
> up memory and caching should prevent any impact on the runtime
> performance.
>
> But if there was a performance problem with a lot of types, at
> what number n would it start to hit hard? And how does it
> increase (linear, quadratic...)?
>
> And would it be better performance-wise to run a MCS-policy with
> say categories c0.cn than to have types c0_t, ... cn_t?
>
> Ole
--
Respectfully,
William C Roberts
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-07 17:07 ` William Roberts
@ 2012-08-07 17:36 ` Daniel J Walsh
0 siblings, 0 replies; 27+ messages in thread
From: Daniel J Walsh @ 2012-08-07 17:36 UTC (permalink / raw)
To: William Roberts; +Cc: Ole Kliemann, selinux
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 08/07/2012 01:07 PM, William Roberts wrote:
> Well as far as caching goes, their are cache misses, so as the amount of
> data increase and your cache size stays fixed, they may be an issue...
>
> On Tue, Aug 7, 2012 at 6:02 AM, Ole Kliemann <ole@plastictree.net> wrote:
>> I read on some locations (Fedora FAQ...) that there is an overall
>> performance impact of about 7% when running with SELinux.
>>
>> Does anyone know if this impact is dependent upon the number of types the
>> policy has? I would assume no: A lot of types only take up memory and
>> caching should prevent any impact on the runtime performance.
>>
>> But if there was a performance problem with a lot of types, at what
>> number n would it start to hit hard? And how does it increase (linear,
>> quadratic...)?
>>
>> And would it be better performance-wise to run a MCS-policy with say
>> categories c0.cn than to have types c0_t, ... cn_t?
>>
>> Ole
>
>
>
Also 7% is ridiculously high. I would do your own measuring of SELinux and I
think for most work loads it is lot closer to unmeasurable difference.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAlAhUgYACgkQrlYvE4MpobMzcACdEccYKiyfZVeTQYF/06aKwc7i
tU4An1JeSEo6qcfNsIBzeZBn01fLN8KM
=Z4gW
-----END PGP SIGNATURE-----
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-07 13:02 SELinux performance depending on type count Ole Kliemann
2012-08-07 14:12 ` David Quigley
2012-08-07 17:07 ` William Roberts
@ 2012-08-08 20:44 ` Ole Kliemann
2012-08-09 10:45 ` Adam Tkac
2012-08-10 12:11 ` Ole Kliemann
2012-08-10 21:38 ` Ole Kliemann
4 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-08 20:44 UTC (permalink / raw)
To: selinux
[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]
I haven't tested any runtime performance just compile-time and
policy size.
All this is done on my old Dell D410 with a 1.73GHz Pentium M.
On Tue, Aug 07, 2012 at 03:02:44PM +0200, Ole Kliemann wrote:
> But if there was a performance problem with a lot of types, at
> what number n would it start to hit hard? And how does it
> increase (linear, quadratic...)?
n=10000, i.e. 20000 types, 10000 attributes and a handful of
allows per type and attribute in one module.
compilation is okay, but inserting the said module with
semodule... well at 18min CPU-time I killed the process... who
knows what size this policy would have had...
n=5000, i.e. 10000 types, 5000 attributes and a handful of
allows per type and attribute in one module.
inserting the module in about 5m30s walltime. policy is 13M of
size.
n=1000, i.e. 2000 types, 1000 attributes and a handful of
allows per type and attribute in one module.
inserting the module in about 9s walltime. policy is 2.5M of
size.
Apparently the runtime of inserting the module is fataly steep in
n. Rough estimation would be at least n^2, could be higher
depending on how long n=10000 would have actually taken.
> And would it be better performance-wise to run a MCS-policy with
> say categories c0.cn than to have types c0_t, ... cn_t?
n=10000, i.e. 10000 categories, one sensitivity and a handful of
mlsconstraints
inserting the module in about 8s walltime. policy is 284K of
size.
Of course this is a very rough and inprecise testing. But I guess
one can say that the policy infrastructure get's into trouble
with high type count whereas a high category count seems to be
handled flawlessly.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-08 20:44 ` Ole Kliemann
@ 2012-08-09 10:45 ` Adam Tkac
2012-08-09 11:56 ` Ole Kliemann
0 siblings, 1 reply; 27+ messages in thread
From: Adam Tkac @ 2012-08-09 10:45 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux
On Wed, Aug 08, 2012 at 10:44:01PM +0200, Ole Kliemann wrote:
> I haven't tested any runtime performance just compile-time and
> policy size.
>
> All this is done on my old Dell D410 with a 1.73GHz Pentium M.
>
> On Tue, Aug 07, 2012 at 03:02:44PM +0200, Ole Kliemann wrote:
> > But if there was a performance problem with a lot of types, at
> > what number n would it start to hit hard? And how does it
> > increase (linear, quadratic...)?
>
> n=10000, i.e. 20000 types, 10000 attributes and a handful of
> allows per type and attribute in one module.
>
> compilation is okay, but inserting the said module with
> semodule... well at 18min CPU-time I killed the process... who
> knows what size this policy would have had...
>
>
> n=5000, i.e. 10000 types, 5000 attributes and a handful of
> allows per type and attribute in one module.
>
> inserting the module in about 5m30s walltime. policy is 13M of
> size.
>
>
> n=1000, i.e. 2000 types, 1000 attributes and a handful of
> allows per type and attribute in one module.
>
> inserting the module in about 9s walltime. policy is 2.5M of
> size.
>
>
> Apparently the runtime of inserting the module is fataly steep in
> n. Rough estimation would be at least n^2, could be higher
> depending on how long n=10000 would have actually taken.
>
> > And would it be better performance-wise to run a MCS-policy with
> > say categories c0.cn than to have types c0_t, ... cn_t?
>
> n=10000, i.e. 10000 categories, one sensitivity and a handful of
> mlsconstraints
>
> inserting the module in about 8s walltime. policy is 284K of
> size.
>
>
> Of course this is a very rough and inprecise testing. But I guess
> one can say that the policy infrastructure get's into trouble
> with high type count whereas a high category count seems to be
> handled flawlessly.
I already wrote a patch for this and sent it to the ML some time ago. It is
currently present in Fedora git tree, you can try it and report if it speeds up
your operations with SELinux.
http://pkgs.fedoraproject.org/cgit/libsepol.git/plain/libsepol-rhat.patch
Regards, Adam
--
Adam Tkac, Red Hat, Inc.
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-09 10:45 ` Adam Tkac
@ 2012-08-09 11:56 ` Ole Kliemann
0 siblings, 0 replies; 27+ messages in thread
From: Ole Kliemann @ 2012-08-09 11:56 UTC (permalink / raw)
To: Adam Tkac; +Cc: selinux
[-- Attachment #1: Type: text/plain, Size: 810 bytes --]
On Thu, Aug 09, 2012 at 12:45:00PM +0200, Adam Tkac wrote:
> On Wed, Aug 08, 2012 at 10:44:01PM +0200, Ole Kliemann wrote:
> > Of course this is a very rough and inprecise testing. But I guess
> > one can say that the policy infrastructure get's into trouble
> > with high type count whereas a high category count seems to be
> > handled flawlessly.
>
> I already wrote a patch for this and sent it to the ML some time ago. It is
> currently present in Fedora git tree, you can try it and report if it speeds up
> your operations with SELinux.
>
> http://pkgs.fedoraproject.org/cgit/libsepol.git/plain/libsepol-rhat.patch
Thanks! But unfortunately this is out of the box incompatible
with Ubuntu's libsepol 2.1.0-1.2 and I don't really have the
overview to try to merge this right now.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-07 13:02 SELinux performance depending on type count Ole Kliemann
` (2 preceding siblings ...)
2012-08-08 20:44 ` Ole Kliemann
@ 2012-08-10 12:11 ` Ole Kliemann
2012-08-10 13:00 ` Stephen Smalley
2012-08-10 21:38 ` Ole Kliemann
4 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 12:11 UTC (permalink / raw)
To: selinux
[-- Attachment #1.1: Type: text/plain, Size: 2255 bytes --]
On Tue, Aug 07, 2012 at 03:02:44PM +0200, Ole Kliemann wrote:
> I read on some locations (Fedora FAQ...) that there is an overall
> performance impact of about 7% when running with SELinux.
>
> Does anyone know if this impact is dependent upon the number of
> types the policy has? I would assume no: A lot of types only take
> up memory and caching should prevent any impact on the runtime
> performance.
>
> But if there was a performance problem with a lot of types, at
> what number n would it start to hit hard? And how does it
> increase (linear, quadratic...)?
>
> And would it be better performance-wise to run a MCS-policy with
> say categories c0.cn than to have types c0_t, ... cn_t?
>
> Ole
I did some runtime test now. I have about 2000 types, 1000 of
them (named xIcJ_t, for 0 <= I <= 9, 0 <= J <= 99) each with his
own role (xIcJ_r) associated to a user_u. Then there is a user_r
and user_t for login. Additionally there is
system_u:system_r:root_t with full access to everything.
I run the attached script. It creates directories for each of the
1000 types, puts something in it, does a find/grep etc.
As system_u:system_r:root_t the script measures an average of
about 6sec walltime over 5 runs. (With very little variance.)
When I change context to user_u:user_r:user_t even things like
'ls' on home dir or 'id' lag consideribly the first time
executed. Just being in this context makes things slow. The
script measures an average of about 15sec walltime over 5 runs.
That's 2.5 times as much. Who thinks 7% is ridiculously high now?
;-)
While it's running the whole system sometimes lags even for just
writing on the terminal. top shows spikes of 50%+ CPU on kworker
threads.
Good side is: It's a clear result and kind of settles the
question. If you want a lot of different types for one user, go
for categories.
But I don't understand this result. Why isn't it slow when root
runs the script? He does the same relabeling to all those types.
It's not like user_u:user_r:user_t would be running in different
type concurrently. Just the fact that user_u is associated with
all those types seems to make it slow to run in any context
user_u:*
[-- Attachment #1.2: x.sh --]
[-- Type: application/x-sh, Size: 844 bytes --]
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 12:11 ` Ole Kliemann
@ 2012-08-10 13:00 ` Stephen Smalley
2012-08-10 14:36 ` Ole Kliemann
0 siblings, 1 reply; 27+ messages in thread
From: Stephen Smalley @ 2012-08-10 13:00 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux
On Fri, 2012-08-10 at 14:11 +0200, Ole Kliemann wrote:
> On Tue, Aug 07, 2012 at 03:02:44PM +0200, Ole Kliemann wrote:
> > I read on some locations (Fedora FAQ...) that there is an overall
> > performance impact of about 7% when running with SELinux.
> >
> > Does anyone know if this impact is dependent upon the number of
> > types the policy has? I would assume no: A lot of types only take
> > up memory and caching should prevent any impact on the runtime
> > performance.
> >
> > But if there was a performance problem with a lot of types, at
> > what number n would it start to hit hard? And how does it
> > increase (linear, quadratic...)?
> >
> > And would it be better performance-wise to run a MCS-policy with
> > say categories c0.cn than to have types c0_t, ... cn_t?
> >
> > Ole
>
> I did some runtime test now. I have about 2000 types, 1000 of
> them (named xIcJ_t, for 0 <= I <= 9, 0 <= J <= 99) each with his
> own role (xIcJ_r) associated to a user_u. Then there is a user_r
> and user_t for login. Additionally there is
> system_u:system_r:root_t with full access to everything.
>
> I run the attached script. It creates directories for each of the
> 1000 types, puts something in it, does a find/grep etc.
>
> As system_u:system_r:root_t the script measures an average of
> about 6sec walltime over 5 runs. (With very little variance.)
>
> When I change context to user_u:user_r:user_t even things like
> 'ls' on home dir or 'id' lag consideribly the first time
> executed. Just being in this context makes things slow. The
> script measures an average of about 15sec walltime over 5 runs.
>
> That's 2.5 times as much. Who thinks 7% is ridiculously high now?
> ;-)
>
> While it's running the whole system sometimes lags even for just
> writing on the terminal. top shows spikes of 50%+ CPU on kworker
> threads.
>
>
> Good side is: It's a clear result and kind of settles the
> question. If you want a lot of different types for one user, go
> for categories.
>
>
> But I don't understand this result. Why isn't it slow when root
> runs the script? He does the same relabeling to all those types.
> It's not like user_u:user_r:user_t would be running in different
> type concurrently. Just the fact that user_u is associated with
> all those types seems to make it slow to run in any context
> user_u:*
Your result doesn't sound right. Wondering whether you are triggering
masses of AVC denials (which could then peg syslog or audit) when
running your script?
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 13:00 ` Stephen Smalley
@ 2012-08-10 14:36 ` Ole Kliemann
2012-08-10 15:05 ` Stephen Smalley
0 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 14:36 UTC (permalink / raw)
To: Stephen Smalley; +Cc: selinux
[-- Attachment #1.1: Type: text/plain, Size: 3159 bytes --]
On Fri, Aug 10, 2012 at 09:00:15AM -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 14:11 +0200, Ole Kliemann wrote:
> > I did some runtime test now. I have about 2000 types, 1000 of
> > them (named xIcJ_t, for 0 <= I <= 9, 0 <= J <= 99) each with his
> > own role (xIcJ_r) associated to a user_u. Then there is a user_r
> > and user_t for login. Additionally there is
> > system_u:system_r:root_t with full access to everything.
> >
> > I run the attached script. It creates directories for each of the
> > 1000 types, puts something in it, does a find/grep etc.
> >
> > As system_u:system_r:root_t the script measures an average of
> > about 6sec walltime over 5 runs. (With very little variance.)
> >
> > When I change context to user_u:user_r:user_t even things like
> > 'ls' on home dir or 'id' lag consideribly the first time
> > executed. Just being in this context makes things slow. The
> > script measures an average of about 15sec walltime over 5 runs.
> >
> > That's 2.5 times as much. Who thinks 7% is ridiculously high now?
> > ;-)
> >
> > While it's running the whole system sometimes lags even for just
> > writing on the terminal. top shows spikes of 50%+ CPU on kworker
> > threads.
> >
> >
> > Good side is: It's a clear result and kind of settles the
> > question. If you want a lot of different types for one user, go
> > for categories.
> >
> >
> > But I don't understand this result. Why isn't it slow when root
> > runs the script? He does the same relabeling to all those types.
> > It's not like user_u:user_r:user_t would be running in different
> > type concurrently. Just the fact that user_u is associated with
> > all those types seems to make it slow to run in any context
> > user_u:*
>
> Your result doesn't sound right. Wondering whether you are triggering
> masses of AVC denials (which could then peg syslog or audit) when
> running your script?
Checked that of course. dmesg showed nothing or just occasional
denials. Just tryed again giving user_t full access to
everything. Changes nothing and dmesg is clear.
Anyway turns out above I forgot to mention something that
actually is the core of the problem:
I build a minimal example which is attached. Of course you have
at modify it to your policy.
Basicly there is one role choke_r with one type choke_t und a
user choke_u with role choke_r. Then are 1000 other types in the
choke role each with a corresponding attribute.
This alone isn't a problem. But if each of these attributes get
attributed to choke_t, the slowdown starts. Not as bad as
mentioned above but still significantly. (10sec in the test
above.)
So we have choke_u:choke_r:choke_t. Although all other types are
in choke_r, that alone causes nothing. But as soon as there are
these 1000 attributes on choke_t the fun starts. Actually you
don't even need the 1000 types. Just 1000 attributes on one type
produces a measurable slowdown (7sec) and lags. Having 1000 types
beside just makes it a lot worse. But there absolutely no
slowdown with just 1000 types and no attributes.
[-- Attachment #1.2: choke.m4 --]
[-- Type: text/plain, Size: 172 bytes --]
define(`mk_choke',`
type choke$1x$2_t, any_type;
role choke_r types { choke$1x$2_t };
attribute dominates_choke$1x$2;
typeattribute choke_t dominates_choke$1x$2;
')
[-- Attachment #1.3: choke.te --]
[-- Type: text/plain, Size: 16277 bytes --]
module choke 1.0.0;
require {
all_kernel_class_perms
all_userspace_class_perms
attribute any_type;
attribute domain_type;
};
type choke_t, any_type;
allow choke_t any_type:all_class_set *;
role choke_r;
role choke_r types { choke_t };
mk_choke(0, 00)
mk_choke(0, 01)
mk_choke(0, 02)
mk_choke(0, 03)
mk_choke(0, 04)
mk_choke(0, 05)
mk_choke(0, 06)
mk_choke(0, 07)
mk_choke(0, 08)
mk_choke(0, 09)
mk_choke(0, 10)
mk_choke(0, 11)
mk_choke(0, 12)
mk_choke(0, 13)
mk_choke(0, 14)
mk_choke(0, 15)
mk_choke(0, 16)
mk_choke(0, 17)
mk_choke(0, 18)
mk_choke(0, 19)
mk_choke(0, 20)
mk_choke(0, 21)
mk_choke(0, 22)
mk_choke(0, 23)
mk_choke(0, 24)
mk_choke(0, 25)
mk_choke(0, 26)
mk_choke(0, 27)
mk_choke(0, 28)
mk_choke(0, 29)
mk_choke(0, 30)
mk_choke(0, 31)
mk_choke(0, 32)
mk_choke(0, 33)
mk_choke(0, 34)
mk_choke(0, 35)
mk_choke(0, 36)
mk_choke(0, 37)
mk_choke(0, 38)
mk_choke(0, 39)
mk_choke(0, 40)
mk_choke(0, 41)
mk_choke(0, 42)
mk_choke(0, 43)
mk_choke(0, 44)
mk_choke(0, 45)
mk_choke(0, 46)
mk_choke(0, 47)
mk_choke(0, 48)
mk_choke(0, 49)
mk_choke(0, 50)
mk_choke(0, 51)
mk_choke(0, 52)
mk_choke(0, 53)
mk_choke(0, 54)
mk_choke(0, 55)
mk_choke(0, 56)
mk_choke(0, 57)
mk_choke(0, 58)
mk_choke(0, 59)
mk_choke(0, 60)
mk_choke(0, 61)
mk_choke(0, 62)
mk_choke(0, 63)
mk_choke(0, 64)
mk_choke(0, 65)
mk_choke(0, 66)
mk_choke(0, 67)
mk_choke(0, 68)
mk_choke(0, 69)
mk_choke(0, 70)
mk_choke(0, 71)
mk_choke(0, 72)
mk_choke(0, 73)
mk_choke(0, 74)
mk_choke(0, 75)
mk_choke(0, 76)
mk_choke(0, 77)
mk_choke(0, 78)
mk_choke(0, 79)
mk_choke(0, 80)
mk_choke(0, 81)
mk_choke(0, 82)
mk_choke(0, 83)
mk_choke(0, 84)
mk_choke(0, 85)
mk_choke(0, 86)
mk_choke(0, 87)
mk_choke(0, 88)
mk_choke(0, 89)
mk_choke(0, 90)
mk_choke(0, 91)
mk_choke(0, 92)
mk_choke(0, 93)
mk_choke(0, 94)
mk_choke(0, 95)
mk_choke(0, 96)
mk_choke(0, 97)
mk_choke(0, 98)
mk_choke(0, 99)
mk_choke(1, 00)
mk_choke(1, 01)
mk_choke(1, 02)
mk_choke(1, 03)
mk_choke(1, 04)
mk_choke(1, 05)
mk_choke(1, 06)
mk_choke(1, 07)
mk_choke(1, 08)
mk_choke(1, 09)
mk_choke(1, 10)
mk_choke(1, 11)
mk_choke(1, 12)
mk_choke(1, 13)
mk_choke(1, 14)
mk_choke(1, 15)
mk_choke(1, 16)
mk_choke(1, 17)
mk_choke(1, 18)
mk_choke(1, 19)
mk_choke(1, 20)
mk_choke(1, 21)
mk_choke(1, 22)
mk_choke(1, 23)
mk_choke(1, 24)
mk_choke(1, 25)
mk_choke(1, 26)
mk_choke(1, 27)
mk_choke(1, 28)
mk_choke(1, 29)
mk_choke(1, 30)
mk_choke(1, 31)
mk_choke(1, 32)
mk_choke(1, 33)
mk_choke(1, 34)
mk_choke(1, 35)
mk_choke(1, 36)
mk_choke(1, 37)
mk_choke(1, 38)
mk_choke(1, 39)
mk_choke(1, 40)
mk_choke(1, 41)
mk_choke(1, 42)
mk_choke(1, 43)
mk_choke(1, 44)
mk_choke(1, 45)
mk_choke(1, 46)
mk_choke(1, 47)
mk_choke(1, 48)
mk_choke(1, 49)
mk_choke(1, 50)
mk_choke(1, 51)
mk_choke(1, 52)
mk_choke(1, 53)
mk_choke(1, 54)
mk_choke(1, 55)
mk_choke(1, 56)
mk_choke(1, 57)
mk_choke(1, 58)
mk_choke(1, 59)
mk_choke(1, 60)
mk_choke(1, 61)
mk_choke(1, 62)
mk_choke(1, 63)
mk_choke(1, 64)
mk_choke(1, 65)
mk_choke(1, 66)
mk_choke(1, 67)
mk_choke(1, 68)
mk_choke(1, 69)
mk_choke(1, 70)
mk_choke(1, 71)
mk_choke(1, 72)
mk_choke(1, 73)
mk_choke(1, 74)
mk_choke(1, 75)
mk_choke(1, 76)
mk_choke(1, 77)
mk_choke(1, 78)
mk_choke(1, 79)
mk_choke(1, 80)
mk_choke(1, 81)
mk_choke(1, 82)
mk_choke(1, 83)
mk_choke(1, 84)
mk_choke(1, 85)
mk_choke(1, 86)
mk_choke(1, 87)
mk_choke(1, 88)
mk_choke(1, 89)
mk_choke(1, 90)
mk_choke(1, 91)
mk_choke(1, 92)
mk_choke(1, 93)
mk_choke(1, 94)
mk_choke(1, 95)
mk_choke(1, 96)
mk_choke(1, 97)
mk_choke(1, 98)
mk_choke(1, 99)
mk_choke(2, 00)
mk_choke(2, 01)
mk_choke(2, 02)
mk_choke(2, 03)
mk_choke(2, 04)
mk_choke(2, 05)
mk_choke(2, 06)
mk_choke(2, 07)
mk_choke(2, 08)
mk_choke(2, 09)
mk_choke(2, 10)
mk_choke(2, 11)
mk_choke(2, 12)
mk_choke(2, 13)
mk_choke(2, 14)
mk_choke(2, 15)
mk_choke(2, 16)
mk_choke(2, 17)
mk_choke(2, 18)
mk_choke(2, 19)
mk_choke(2, 20)
mk_choke(2, 21)
mk_choke(2, 22)
mk_choke(2, 23)
mk_choke(2, 24)
mk_choke(2, 25)
mk_choke(2, 26)
mk_choke(2, 27)
mk_choke(2, 28)
mk_choke(2, 29)
mk_choke(2, 30)
mk_choke(2, 31)
mk_choke(2, 32)
mk_choke(2, 33)
mk_choke(2, 34)
mk_choke(2, 35)
mk_choke(2, 36)
mk_choke(2, 37)
mk_choke(2, 38)
mk_choke(2, 39)
mk_choke(2, 40)
mk_choke(2, 41)
mk_choke(2, 42)
mk_choke(2, 43)
mk_choke(2, 44)
mk_choke(2, 45)
mk_choke(2, 46)
mk_choke(2, 47)
mk_choke(2, 48)
mk_choke(2, 49)
mk_choke(2, 50)
mk_choke(2, 51)
mk_choke(2, 52)
mk_choke(2, 53)
mk_choke(2, 54)
mk_choke(2, 55)
mk_choke(2, 56)
mk_choke(2, 57)
mk_choke(2, 58)
mk_choke(2, 59)
mk_choke(2, 60)
mk_choke(2, 61)
mk_choke(2, 62)
mk_choke(2, 63)
mk_choke(2, 64)
mk_choke(2, 65)
mk_choke(2, 66)
mk_choke(2, 67)
mk_choke(2, 68)
mk_choke(2, 69)
mk_choke(2, 70)
mk_choke(2, 71)
mk_choke(2, 72)
mk_choke(2, 73)
mk_choke(2, 74)
mk_choke(2, 75)
mk_choke(2, 76)
mk_choke(2, 77)
mk_choke(2, 78)
mk_choke(2, 79)
mk_choke(2, 80)
mk_choke(2, 81)
mk_choke(2, 82)
mk_choke(2, 83)
mk_choke(2, 84)
mk_choke(2, 85)
mk_choke(2, 86)
mk_choke(2, 87)
mk_choke(2, 88)
mk_choke(2, 89)
mk_choke(2, 90)
mk_choke(2, 91)
mk_choke(2, 92)
mk_choke(2, 93)
mk_choke(2, 94)
mk_choke(2, 95)
mk_choke(2, 96)
mk_choke(2, 97)
mk_choke(2, 98)
mk_choke(2, 99)
mk_choke(3, 00)
mk_choke(3, 01)
mk_choke(3, 02)
mk_choke(3, 03)
mk_choke(3, 04)
mk_choke(3, 05)
mk_choke(3, 06)
mk_choke(3, 07)
mk_choke(3, 08)
mk_choke(3, 09)
mk_choke(3, 10)
mk_choke(3, 11)
mk_choke(3, 12)
mk_choke(3, 13)
mk_choke(3, 14)
mk_choke(3, 15)
mk_choke(3, 16)
mk_choke(3, 17)
mk_choke(3, 18)
mk_choke(3, 19)
mk_choke(3, 20)
mk_choke(3, 21)
mk_choke(3, 22)
mk_choke(3, 23)
mk_choke(3, 24)
mk_choke(3, 25)
mk_choke(3, 26)
mk_choke(3, 27)
mk_choke(3, 28)
mk_choke(3, 29)
mk_choke(3, 30)
mk_choke(3, 31)
mk_choke(3, 32)
mk_choke(3, 33)
mk_choke(3, 34)
mk_choke(3, 35)
mk_choke(3, 36)
mk_choke(3, 37)
mk_choke(3, 38)
mk_choke(3, 39)
mk_choke(3, 40)
mk_choke(3, 41)
mk_choke(3, 42)
mk_choke(3, 43)
mk_choke(3, 44)
mk_choke(3, 45)
mk_choke(3, 46)
mk_choke(3, 47)
mk_choke(3, 48)
mk_choke(3, 49)
mk_choke(3, 50)
mk_choke(3, 51)
mk_choke(3, 52)
mk_choke(3, 53)
mk_choke(3, 54)
mk_choke(3, 55)
mk_choke(3, 56)
mk_choke(3, 57)
mk_choke(3, 58)
mk_choke(3, 59)
mk_choke(3, 60)
mk_choke(3, 61)
mk_choke(3, 62)
mk_choke(3, 63)
mk_choke(3, 64)
mk_choke(3, 65)
mk_choke(3, 66)
mk_choke(3, 67)
mk_choke(3, 68)
mk_choke(3, 69)
mk_choke(3, 70)
mk_choke(3, 71)
mk_choke(3, 72)
mk_choke(3, 73)
mk_choke(3, 74)
mk_choke(3, 75)
mk_choke(3, 76)
mk_choke(3, 77)
mk_choke(3, 78)
mk_choke(3, 79)
mk_choke(3, 80)
mk_choke(3, 81)
mk_choke(3, 82)
mk_choke(3, 83)
mk_choke(3, 84)
mk_choke(3, 85)
mk_choke(3, 86)
mk_choke(3, 87)
mk_choke(3, 88)
mk_choke(3, 89)
mk_choke(3, 90)
mk_choke(3, 91)
mk_choke(3, 92)
mk_choke(3, 93)
mk_choke(3, 94)
mk_choke(3, 95)
mk_choke(3, 96)
mk_choke(3, 97)
mk_choke(3, 98)
mk_choke(3, 99)
mk_choke(4, 00)
mk_choke(4, 01)
mk_choke(4, 02)
mk_choke(4, 03)
mk_choke(4, 04)
mk_choke(4, 05)
mk_choke(4, 06)
mk_choke(4, 07)
mk_choke(4, 08)
mk_choke(4, 09)
mk_choke(4, 10)
mk_choke(4, 11)
mk_choke(4, 12)
mk_choke(4, 13)
mk_choke(4, 14)
mk_choke(4, 15)
mk_choke(4, 16)
mk_choke(4, 17)
mk_choke(4, 18)
mk_choke(4, 19)
mk_choke(4, 20)
mk_choke(4, 21)
mk_choke(4, 22)
mk_choke(4, 23)
mk_choke(4, 24)
mk_choke(4, 25)
mk_choke(4, 26)
mk_choke(4, 27)
mk_choke(4, 28)
mk_choke(4, 29)
mk_choke(4, 30)
mk_choke(4, 31)
mk_choke(4, 32)
mk_choke(4, 33)
mk_choke(4, 34)
mk_choke(4, 35)
mk_choke(4, 36)
mk_choke(4, 37)
mk_choke(4, 38)
mk_choke(4, 39)
mk_choke(4, 40)
mk_choke(4, 41)
mk_choke(4, 42)
mk_choke(4, 43)
mk_choke(4, 44)
mk_choke(4, 45)
mk_choke(4, 46)
mk_choke(4, 47)
mk_choke(4, 48)
mk_choke(4, 49)
mk_choke(4, 50)
mk_choke(4, 51)
mk_choke(4, 52)
mk_choke(4, 53)
mk_choke(4, 54)
mk_choke(4, 55)
mk_choke(4, 56)
mk_choke(4, 57)
mk_choke(4, 58)
mk_choke(4, 59)
mk_choke(4, 60)
mk_choke(4, 61)
mk_choke(4, 62)
mk_choke(4, 63)
mk_choke(4, 64)
mk_choke(4, 65)
mk_choke(4, 66)
mk_choke(4, 67)
mk_choke(4, 68)
mk_choke(4, 69)
mk_choke(4, 70)
mk_choke(4, 71)
mk_choke(4, 72)
mk_choke(4, 73)
mk_choke(4, 74)
mk_choke(4, 75)
mk_choke(4, 76)
mk_choke(4, 77)
mk_choke(4, 78)
mk_choke(4, 79)
mk_choke(4, 80)
mk_choke(4, 81)
mk_choke(4, 82)
mk_choke(4, 83)
mk_choke(4, 84)
mk_choke(4, 85)
mk_choke(4, 86)
mk_choke(4, 87)
mk_choke(4, 88)
mk_choke(4, 89)
mk_choke(4, 90)
mk_choke(4, 91)
mk_choke(4, 92)
mk_choke(4, 93)
mk_choke(4, 94)
mk_choke(4, 95)
mk_choke(4, 96)
mk_choke(4, 97)
mk_choke(4, 98)
mk_choke(4, 99)
mk_choke(5, 00)
mk_choke(5, 01)
mk_choke(5, 02)
mk_choke(5, 03)
mk_choke(5, 04)
mk_choke(5, 05)
mk_choke(5, 06)
mk_choke(5, 07)
mk_choke(5, 08)
mk_choke(5, 09)
mk_choke(5, 10)
mk_choke(5, 11)
mk_choke(5, 12)
mk_choke(5, 13)
mk_choke(5, 14)
mk_choke(5, 15)
mk_choke(5, 16)
mk_choke(5, 17)
mk_choke(5, 18)
mk_choke(5, 19)
mk_choke(5, 20)
mk_choke(5, 21)
mk_choke(5, 22)
mk_choke(5, 23)
mk_choke(5, 24)
mk_choke(5, 25)
mk_choke(5, 26)
mk_choke(5, 27)
mk_choke(5, 28)
mk_choke(5, 29)
mk_choke(5, 30)
mk_choke(5, 31)
mk_choke(5, 32)
mk_choke(5, 33)
mk_choke(5, 34)
mk_choke(5, 35)
mk_choke(5, 36)
mk_choke(5, 37)
mk_choke(5, 38)
mk_choke(5, 39)
mk_choke(5, 40)
mk_choke(5, 41)
mk_choke(5, 42)
mk_choke(5, 43)
mk_choke(5, 44)
mk_choke(5, 45)
mk_choke(5, 46)
mk_choke(5, 47)
mk_choke(5, 48)
mk_choke(5, 49)
mk_choke(5, 50)
mk_choke(5, 51)
mk_choke(5, 52)
mk_choke(5, 53)
mk_choke(5, 54)
mk_choke(5, 55)
mk_choke(5, 56)
mk_choke(5, 57)
mk_choke(5, 58)
mk_choke(5, 59)
mk_choke(5, 60)
mk_choke(5, 61)
mk_choke(5, 62)
mk_choke(5, 63)
mk_choke(5, 64)
mk_choke(5, 65)
mk_choke(5, 66)
mk_choke(5, 67)
mk_choke(5, 68)
mk_choke(5, 69)
mk_choke(5, 70)
mk_choke(5, 71)
mk_choke(5, 72)
mk_choke(5, 73)
mk_choke(5, 74)
mk_choke(5, 75)
mk_choke(5, 76)
mk_choke(5, 77)
mk_choke(5, 78)
mk_choke(5, 79)
mk_choke(5, 80)
mk_choke(5, 81)
mk_choke(5, 82)
mk_choke(5, 83)
mk_choke(5, 84)
mk_choke(5, 85)
mk_choke(5, 86)
mk_choke(5, 87)
mk_choke(5, 88)
mk_choke(5, 89)
mk_choke(5, 90)
mk_choke(5, 91)
mk_choke(5, 92)
mk_choke(5, 93)
mk_choke(5, 94)
mk_choke(5, 95)
mk_choke(5, 96)
mk_choke(5, 97)
mk_choke(5, 98)
mk_choke(5, 99)
mk_choke(6, 00)
mk_choke(6, 01)
mk_choke(6, 02)
mk_choke(6, 03)
mk_choke(6, 04)
mk_choke(6, 05)
mk_choke(6, 06)
mk_choke(6, 07)
mk_choke(6, 08)
mk_choke(6, 09)
mk_choke(6, 10)
mk_choke(6, 11)
mk_choke(6, 12)
mk_choke(6, 13)
mk_choke(6, 14)
mk_choke(6, 15)
mk_choke(6, 16)
mk_choke(6, 17)
mk_choke(6, 18)
mk_choke(6, 19)
mk_choke(6, 20)
mk_choke(6, 21)
mk_choke(6, 22)
mk_choke(6, 23)
mk_choke(6, 24)
mk_choke(6, 25)
mk_choke(6, 26)
mk_choke(6, 27)
mk_choke(6, 28)
mk_choke(6, 29)
mk_choke(6, 30)
mk_choke(6, 31)
mk_choke(6, 32)
mk_choke(6, 33)
mk_choke(6, 34)
mk_choke(6, 35)
mk_choke(6, 36)
mk_choke(6, 37)
mk_choke(6, 38)
mk_choke(6, 39)
mk_choke(6, 40)
mk_choke(6, 41)
mk_choke(6, 42)
mk_choke(6, 43)
mk_choke(6, 44)
mk_choke(6, 45)
mk_choke(6, 46)
mk_choke(6, 47)
mk_choke(6, 48)
mk_choke(6, 49)
mk_choke(6, 50)
mk_choke(6, 51)
mk_choke(6, 52)
mk_choke(6, 53)
mk_choke(6, 54)
mk_choke(6, 55)
mk_choke(6, 56)
mk_choke(6, 57)
mk_choke(6, 58)
mk_choke(6, 59)
mk_choke(6, 60)
mk_choke(6, 61)
mk_choke(6, 62)
mk_choke(6, 63)
mk_choke(6, 64)
mk_choke(6, 65)
mk_choke(6, 66)
mk_choke(6, 67)
mk_choke(6, 68)
mk_choke(6, 69)
mk_choke(6, 70)
mk_choke(6, 71)
mk_choke(6, 72)
mk_choke(6, 73)
mk_choke(6, 74)
mk_choke(6, 75)
mk_choke(6, 76)
mk_choke(6, 77)
mk_choke(6, 78)
mk_choke(6, 79)
mk_choke(6, 80)
mk_choke(6, 81)
mk_choke(6, 82)
mk_choke(6, 83)
mk_choke(6, 84)
mk_choke(6, 85)
mk_choke(6, 86)
mk_choke(6, 87)
mk_choke(6, 88)
mk_choke(6, 89)
mk_choke(6, 90)
mk_choke(6, 91)
mk_choke(6, 92)
mk_choke(6, 93)
mk_choke(6, 94)
mk_choke(6, 95)
mk_choke(6, 96)
mk_choke(6, 97)
mk_choke(6, 98)
mk_choke(6, 99)
mk_choke(7, 00)
mk_choke(7, 01)
mk_choke(7, 02)
mk_choke(7, 03)
mk_choke(7, 04)
mk_choke(7, 05)
mk_choke(7, 06)
mk_choke(7, 07)
mk_choke(7, 08)
mk_choke(7, 09)
mk_choke(7, 10)
mk_choke(7, 11)
mk_choke(7, 12)
mk_choke(7, 13)
mk_choke(7, 14)
mk_choke(7, 15)
mk_choke(7, 16)
mk_choke(7, 17)
mk_choke(7, 18)
mk_choke(7, 19)
mk_choke(7, 20)
mk_choke(7, 21)
mk_choke(7, 22)
mk_choke(7, 23)
mk_choke(7, 24)
mk_choke(7, 25)
mk_choke(7, 26)
mk_choke(7, 27)
mk_choke(7, 28)
mk_choke(7, 29)
mk_choke(7, 30)
mk_choke(7, 31)
mk_choke(7, 32)
mk_choke(7, 33)
mk_choke(7, 34)
mk_choke(7, 35)
mk_choke(7, 36)
mk_choke(7, 37)
mk_choke(7, 38)
mk_choke(7, 39)
mk_choke(7, 40)
mk_choke(7, 41)
mk_choke(7, 42)
mk_choke(7, 43)
mk_choke(7, 44)
mk_choke(7, 45)
mk_choke(7, 46)
mk_choke(7, 47)
mk_choke(7, 48)
mk_choke(7, 49)
mk_choke(7, 50)
mk_choke(7, 51)
mk_choke(7, 52)
mk_choke(7, 53)
mk_choke(7, 54)
mk_choke(7, 55)
mk_choke(7, 56)
mk_choke(7, 57)
mk_choke(7, 58)
mk_choke(7, 59)
mk_choke(7, 60)
mk_choke(7, 61)
mk_choke(7, 62)
mk_choke(7, 63)
mk_choke(7, 64)
mk_choke(7, 65)
mk_choke(7, 66)
mk_choke(7, 67)
mk_choke(7, 68)
mk_choke(7, 69)
mk_choke(7, 70)
mk_choke(7, 71)
mk_choke(7, 72)
mk_choke(7, 73)
mk_choke(7, 74)
mk_choke(7, 75)
mk_choke(7, 76)
mk_choke(7, 77)
mk_choke(7, 78)
mk_choke(7, 79)
mk_choke(7, 80)
mk_choke(7, 81)
mk_choke(7, 82)
mk_choke(7, 83)
mk_choke(7, 84)
mk_choke(7, 85)
mk_choke(7, 86)
mk_choke(7, 87)
mk_choke(7, 88)
mk_choke(7, 89)
mk_choke(7, 90)
mk_choke(7, 91)
mk_choke(7, 92)
mk_choke(7, 93)
mk_choke(7, 94)
mk_choke(7, 95)
mk_choke(7, 96)
mk_choke(7, 97)
mk_choke(7, 98)
mk_choke(7, 99)
mk_choke(8, 00)
mk_choke(8, 01)
mk_choke(8, 02)
mk_choke(8, 03)
mk_choke(8, 04)
mk_choke(8, 05)
mk_choke(8, 06)
mk_choke(8, 07)
mk_choke(8, 08)
mk_choke(8, 09)
mk_choke(8, 10)
mk_choke(8, 11)
mk_choke(8, 12)
mk_choke(8, 13)
mk_choke(8, 14)
mk_choke(8, 15)
mk_choke(8, 16)
mk_choke(8, 17)
mk_choke(8, 18)
mk_choke(8, 19)
mk_choke(8, 20)
mk_choke(8, 21)
mk_choke(8, 22)
mk_choke(8, 23)
mk_choke(8, 24)
mk_choke(8, 25)
mk_choke(8, 26)
mk_choke(8, 27)
mk_choke(8, 28)
mk_choke(8, 29)
mk_choke(8, 30)
mk_choke(8, 31)
mk_choke(8, 32)
mk_choke(8, 33)
mk_choke(8, 34)
mk_choke(8, 35)
mk_choke(8, 36)
mk_choke(8, 37)
mk_choke(8, 38)
mk_choke(8, 39)
mk_choke(8, 40)
mk_choke(8, 41)
mk_choke(8, 42)
mk_choke(8, 43)
mk_choke(8, 44)
mk_choke(8, 45)
mk_choke(8, 46)
mk_choke(8, 47)
mk_choke(8, 48)
mk_choke(8, 49)
mk_choke(8, 50)
mk_choke(8, 51)
mk_choke(8, 52)
mk_choke(8, 53)
mk_choke(8, 54)
mk_choke(8, 55)
mk_choke(8, 56)
mk_choke(8, 57)
mk_choke(8, 58)
mk_choke(8, 59)
mk_choke(8, 60)
mk_choke(8, 61)
mk_choke(8, 62)
mk_choke(8, 63)
mk_choke(8, 64)
mk_choke(8, 65)
mk_choke(8, 66)
mk_choke(8, 67)
mk_choke(8, 68)
mk_choke(8, 69)
mk_choke(8, 70)
mk_choke(8, 71)
mk_choke(8, 72)
mk_choke(8, 73)
mk_choke(8, 74)
mk_choke(8, 75)
mk_choke(8, 76)
mk_choke(8, 77)
mk_choke(8, 78)
mk_choke(8, 79)
mk_choke(8, 80)
mk_choke(8, 81)
mk_choke(8, 82)
mk_choke(8, 83)
mk_choke(8, 84)
mk_choke(8, 85)
mk_choke(8, 86)
mk_choke(8, 87)
mk_choke(8, 88)
mk_choke(8, 89)
mk_choke(8, 90)
mk_choke(8, 91)
mk_choke(8, 92)
mk_choke(8, 93)
mk_choke(8, 94)
mk_choke(8, 95)
mk_choke(8, 96)
mk_choke(8, 97)
mk_choke(8, 98)
mk_choke(8, 99)
mk_choke(9, 00)
mk_choke(9, 01)
mk_choke(9, 02)
mk_choke(9, 03)
mk_choke(9, 04)
mk_choke(9, 05)
mk_choke(9, 06)
mk_choke(9, 07)
mk_choke(9, 08)
mk_choke(9, 09)
mk_choke(9, 10)
mk_choke(9, 11)
mk_choke(9, 12)
mk_choke(9, 13)
mk_choke(9, 14)
mk_choke(9, 15)
mk_choke(9, 16)
mk_choke(9, 17)
mk_choke(9, 18)
mk_choke(9, 19)
mk_choke(9, 20)
mk_choke(9, 21)
mk_choke(9, 22)
mk_choke(9, 23)
mk_choke(9, 24)
mk_choke(9, 25)
mk_choke(9, 26)
mk_choke(9, 27)
mk_choke(9, 28)
mk_choke(9, 29)
mk_choke(9, 30)
mk_choke(9, 31)
mk_choke(9, 32)
mk_choke(9, 33)
mk_choke(9, 34)
mk_choke(9, 35)
mk_choke(9, 36)
mk_choke(9, 37)
mk_choke(9, 38)
mk_choke(9, 39)
mk_choke(9, 40)
mk_choke(9, 41)
mk_choke(9, 42)
mk_choke(9, 43)
mk_choke(9, 44)
mk_choke(9, 45)
mk_choke(9, 46)
mk_choke(9, 47)
mk_choke(9, 48)
mk_choke(9, 49)
mk_choke(9, 50)
mk_choke(9, 51)
mk_choke(9, 52)
mk_choke(9, 53)
mk_choke(9, 54)
mk_choke(9, 55)
mk_choke(9, 56)
mk_choke(9, 57)
mk_choke(9, 58)
mk_choke(9, 59)
mk_choke(9, 60)
mk_choke(9, 61)
mk_choke(9, 62)
mk_choke(9, 63)
mk_choke(9, 64)
mk_choke(9, 65)
mk_choke(9, 66)
mk_choke(9, 67)
mk_choke(9, 68)
mk_choke(9, 69)
mk_choke(9, 70)
mk_choke(9, 71)
mk_choke(9, 72)
mk_choke(9, 73)
mk_choke(9, 74)
mk_choke(9, 75)
mk_choke(9, 76)
mk_choke(9, 77)
mk_choke(9, 78)
mk_choke(9, 79)
mk_choke(9, 80)
mk_choke(9, 81)
mk_choke(9, 82)
mk_choke(9, 83)
mk_choke(9, 84)
mk_choke(9, 85)
mk_choke(9, 86)
mk_choke(9, 87)
mk_choke(9, 88)
mk_choke(9, 89)
mk_choke(9, 90)
mk_choke(9, 91)
mk_choke(9, 92)
mk_choke(9, 93)
mk_choke(9, 94)
mk_choke(9, 95)
mk_choke(9, 96)
mk_choke(9, 97)
mk_choke(9, 98)
mk_choke(9, 99)
user choke_u roles { choke_r };
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 14:36 ` Ole Kliemann
@ 2012-08-10 15:05 ` Stephen Smalley
2012-08-10 15:43 ` Ole Kliemann
2012-08-10 15:44 ` Ole Kliemann
0 siblings, 2 replies; 27+ messages in thread
From: Stephen Smalley @ 2012-08-10 15:05 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Fri, 2012-08-10 at 16:36 +0200, Ole Kliemann wrote:
> On Fri, Aug 10, 2012 at 09:00:15AM -0400, Stephen Smalley wrote:
> > On Fri, 2012-08-10 at 14:11 +0200, Ole Kliemann wrote:
> > > I did some runtime test now. I have about 2000 types, 1000 of
> > > them (named xIcJ_t, for 0 <= I <= 9, 0 <= J <= 99) each with his
> > > own role (xIcJ_r) associated to a user_u. Then there is a user_r
> > > and user_t for login. Additionally there is
> > > system_u:system_r:root_t with full access to everything.
> > >
> > > I run the attached script. It creates directories for each of the
> > > 1000 types, puts something in it, does a find/grep etc.
> > >
> > > As system_u:system_r:root_t the script measures an average of
> > > about 6sec walltime over 5 runs. (With very little variance.)
> > >
> > > When I change context to user_u:user_r:user_t even things like
> > > 'ls' on home dir or 'id' lag consideribly the first time
> > > executed. Just being in this context makes things slow. The
> > > script measures an average of about 15sec walltime over 5 runs.
> > >
> > > That's 2.5 times as much. Who thinks 7% is ridiculously high now?
> > > ;-)
> > >
> > > While it's running the whole system sometimes lags even for just
> > > writing on the terminal. top shows spikes of 50%+ CPU on kworker
> > > threads.
> > >
> > >
> > > Good side is: It's a clear result and kind of settles the
> > > question. If you want a lot of different types for one user, go
> > > for categories.
> > >
> > >
> > > But I don't understand this result. Why isn't it slow when root
> > > runs the script? He does the same relabeling to all those types.
> > > It's not like user_u:user_r:user_t would be running in different
> > > type concurrently. Just the fact that user_u is associated with
> > > all those types seems to make it slow to run in any context
> > > user_u:*
> >
> > Your result doesn't sound right. Wondering whether you are triggering
> > masses of AVC denials (which could then peg syslog or audit) when
> > running your script?
>
> Checked that of course. dmesg showed nothing or just occasional
> denials. Just tryed again giving user_t full access to
> everything. Changes nothing and dmesg is clear.
>
> Anyway turns out above I forgot to mention something that
> actually is the core of the problem:
>
> I build a minimal example which is attached. Of course you have
> at modify it to your policy.
>
> Basicly there is one role choke_r with one type choke_t und a
> user choke_u with role choke_r. Then are 1000 other types in the
> choke role each with a corresponding attribute.
>
> This alone isn't a problem. But if each of these attributes get
> attributed to choke_t, the slowdown starts. Not as bad as
> mentioned above but still significantly. (10sec in the test
> above.)
>
> So we have choke_u:choke_r:choke_t. Although all other types are
> in choke_r, that alone causes nothing. But as soon as there are
> these 1000 attributes on choke_t the fun starts. Actually you
> don't even need the 1000 types. Just 1000 attributes on one type
> produces a measurable slowdown (7sec) and lags. Having 1000 types
> beside just makes it a lot worse. But there absolutely no
> slowdown with just 1000 types and no attributes.
Interesting. We would expect some slowdown on an AVC cache miss in that
situation, although the amount seems troubling. Things to consider:
- Does the AVC cache need to be increased or otherwise tuned? You can
see some information via the avcstat utility or by directly looking
at /sys/fs/selinux/avc.
- Does security_compute_av(), which is called on a cache miss, need some
profiling and tuning? Particularly the logic within
context_struct_compute_av(), where we are iterating through the type
attribute ebitmaps.
In Fedora 17, most types only have a few attributes (as shown by seinfo
-t -x). unconfined_t has a larger number of attributes than most, but
even it only has 46 attributes. So we likely don't see this behavior
there.
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 15:05 ` Stephen Smalley
@ 2012-08-10 15:43 ` Ole Kliemann
2012-08-10 15:44 ` Ole Kliemann
1 sibling, 0 replies; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 15:43 UTC (permalink / raw)
To: Stephen Smalley; +Cc: selinux, Eric Paris
[-- Attachment #1: Type: text/plain, Size: 1865 bytes --]
On Fri, Aug 10, 2012 at 11:05:47AM -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 16:36 +0200, Ole Kliemann wrote:
> > So we have choke_u:choke_r:choke_t. Although all other types are
> > in choke_r, that alone causes nothing. But as soon as there are
> > these 1000 attributes on choke_t the fun starts. Actually you
> > don't even need the 1000 types. Just 1000 attributes on one type
> > produces a measurable slowdown (7sec) and lags. Having 1000 types
> > beside just makes it a lot worse. But there absolutely no
> > slowdown with just 1000 types and no attributes.
>
> Interesting. We would expect some slowdown on an AVC cache miss in that
> situation, although the amount seems troubling. Things to consider:
> - Does the AVC cache need to be increased or otherwise tuned? You can
> see some information via the avcstat utility or by directly looking
> at /sys/fs/selinux/avc.
Just having 'avcstat 3' running and peeping at it:
calling 'id' without lag takes about 400 lookups, all in cache.
If I reload the policy, the first call to 'id' lags for maybe
500ms. avcstat shows for that period 400 lookups and maybe 20
misses.
Consequent calls to id do not lag and have no misses.
About the same is for 'ls ~'
But: calling 'id' from a not attribute-spammed domain after
reload shows about 5 misses too. But there is absolutely no lag.
>
> - Does security_compute_av(), which is called on a cache miss, need some
> profiling and tuning? Particularly the logic within
> context_struct_compute_av(), where we are iterating through the type
> attribute ebitmaps.
>
> In Fedora 17, most types only have a few attributes (as shown by seinfo
> -t -x). unconfined_t has a larger number of attributes than most, but
> even it only has 46 attributes. So we likely don't see this behavior
> there.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 15:05 ` Stephen Smalley
2012-08-10 15:43 ` Ole Kliemann
@ 2012-08-10 15:44 ` Ole Kliemann
2012-08-10 16:08 ` Stephen Smalley
1 sibling, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 15:44 UTC (permalink / raw)
To: Stephen Smalley; +Cc: selinux, Eric Paris
[-- Attachment #1: Type: text/plain, Size: 4513 bytes --]
PS: Have you actually reproduced this problem? Could still be
something else broken on my system...
On Fri, Aug 10, 2012 at 11:05:47AM -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 16:36 +0200, Ole Kliemann wrote:
> > On Fri, Aug 10, 2012 at 09:00:15AM -0400, Stephen Smalley wrote:
> > > On Fri, 2012-08-10 at 14:11 +0200, Ole Kliemann wrote:
> > > > I did some runtime test now. I have about 2000 types, 1000 of
> > > > them (named xIcJ_t, for 0 <= I <= 9, 0 <= J <= 99) each with his
> > > > own role (xIcJ_r) associated to a user_u. Then there is a user_r
> > > > and user_t for login. Additionally there is
> > > > system_u:system_r:root_t with full access to everything.
> > > >
> > > > I run the attached script. It creates directories for each of the
> > > > 1000 types, puts something in it, does a find/grep etc.
> > > >
> > > > As system_u:system_r:root_t the script measures an average of
> > > > about 6sec walltime over 5 runs. (With very little variance.)
> > > >
> > > > When I change context to user_u:user_r:user_t even things like
> > > > 'ls' on home dir or 'id' lag consideribly the first time
> > > > executed. Just being in this context makes things slow. The
> > > > script measures an average of about 15sec walltime over 5 runs.
> > > >
> > > > That's 2.5 times as much. Who thinks 7% is ridiculously high now?
> > > > ;-)
> > > >
> > > > While it's running the whole system sometimes lags even for just
> > > > writing on the terminal. top shows spikes of 50%+ CPU on kworker
> > > > threads.
> > > >
> > > >
> > > > Good side is: It's a clear result and kind of settles the
> > > > question. If you want a lot of different types for one user, go
> > > > for categories.
> > > >
> > > >
> > > > But I don't understand this result. Why isn't it slow when root
> > > > runs the script? He does the same relabeling to all those types.
> > > > It's not like user_u:user_r:user_t would be running in different
> > > > type concurrently. Just the fact that user_u is associated with
> > > > all those types seems to make it slow to run in any context
> > > > user_u:*
> > >
> > > Your result doesn't sound right. Wondering whether you are triggering
> > > masses of AVC denials (which could then peg syslog or audit) when
> > > running your script?
> >
> > Checked that of course. dmesg showed nothing or just occasional
> > denials. Just tryed again giving user_t full access to
> > everything. Changes nothing and dmesg is clear.
> >
> > Anyway turns out above I forgot to mention something that
> > actually is the core of the problem:
> >
> > I build a minimal example which is attached. Of course you have
> > at modify it to your policy.
> >
> > Basicly there is one role choke_r with one type choke_t und a
> > user choke_u with role choke_r. Then are 1000 other types in the
> > choke role each with a corresponding attribute.
> >
> > This alone isn't a problem. But if each of these attributes get
> > attributed to choke_t, the slowdown starts. Not as bad as
> > mentioned above but still significantly. (10sec in the test
> > above.)
> >
> > So we have choke_u:choke_r:choke_t. Although all other types are
> > in choke_r, that alone causes nothing. But as soon as there are
> > these 1000 attributes on choke_t the fun starts. Actually you
> > don't even need the 1000 types. Just 1000 attributes on one type
> > produces a measurable slowdown (7sec) and lags. Having 1000 types
> > beside just makes it a lot worse. But there absolutely no
> > slowdown with just 1000 types and no attributes.
>
> Interesting. We would expect some slowdown on an AVC cache miss in that
> situation, although the amount seems troubling. Things to consider:
> - Does the AVC cache need to be increased or otherwise tuned? You can
> see some information via the avcstat utility or by directly looking
> at /sys/fs/selinux/avc.
>
> - Does security_compute_av(), which is called on a cache miss, need some
> profiling and tuning? Particularly the logic within
> context_struct_compute_av(), where we are iterating through the type
> attribute ebitmaps.
>
> In Fedora 17, most types only have a few attributes (as shown by seinfo
> -t -x). unconfined_t has a larger number of attributes than most, but
> even it only has 46 attributes. So we likely don't see this behavior
> there.
>
> --
> Stephen Smalley
> National Security Agency
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 15:44 ` Ole Kliemann
@ 2012-08-10 16:08 ` Stephen Smalley
2012-08-10 16:18 ` Stephen Smalley
0 siblings, 1 reply; 27+ messages in thread
From: Stephen Smalley @ 2012-08-10 16:08 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Fri, 2012-08-10 at 17:44 +0200, Ole Kliemann wrote:
> PS: Have you actually reproduced this problem? Could still be
> something else broken on my system...
No, I haven't tried, as you didn't supply a complete policy.
Two other items to double check:
- Are you running auditd, and if so, did you check that you aren't
flooding it? That won't show up in dmesg, only
in /var/log/audit/audit.log.
- Are you running mcstrans? If so, disable it.
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 16:08 ` Stephen Smalley
@ 2012-08-10 16:18 ` Stephen Smalley
2012-08-10 17:00 ` Ole Kliemann
0 siblings, 1 reply; 27+ messages in thread
From: Stephen Smalley @ 2012-08-10 16:18 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Fri, 2012-08-10 at 12:08 -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 17:44 +0200, Ole Kliemann wrote:
> > PS: Have you actually reproduced this problem? Could still be
> > something else broken on my system...
>
> No, I haven't tried, as you didn't supply a complete policy.
>
> Two other items to double check:
> - Are you running auditd, and if so, did you check that you aren't
> flooding it? That won't show up in dmesg, only
> in /var/log/audit/audit.log.
>
> - Are you running mcstrans? If so, disable it.
Also, what does cat /sys/fs/selinux/policyvers show and what is the
version suffix on the policy file under /etc/selinux/.../policy? And
what is your kernel version?
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 16:18 ` Stephen Smalley
@ 2012-08-10 17:00 ` Ole Kliemann
2012-08-10 18:08 ` Stephen Smalley
0 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 17:00 UTC (permalink / raw)
To: Stephen Smalley; +Cc: selinux, Eric Paris
[-- Attachment #1.1: Type: text/plain, Size: 1227 bytes --]
On Fri, Aug 10, 2012 at 12:18:05PM -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 12:08 -0400, Stephen Smalley wrote:
> > On Fri, 2012-08-10 at 17:44 +0200, Ole Kliemann wrote:
> > > PS: Have you actually reproduced this problem? Could still be
> > > something else broken on my system...
> >
> > No, I haven't tried, as you didn't supply a complete policy.
> >
> > Two other items to double check:
> > - Are you running auditd, and if so, did you check that you aren't
> > flooding it? That won't show up in dmesg, only
> > in /var/log/audit/audit.log.
> >
> > - Are you running mcstrans? If so, disable it.
>
> Also, what does cat /sys/fs/selinux/policyvers show and what is the
> version suffix on the policy file under /etc/selinux/.../policy? And
> what is your kernel version?
I don't have an auditd, not running mcstransd and also had
disabled restorecond.
I take it, /sys/fs/selinux is equivalent to /selinux?
/sys/fs/selinux is empty on both my Ubuntu systems.
/selinux/policyver in 26 as is the suffix of the policy file.
Complete policy is attached. choke/src/support/choke.spt can be tuned
to suck even more. Do 'make load' in choke/src/ and you are good
to go.
[-- Attachment #1.2: choke.tar.bz2 --]
[-- Type: application/octet-stream, Size: 8877 bytes --]
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 17:00 ` Ole Kliemann
@ 2012-08-10 18:08 ` Stephen Smalley
2012-08-10 18:46 ` Ole Kliemann
0 siblings, 1 reply; 27+ messages in thread
From: Stephen Smalley @ 2012-08-10 18:08 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Fri, 2012-08-10 at 19:00 +0200, Ole Kliemann wrote:
> I don't have an auditd, not running mcstransd and also had
> disabled restorecond.
>
> I take it, /sys/fs/selinux is equivalent to /selinux?
Yes. /selinux moved to /sys/fs/selinux in more modern distro versions.
> /sys/fs/selinux is empty on both my Ubuntu systems.
>
> /selinux/policyver in 26 as is the suffix of the policy file.
>
> Complete policy is attached. choke/src/support/choke.spt can be tuned
> to suck even more. Do 'make load' in choke/src/ and you are good
> to go.
Ok, loaded. Now what exactly are you doing to test it?
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 18:08 ` Stephen Smalley
@ 2012-08-10 18:46 ` Ole Kliemann
2012-08-10 18:55 ` Stephen Smalley
0 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 18:46 UTC (permalink / raw)
To: Stephen Smalley; +Cc: selinux, Eric Paris
[-- Attachment #1.1: Type: text/plain, Size: 1215 bytes --]
On Fri, Aug 10, 2012 at 02:08:26PM -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 19:00 +0200, Ole Kliemann wrote:
> > I don't have an auditd, not running mcstransd and also had
> > disabled restorecond.
> >
> > I take it, /sys/fs/selinux is equivalent to /selinux?
>
> Yes. /selinux moved to /sys/fs/selinux in more modern distro versions.
>
> > /sys/fs/selinux is empty on both my Ubuntu systems.
> >
> > /selinux/policyver in 26 as is the suffix of the policy file.
> >
> > Complete policy is attached. choke/src/support/choke.spt can be tuned
> > to suck even more. Do 'make load' in choke/src/ and you are good
> > to go.
>
> Ok, loaded. Now what exactly are you doing to test it?
$ runcon choke_u:choke_r:choke_t ksh -l
$ id
Then witness the lag.
If you want hard numbers, use the attached script. First start
off in system_r:unconfined_r:unconfined_t. Run the script
somewhere, /tmp e.g. For proper average value computation you
need 'bc' installed, otherwise it's rounded but doesn't matter.
Then switch to choke_u:choke_r:choke_t. Run the script here. If
it's inconclusive, start uncommenting additional attributes in
choke/src/support/choke.spt.
[-- Attachment #1.2: x.sh --]
[-- Type: application/x-sh, Size: 1122 bytes --]
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 18:46 ` Ole Kliemann
@ 2012-08-10 18:55 ` Stephen Smalley
2012-08-10 19:11 ` Ole Kliemann
0 siblings, 1 reply; 27+ messages in thread
From: Stephen Smalley @ 2012-08-10 18:55 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Fri, 2012-08-10 at 20:46 +0200, Ole Kliemann wrote:
> On Fri, Aug 10, 2012 at 02:08:26PM -0400, Stephen Smalley wrote:
> > On Fri, 2012-08-10 at 19:00 +0200, Ole Kliemann wrote:
> > > I don't have an auditd, not running mcstransd and also had
> > > disabled restorecond.
> > >
> > > I take it, /sys/fs/selinux is equivalent to /selinux?
> >
> > Yes. /selinux moved to /sys/fs/selinux in more modern distro versions.
> >
> > > /sys/fs/selinux is empty on both my Ubuntu systems.
> > >
> > > /selinux/policyver in 26 as is the suffix of the policy file.
> > >
> > > Complete policy is attached. choke/src/support/choke.spt can be tuned
> > > to suck even more. Do 'make load' in choke/src/ and you are good
> > > to go.
> >
> > Ok, loaded. Now what exactly are you doing to test it?
>
> $ runcon choke_u:choke_r:choke_t ksh -l
> $ id
>
> Then witness the lag.
Not seeing it.
> If you want hard numbers, use the attached script. First start
> off in system_r:unconfined_r:unconfined_t. Run the script
> somewhere, /tmp e.g. For proper average value computation you
> need 'bc' installed, otherwise it's rounded but doesn't matter.
Triggers a ton of error messages in dmesg from SELinux about unmapped
security contexts?
> Then switch to choke_u:choke_r:choke_t. Run the script here. If
> it's inconclusive, start uncommenting additional attributes in
> choke/src/support/choke.spt.
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 18:55 ` Stephen Smalley
@ 2012-08-10 19:11 ` Ole Kliemann
2012-08-10 19:19 ` Stephen Smalley
0 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 19:11 UTC (permalink / raw)
To: Stephen Smalley; +Cc: selinux, Eric Paris
[-- Attachment #1.1: Type: text/plain, Size: 1679 bytes --]
On Fri, Aug 10, 2012 at 02:55:30PM -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 20:46 +0200, Ole Kliemann wrote:
> > On Fri, Aug 10, 2012 at 02:08:26PM -0400, Stephen Smalley wrote:
> > > On Fri, 2012-08-10 at 19:00 +0200, Ole Kliemann wrote:
> > > > I don't have an auditd, not running mcstransd and also had
> > > > disabled restorecond.
> > > >
> > > > I take it, /sys/fs/selinux is equivalent to /selinux?
> > >
> > > Yes. /selinux moved to /sys/fs/selinux in more modern distro versions.
> > >
> > > > /sys/fs/selinux is empty on both my Ubuntu systems.
> > > >
> > > > /selinux/policyver in 26 as is the suffix of the policy file.
> > > >
> > > > Complete policy is attached. choke/src/support/choke.spt can be tuned
> > > > to suck even more. Do 'make load' in choke/src/ and you are good
> > > > to go.
> > >
> > > Ok, loaded. Now what exactly are you doing to test it?
> >
> > $ runcon choke_u:choke_r:choke_t ksh -l
> > $ id
> >
> > Then witness the lag.
>
> Not seeing it.
>
> > If you want hard numbers, use the attached script. First start
> > off in system_r:unconfined_r:unconfined_t. Run the script
> > somewhere, /tmp e.g. For proper average value computation you
> > need 'bc' installed, otherwise it's rounded but doesn't matter.
>
> Triggers a ton of error messages in dmesg from SELinux about unmapped
> security contexts?
>
> > Then switch to choke_u:choke_r:choke_t. Run the script here. If
> > it's inconclusive, start uncommenting additional attributes in
> > choke/src/support/choke.spt.
Sorry, my mistake, got confused. Here's the right stuff now.
The script is in choke/test/
[-- Attachment #1.2: choke.tar.bz2 --]
[-- Type: application/octet-stream, Size: 9312 bytes --]
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 19:11 ` Ole Kliemann
@ 2012-08-10 19:19 ` Stephen Smalley
2012-08-10 19:26 ` Ole Kliemann
0 siblings, 1 reply; 27+ messages in thread
From: Stephen Smalley @ 2012-08-10 19:19 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Fri, 2012-08-10 at 21:11 +0200, Ole Kliemann wrote:
> On Fri, Aug 10, 2012 at 02:55:30PM -0400, Stephen Smalley wrote:
> > > If you want hard numbers, use the attached script. First start
> > > off in system_r:unconfined_r:unconfined_t. Run the script
> > > somewhere, /tmp e.g. For proper average value computation you
> > > need 'bc' installed, otherwise it's rounded but doesn't matter.
> >
> > Triggers a ton of error messages in dmesg from SELinux about unmapped
> > security contexts?
> >
> > > Then switch to choke_u:choke_r:choke_t. Run the script here. If
> > > it's inconclusive, start uncommenting additional attributes in
> > > choke/src/support/choke.spt.
>
> Sorry, my mistake, got confused. Here's the right stuff now.
> The script is in choke/test/
Well, that certainly yielded very different numbers but also lots of AVC
denials, all of which look like this:
time->Fri Aug 10 15:12:33 2012
type=SYSCALL msg=audit(1344625953.002:10135): arch=c000003e syscall=188
success=yes exit=0 a0=125a0e0 a1=311e81646b a2=125b5b0 a3=1d items=0
ppid=10903 pid=18574 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
sgid=0 fsgid=0 tty=pts0 ses=1 comm="chcon" exe="/usr/bin/chcon"
subj=choke_u:choke_r:choke_t key=(null)
type=AVC msg=audit(1344625953.002:10135): avc: denied { associate }
for pid=18574 comm="chcon" name="9448e490-297f-4856-8022-da19d91db9a4"
dev="dm-2" ino=1706648 scontext=choke_u:object_r:choke9x55_t
tcontext=system_u:object_r:unconfined_t tclass=filesystem
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 19:19 ` Stephen Smalley
@ 2012-08-10 19:26 ` Ole Kliemann
2012-08-10 19:50 ` Stephen Smalley
0 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 19:26 UTC (permalink / raw)
To: Stephen Smalley; +Cc: selinux, Eric Paris
[-- Attachment #1: Type: text/plain, Size: 1831 bytes --]
On Fri, Aug 10, 2012 at 03:19:48PM -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 21:11 +0200, Ole Kliemann wrote:
> > On Fri, Aug 10, 2012 at 02:55:30PM -0400, Stephen Smalley wrote:
> > > > If you want hard numbers, use the attached script. First start
> > > > off in system_r:unconfined_r:unconfined_t. Run the script
> > > > somewhere, /tmp e.g. For proper average value computation you
> > > > need 'bc' installed, otherwise it's rounded but doesn't matter.
> > >
> > > Triggers a ton of error messages in dmesg from SELinux about unmapped
> > > security contexts?
> > >
> > > > Then switch to choke_u:choke_r:choke_t. Run the script here. If
> > > > it's inconclusive, start uncommenting additional attributes in
> > > > choke/src/support/choke.spt.
> >
> > Sorry, my mistake, got confused. Here's the right stuff now.
> > The script is in choke/test/
>
> Well, that certainly yielded very different numbers but also lots of AVC
> denials, all of which look like this:
> time->Fri Aug 10 15:12:33 2012
> type=SYSCALL msg=audit(1344625953.002:10135): arch=c000003e syscall=188
> success=yes exit=0 a0=125a0e0 a1=311e81646b a2=125b5b0 a3=1d items=0
> ppid=10903 pid=18574 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
> sgid=0 fsgid=0 tty=pts0 ses=1 comm="chcon" exe="/usr/bin/chcon"
> subj=choke_u:choke_r:choke_t key=(null)
> type=AVC msg=audit(1344625953.002:10135): avc: denied { associate }
> for pid=18574 comm="chcon" name="9448e490-297f-4856-8022-da19d91db9a4"
> dev="dm-2" ino=1706648 scontext=choke_u:object_r:choke9x55_t
> tcontext=system_u:object_r:unconfined_t tclass=filesystem
Forgot to mention, I added that associate rule in the policy. You
have to use the one I sent last and rebuild.
But you'll see the AVC denials are not causing the slowdown.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 19:26 ` Ole Kliemann
@ 2012-08-10 19:50 ` Stephen Smalley
0 siblings, 0 replies; 27+ messages in thread
From: Stephen Smalley @ 2012-08-10 19:50 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Fri, 2012-08-10 at 21:26 +0200, Ole Kliemann wrote:
> On Fri, Aug 10, 2012 at 03:19:48PM -0400, Stephen Smalley wrote:
> > On Fri, 2012-08-10 at 21:11 +0200, Ole Kliemann wrote:
> > > On Fri, Aug 10, 2012 at 02:55:30PM -0400, Stephen Smalley wrote:
> > > > > If you want hard numbers, use the attached script. First start
> > > > > off in system_r:unconfined_r:unconfined_t. Run the script
> > > > > somewhere, /tmp e.g. For proper average value computation you
> > > > > need 'bc' installed, otherwise it's rounded but doesn't matter.
> > > >
> > > > Triggers a ton of error messages in dmesg from SELinux about unmapped
> > > > security contexts?
> > > >
> > > > > Then switch to choke_u:choke_r:choke_t. Run the script here. If
> > > > > it's inconclusive, start uncommenting additional attributes in
> > > > > choke/src/support/choke.spt.
> > >
> > > Sorry, my mistake, got confused. Here's the right stuff now.
> > > The script is in choke/test/
> >
> > Well, that certainly yielded very different numbers but also lots of AVC
> > denials, all of which look like this:
> > time->Fri Aug 10 15:12:33 2012
> > type=SYSCALL msg=audit(1344625953.002:10135): arch=c000003e syscall=188
> > success=yes exit=0 a0=125a0e0 a1=311e81646b a2=125b5b0 a3=1d items=0
> > ppid=10903 pid=18574 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
> > sgid=0 fsgid=0 tty=pts0 ses=1 comm="chcon" exe="/usr/bin/chcon"
> > subj=choke_u:choke_r:choke_t key=(null)
> > type=AVC msg=audit(1344625953.002:10135): avc: denied { associate }
> > for pid=18574 comm="chcon" name="9448e490-297f-4856-8022-da19d91db9a4"
> > dev="dm-2" ino=1706648 scontext=choke_u:object_r:choke9x55_t
> > tcontext=system_u:object_r:unconfined_t tclass=filesystem
>
> Forgot to mention, I added that associate rule in the policy. You
> have to use the one I sent last and rebuild.
>
> But you'll see the AVC denials are not causing the slowdown.
Ok, I can now reproduce without any AVC denials or other SELinux error
messages. On Fedora using your policy, after a complete filesystem
relabel.
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-07 13:02 SELinux performance depending on type count Ole Kliemann
` (3 preceding siblings ...)
2012-08-10 12:11 ` Ole Kliemann
@ 2012-08-10 21:38 ` Ole Kliemann
2012-08-13 12:35 ` Stephen Smalley
4 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-10 21:38 UTC (permalink / raw)
To: selinux; +Cc: Stephen Smalley
[-- Attachment #1.1: Type: text/plain, Size: 1714 bytes --]
On Tue, Aug 07, 2012 at 03:02:44PM +0200, Ole Kliemann wrote:
> And would it be better performance-wise to run a MCS-policy with
> say categories c0.cn than to have types c0_t, ... cn_t?
I can measure a performance impact here too using a test analog
to the one used to measure the attribute-spam.
With 10000 categories:
I reside in system_u:system_r:unconfined_t:s0.
I run the script. Average walltime is about 6.67sec.
$ runcon -l s0:c0.c9999
Now I'm system_u:system_r:unconfined_t:s0:c0.c9999.
I rerun the script. Average walltime is about 39sec.
Ouch! :-/
$ runcon -l s0:c0.c999
Now I'm system_u:system_r:unconfined_t:s0:c0.c999.
I rerun the script. Average walltime is about 7.89sec.
With 1000 categories:
system_u:system_r:unconfined_t:s0 6.53sec
system_u:system_r:unconfined_t:s0:c0.c9 6.63sec
system_u:system_r:unconfined_t:s0:c0.c99 6.73sec
system_u:system_r:unconfined_t:s0:c0.c999 7.89sec
That's almost 19% increase still at full range!
But several points:
It's different than with attribute-spam. There is no lag, no
CPU spikes in kworker threads. It's just a smooth increase in
runtime, even at 10k.
It only matters in what range you run. You seldom will be running
something in the full range. The results for c0.c9 will be the
most realistic for everyday usage. There is no big difference
measurable. (At this point variance comes into play, would need
bigger data base to say something.)
The test script is kind of far away from everyday usage.
So bottom line is: Unless one goes berserk with 10k running in
full range, this looks like no problem.
I attached both versions.
[-- Attachment #1.2: choke1000.tar.bz2 --]
[-- Type: application/octet-stream, Size: 9860 bytes --]
[-- Attachment #1.3: choke10000.tar.bz2 --]
[-- Type: application/octet-stream, Size: 19838 bytes --]
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-10 21:38 ` Ole Kliemann
@ 2012-08-13 12:35 ` Stephen Smalley
2012-08-27 15:28 ` Ole Kliemann
0 siblings, 1 reply; 27+ messages in thread
From: Stephen Smalley @ 2012-08-13 12:35 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Fri, 2012-08-10 at 23:38 +0200, Ole Kliemann wrote:
> On Tue, Aug 07, 2012 at 03:02:44PM +0200, Ole Kliemann wrote:
> > And would it be better performance-wise to run a MCS-policy with
> > say categories c0.cn than to have types c0_t, ... cn_t?
>
> I can measure a performance impact here too using a test analog
> to the one used to measure the attribute-spam.
>
> With 10000 categories:
>
> I reside in system_u:system_r:unconfined_t:s0.
> I run the script. Average walltime is about 6.67sec.
>
>
> $ runcon -l s0:c0.c9999
>
> Now I'm system_u:system_r:unconfined_t:s0:c0.c9999.
> I rerun the script. Average walltime is about 39sec.
>
> Ouch! :-/
>
>
> $ runcon -l s0:c0.c999
> Now I'm system_u:system_r:unconfined_t:s0:c0.c999.
> I rerun the script. Average walltime is about 7.89sec.
>
>
> With 1000 categories:
>
> system_u:system_r:unconfined_t:s0 6.53sec
> system_u:system_r:unconfined_t:s0:c0.c9 6.63sec
> system_u:system_r:unconfined_t:s0:c0.c99 6.73sec
> system_u:system_r:unconfined_t:s0:c0.c999 7.89sec
>
> That's almost 19% increase still at full range!
>
>
> But several points:
>
> It's different than with attribute-spam. There is no lag, no
> CPU spikes in kworker threads. It's just a smooth increase in
> runtime, even at 10k.
>
> It only matters in what range you run. You seldom will be running
> something in the full range. The results for c0.c9 will be the
> most realistic for everyday usage. There is no big difference
> measurable. (At this point variance comes into play, would need
> bigger data base to say something.)
>
> The test script is kind of far away from everyday usage.
>
>
> So bottom line is: Unless one goes berserk with 10k running in
> full range, this looks like no problem.
>
>
> I attached both versions.
I wonder how much of that time is spent on the chcon calls (i.e.
getxattr + setxattr) vs the actual accessing of the files.
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-13 12:35 ` Stephen Smalley
@ 2012-08-27 15:28 ` Ole Kliemann
2012-08-27 16:24 ` Stephen Smalley
0 siblings, 1 reply; 27+ messages in thread
From: Ole Kliemann @ 2012-08-27 15:28 UTC (permalink / raw)
To: Stephen Smalley; +Cc: selinux, Eric Paris
[-- Attachment #1.1: Type: text/plain, Size: 757 bytes --]
On Mon, Aug 13, 2012 at 08:35:29AM -0400, Stephen Smalley wrote:
> On Fri, 2012-08-10 at 23:38 +0200, Ole Kliemann wrote:
> > [...]
> >
> > $ runcon -l s0:c0.c9999
> >
> > Now I'm system_u:system_r:unconfined_t:s0:c0.c9999.
> > I rerun the script. Average walltime is about 39sec.
> >
> > Ouch! :-/
> >
> > [...]
>
> I wonder how much of that time is spent on the chcon calls (i.e.
> getxattr + setxattr) vs the actual accessing of the files.
I was away for some time, but in case you are still interested:
As can be seen with the attached script, the time it takes
accessing the files remains stable. Only creating and chcon'ing
files and dirs seems to be the problem.
So for all of my purposes I consider this a non-issue.
[-- Attachment #1.2: x.sh --]
[-- Type: application/x-sh, Size: 982 bytes --]
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: SELinux performance depending on type count
2012-08-27 15:28 ` Ole Kliemann
@ 2012-08-27 16:24 ` Stephen Smalley
0 siblings, 0 replies; 27+ messages in thread
From: Stephen Smalley @ 2012-08-27 16:24 UTC (permalink / raw)
To: Ole Kliemann; +Cc: selinux, Eric Paris
On Mon, 2012-08-27 at 17:28 +0200, Ole Kliemann wrote:
> On Mon, Aug 13, 2012 at 08:35:29AM -0400, Stephen Smalley wrote:
> > On Fri, 2012-08-10 at 23:38 +0200, Ole Kliemann wrote:
> > > [...]
> > >
> > > $ runcon -l s0:c0.c9999
> > >
> > > Now I'm system_u:system_r:unconfined_t:s0:c0.c9999.
> > > I rerun the script. Average walltime is about 39sec.
> > >
> > > Ouch! :-/
> > >
> > > [...]
> >
> > I wonder how much of that time is spent on the chcon calls (i.e.
> > getxattr + setxattr) vs the actual accessing of the files.
>
> I was away for some time, but in case you are still interested:
>
> As can be seen with the attached script, the time it takes
> accessing the files remains stable. Only creating and chcon'ing
> files and dirs seems to be the problem.
>
> So for all of my purposes I consider this a non-issue.
Yes, that makes more sense to me.
--
Stephen Smalley
National Security Agency
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2012-08-27 16:24 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-08-07 13:02 SELinux performance depending on type count Ole Kliemann
2012-08-07 14:12 ` David Quigley
2012-08-07 17:07 ` William Roberts
2012-08-07 17:36 ` Daniel J Walsh
2012-08-08 20:44 ` Ole Kliemann
2012-08-09 10:45 ` Adam Tkac
2012-08-09 11:56 ` Ole Kliemann
2012-08-10 12:11 ` Ole Kliemann
2012-08-10 13:00 ` Stephen Smalley
2012-08-10 14:36 ` Ole Kliemann
2012-08-10 15:05 ` Stephen Smalley
2012-08-10 15:43 ` Ole Kliemann
2012-08-10 15:44 ` Ole Kliemann
2012-08-10 16:08 ` Stephen Smalley
2012-08-10 16:18 ` Stephen Smalley
2012-08-10 17:00 ` Ole Kliemann
2012-08-10 18:08 ` Stephen Smalley
2012-08-10 18:46 ` Ole Kliemann
2012-08-10 18:55 ` Stephen Smalley
2012-08-10 19:11 ` Ole Kliemann
2012-08-10 19:19 ` Stephen Smalley
2012-08-10 19:26 ` Ole Kliemann
2012-08-10 19:50 ` Stephen Smalley
2012-08-10 21:38 ` Ole Kliemann
2012-08-13 12:35 ` Stephen Smalley
2012-08-27 15:28 ` Ole Kliemann
2012-08-27 16:24 ` Stephen Smalley
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.