Unit testing userspace

Unit testing userspace

[Joshua Brindle]

So, we have a few "unit" tests for parts of userspace. I use quotes because many 
of them aren't unit tests at all but tests wrapping around a giant black box 
masquerader as a unit test (e.g., all the linker/expander tests).

Some problems with the current tests are 1) they aren't run by default, 2) they 
use a unit test framework now in yum, 3) they have interdependencies within the 
tree and 4) a lot of the things in the repo are really hard to unit test.

We'd like to spend some cycles here doing unit tests but we need to address the 
above problems first.

The requirements I'd throw out are:

1) Each component (libsepol, libselinux, etc) must be testable outside of the 
main repo (so that tests can be run from rpm, portage, etc)
2) Must not require anything outside of yum
3) Each test should test a single unit (see 
<http://www.javaranch.com/unit-testing.jsp> for an explanation)
4) Must be run in the default build target
5) Be useful tests

For unit tests to be useful they must get run, unfortunately Fedora doesn't seem 
to like to package unit test frameworks so we'd need to put one in the tree and 
use it directly.

We could:

- write our own and put it in the repo
- use a simple single .c file solution like CuTest
- try to get one packaged up in Fedora
- use something more powerful and add it to our repo

All varying levels of bad.

Further, to meet requirement #1 the test framework would have to be duplicated 
once per framework.

Now, to throw more wrenches in the machinery, a huge amount of the libraries 
(particularly libselinux) aren't directly unit testable. They'd need to use a 
framework that can mock functions (eg., to test is_selinux_enabled() you'd need 
to mock the fopen() and read() functions). This makes the more simple solutions 
not as useful.

I've recently worked with a framework called test-dept 
<http://code.google.com/p/test-dept/> that can do mocking and assertion testing. 
It is a pain to use and requires a bunch of headers, Makefiles and a program to 
manipulate the object files to run.

After thinking about this quite a bit I don't know if we can have reasonable 
test coverage without using something capable of mocking. I've also looked at 
other frameworks that provide mocking and they aren't suitable for a variety of 
reasons.

So... this is starting to sound like rambling but the only thing reasonable I 
can come up with (and reasonable is a stretch) is to remove all the CUnit tests, 
start doing unit tests with test-dept, checked in to the test directory of each 
component. We could start with simple tests and eventually graduate to mock 
tests. The reason I'd rather do this than start with something like CuTest and 
move to test-dept when we need its capabilities is because I did that very thing 
on a recent project and the two systems are very incompatible and we now have 2 
completely separate sets of tests and difficulty finding out what is tested and 
where and by what.


Opinions? Compromise? Should we just give up on unit testing altogether?

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Unit testing userspace

[Chad Sellers]

On 12/13/10 2:59 PM, "Joshua Brindle" <method@manicmethod.com> wrote:

> So, we have a few "unit" tests for parts of userspace. I use quotes because
> many 
> of them aren't unit tests at all but tests wrapping around a giant black box
> masquerader as a unit test (e.g., all the linker/expander tests).
> 
> Some problems with the current tests are 1) they aren't run by default, 2)
> they 
> use a unit test framework now in yum, 3) they have interdependencies within
> the 
> tree and 4) a lot of the things in the repo are really hard to unit test.
> 
> We'd like to spend some cycles here doing unit tests but we need to address
> the 
> above problems first.
> 
> The requirements I'd throw out are:
> 
> 1) Each component (libsepol, libselinux, etc) must be testable outside of the
> main repo (so that tests can be run from rpm, portage, etc)
> 2) Must not require anything outside of yum
> 3) Each test should test a single unit (see
> <http://www.javaranch.com/unit-testing.jsp> for an explanation)
> 4) Must be run in the default build target
> 5) Be useful tests
> 
> For unit tests to be useful they must get run, unfortunately Fedora doesn't
> seem 
> to like to package unit test frameworks so we'd need to put one in the tree
> and 
> use it directly.
> 
> We could:
> 
> - write our own and put it in the repo
> - use a simple single .c file solution like CuTest
> - try to get one packaged up in Fedora
> - use something more powerful and add it to our repo
> 
> All varying levels of bad.
> 
> Further, to meet requirement #1 the test framework would have to be duplicated
> once per framework.
> 
> Now, to throw more wrenches in the machinery, a huge amount of the libraries
> (particularly libselinux) aren't directly unit testable. They'd need to use a
> framework that can mock functions (eg., to test is_selinux_enabled() you'd
> need 
> to mock the fopen() and read() functions). This makes the more simple
> solutions 
> not as useful.
> 
> I've recently worked with a framework called test-dept
> <http://code.google.com/p/test-dept/> that can do mocking and assertion
> testing. 
> It is a pain to use and requires a bunch of headers, Makefiles and a program
> to 
> manipulate the object files to run.
> 
> After thinking about this quite a bit I don't know if we can have reasonable
> test coverage without using something capable of mocking. I've also looked at
> other frameworks that provide mocking and they aren't suitable for a variety
> of 
> reasons.
> 
> So... this is starting to sound like rambling but the only thing reasonable I
> can come up with (and reasonable is a stretch) is to remove all the CUnit
> tests, 
> start doing unit tests with test-dept, checked in to the test directory of
> each 
> component. We could start with simple tests and eventually graduate to mock
> tests. The reason I'd rather do this than start with something like CuTest and
> move to test-dept when we need its capabilities is because I did that very
> thing 
> on a recent project and the two systems are very incompatible and we now have
> 2 
> completely separate sets of tests and difficulty finding out what is tested
> and 
> where and by what.
> 
> 
> Opinions? Compromise? Should we just give up on unit testing altogether?

My vote is to go the CuTest route. My primary reason is that writing CuTest
tests is easy. We've been really bad about creating unit tests so far, and I
doubt that will change. Something as painful to use as test-dept will just
encourage developers to not write unit tests. I realize that this will mean
we will never get near 100% test coverage for things like libselinux due to
the lack of mocking support, but I think it's the lesser of two evils. Given
our lack of diligence in this area historically, I'd rather make things easy
to do than shoot for full coverage.

Note that we've been using CuTest in CIL, and it is very simple and easy to
use. There's almost no spin up to using it.

Chad
 


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Unit testing userspace

[Jeff Johnson]

On Dec 14, 2010, at 2:07 PM, Chad Sellers wrote:
> 
> Note that we've been using CuTest in CIL, and it is very simple and easy to
> use. There's almost no spin up to using it.
> 

Try the ISPRAS api-sanity-autotest.pl (same people doing LSB testing).

There is zero spin-up to shallow testing using ISPRAS, and some modest thought
writing some XML templates can go a long ways towards reasonable
unit testing with significant coverage.

OTOH, if you truly need mocking with symbol replacemnet, well ... good luck! That's hard.

hth

733


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Unit testing userspace

[Joshua Brindle]

Jeff Johnson wrote:
> On Dec 14, 2010, at 2:07 PM, Chad Sellers wrote:
>> Note that we've been using CuTest in CIL, and it is very simple and easy to
>> use. There's almost no spin up to using it.
>>
>
> Try the ISPRAS api-sanity-autotest.pl (same people doing LSB testing).

If CIL is already using cutest that would be helpful. AFAIK Perl isn't installed 
by default on fedora or gentoo, and we'd want to have the tests run in the 
default make target in a clean build environment (for RPM builds and such). FWIW 
I discounted other test frameworks outright because they use ruby or other 
non-default packages.

>
> There is zero spin-up to shallow testing using ISPRAS, and some modest thought
> writing some XML templates can go a long ways towards reasonable
> unit testing with significant coverage.
>
> OTOH, if you truly need mocking with symbol replacemnet, well ... good luck! That's hard.
>

test-dept does mocking with symbol replacement and all that. Yes it is hard (not 
nearly as hard as doing it without something like test-dept) but Chad may be 
right, it could be just too hard to encourage unit tests to be written.

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Unit testing userspace

[Michal Svoboda]

[-- Attachment #1: Type: text/plain, Size: 1485 bytes --]

Joshua Brindle wrote:
> to test is_selinux_enabled() you'd need to mock the fopen() and read()
> functions

You need to realize how much important mocking is. It's not about
mocking the system calls. Instead, it should be used rigorously in all
the tests.

Example: IMO the problem with is_selinux_enabled is that it tries to do
too much in one block of code. To effectively test all the paths through
that function you would get crazy in simulating the right conditions.

Now consider this variant instead:

int is_selinux_enabled(void) {
    if (is_selinux_filesystem_mounted()) {
        return !no_policy_loaded();
    }
    else {
        return 0;
    }
}

int is_selinux_filesystem_mounted() {
    return selinux_mnt || filesystem_detected_the_long_way();
}

int no_policy_loaded() {
    ... the stuff with getcon (notice it is currently duplicated)
}

...

Now each of those isn't too hard to test given you can mock the result
of the individual sub-functions! (And the logical or one liner is so
brutally simple & obvious that it even does not need any testing.)

Given this I would seriously consider a framework that gives this
ability, otherwise you are stuck with doing it manually through
#define's and that would seriously discourage people to unit-test. They
will black-box-test or integration-test instead which will not cover all
the nuances of the function logic and is likely to miss half of the
corner cases, improper assumptions, etc., ie. bugs.

Michal Svoboda


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]