Re: [LTP] network namespaces tests cleanup

Re: [LTP] network namespaces tests cleanup

[Cyril Hrubis]

Hi!
> As network namespaces tests (ltp/testcases/kernel/containers/netns) code is
> a bit messy and it's hard to figure out how return values are passed in the code
> I propose to remove all test cases and leave only the ones specified below.

Sounds good to me.

> Implementation:
> ===============
> As ip command cannot be used (because of older systems), we can use this generic
> functions to create/delete namespace handle:

Maybe we can do even better, make the testcases work both with unshare
and ip and run them twice (and get TCONF with ip on older systems) so
that we cover ip on newer systems as well.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] network namespaces tests cleanup

[Cyril Hrubis]

Hi!
> Alternatively, you could approach it a bit differently - test the
> actual kernel namespacing functionality using unshare/setns in
> *kernel*/containers/ and verify that iproute2 uses it in
> network/iproute/ (or perhaps commands/ip/ ?).

Right what needs to be tested is the kernel functionality and that the
tools call correct syscall with correct paramters, these are two
different things. But still as far as I can see the easiest way is to
run the whole test and switch between the tool it uses which is just a
few lines of code.

How do you plan to test that ip does what it should? Run it under strace
and grep for the syscall and check the parameters? That sound fragile to
me. Or is there a better way?

> The idea being that you don't exactly need to "brute force" test
> everything everytime - the kernel functionality doesn't change
> based on what utility uses the unshare() syscall.
> I (in ltp-unrelated activities) found out that this approach gives
> the best coverage / time / maintenance ratio.

Agree that the kernel functionality is the same. But how does having two
different testcases one for kernel functionality and one for ip decrease
maintenance ratio? This soulution would IMHO be more complicated. The
coverate should be same. The only aspect in which this solution is
likely better is test runtime, but that shouldn't be a problem because
these testcases does not take long time. Or am I mistaken?

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] network namespaces tests cleanup

[Cyril Hrubis]

Hi!
> $ ip netns add testns
> 
> and the following two have to return different IDs:
> 
> $ readlink /proc/self/ns/net
> net:[4026531969]
> 
> $ ip netns exec testns readlink /proc/self/ns/net
> net:[4026532368]
> 
> (And the kernel tests would ideally ensure that different procfs/ns
>  values *really* result in different namespaces based on tested
>  behavior.)

That sounds reasonable and easy enough.

> > Agree that the kernel functionality is the same. But how does having two
> > different testcases one for kernel functionality and one for ip decrease
> > maintenance ratio? This soulution would IMHO be more complicated. The
> > coverate should be same. The only aspect in which this solution is
> > likely better is test runtime, but that shouldn't be a problem because
> > these testcases does not take long time. Or am I mistaken?
> 
> Right now, they don't (IIRC), but will you document it? Will you watch
> for future tests added to this area, to make sure they don't take a long
> time because of this "hack"? I guess these are the maintenance costs
> I'm trying to point out.

This sounds reasonable as well.

> In the end, my suggestion was really just a suggestion - each approach
> has pros and cons and it's up to the author (and you, as an upstream
> maintainer) to pick the one you want.

And I'm certainly willing to listen to anybody with good ideas ;).

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] network namespaces tests cleanup

[Jiri Jaburek]

On 02/10/2015 04:17 PM, Matus Marhefka wrote:
> As network namespaces tests (ltp/testcases/kernel/containers/netns) code is
> a bit messy and it's hard to figure out how return values are passed in the code
> I propose to remove all test cases and leave only the ones specified below.

Just few notes about history / backwards compatibility; the proposed
replacements are not exactly true replacements for the original code
regarding kernel/userspace requirements.

2002: (pre-git) Linux namespaces support in the kernel
2007/09: kernel supports moving interfaces between network namespaces
2008/06: 'ip link .. netns <pid>' support in iproute2
2010/03: /proc/<pid>/ns/* support in the kernel
2011/05: /proc/<pid>/ns/* usable as file descriptors (mountable)
2011/07: 'ip netns' support in iproute2, incl. 'ip link .. netns <file>'

The point being how far are you willing to go to preserve the
functionality on older kernels/userspace.

The proposed code uses /proc/<pid>/ns/* as file descriptors, so it needs
at least 2.6.39 kernel and related iproute version or /usr/include
recent enough to have IFLA_NET_NS_FD (so you can write your own netlink
setup utility).

The original code uses only fork/clone as far as I see, so it goes much
more "back" in terms of compatibility, quite likely at the cost of
readability, though. The bash portions of the original code use iproute2
with 'netns <pid>' only, so their compatibility goes as back as 2008.

The ultimate question is therefore "are we willing to TCONF the new
tests on anything older than (upstream) 2.6.39?".

Some (most?) enterprise distributions may have the functionality
backported, for example RHEL-6.6 fully supports mounting
/proc/<pid>/ns/* and even has util-linux support (like nsenter(1)) for
the mounted file descriptors. It is, however, missing any 'ip netns'
support as well as 'ip link .. netns <file>' (supports only pid).

What about others?

# kernel fd (setns(2)) + util-linux support
touch netnstest
unshare --net mount --bind /proc/self/ns/net netnstest
nsenter --net=netnstest ip link show

# iproute pid support (possible bashisms follow)
ip link add dummy123 type dummy
nsenter --net=netnstest sleep 10 &
ip link set dummy123 netns $!
wait
nsenter --net=netnstest ip link show

# iproute fd support
ip link add dummy234 type dummy
ip link set dummy234 netns ./netnstest
nsenter --net=netnstest ip link show

# iproute 'ip netns' support
ip netns help

# cleanup
ip link del dummy123
ip link del dummy234
umount netnstest
rm -f netnstest


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] network namespaces tests cleanup

[Jiri Jaburek]

On 02/18/2015 05:54 PM, Jiri Jaburek wrote:
> 
> # iproute pid support (possible bashisms follow)
> ip link add dummy123 type dummy
> nsenter --net=netnstest sleep 10 &
> ip link set dummy123 netns $!
> wait
> nsenter --net=netnstest ip link show

Should be rather:

# iproute pid support (possible bashisms follow)
ip link add dummy123 type dummy
nsenter --net=netnstest sleep 10 &
sleep 1
ip link set dummy123 netns $!
wait
nsenter --net=netnstest ip link show

(or so, I forgot about the race between nsenter calling setns(2) and
 ip-link reading the namespace of the child)


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] network namespaces tests cleanup

[Cyril Hrubis]

Hi!
> > As network namespaces tests (ltp/testcases/kernel/containers/netns) code is
> > a bit messy and it's hard to figure out how return values are passed in the code
> > I propose to remove all test cases and leave only the ones specified below.
> 
> Just few notes about history / backwards compatibility; the proposed
> replacements are not exactly true replacements for the original code
> regarding kernel/userspace requirements.
> 
> 2002: (pre-git) Linux namespaces support in the kernel
> 2007/09: kernel supports moving interfaces between network namespaces
> 2008/06: 'ip link .. netns <pid>' support in iproute2
> 2010/03: /proc/<pid>/ns/* support in the kernel
> 2011/05: /proc/<pid>/ns/* usable as file descriptors (mountable)
> 2011/07: 'ip netns' support in iproute2, incl. 'ip link .. netns <file>'
> 
> The point being how far are you willing to go to preserve the
> functionality on older kernels/userspace.
> 
> The proposed code uses /proc/<pid>/ns/* as file descriptors, so it needs
> at least 2.6.39 kernel and related iproute version or /usr/include
> recent enough to have IFLA_NET_NS_FD (so you can write your own netlink
> setup utility).

Does not seem to be the case for SLES11 SP3 nor SP4. The kernel should
be new enough since SP2 though.

> The original code uses only fork/clone as far as I see, so it goes much
> more "back" in terms of compatibility, quite likely at the cost of
> readability, though. The bash portions of the original code use iproute2
> with 'netns <pid>' only, so their compatibility goes as back as 2008.
>
> The ultimate question is therefore "are we willing to TCONF the new
> tests on anything older than (upstream) 2.6.39?".

Well that may be one possibility. Unfortunatelly 2.6.39 is quite new, if
it were at least ten years old I would not hesitate. On the other hand
the old testcases are messy and basically unmaintainable and they tend
to fail if some subtle details in kernel change or even randomly from
time to time.

Ideal solution would be to fix the new testcases to work even on older
kernels but if that is too much work I would rather see us concentrate
on more recent code.

> Some (most?) enterprise distributions may have the functionality
> backported, for example RHEL-6.6 fully supports mounting
> /proc/<pid>/ns/* and even has util-linux support (like nsenter(1)) for
> the mounted file descriptors. It is, however, missing any 'ip netns'
> support as well as 'ip link .. netns <file>' (supports only pid).
> 
> What about others?
> 
> # kernel fd (setns(2)) + util-linux support
> touch netnstest
> unshare --net mount --bind /proc/self/ns/net netnstest
> nsenter --net=netnstest ip link show

SLES11 unshare does not support --bind and does not have nsenter likely
util-linux is too old (2.19.1).

> # iproute pid support (possible bashisms follow)
> ip link add dummy123 type dummy
> nsenter --net=netnstest sleep 10 &
> ip link set dummy123 netns $!
> wait
> nsenter --net=netnstest ip link show

Same here no nesenter.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] network namespaces tests cleanup

[Cyril Hrubis]

Hi!
> >> The proposed code uses /proc/<pid>/ns/* as file descriptors, so it needs
> >> at least 2.6.39 kernel and related iproute version or /usr/include
> >> recent enough to have IFLA_NET_NS_FD (so you can write your own netlink
> >> setup utility).
> > 
> > Does not seem to be the case for SLES11 SP3 nor SP4. The kernel should
> > be new enough since SP2 though.
> 
> Well, this would be a "nice to have", we could still use
> IFLA_NET_NS_PID, which you should have (also used by current test code).

This one is present.

> Your iproute could also have support for it,
>   $ ip link help 2>&1 | grep netns

                          [ netns PID ]
                          [ netns NAME ]

Seems to be the case as well.

> >> Some (most?) enterprise distributions may have the functionality
> >> backported, for example RHEL-6.6 fully supports mounting
> >> /proc/<pid>/ns/* and even has util-linux support (like nsenter(1)) for
> >> the mounted file descriptors. It is, however, missing any 'ip netns'
> >> support as well as 'ip link .. netns <file>' (supports only pid).
> >>
> >> What about others?
> >>
> >> # kernel fd (setns(2)) + util-linux support
> >> touch netnstest
> >> unshare --net mount --bind /proc/self/ns/net netnstest
> >> nsenter --net=netnstest ip link show
> > 
> > SLES11 unshare does not support --bind and does not have nsenter likely
> > util-linux is too old (2.19.1).
> 
> The --bind is not for unshare, but for mount(8). If you don't have
> unshare(1) from util-linux or have some old one (without netns support),
> try this as a quick replacement for creating the namespace, feel free
> to use mount(2) with MS_BIND (linux 2.4+) if your mount(8) doesn't
> support --bind:

Ah, right, mount supports bind without problem.

> #define _GNU_SOURCE
> #include <sched.h>
> #include <unistd.h>
> int main(int argc, char **argv)
> {
>     char *ma[] = {"mount","--bind","/proc/self/ns/net","netnstest",NULL};
>     unshare(CLONE_NEWNET);
>     execvp("mount", ma);
>     return 1;
> }

What I got:

mount("/proc/15413/ns/net", "netnstest", 0x410848, MS_MGC_VAL|MS_BIND, NULL) = -1 ENOTDIR (Not a directory)

ls -l /proc/self/ns/net
-r-------- 1 root root 0 Feb 25 13:23 /proc/self/ns/net

> and this as a replacement for nsenter:
> 
> #include <sched.h>
> #include <fcntl.h>
> #include <unistd.h>
> int main(int argc, char **argv)
> {
>     int i, fd;
> 
>     fd = open(argv[1], O_RDONLY);
>     /* provide NULL-terminated array for execve */
>     for (i = 1; i < argc-1; i++)
>         argv[i] = argv[i+1];
>     argv[i] = NULL;
> 
>     setns(fd, 0);
>     execvp(argv[1], argv+1);
>     return 1;
> }
> 
> (use as ./a.out netnstest <cmd> [args])
> 
> If it doesn't work, please try strace, doing proper error detection
> for everything above would double the necessary code.

This works after replacing setns(fd, 0) with the syscall(__NR_setns, fd, 0).

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] network namespaces tests cleanup

[Cyril Hrubis]

Hi!
> > What I got:
> > 
> > mount("/proc/15413/ns/net", "netnstest", 0x410848, MS_MGC_VAL|MS_BIND, NULL) = -1 ENOTDIR (Not a directory)
> 
> Right, "netnstest" already needs to exist as a *file*, not a directory
> and the second argument needs to be without a trailing '/' (which it
> is). Then it works for me. The posted C snippets are meant to fit in
> the example setup I posted earlier (just acting in place of unshare or
> nsenter).

My bad I wasn't reading the instruction carefully enough and did mkdir
instead of touch. Now it works fine for me.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list