Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

I have encountered a system hang with my Smack
networking tests that bisects to the change below.
I can't say that I have any idea why the change
would impact the Smack processing, but there appears
to be some serious packet processing going on. The
Smack code is using CIPSO on the loopback interface.
The test is supposed to verify that labels can be
set on the packets using CIPSO. Unlabeled packets
do not appear to be impacted. I do not know if SELinux
is affected, and if not, why not. Smack and SELinux
use CIPSO differently.


c3f1010b30f7fc611139cfb702a8685741aa6827

commit c3f1010b30f7fc611139cfb702a8685741aa6827
Merge: ca4aa97 0b922b7
Author: David S. Miller <davem@davemloft.net>
Date:   Wed May 11 19:31:40 2016 -0400

    Merge branch 'vrf-pktinfo'
    
    David Ahern says:
    
    ====================
    net: vrf: Fixup PKTINFO to return enslaved device index
    
    Applications such as OSPF and BFD need the original ingress device not
    the VRF device; the latter can be derived from the former. To that end
    move the packet intercept from an rx handler that is invoked by
    __netif_receive_skb_core to the ipv4 and ipv6 receive processing.
    
    IPv6 already saves the skb_iif to the control buffer in ipv6_rcv. Since
    the skb->dev has not been switched the cb has the enslaved device. Make
    the same happen for IPv4 by adding the skb_iif to inet_skb_parm and set
    it in ipv4 code after clearing the skb control buffer similar to IPv6.
    From there the pktinfo can just pull it from cb with the PKTINFO_SKB_CB
    cast.
    ====================
    
    Signed-off-by: David S. Miller <davem@davemloft.net>

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[David Ahern]

On 7/5/16 5:38 PM, Casey Schaufler wrote:
> I have encountered a system hang with my Smack
> networking tests that bisects to the change below.
> I can't say that I have any idea why the change
> would impact the Smack processing, but there appears
> to be some serious packet processing going on. The
> Smack code is using CIPSO on the loopback interface.
> The test is supposed to verify that labels can be
> set on the packets using CIPSO. Unlabeled packets
> do not appear to be impacted. I do not know if SELinux
> is affected, and if not, why not. Smack and SELinux
> use CIPSO differently.

What are the commands to repeat the test?

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

On 7/5/2016 5:49 PM, David Ahern wrote:
> On 7/5/16 5:38 PM, Casey Schaufler wrote:
>> I have encountered a system hang with my Smack
>> networking tests that bisects to the change below.
>> I can't say that I have any idea why the change
>> would impact the Smack processing, but there appears
>> to be some serious packet processing going on. The
>> Smack code is using CIPSO on the loopback interface.
>> The test is supposed to verify that labels can be
>> set on the packets using CIPSO. Unlabeled packets
>> do not appear to be impacted. I do not know if SELinux
>> is affected, and if not, why not. Smack and SELinux
>> use CIPSO differently.
>
> What are the commands to repeat the test?
>
There is a tar file attached with the tests.
Put the etc/smack/user file into /etc/smack/user.
In the tools-2012 directory run make to build
the tools. The test in question is called
testnetworking.sh and needs to be run as root.
You will need to configure Smack in the kernel,
of course.


[-- Attachment #2: smack-tests.tar --]
[-- Type: application/octet-stream, Size: 102400 bytes --]

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Paul Moore]

On Tue, Jul 5, 2016 at 8:38 PM, Casey Schaufler <casey@schaufler-ca.com> wrote:
> I have encountered a system hang with my Smack
> networking tests that bisects to the change below.
> I can't say that I have any idea why the change
> would impact the Smack processing, but there appears
> to be some serious packet processing going on. The
> Smack code is using CIPSO on the loopback interface.
> The test is supposed to verify that labels can be
> set on the packets using CIPSO. Unlabeled packets
> do not appear to be impacted. I do not know if SELinux
> is affected, and if not, why not. Smack and SELinux
> use CIPSO differently.

For the past several months I've been running the SELinux testsuite on
a weekly basis against Linus' kernel plus the SELinux and audit
development trees and I haven't noticed any problems that haven't
already been reported.  While not exhaustive, the testsuite does
exercise the NetLabel/CIPSO code.  I'll see if I can take a closer
look at the Smack code, but do you rely on the inet_skb_param values
in Smack?  We did have a similar problem in the NetLabel core code
that we fixed with 04f81f0154e4bf002be6f4d85668ce1257efa4d9; it's
possible there is a similar problem in code that we just aren't
exercising with SELinux at the moment.

* https://github.com/SELinuxProject/selinux-testsuite
* https://copr.fedorainfracloud.org/coprs/pcmoore/kernel-secnext

> commit c3f1010b30f7fc611139cfb702a8685741aa6827
> Merge: ca4aa97 0b922b7
> Author: David S. Miller <davem@davemloft.net>
> Date:   Wed May 11 19:31:40 2016 -0400
>
>     Merge branch 'vrf-pktinfo'
>
>     David Ahern says:
>
>     ====================
>     net: vrf: Fixup PKTINFO to return enslaved device index
>
>     Applications such as OSPF and BFD need the original ingress device not
>     the VRF device; the latter can be derived from the former. To that end
>     move the packet intercept from an rx handler that is invoked by
>     __netif_receive_skb_core to the ipv4 and ipv6 receive processing.
>
>     IPv6 already saves the skb_iif to the control buffer in ipv6_rcv. Since
>     the skb->dev has not been switched the cb has the enslaved device. Make
>     the same happen for IPv4 by adding the skb_iif to inet_skb_parm and set
>     it in ipv4 code after clearing the skb control buffer similar to IPv6.
>     From there the pktinfo can just pull it from cb with the PKTINFO_SKB_CB
>     cast.
>     ====================
>
>     Signed-off-by: David S. Miller <davem@davemloft.net>

-- 
paul moore
www.paul-moore.com

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Paul Moore]

On Wed, Jul 6, 2016 at 8:50 AM, Paul Moore <paul@paul-moore.com> wrote:
> On Tue, Jul 5, 2016 at 8:38 PM, Casey Schaufler <casey@schaufler-ca.com> wrote:
>> I have encountered a system hang with my Smack
>> networking tests that bisects to the change below.
>> I can't say that I have any idea why the change
>> would impact the Smack processing, but there appears
>> to be some serious packet processing going on. The
>> Smack code is using CIPSO on the loopback interface.
>> The test is supposed to verify that labels can be
>> set on the packets using CIPSO. Unlabeled packets
>> do not appear to be impacted. I do not know if SELinux
>> is affected, and if not, why not. Smack and SELinux
>> use CIPSO differently.
>
> For the past several months I've been running the SELinux testsuite on
> a weekly basis against Linus' kernel plus the SELinux and audit
> development trees and I haven't noticed any problems that haven't
> already been reported.  While not exhaustive, the testsuite does
> exercise the NetLabel/CIPSO code.  I'll see if I can take a closer
> look at the Smack code, but do you rely on the inet_skb_param values
> in Smack?  We did have a similar problem in the NetLabel core code
> that we fixed with 04f81f0154e4bf002be6f4d85668ce1257efa4d9; it's
> possible there is a similar problem in code that we just aren't
> exercising with SELinux at the moment.
>
> * https://github.com/SELinuxProject/selinux-testsuite
> * https://copr.fedorainfracloud.org/coprs/pcmoore/kernel-secnext

I just ran some additional SELinux/NetLabel tests on 4.7-rc6
(+selinux#next +audit#next) and everything worked as expected.  This
is looking more and more like a Smack specific bug.  I took a quick
look at the NetLabel/CIPSO code as well as some of the Smack code and
nothing jumped out at me as obviously wrong, but it has been years
since I've looked at the Smack code in any detail.

I'm happy to help debug, but I think it might be more helpful if you
take a closer look at the Smack code first.

-- 
paul moore
www.paul-moore.com

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

On 7/6/2016 5:50 AM, Paul Moore wrote:
> On Tue, Jul 5, 2016 at 8:38 PM, Casey Schaufler <casey@schaufler-ca.com> wrote:
>> I have encountered a system hang with my Smack
>> networking tests that bisects to the change below.
>> I can't say that I have any idea why the change
>> would impact the Smack processing, but there appears
>> to be some serious packet processing going on. The
>> Smack code is using CIPSO on the loopback interface.
>> The test is supposed to verify that labels can be
>> set on the packets using CIPSO. Unlabeled packets
>> do not appear to be impacted. I do not know if SELinux
>> is affected, and if not, why not. Smack and SELinux
>> use CIPSO differently.
> For the past several months I've been running the SELinux testsuite on
> a weekly basis against Linus' kernel plus the SELinux and audit
> development trees and I haven't noticed any problems that haven't
> already been reported.  While not exhaustive, the testsuite does
> exercise the NetLabel/CIPSO code.  I'll see if I can take a closer
> look at the Smack code, but do you rely on the inet_skb_param values
> in Smack?  We did have a similar problem in the NetLabel core code
> that we fixed with 04f81f0154e4bf002be6f4d85668ce1257efa4d9; it's
> possible there is a similar problem in code that we just aren't
> exercising with SELinux at the moment.

I reported that problem, and that problem was fixed.

I'm looking at the Linus tree and see no structure inet_skb_param.

> * https://github.com/SELinuxProject/selinux-testsuite
> * https://copr.fedorainfracloud.org/coprs/pcmoore/kernel-secnext
>
>> commit c3f1010b30f7fc611139cfb702a8685741aa6827
>> Merge: ca4aa97 0b922b7
>> Author: David S. Miller <davem@davemloft.net>
>> Date:   Wed May 11 19:31:40 2016 -0400
>>
>>     Merge branch 'vrf-pktinfo'
>>
>>     David Ahern says:
>>
>>     ====================
>>     net: vrf: Fixup PKTINFO to return enslaved device index
>>
>>     Applications such as OSPF and BFD need the original ingress device not
>>     the VRF device; the latter can be derived from the former. To that end
>>     move the packet intercept from an rx handler that is invoked by
>>     __netif_receive_skb_core to the ipv4 and ipv6 receive processing.
>>
>>     IPv6 already saves the skb_iif to the control buffer in ipv6_rcv. Since
>>     the skb->dev has not been switched the cb has the enslaved device. Make
>>     the same happen for IPv4 by adding the skb_iif to inet_skb_parm and set
>>     it in ipv4 code after clearing the skb control buffer similar to IPv6.
>>     From there the pktinfo can just pull it from cb with the PKTINFO_SKB_CB
>>     cast.
>>     ====================
>>
>>     Signed-off-by: David S. Miller <davem@davemloft.net>

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Paul Moore]

On Wed, Jul 6, 2016 at 10:15 AM, Casey Schaufler <casey@schaufler-ca.com> wrote:
> On 7/6/2016 5:50 AM, Paul Moore wrote:
>> On Tue, Jul 5, 2016 at 8:38 PM, Casey Schaufler <casey@schaufler-ca.com> wrote:
>>> I have encountered a system hang with my Smack
>>> networking tests that bisects to the change below.
>>> I can't say that I have any idea why the change
>>> would impact the Smack processing, but there appears
>>> to be some serious packet processing going on. The
>>> Smack code is using CIPSO on the loopback interface.
>>> The test is supposed to verify that labels can be
>>> set on the packets using CIPSO. Unlabeled packets
>>> do not appear to be impacted. I do not know if SELinux
>>> is affected, and if not, why not. Smack and SELinux
>>> use CIPSO differently.
>> For the past several months I've been running the SELinux testsuite on
>> a weekly basis against Linus' kernel plus the SELinux and audit
>> development trees and I haven't noticed any problems that haven't
>> already been reported.  While not exhaustive, the testsuite does
>> exercise the NetLabel/CIPSO code.  I'll see if I can take a closer
>> look at the Smack code, but do you rely on the inet_skb_param values
>> in Smack?  We did have a similar problem in the NetLabel core code
>> that we fixed with 04f81f0154e4bf002be6f4d85668ce1257efa4d9; it's
>> possible there is a similar problem in code that we just aren't
>> exercising with SELinux at the moment.
>
> I reported that problem, and that problem was fixed.

Yep, I know, that is why I included the commit ID.

> I'm looking at the Linus tree and see no structure inet_skb_param.

I mistakenly added an extra "a", look for "struct inet_skb_parm" near
the top of include/net/ip.h; it was modified as part of the
pull-request/merge you mentioned.

-- 
paul moore
www.paul-moore.com

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

On 7/6/2016 7:03 AM, Paul Moore wrote:
> On Wed, Jul 6, 2016 at 8:50 AM, Paul Moore <paul@paul-moore.com> wrote:
>> On Tue, Jul 5, 2016 at 8:38 PM, Casey Schaufler <casey@schaufler-ca.com> wrote:
>>> I have encountered a system hang with my Smack
>>> networking tests that bisects to the change below.
>>> I can't say that I have any idea why the change
>>> would impact the Smack processing, but there appears
>>> to be some serious packet processing going on. The
>>> Smack code is using CIPSO on the loopback interface.
>>> The test is supposed to verify that labels can be
>>> set on the packets using CIPSO. Unlabeled packets
>>> do not appear to be impacted. I do not know if SELinux
>>> is affected, and if not, why not. Smack and SELinux
>>> use CIPSO differently.
>> For the past several months I've been running the SELinux testsuite on
>> a weekly basis against Linus' kernel plus the SELinux and audit
>> development trees and I haven't noticed any problems that haven't
>> already been reported.  While not exhaustive, the testsuite does
>> exercise the NetLabel/CIPSO code.  I'll see if I can take a closer
>> look at the Smack code, but do you rely on the inet_skb_param values
>> in Smack?  We did have a similar problem in the NetLabel core code
>> that we fixed with 04f81f0154e4bf002be6f4d85668ce1257efa4d9; it's
>> possible there is a similar problem in code that we just aren't
>> exercising with SELinux at the moment.
>>
>> * https://github.com/SELinuxProject/selinux-testsuite
>> * https://copr.fedorainfracloud.org/coprs/pcmoore/kernel-secnext
> I just ran some additional SELinux/NetLabel tests on 4.7-rc6
> (+selinux#next +audit#next) and everything worked as expected.  This
> is looking more and more like a Smack specific bug.

That would not surprise me at all. Smack uses CIPSO more
extensively than SELinux.

>   I took a quick
> look at the NetLabel/CIPSO code as well as some of the Smack code and
> nothing jumped out at me as obviously wrong, but it has been years
> since I've looked at the Smack code in any detail.
>
> I'm happy to help debug, but I think it might be more helpful if you
> take a closer look at the Smack code first.

That was where my long weekend was spent. Except for a trivial
change in the signal delivery code, there haven't been changes
in Smack recently.

That's why I took the bisect route to try to find the problem.

I am perfectly willing to accept that there's something that
I'm doing wrong (I often do things wrong). In this case, it's
got to be something that I've been getting away with for quite
some time.

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[David Ahern]

On 7/5/16 6:31 PM, Casey Schaufler wrote:
> On 7/5/2016 5:49 PM, David Ahern wrote:
>> On 7/5/16 5:38 PM, Casey Schaufler wrote:
>>> I have encountered a system hang with my Smack
>>> networking tests that bisects to the change below.
>>> I can't say that I have any idea why the change
>>> would impact the Smack processing, but there appears
>>> to be some serious packet processing going on. The
>>> Smack code is using CIPSO on the loopback interface.
>>> The test is supposed to verify that labels can be
>>> set on the packets using CIPSO. Unlabeled packets
>>> do not appear to be impacted. I do not know if SELinux
>>> is affected, and if not, why not. Smack and SELinux
>>> use CIPSO differently.
>>
>> What are the commands to repeat the test?
>>
> There is a tar file attached with the tests.
> Put the etc/smack/user file into /etc/smack/user.
> In the tools-2012 directory run make to build
> the tools. The test in question is called
> testnetworking.sh and needs to be run as root.
> You will need to configure Smack in the kernel,
> of course.
>

I understand Paul's point and glad to see the changes are not causing 
issues with his SELinux testing.

I have tried to reproduce the lockup you are seeing but not successful. 
I am using a jessie based VM with latest net-next kernel.

root@kenny-jessie3:~/smack/tools-2012# zcat /proc/config.gz | grep SMACK
CONFIG_SECURITY_SMACK=y
CONFIG_SECURITY_SMACK_BRINGUP=y
CONFIG_SECURITY_SMACK_NETFILTER=y
# CONFIG_DEFAULT_SECURITY_SMACK is not set

root@kenny-jessie3:~/smack/tools-2012# bash -x ./testnetworking.sh
<no lockup>

Send me your kernel config. Perhaps I do not have some config enabled.

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

On 7/6/2016 9:28 AM, David Ahern wrote:
> On 7/5/16 6:31 PM, Casey Schaufler wrote:
>> On 7/5/2016 5:49 PM, David Ahern wrote:
>>> On 7/5/16 5:38 PM, Casey Schaufler wrote:
>>>> I have encountered a system hang with my Smack
>>>> networking tests that bisects to the change below.
>>>> I can't say that I have any idea why the change
>>>> would impact the Smack processing, but there appears
>>>> to be some serious packet processing going on. The
>>>> Smack code is using CIPSO on the loopback interface.
>>>> The test is supposed to verify that labels can be
>>>> set on the packets using CIPSO. Unlabeled packets
>>>> do not appear to be impacted. I do not know if SELinux
>>>> is affected, and if not, why not. Smack and SELinux
>>>> use CIPSO differently.
>>>
>>> What are the commands to repeat the test?
>>>
>> There is a tar file attached with the tests.
>> Put the etc/smack/user file into /etc/smack/user.
>> In the tools-2012 directory run make to build
>> the tools. The test in question is called
>> testnetworking.sh and needs to be run as root.
>> You will need to configure Smack in the kernel,
>> of course.
>>
>
> I understand Paul's point and glad to see the changes are not causing issues with his SELinux testing.
>
> I have tried to reproduce the lockup you are seeing but not successful. I am using a jessie based VM with latest net-next kernel.
>
> root@kenny-jessie3:~/smack/tools-2012# zcat /proc/config.gz | grep SMACK
> CONFIG_SECURITY_SMACK=y
> CONFIG_SECURITY_SMACK_BRINGUP=y
> CONFIG_SECURITY_SMACK_NETFILTER=y
> # CONFIG_DEFAULT_SECURITY_SMACK is not set

CONFIG_DEFAULT_SECURITY_SMACK needs to be set.

>
> root@kenny-jessie3:~/smack/tools-2012# bash -x ./testnetworking.sh
> <no lockup>
>
> Send me your kernel config. Perhaps I do not have some config enabled.
>

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[David Ahern]

On 7/6/16 10:24 AM, Casey Schaufler wrote:
> On 7/6/2016 9:28 AM, David Ahern wrote:
>> On 7/5/16 6:31 PM, Casey Schaufler wrote:
>>> On 7/5/2016 5:49 PM, David Ahern wrote:
>>>> On 7/5/16 5:38 PM, Casey Schaufler wrote:
>>>>> I have encountered a system hang with my Smack
>>>>> networking tests that bisects to the change below.
>>>>> I can't say that I have any idea why the change
>>>>> would impact the Smack processing, but there appears
>>>>> to be some serious packet processing going on. The
>>>>> Smack code is using CIPSO on the loopback interface.
>>>>> The test is supposed to verify that labels can be
>>>>> set on the packets using CIPSO. Unlabeled packets
>>>>> do not appear to be impacted. I do not know if SELinux
>>>>> is affected, and if not, why not. Smack and SELinux
>>>>> use CIPSO differently.
>>>>
>>>> What are the commands to repeat the test?
>>>>
>>> There is a tar file attached with the tests.
>>> Put the etc/smack/user file into /etc/smack/user.
>>> In the tools-2012 directory run make to build
>>> the tools. The test in question is called
>>> testnetworking.sh and needs to be run as root.
>>> You will need to configure Smack in the kernel,
>>> of course.
>>>
>>
>> I understand Paul's point and glad to see the changes are not causing issues with his SELinux testing.
>>
>> I have tried to reproduce the lockup you are seeing but not successful. I am using a jessie based VM with latest net-next kernel.
>>
>> root@kenny-jessie3:~/smack/tools-2012# zcat /proc/config.gz | grep SMACK
>> CONFIG_SECURITY_SMACK=y
>> CONFIG_SECURITY_SMACK_BRINGUP=y
>> CONFIG_SECURITY_SMACK_NETFILTER=y
>> # CONFIG_DEFAULT_SECURITY_SMACK is not set
>
> CONFIG_DEFAULT_SECURITY_SMACK needs to be set.

ok. That caused more interesting output but still no hang after all of 
the snap-crackle-and-pops.

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

On 7/6/2016 10:40 AM, David Ahern wrote:
> On 7/6/16 10:24 AM, Casey Schaufler wrote:
>> On 7/6/2016 9:28 AM, David Ahern wrote:
>>> On 7/5/16 6:31 PM, Casey Schaufler wrote:
>>>> On 7/5/2016 5:49 PM, David Ahern wrote:
>>>>> On 7/5/16 5:38 PM, Casey Schaufler wrote:
>>>>>> I have encountered a system hang with my Smack
>>>>>> networking tests that bisects to the change below.
>>>>>> I can't say that I have any idea why the change
>>>>>> would impact the Smack processing, but there appears
>>>>>> to be some serious packet processing going on. The
>>>>>> Smack code is using CIPSO on the loopback interface.
>>>>>> The test is supposed to verify that labels can be
>>>>>> set on the packets using CIPSO. Unlabeled packets
>>>>>> do not appear to be impacted. I do not know if SELinux
>>>>>> is affected, and if not, why not. Smack and SELinux
>>>>>> use CIPSO differently.
>>>>>
>>>>> What are the commands to repeat the test?
>>>>>
>>>> There is a tar file attached with the tests.
>>>> Put the etc/smack/user file into /etc/smack/user.
>>>> In the tools-2012 directory run make to build
>>>> the tools. The test in question is called
>>>> testnetworking.sh and needs to be run as root.
>>>> You will need to configure Smack in the kernel,
>>>> of course.
>>>>
>>>
>>> I understand Paul's point and glad to see the changes are not causing issues with his SELinux testing.
>>>
>>> I have tried to reproduce the lockup you are seeing but not successful. I am using a jessie based VM with latest net-next kernel.
>>>
>>> root@kenny-jessie3:~/smack/tools-2012# zcat /proc/config.gz | grep SMACK
>>> CONFIG_SECURITY_SMACK=y
>>> CONFIG_SECURITY_SMACK_BRINGUP=y
>>> CONFIG_SECURITY_SMACK_NETFILTER=y
>>> # CONFIG_DEFAULT_SECURITY_SMACK is not set
>>
>> CONFIG_DEFAULT_SECURITY_SMACK needs to be set.
>
> ok. That caused more interesting output but still no hang after all of the snap-crackle-and-pops.
>
I find the test occasionally passes without hanging, but will
hang the system if repeated. I am running on Ubuntu and Fedora,
both with systemd, which may be a contributing factor. I run
under qemu, and am based on Linus' tree.

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[David Ahern]

On 7/6/16 11:01 AM, Casey Schaufler wrote:
> I find the test occasionally passes without hanging, but will
> hang the system if repeated. I am running on Ubuntu and Fedora,
> both with systemd, which may be a contributing factor. I run
> under qemu, and am based on Linus' tree.
>

With this:

for n in $(seq 1 10); do
     bash -x ./testnetworking.sh
     sleep 10
done

I do get the VM to loop where I can not kill the test. dmesg has this splat:

[ 3576.504715] general protection fault: 0000 [#21] SMP
[ 3576.505322] Modules linked in: 8021q garp mrp stp llc
[ 3576.506007] CPU: 3 PID: 2938 Comm: killall Tainted: G      D 
4.7.0-rc5+ #20
[ 3576.506881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[ 3576.508048] task: ffff8800b4e72340 ti: ffff880138a48000 task.ti: 
ffff880138a48000
[ 3576.508894] RIP: 0010:[<ffffffff81184dc2>]  [<ffffffff81184dc2>] 
next_tgid+0x53/0x99
[ 3576.509803] RSP: 0018:ffff880138a4bde8  EFLAGS: 00010206
[ 3576.510410] RAX: 4100646e4100608e RBX: 00000000000007f2 RCX: 
ffff8800b98c9bb0
[ 3576.511218] RDX: 4100646e4100608e RSI: 00000000000003e0 RDI: 
ffff8800b98c9b80
[ 3576.512024] RBP: ffff880138a4be10 R08: 0000000000000032 R09: 
0000000000000000
[ 3576.512833] R10: 0000000000000000 R11: 0000000000000200 R12: 
00000000000007e5
[ 3576.513647] R13: ffff8800b98c9b80 R14: ffffffff81a27900 R15: 
00000000000007e4
[ 3576.514453] FS:  00007fc084469700(0000) GS:ffff88013fd80000(0000) 
knlGS:0000000000000000
[ 3576.515361] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3576.516009] CR2: 0000000001947000 CR3: 00000000b5449000 CR4: 
00000000000406e0
[ 3576.516818] Stack:
[ 3576.517057]  00000000000007e5 0000000000000000 ffff880138a4bee0 
ffff8800b982a140
[ 3576.517963]  ffffffff81a27900 ffff880138a4be68 ffffffff81187090 
ffff8800b1d9d300
[ 3576.518857]  0030323032000001 ffff880138a4bee0 ffff880138a4bee0 
0000000000000000
[ 3576.519754] Call Trace:
[ 3576.520044]  [<ffffffff81187090>] proc_pid_readdir+0xd4/0x18b
[ 3576.520697]  [<ffffffff81183d6b>] proc_root_readdir+0x35/0x3a
[ 3576.521352]  [<ffffffff8114951a>] iterate_dir+0xac/0x148
[ 3576.521966]  [<ffffffff811513ad>] ? __fget_light+0x27/0x48
[ 3576.522587]  [<ffffffff81149892>] SyS_getdents+0x8a/0xdc
[ 3576.523189]  [<ffffffff8114967d>] ? fillonedir+0xc7/0xc7
[ 3576.523794]  [<ffffffff814a2172>] entry_SYSCALL_64_fastpath+0x1a/0xa4
[ 3576.524524]  [<ffffffff814a2172>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
[ 3576.525276] Code: d6 aa ed ff 48 85 c0 49 89 c5 74 40 4c 89 f6 48 89 
c7 e8 8b a2 ed ff 31 f6 4c 89 ef 89 c3 e8 15 a2 ed ff 48 85 c0 48 89 c2 
74 17 <48> 8b 80 78 05 00 00 48 8b 80 c8 00 00 00 48 39 82 f0 03 00 00
[ 3576.528359] RIP  [<ffffffff81184dc2>] next_tgid+0x53/0x99
[ 3576.528991]  RSP <ffff880138a4bde8>
[ 3576.529452] ---[ end trace a6f0cb9bfb70d9e6 ]---

And then I can no longer run commands:

root@kenny-jessie3:~# top -d1
Segmentation fault

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

On 7/6/2016 11:43 AM, David Ahern wrote:
> On 7/6/16 11:01 AM, Casey Schaufler wrote:
>> I find the test occasionally passes without hanging, but will
>> hang the system if repeated. I am running on Ubuntu and Fedora,
>> both with systemd, which may be a contributing factor. I run
>> under qemu, and am based on Linus' tree.
>>
>
> With this:
>
> for n in $(seq 1 10); do
>     bash -x ./testnetworking.sh
>     sleep 10
> done
>
> I do get the VM to loop where I can not kill the test. dmesg has this splat:
>
> [ 3576.504715] general protection fault: 0000 [#21] SMP
> [ 3576.505322] Modules linked in: 8021q garp mrp stp llc
> [ 3576.506007] CPU: 3 PID: 2938 Comm: killall Tainted: G      D 4.7.0-rc5+ #20
> [ 3576.506881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
> [ 3576.508048] task: ffff8800b4e72340 ti: ffff880138a48000 task.ti: ffff880138a48000
> [ 3576.508894] RIP: 0010:[<ffffffff81184dc2>]  [<ffffffff81184dc2>] next_tgid+0x53/0x99
> [ 3576.509803] RSP: 0018:ffff880138a4bde8  EFLAGS: 00010206
> [ 3576.510410] RAX: 4100646e4100608e RBX: 00000000000007f2 RCX: ffff8800b98c9bb0
> [ 3576.511218] RDX: 4100646e4100608e RSI: 00000000000003e0 RDI: ffff8800b98c9b80
> [ 3576.512024] RBP: ffff880138a4be10 R08: 0000000000000032 R09: 0000000000000000
> [ 3576.512833] R10: 0000000000000000 R11: 0000000000000200 R12: 00000000000007e5
> [ 3576.513647] R13: ffff8800b98c9b80 R14: ffffffff81a27900 R15: 00000000000007e4
> [ 3576.514453] FS:  00007fc084469700(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
> [ 3576.515361] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 3576.516009] CR2: 0000000001947000 CR3: 00000000b5449000 CR4: 00000000000406e0
> [ 3576.516818] Stack:
> [ 3576.517057]  00000000000007e5 0000000000000000 ffff880138a4bee0 ffff8800b982a140
> [ 3576.517963]  ffffffff81a27900 ffff880138a4be68 ffffffff81187090 ffff8800b1d9d300
> [ 3576.518857]  0030323032000001 ffff880138a4bee0 ffff880138a4bee0 0000000000000000
> [ 3576.519754] Call Trace:
> [ 3576.520044]  [<ffffffff81187090>] proc_pid_readdir+0xd4/0x18b
> [ 3576.520697]  [<ffffffff81183d6b>] proc_root_readdir+0x35/0x3a
> [ 3576.521352]  [<ffffffff8114951a>] iterate_dir+0xac/0x148
> [ 3576.521966]  [<ffffffff811513ad>] ? __fget_light+0x27/0x48
> [ 3576.522587]  [<ffffffff81149892>] SyS_getdents+0x8a/0xdc
> [ 3576.523189]  [<ffffffff8114967d>] ? fillonedir+0xc7/0xc7
> [ 3576.523794]  [<ffffffff814a2172>] entry_SYSCALL_64_fastpath+0x1a/0xa4
> [ 3576.524524]  [<ffffffff814a2172>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
> [ 3576.525276] Code: d6 aa ed ff 48 85 c0 49 89 c5 74 40 4c 89 f6 48 89 c7 e8 8b a2 ed ff 31 f6 4c 89 ef 89 c3 e8 15 a2 ed ff 48 85 c0 48 89 c2 74 17 <48> 8b 80 78 05 00 00 48 8b 80 c8 00 00 00 48 39 82 f0 03 00 00
> [ 3576.528359] RIP  [<ffffffff81184dc2>] next_tgid+0x53/0x99
> [ 3576.528991]  RSP <ffff880138a4bde8>
> [ 3576.529452] ---[ end trace a6f0cb9bfb70d9e6 ]---
>
> And then I can no longer run commands:
>
> root@kenny-jessie3:~# top -d1
> Segmentation fault
>
My thought is that there's a locking issue on a resource
somewhere in the TCP stack, and that a freed but still in
use buffer is getting into the filesystem code somehow.

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

On 7/6/2016 11:56 AM, Casey Schaufler wrote:
> On 7/6/2016 11:43 AM, David Ahern wrote:
>> On 7/6/16 11:01 AM, Casey Schaufler wrote:
>>> I find the test occasionally passes without hanging, but will
>>> hang the system if repeated. I am running on Ubuntu and Fedora,
>>> both with systemd, which may be a contributing factor. I run
>>> under qemu, and am based on Linus' tree.
>>>
>> With this:
>>
>> for n in $(seq 1 10); do
>>     bash -x ./testnetworking.sh
>>     sleep 10
>> done
>>
>> I do get the VM to loop where I can not kill the test. dmesg has this splat:
>>
>> [ 3576.504715] general protection fault: 0000 [#21] SMP
>> [ 3576.505322] Modules linked in: 8021q garp mrp stp llc
>> [ 3576.506007] CPU: 3 PID: 2938 Comm: killall Tainted: G      D 4.7.0-rc5+ #20
>> [ 3576.506881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
>> [ 3576.508048] task: ffff8800b4e72340 ti: ffff880138a48000 task.ti: ffff880138a48000
>> [ 3576.508894] RIP: 0010:[<ffffffff81184dc2>]  [<ffffffff81184dc2>] next_tgid+0x53/0x99
>> [ 3576.509803] RSP: 0018:ffff880138a4bde8  EFLAGS: 00010206
>> [ 3576.510410] RAX: 4100646e4100608e RBX: 00000000000007f2 RCX: ffff8800b98c9bb0
>> [ 3576.511218] RDX: 4100646e4100608e RSI: 00000000000003e0 RDI: ffff8800b98c9b80
>> [ 3576.512024] RBP: ffff880138a4be10 R08: 0000000000000032 R09: 0000000000000000
>> [ 3576.512833] R10: 0000000000000000 R11: 0000000000000200 R12: 00000000000007e5
>> [ 3576.513647] R13: ffff8800b98c9b80 R14: ffffffff81a27900 R15: 00000000000007e4
>> [ 3576.514453] FS:  00007fc084469700(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
>> [ 3576.515361] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 3576.516009] CR2: 0000000001947000 CR3: 00000000b5449000 CR4: 00000000000406e0
>> [ 3576.516818] Stack:
>> [ 3576.517057]  00000000000007e5 0000000000000000 ffff880138a4bee0 ffff8800b982a140
>> [ 3576.517963]  ffffffff81a27900 ffff880138a4be68 ffffffff81187090 ffff8800b1d9d300
>> [ 3576.518857]  0030323032000001 ffff880138a4bee0 ffff880138a4bee0 0000000000000000
>> [ 3576.519754] Call Trace:
>> [ 3576.520044]  [<ffffffff81187090>] proc_pid_readdir+0xd4/0x18b
>> [ 3576.520697]  [<ffffffff81183d6b>] proc_root_readdir+0x35/0x3a
>> [ 3576.521352]  [<ffffffff8114951a>] iterate_dir+0xac/0x148
>> [ 3576.521966]  [<ffffffff811513ad>] ? __fget_light+0x27/0x48
>> [ 3576.522587]  [<ffffffff81149892>] SyS_getdents+0x8a/0xdc
>> [ 3576.523189]  [<ffffffff8114967d>] ? fillonedir+0xc7/0xc7
>> [ 3576.523794]  [<ffffffff814a2172>] entry_SYSCALL_64_fastpath+0x1a/0xa4
>> [ 3576.524524]  [<ffffffff814a2172>] ? entry_SYSCALL_64_fastpath+0x1a/0xa4
>> [ 3576.525276] Code: d6 aa ed ff 48 85 c0 49 89 c5 74 40 4c 89 f6 48 89 c7 e8 8b a2 ed ff 31 f6 4c 89 ef 89 c3 e8 15 a2 ed ff 48 85 c0 48 89 c2 74 17 <48> 8b 80 78 05 00 00 48 8b 80 c8 00 00 00 48 39 82 f0 03 00 00
>> [ 3576.528359] RIP  [<ffffffff81184dc2>] next_tgid+0x53/0x99
>> [ 3576.528991]  RSP <ffff880138a4bde8>
>> [ 3576.529452] ---[ end trace a6f0cb9bfb70d9e6 ]---
>>
>> And then I can no longer run commands:
>>
>> root@kenny-jessie3:~# top -d1
>> Segmentation fault
>>
> My thought is that there's a locking issue on a resource
> somewhere in the TCP stack, and that a freed but still in
> use buffer is getting into the filesystem code somehow.

Digging into this further I have determined that the
circumstances leading to this issue are somewhat complex.
The good news is that there seems to be a very limited
circumstances under which the problem manifests.

I have a socket, and change the Smack attributes on the
socket (security_inode_setsecurity) before connecting to
a server. The connect succeeds. The client sends a packet,
also successfully. The response is received. Now here's
where it gets interesting. I instrumented the code to print
the Smack attributes on the socket both before and after
the Smack access check. Before the check is made the Smack
data reflects the initial values from when the socket was
created. After the check, they reflect the explicit change
made earlier. The check reports failure based on the initial
values. As a result, an attempt to notify the caller that
the action failed is made (netlbl_skbuff_err) which results
in a call to icmp_send that frees already freed memory.

If the Smack attributes in the sk_security blob are not
explicitly set the problem does not occur. I have the same
result if I change the Smack attributes within the socket
security blob as I do if I replace the security blob.

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Paul Moore]

On Tue, Jul 19, 2016 at 7:37 PM, Casey Schaufler <casey@schaufler-ca.com> wrote:
> Digging into this further I have determined that the
> circumstances leading to this issue are somewhat complex.
> The good news is that there seems to be a very limited
> circumstances under which the problem manifests.
>
> I have a socket, and change the Smack attributes on the
> socket (security_inode_setsecurity) before connecting to
> a server.

This is a minty fresh, disconnected socket, yes?

> The connect succeeds. The client sends a packet,
> also successfully. The response is received. Now here's
> where it gets interesting. I instrumented the code to print
> the Smack attributes on the socket both before and after
> the Smack access check.

I'm assuming that when you say "access check" you are talking about
the smk_access() call in smack_socket_sock_rcv_skb(), yes?

(as a totally unrelated side note, you really went nuts on the cpp
conditionals in there, was there a sale on #ifdefs that I missed? <g>)

> Before the check is made the Smack
> data reflects the initial values from when the socket was
> created. After the check, they reflect the explicit change
> made earlier.

It has been too long since I looked at how Smack handled network
packets, I assume this is not the intended behavior?

> The check reports failure based on the initial
> values. As a result, an attempt to notify the caller that
> the action failed is made (netlbl_skbuff_err) which results
> in a call to icmp_send that frees already freed memory.

What memory is being double freed?  The original skb?  I don't believe
netlbl_skbuff_err(), cipso_v4_error(), or icmp_send() frees the
original skb ... or rather it shouldn't, perhaps I'm missing
something.

I'm not arguing, you saw what you saw, I'm just trying to understand
and make sense of it.  Can you elaborate on what you saw, using very
small words, and concrete descriptions (I'm much more stupider than
everyone here so you have to make it easy for me to understand)?

> If the Smack attributes in the sk_security blob are not
> explicitly set the problem does not occur. I have the same
> result if I change the Smack attributes within the socket
> security blob as I do if I replace the security blob.

-- 
paul moore
www.paul-moore.com

Re: Network hang after c3f1010b30f7fc611139cfb702a8685741aa6827 with CIPSO & Smack

[Casey Schaufler]

On 7/20/2016 1:13 PM, Paul Moore wrote:
> On Tue, Jul 19, 2016 at 7:37 PM, Casey Schaufler <casey@schaufler-ca.com> wrote:
>> Digging into this further I have determined that the
>> circumstances leading to this issue are somewhat complex.
>> The good news is that there seems to be a very limited
>> circumstances under which the problem manifests.
>>
>> I have a socket, and change the Smack attributes on the
>> socket (security_inode_setsecurity) before connecting to
>> a server.
> This is a minty fresh, disconnected socket, yes?
>
>> The connect succeeds. The client sends a packet,
>> also successfully. The response is received. Now here's
>> where it gets interesting. I instrumented the code to print
>> the Smack attributes on the socket both before and after
>> the Smack access check.
> I'm assuming that when you say "access check" you are talking about
> the smk_access() call in smack_socket_sock_rcv_skb(), yes?

I have been able to track this down to my careless
use of netlbl_skbuff_err(). Because the socket started
life as unlabeled, and changed to labeled, calling
netlbl_skbuff_err() resulted in multiple frees of some
netlabel data under some circumstances. I don't know
why it worked before, but the code certainly shouldn't
have been making that call. I have a patch in final test.

>
> (as a totally unrelated side note, you really went nuts on the cpp
> conditionals in there, was there a sale on #ifdefs that I missed? <g>)

I can't say that I'm happy about how that code ended up.
I hope to do a clean up in association with switching away
from CIPSO to secmark for local access controls. That's something
I have to do for Extreme Security Module Stacking.

>
>> Before the check is made the Smack
>> data reflects the initial values from when the socket was
>> created. After the check, they reflect the explicit change
>> made earlier.
> It has been too long since I looked at how Smack handled network
> packets, I assume this is not the intended behavior?
>
>> The check reports failure based on the initial
>> values. As a result, an attempt to notify the caller that
>> the action failed is made (netlbl_skbuff_err) which results
>> in a call to icmp_send that frees already freed memory.
> What memory is being double freed?  The original skb?  I don't believe
> netlbl_skbuff_err(), cipso_v4_error(), or icmp_send() frees the
> original skb ... or rather it shouldn't, perhaps I'm missing
> something.
>
> I'm not arguing, you saw what you saw, I'm just trying to understand
> and make sense of it.  Can you elaborate on what you saw, using very
> small words, and concrete descriptions (I'm much more stupider than
> everyone here so you have to make it easy for me to understand)?
>
>> If the Smack attributes in the sk_security blob are not
>> explicitly set the problem does not occur. I have the same
>> result if I change the Smack attributes within the socket
>> security blob as I do if I replace the security blob.