Re: 2.6.24-rc2-mm1 -- strange apparent network failures

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Andy Whitcroft]

When testing some of the later 2.6.24-rc2-mm1+hotfix combinations on three
of our test systems one job from each batch (1/4) failed.  In each case the
machine appears to have booted normally all the way to a login: prompt.
However in the failed boots the networking though apparently initialised
completely and correctly (as far as I can tell from the console output), is
reported as not responding to ssh connections.  The network interface seems
to have been initialised on the right port, and the ssh daemons started.

Two of the machines are powerpc boxes, the other an older x86_64.
One machine is 4/4 in testing, just one.  Most of the other machines are
still not able to compile this stack so do not contribute to our knowledge.

Any ideas?

-apw

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Kevin Winchester]

On November 15, 2007 06:02:09 am Andy Whitcroft wrote:
> When testing some of the later 2.6.24-rc2-mm1+hotfix combinations on three
> of our test systems one job from each batch (1/4) failed.  In each case the
> machine appears to have booted normally all the way to a login: prompt.
> However in the failed boots the networking though apparently initialised
> completely and correctly (as far as I can tell from the console output), is
> reported as not responding to ssh connections.  The network interface seems
> to have been initialised on the right port, and the ssh daemons started.
>
> Two of the machines are powerpc boxes, the other an older x86_64.
> One machine is 4/4 in testing, just one.  Most of the other machines are
> still not able to compile this stack so do not contribute to our knowledge.
>
> Any ideas?
>

I see this as well - the computer boots fine but no network.  The only clues 
in the dmesg are:

[  294.097876] warning: process `dhclient' gets w/ old libcap
[  294.097893] warning: process `dhclient' sets w/ old libcap

So I'll try backing up the patch series to before:

add-64-bit-capability-support-to-the-kernel.patch

or so, and see if that's the problem.  If anyone has any other ideas, let me 
know.

-- 
Kevin Winchester

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Andrew Morton]

On Thu, 15 Nov 2007 20:28:29 -0400
Kevin Winchester <kjwinchester@gmail.com> wrote:

> On November 15, 2007 06:02:09 am Andy Whitcroft wrote:
> > When testing some of the later 2.6.24-rc2-mm1+hotfix combinations on three
> > of our test systems one job from each batch (1/4) failed.  In each case the
> > machine appears to have booted normally all the way to a login: prompt.
> > However in the failed boots the networking though apparently initialised
> > completely and correctly (as far as I can tell from the console output), is
> > reported as not responding to ssh connections.  The network interface seems
> > to have been initialised on the right port, and the ssh daemons started.
> >
> > Two of the machines are powerpc boxes, the other an older x86_64.
> > One machine is 4/4 in testing, just one.  Most of the other machines are
> > still not able to compile this stack so do not contribute to our knowledge.
> >
> > Any ideas?
> >
> 
> I see this as well - the computer boots fine but no network.  The only clues 
> in the dmesg are:
> 
> [  294.097876] warning: process `dhclient' gets w/ old libcap
> [  294.097893] warning: process `dhclient' sets w/ old libcap
> 
> So I'll try backing up the patch series to before:
> 
> add-64-bit-capability-support-to-the-kernel.patch

Yes, that's a good one to suspect.

What a peculiar error message.

> or so, and see if that's the problem.  If anyone has any other ideas, let me 
> know.

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Andrew Morgan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

This warning is just saying that you might want to reconsider
recompiling your dhclient with a newer libcap - which has native support
for 64-bit capabilities. This is supposed to be informative, and not be
associated with any particular error.

- From your comments, you believe that this patch causes something in your
boot process to fail. Can you supply some detail about the version of
dhclient you are using? I'd like to understand exactly what it is doing
(via libcap).

Thanks

Andrew

Andrew Morton wrote:
> On Thu, 15 Nov 2007 20:28:29 -0400
> Kevin Winchester <kjwinchester@gmail.com> wrote:
> 
>> On November 15, 2007 06:02:09 am Andy Whitcroft wrote:
>>> When testing some of the later 2.6.24-rc2-mm1+hotfix combinations on three
>>> of our test systems one job from each batch (1/4) failed.  In each case the
>>> machine appears to have booted normally all the way to a login: prompt.
>>> However in the failed boots the networking though apparently initialised
>>> completely and correctly (as far as I can tell from the console output), is
>>> reported as not responding to ssh connections.  The network interface seems
>>> to have been initialised on the right port, and the ssh daemons started.
>>>
>>> Two of the machines are powerpc boxes, the other an older x86_64.
>>> One machine is 4/4 in testing, just one.  Most of the other machines are
>>> still not able to compile this stack so do not contribute to our knowledge.
>>>
>>> Any ideas?
>>>
>> I see this as well - the computer boots fine but no network.  The only clues 
>> in the dmesg are:
>>
>> [  294.097876] warning: process `dhclient' gets w/ old libcap
>> [  294.097893] warning: process `dhclient' sets w/ old libcap
>>
>> So I'll try backing up the patch series to before:
>>
>> add-64-bit-capability-support-to-the-kernel.patch
> 
> Yes, that's a good one to suspect.
> 
> What a peculiar error message.
> 
>> or so, and see if that's the problem.  If anyone has any other ideas, let me 
>> know.
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFHPnlIQheEq9QabfIRAlglAKCG2NG1xnwMT8G/Lk8GoEPwtBzq9QCdFLYi
k+pt5Sd2AdtOJ+TjMIt1y6g=
=5wpX
-----END PGP SIGNATURE-----

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Kevin Winchester]

On November 17, 2007 01:16:58 am Andrew Morgan wrote:
> Hi,
>
> This warning is just saying that you might want to reconsider
> recompiling your dhclient with a newer libcap - which has native support
> for 64-bit capabilities. This is supposed to be informative, and not be
> associated with any particular error.
>
> From your comments, you believe that this patch causes something in your
> boot process to fail. Can you supply some detail about the version of
> dhclient you are using? I'd like to understand exactly what it is doing
> (via libcap).
>
> Thanks
>

The boot succeeds (and appears to bring initialize the network adapter 
properly - it autonegotiates a 100Mbps link speed), but the dhcp client is 
never able to get an address.  However, applying the rc2-mm1 patch series up 
to just before:

  add-64-bit-capability-support-to-the-kernel.patch

results in a working kernel.  Applying just this patch causes the failure.  To 
be sure, I also tried applying the above patch plus the following ones:

  add-64-bit-capability-support-to-the-kernel-checkpatch-fixes.patch
  add-64-bit-capability-support-to-the-kernel-fix.patch
  add-64-bit-capability-support-to-the-kernel-fix-fix.patch
  remove-unnecessary-include-from-include-linux-capabilityh.patch

but the problem still occurs even with all of these.

As to versions, I'm running Kubuntu gutsy, so I have the default:

dhcp3-client           3.0.5-3ubuntu4
libcap1                1:1.10-14build1

packages installed.

Let me know if you need any other information, or if you have a patch you 
would like tested.

-- 
Kevin Winchester

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Andy Whitcroft]

On Fri, Nov 16, 2007 at 09:16:58PM -0800, Andrew Morgan wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi,
> 
> This warning is just saying that you might want to reconsider
> recompiling your dhclient with a newer libcap - which has native support
> for 64-bit capabilities. This is supposed to be informative, and not be
> associated with any particular error.
> 
> - From your comments, you believe that this patch causes something in your
> boot process to fail. Can you supply some detail about the version of
> dhclient you are using? I'd like to understand exactly what it is doing
> (via libcap).
> 
> Thanks

The machine which show this problem for me are using static network
configurations, so I don't know if libcap is still in the mix there.

I've just compared the boot logs from a successful and unsuccessful boot
on this kernel, and I don't see that particular message, nor do I see
any significant differences overall.

Perlexed.

-apw

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Andrew Morgan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kevin,

Can you try this quick hack?

diff --git a/kernel/capability.c b/kernel/capability.c
index e57d1aa..4088610 100644
- --- a/kernel/capability.c
+++ b/kernel/capability.c
@@ -109,7 +109,7 @@ out:
                        kdata[i].permitted = pP.cap[i];
                        kdata[i].inheritable = pI.cap[i];
                }
- -               while (i < _LINUX_CAPABILITY_U32S) {
+               while (0 && (i < _LINUX_CAPABILITY_U32S)) {
                        if (pE.cap[i] || pP.cap[i] || pP.cap[i]) {
                                /* Cannot represent w/ legacy structure */
                                return -ERANGE;

Thanks

Andrew

Kevin Winchester wrote:
> On November 17, 2007 01:16:58 am Andrew Morgan wrote:
>> Hi,
>>
>> This warning is just saying that you might want to reconsider
>> recompiling your dhclient with a newer libcap - which has native support
>> for 64-bit capabilities. This is supposed to be informative, and not be
>> associated with any particular error.
>>
>> From your comments, you believe that this patch causes something in your
>> boot process to fail. Can you supply some detail about the version of
>> dhclient you are using? I'd like to understand exactly what it is doing
>> (via libcap).
>>
>> Thanks
>>
> 
> The boot succeeds (and appears to bring initialize the network adapter 
> properly - it autonegotiates a 100Mbps link speed), but the dhcp client is 
> never able to get an address.  However, applying the rc2-mm1 patch series up 
> to just before:
> 
>   add-64-bit-capability-support-to-the-kernel.patch
> 
> results in a working kernel.  Applying just this patch causes the failure.  To 
> be sure, I also tried applying the above patch plus the following ones:
> 
>   add-64-bit-capability-support-to-the-kernel-checkpatch-fixes.patch
>   add-64-bit-capability-support-to-the-kernel-fix.patch
>   add-64-bit-capability-support-to-the-kernel-fix-fix.patch
>   remove-unnecessary-include-from-include-linux-capabilityh.patch
> 
> but the problem still occurs even with all of these.
> 
> As to versions, I'm running Kubuntu gutsy, so I have the default:
> 
> dhcp3-client           3.0.5-3ubuntu4
> libcap1                1:1.10-14build1
> 
> packages installed.
> 
> Let me know if you need any other information, or if you have a patch you 
> would like tested.
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFHP37LQheEq9QabfIRAst5AJ9Nsw0RtF2NDuUAMvQZh5OFWEB4ugCeIxMH
lp5/Ka7SJZLIrQpZDijrd1E=
=GN18
-----END PGP SIGNATURE-----

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Kevin Winchester]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Morgan wrote:
> Kevin,
> 
> Can you try this quick hack?
> 
> diff --git a/kernel/capability.c b/kernel/capability.c
> index e57d1aa..4088610 100644
> --- a/kernel/capability.c
> +++ b/kernel/capability.c
> @@ -109,7 +109,7 @@ out:
>                         kdata[i].permitted = pP.cap[i];
>                         kdata[i].inheritable = pI.cap[i];
>                 }
> -               while (i < _LINUX_CAPABILITY_U32S) {
> +               while (0 && (i < _LINUX_CAPABILITY_U32S)) {
>                         if (pE.cap[i] || pP.cap[i] || pP.cap[i]) {
>                                 /* Cannot represent w/ legacy structure */
>                                 return -ERANGE;
> 

Well, something went wrong with the patch - it has extra negative signs
in my mail reader, and on lkml, but now that I've hit reply and it's
been quoted, it looks fine in my mail client.  So I have no idea what
went on.

However, I got around the problem by making the code change manually -
and my network connection is now working.  Looking at the code being
bypassed:

    if (pE.cap[i] || pP.cap[i] || pP.cap[i])

looks somewhat weird as it is testing the same condition twice.  Should
it have been:

    if (pE.cap[i] || pP.cap[i] || pI.cap[i])

?

I'm about to test that change instead of bypassing the loop, so I'll let
you know the results.

- --
Kevin Winchester


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHP4xGKPGFQbiQ3tQRAooWAJ9c6exhOiD4VUZ04hS9z77/RmERUACfauTE
BV/JAexzlm2zSmG4laYi+HQ=
=IPkA
-----END PGP SIGNATURE-----

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Kevin Winchester]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kevin Winchester wrote:
> Looking at the code being bypassed:
> 
>     if (pE.cap[i] || pP.cap[i] || pP.cap[i])
> 
> looks somewhat weird as it is testing the same condition twice.  Should
> it have been:
> 
>     if (pE.cap[i] || pP.cap[i] || pI.cap[i])
> 
> ?
> 
> I'm about to test that change instead of bypassing the loop, so I'll let
> you know the results.
> 

No, this still results in a dead network connection, although it is
probably a correct change.  I suppose giving the loop even more reasons
to return -ERANGE wasn't going to be helpful.

- --
Kevin Winchester

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHP5KXKPGFQbiQ3tQRAilbAJ9h3qtO9sb9+ctVU0pxzCBjysy06QCdE1Wd
M5V3+0BWyn04p0UeUq/KSlw=
=663t
-----END PGP SIGNATURE-----

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Kevin Winchester]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Morgan wrote:
> Kevin,
> 
> Can you try this quick hack?
> 
> diff --git a/kernel/capability.c b/kernel/capability.c
> index e57d1aa..4088610 100644
> --- a/kernel/capability.c
> +++ b/kernel/capability.c
> @@ -109,7 +109,7 @@ out:
>                         kdata[i].permitted = pP.cap[i];
>                         kdata[i].inheritable = pI.cap[i];
>                 }
> -               while (i < _LINUX_CAPABILITY_U32S) {
> +               while (0 && (i < _LINUX_CAPABILITY_U32S)) {
>                         if (pE.cap[i] || pP.cap[i] || pP.cap[i]) {
>                                 /* Cannot represent w/ legacy structure */
>                                 return -ERANGE;
> 


Oh, and the reason your patch turned up incorrect in my mailer and on
lkml seems to be the PGP signature.  I didn't have your public key, so
my mail client just left the full PGP-signed text in, which includes
escaping of '-' characters.  LKML must also ignore the signature.  Once
I added your public key, the patch shows up correctly in my client at least.

(I guess everyone else probably knew this already...but at least I
learned something new today)

- --
Kevin Winchester
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHP5QdKPGFQbiQ3tQRAqimAJwOSGDSM2wXeLbm+sBKehGf/haNpACfX7Cb
IALnPxwlgShR6Xb+XQclBro=
=xFUp
-----END PGP SIGNATURE-----

Re: 2.6.24-rc2-mm1 -- strange apparent network failures

[Andrew Morgan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kevin Winchester wrote:
> However, I got around the problem by making the code change manually -
> and my network connection is now working.  Looking at the code being
> bypassed:
> 
>     if (pE.cap[i] || pP.cap[i] || pP.cap[i])
> 
> looks somewhat weird as it is testing the same condition twice.  Should
> it have been:
> 
>     if (pE.cap[i] || pP.cap[i] || pI.cap[i])

Yes, that was also a bug. However, upon reflection (and as per my "0 &&"
hack), I now believe these few lines of code are problematic in general.

Thanks for reporting this bug. I'll post a more clear patch (that isn't
GPG'd).

Cheers

Andrew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHP5vy+bHCR3gb8jsRAliTAKCvCsfZuNN7Og57S0s8O4SZNveSUwCgq4VP
vHUE/S+x09l5I24E2/rmLj4=
=JaWT
-----END PGP SIGNATURE-----