pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Jorge Daniel Sequeira Matias]

  Hi everyone,

  I'm running a PPTP VPN Server for almost 4 years now. This server is 
running in a Debian Linux (unstable).
  The configuration for the last 1 year and a half was the following:

  Computer 2x Intel Xeon 2.6GHz and 512MB RAM
  Linux v2.6.11.6 (vanilla) with MPPE patch
  Debian distribution with the following packages
     - pppd 2.4.3-20050321
     - libradius1 0.3.2-8
     - radiusclient1 0.3.2-8

  With this configuration I had reach a peek of aprox. 600 hundred PPTP 
connections simultaneously. The computer gets heavy loaded doing MPPE 
compression/decompression with 40Mbits/sec of PPP traffic but didn't have 
any problem with this setup. The authentication process is using a pool of 
RADIUS servers.

   This machine is also a firewall, and now a new feature was needed: PPTP 
VPN connections from the inside.

  The problem:

   After the upgrade to a Linux v2.6.15.4 (vanilla) with no patches to use 
the new "ip_conntrack_pptp" module, I started to get "pppd" processes 
hanging in "S" state and consuming CPU power.

---- Here is a bit of debug commands ----
machine:# strace -p 18859
Process 18859 attached - interrupt to quit
ioctl(8, TIOCSETD (Issued Ctrl+C here after a few seconds)<unfinished ...>
machine:# ps aux | grep 18859
root     18859  2.1  0.2  2696 1360 ?        S    09:52  17:27 
/usr/sbin/pppd local file /etc/ppp/options.pptpd 115200 10.2.255.254:
machine:# ps ax | grep 18859
18859 ?        S     17:30 /usr/sbin/pppd local file /etc/ppp/options.pptpd 
115200 10.2.255.254:10.2.0.71 ipparam 10.1.7.232 plugin 
/usr/lib/pptpd/pptpd-logwtmp.so pptpd-original-ip 10.1.7.232
machine:# lsof -p 18859
COMMAND   PID USER   FD   TYPE     DEVICE    SIZE     NODE NAME
pppd    18859 root  cwd    DIR        9,1    4096        2 /
pppd    18859 root  rtd    DIR        9,1    4096        2 /
pppd    18859 root  txt    REG        9,1  265880  4424041 /usr/sbin/pppd
pppd    18859 root  mem    REG        0,0                0 [heap] (stat: No 
such file or directory)
pppd    18859 root  mem    REG        9,1   64924  1720713 
/lib/tls/libresolv-2.3.2.so
pppd    18859 root  mem    REG        9,1   13976  1720706 
/lib/tls/libnss_dns-2.3.2.so
pppd    18859 root  mem    REG        9,2  147456   711080 
/var/run/pppd2.tdb
pppd    18859 root  mem    REG        9,1   47024  5095932 
/usr/lib/pppd/2.4.3/radius.so
pppd    18859 root  mem    REG        9,1   34748  1720707 
/lib/tls/libnss_files-2.3.2.so
pppd    18859 root  mem    REG        9,1   33440  1720709 
/lib/tls/libnss_nis-2.3.2.so
pppd    18859 root  mem    REG        9,1   73304  1720704 
/lib/tls/libnsl-2.3.2.so
pppd    18859 root  mem    REG        9,1   28616  1720705 
/lib/tls/libnss_compat-2.3.2.so
pppd    18859 root  mem    REG        9,1 1254468  1720699 
/lib/tls/libc-2.3.2.so
pppd    18859 root  mem    REG        9,1  113040  3621129 
/usr/lib/libpcap.so.0.7.1
pppd    18859 root  mem    REG        9,1    9872  1720701 
/lib/tls/libdl-2.3.2.so
pppd    18859 root  mem    REG        9,1   30360  4637043 
/lib/libpam.so.0.76
pppd    18859 root  mem    REG        9,1   18876  1720700 
/lib/tls/libcrypt-2.3.2.so
pppd    18859 root  mem    REG        9,1    4272  3621647 
/usr/lib/pptpd/pptpd-logwtmp.so
pppd    18859 root  mem    REG        9,1   90024  4637078 /lib/ld-2.3.2.so
pppd    18859 root    0u   CHR     136,64               66 /dev/pts/64
pppd    18859 root    1u   CHR     136,64               66 /dev/pts/64
pppd    18859 root    2u   CHR        1,3          4472977 /dev/null
pppd    18859 root    3u  unix 0xd5341500         34918748 socket
pppd    18859 root    4u   CHR        1,3          4472977 /dev/null
pppd    18859 root    5u  sock        0,4         34918753 can't identify 
protocol
pppd    18859 root    6u  sock        0,4         34918754 can't identify 
protocol
pppd    18859 root    7u   REG        9,2  147456   711080 
/var/run/pppd2.tdb
pppd    18859 root    8u   CHR     136,64               66 /dev/pts/64
pppd    18859 root    9u   CHR      108,0          4474356 /dev/ppp
pppd    18859 root   10u   CHR      108,0          4474356 /dev/ppp
------ End of debug info ----

Every day I have about 5 new "pppd" processes hanging in this state.

I upgraded to a Linux 2.6.16.16 with no patches and the "ip_conntrack_pptp" 
enable. Every other piece of software remains the same. The problem is the 
same. I still have compiled a Kernel with no "ip_conntrack_pptp" to see this 
was causing the problem but appears not to be the case.

This is kernel config:
#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=y
# CONFIG_PPP_MULTILINK is not set
# CONFIG_PPP_FILTER is not set
CONFIG_PPP_ASYNC=y
CONFIG_PPP_SYNC_TTY=y
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
CONFIG_PPP_MPPE=y
# CONFIG_PPPOE is not set
# CONFIG_SLIP is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set

------------------------------------------------------------------------------------------------------

Anyone else has this problem?

Best Regards,
Jorge Matias
CIIST - Centro de Informática do
Instituto Superior Técnico
Universidade Técnica de Lisboa

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Patrick McHardy]

Jorge Daniel Sequeira Matias wrote:
> 
>  Hi everyone,
> 
>  I'm running a PPTP VPN Server for almost 4 years now. This server is
> running in a Debian Linux (unstable).
>  The configuration for the last 1 year and a half was the following:
> 
>  Computer 2x Intel Xeon 2.6GHz and 512MB RAM
>  Linux v2.6.11.6 (vanilla) with MPPE patch
>  Debian distribution with the following packages
>     - pppd 2.4.3-20050321
>     - libradius1 0.3.2-8
>     - radiusclient1 0.3.2-8
> 
>  With this configuration I had reach a peek of aprox. 600 hundred PPTP
> connections simultaneously. The computer gets heavy loaded doing MPPE
> compression/decompression with 40Mbits/sec of PPP traffic but didn't
> have any problem with this setup. The authentication process is using a
> pool of RADIUS servers.
> 
>   This machine is also a firewall, and now a new feature was needed:
> PPTP VPN connections from the inside.
> 
>  The problem:
> 
>   After the upgrade to a Linux v2.6.15.4 (vanilla) with no patches to
> use the new "ip_conntrack_pptp" module, I started to get "pppd"
> processes hanging in "S" state and consuming CPU power.

Does the problem go away without the ip_conntrack_pptp module loaded?

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Jorge Daniel Sequeira Matias]

----- Original Message ----- 
From: "Patrick McHardy" <kaber@trash.net>
To: "Jorge Daniel Sequeira Matias" <martin@andorinha.ist.utl.pt>
Cc: <linux-ppp@vger.kernel.org>; "Netfilter Development Mailinglist" 
<netfilter-devel@lists.netfilter.org>
Sent: Tuesday, June 27, 2006 11:43 PM
Subject: Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+



>> [...]
> Does the problem go away without the ip_conntrack_pptp module loaded?

  When I had the 2.6.15.4 installed I have recompiled without this module. 
The problem didn't go away.
  I know that 2.6.16.x has a different Netfilter structure but I have not 
tried yet to remove "ip_conntrack_pptp" with this version.
  Right now I have 2.6.16.16 with the module loaded.

Regards,
Jorge Matias

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Phil Mayers]

Jorge Daniel Sequeira Matias wrote:
> 
>  Hi everyone,
> 
>  I'm running a PPTP VPN Server for almost 4 years now. This server is 
> running in a Debian Linux (unstable).
>  The configuration for the last 1 year and a half was the following:
> 
>  Computer 2x Intel Xeon 2.6GHz and 512MB RAM
>  Linux v2.6.11.6 (vanilla) with MPPE patch

>   After the upgrade to a Linux v2.6.15.4 (vanilla) with no patches to 

There were a couple of bugs recently in the TTY layer related to poptop 
and running pppd on PTYs, but I believe 2.6.15 did not contain them 
(though I find the git logs incomprehensible so could be wrong). See:

http://bugzilla.kernel.org/show_bug.cgi?idd02
http://bugzilla.kernel.org/show_bug.cgi?ide30

Just in case, you should definitely try the latest kernel - 2.6.17.1 is 
stable.

If that doesn't solve it and if you can reliably reproduce it you could 
do a "git bisect" to track the bug down (tedious though):

http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
http://www.kernel.org/pub/software/scm/git/docs/howto/isolate-bugs-with-bisect.txt
http://www.reactivated.net/weblog/archives/2006/01/using-git-bisect-to-find-buggy-kernel-patches/

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Jorge Daniel Sequeira Matias]

  I forgot to mention that I have analyzed the pppd logs and found out that 
it enters in this state when closing connection with the other peer. In 
particular, when the peer doesn't reply for N LCP Requests and our pppd 
gives up and terminate connection.
  What I mean is that, this happens when pppd is restoring TTY line 
discipline, after tcflush(). I saw this in the source code of pppd 2.4.3 
which is what I'm using.

  Anyway, I'll give the 2.6.17.1 a try and report again tomorrow.

Regards,
Jorge Matias


----- Original Message ----- 
From: "Phil Mayers" <p.mayers@imperial.ac.uk>
To: <linux-ppp@vger.kernel.org>
Sent: Wednesday, June 28, 2006 12:03 AM
Subject: Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+


> Jorge Daniel Sequeira Matias wrote:
>>
>>  Hi everyone,
>>
>>  I'm running a PPTP VPN Server for almost 4 years now. This server is 
>> running in a Debian Linux (unstable).
>>  The configuration for the last 1 year and a half was the following:
>>
>>  Computer 2x Intel Xeon 2.6GHz and 512MB RAM
>>  Linux v2.6.11.6 (vanilla) with MPPE patch
>
>>   After the upgrade to a Linux v2.6.15.4 (vanilla) with no patches to
>
> There were a couple of bugs recently in the TTY layer related to poptop 
> and running pppd on PTYs, but I believe 2.6.15 did not contain them 
> (though I find the git logs incomprehensible so could be wrong). See:
>
> http://bugzilla.kernel.org/show_bug.cgi?idd02
> http://bugzilla.kernel.org/show_bug.cgi?ide30
>
> Just in case, you should definitely try the latest kernel - 2.6.17.1 is 
> stable.
>
> If that doesn't solve it and if you can reliably reproduce it you could do 
> a "git bisect" to track the bug down (tedious though):
>
> http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
> http://www.kernel.org/pub/software/scm/git/docs/howto/isolate-bugs-with-bisect.txt
> http://www.reactivated.net/weblog/archives/2006/01/using-git-bisect-to-find-buggy-kernel-patches/
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Jorge Daniel Sequeira Matias]

On Wednesday 28 June 2006 00:03, you wrote:
> There were a couple of bugs recently in the TTY layer related to poptop
> and running pppd on PTYs, but I believe 2.6.15 did not contain them
> (though I find the git logs incomprehensible so could be wrong). See:
>
> http://bugzilla.kernel.org/show_bug.cgi?idd02
> http://bugzilla.kernel.org/show_bug.cgi?ide30
>
> Just in case, you should definitely try the latest kernel - 2.6.17.1 is
> stable.

  It didn't solve. After running 2.6.17.1 for almost 48 hours I already have 4 
pppd processes stuck in "ioctl(TIOCSETD..." in the connection finish section.
  Here's a log of one PPPd process:

--- Log ---
[...]
Jun 29 16:01:36 wifi pppd[18575]: No response to 10 echo-requests
Jun 29 16:01:36 wifi pppd[18575]: Serial link appears to be disconnected.
Jun 29 16:01:36 wifi pppd[18575]: pptpd-logwtmp.so ip-down ppp99
Jun 29 16:01:36 wifi pppd[18575]: Connect time 65.5 minutes.
Jun 29 16:01:36 wifi pppd[18575]: Sent 1927619 bytes, received 620442 bytes.
Jun 29 16:01:36 wifi pppd[18575]: Script /etc/ppp/ip-down started (pid 20428)
Jun 29 16:01:36 wifi pppd[18575]: MPPE disabled
Jun 29 16:01:36 wifi pppd[18575]: sent [LCP TermReq id=0x2 "MPPE disabled"]
Jun 29 16:01:36 wifi pppd[18575]: sent [LCP TermReq id=0x3 "MPPE disabled"]
Jun 29 16:01:36 wifi pppd[18575]: Script /etc/ppp/ip-down finished (pid 
20428), status = 0x0
Jun 29 16:01:39 wifi pppd[18575]: sent [LCP TermReq id=0x4 "MPPE disabled"]
Jun 29 16:01:39 wifi pppd[18575]: Connection terminated.
--- End of Log ---

  The process has not sent the "Exit." string to the syslog yet.
  One thing that all pppd processes have in common is that the end peer didn't 
respond to the our max number of echo-requests. Though not all pppd processes 
end up in this lock scenario.


>
> If that doesn't solve it and if you can reliably reproduce it you could
> do a "git bisect" to track the bug down (tedious though):
>
> http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
> http://www.kernel.org/pub/software/scm/git/docs/howto/isolate-bugs-with-bis
>ect.txt
> http://www.reactivated.net/weblog/archives/2006/01/using-git-bisect-to-find
>-buggy-kernel-patches/ -

  It is not easy to reboot the machine every day but I'll try it out.

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Paul Mackerras]

Jorge Daniel Sequeira Matias writes:

>   It didn't solve. After running 2.6.17.1 for almost 48 hours I already have 4 
> pppd processes stuck in "ioctl(TIOCSETD..." in the connection finish section.

Try the newly-released version 2.4.4 of pppd, please.

Paul.

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Jorge Daniel Sequeira Matias]

On Friday 30 June 2006 00:26, Paul Mackerras wrote:
> Jorge Daniel Sequeira Matias writes:
> >   It didn't solve. After running 2.6.17.1 for almost 48 hours I already
> > have 4 pppd processes stuck in "ioctl(TIOCSETD..." in the connection
> > finish section.
>
> Try the newly-released version 2.4.4 of pppd, please.

  I have tried pppd v2.4.4 and didn't see any problem because with 2.6.17.1 I 
have a new problem.
  With 2.6.17.1, the "pptpctrl" processes, are heating all the CPU even if the 
VPN user traffic is low (like 40kbits/sec).
  I think this new problem is hiding the "pppd" problem because the VPN server 
is so slow that less people establishes connections. The OS network is now 
very lagged ...

Regards,
Jorge Matias

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[James Cameron]

On Tue, Jul 11, 2006 at 01:18:33PM +0100, Jorge Daniel Sequeira Matias wrote:
> I have tried pppd v2.4.4 and didn't see any problem because with
> 2.6.17.1 I have a new problem.
> With 2.6.17.1, the "pptpctrl" processes, are heating all the CPU even
> if the VPN user traffic is low (like 40kbits/sec).

I'm the maintainer of pptpctrl.  Could you describe this problem
technically please?  Write to poptop-server@lists.sourceforge.net.
We do not have a goal of cooling a CPU, so heating is expected.

>   I think this new problem is hiding the "pppd" problem because the
>   VPN server is so slow that less people establishes connections. The
>   OS network is now very lagged ...

You've given your perceptions but not any measurements.  To obtain
measurements, downgrade to the previous configuration and do some tests.
We have test scripts in pptpd and pptp CVS.

-- 
James Cameron
http://ftp.hp.com.au/sigs/jc/

Re: pppd stuck in ioctl(TIOCSETD) in Linux 2.6.15.4+

[Jorge Daniel Sequeira Matias]

On Tuesday 11 July 2006 23:53, James Cameron wrote:
> On Tue, Jul 11, 2006 at 01:18:33PM +0100, Jorge Daniel Sequeira Matias 
wrote:
> > I have tried pppd v2.4.4 and didn't see any problem because with
> > 2.6.17.1 I have a new problem.
> > With 2.6.17.1, the "pptpctrl" processes, are heating all the CPU even
> > if the VPN user traffic is low (like 40kbits/sec).
>
> I'm the maintainer of pptpctrl.  Could you describe this problem
> technically please?  Write to poptop-server@lists.sourceforge.net.
> We do not have a goal of cooling a CPU, so heating is expected.

  Sorry for my English. What I meant was it as consuming all the CPU process 
time.
  Meanwhile I saw a reference to this problem in this e-mail

  "2.6.18-rc1-mm1 - VPN chewing CPU like crazy.." in
     http://lkml.org/lkml/2006/7/11/8

  But I have only seen the problem starting with 2.6.17.1

> >   I think this new problem is hiding the "pppd" problem because the
> >   VPN server is so slow that less people establishes connections. The
> >   OS network is now very lagged ...
>
> You've given your perceptions but not any measurements.  To obtain
> measurements, downgrade to the previous configuration and do some tests.
> We have test scripts in pptpd and pptp CVS.

  Unfortunatly I have no time at the moment to do any tests. For now I will be 
with my old setup. I'll get back to this subject during August.
  Thank you for your attention.

Regards,
Jorge Matias