Using PPPD pty option and script: controlling stdin buffer size?

Using PPPD pty option and script: controlling stdin buffer size?

[Arne Lie]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1254", Size: 1278 bytes --]

Hi,

We have a HW/SW set-up using Linux PPP over slow links, and have recently abandoned a solution with a dedicated kernel module driver for our specific hardware, and are now trying a pure user-mode solution using the pty option in pppd. Most if its performance is quite OK, but we see that we do not have complete control of the buffering along the packet signal path. From the PPPD MAN page it says that pty script starts a pseudo-tty instance, and that our script shall read data from it as it was from stdin using read(0,...). THIS WORKS. HOWEVER, since our link is slow we have implemented flow control, and in periods we do not read from stdio. In that period data builds up in stdio, but we see no sigh of data being dropped in the ppp0 interface queue (which is set to 3 packets as is the default value). 

Question: It seems that the stdin interface has a large buffer, much larger than we appreciate. Is there any way we can configure its size, making it smaller?

Best regards,


Arne Lie
SINTEF ICT, Dept. of Communication Systems
NO-7465 Trondheim, NORWAY
Dir. phone +47 73 59 27 03 - mobile +47 450 20 040
e-mail: arne.lie@sintef.no 

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þšiþ)í…æèw*\x1fjg¬±¨\x1e¶‰šŽŠÝ¢jÿ¾\a«þG«éÿ¢¸\f¢·¦j:+v‰¨ŠwèjØm¶Ÿÿþø\x1e¯ù\x1e®w¥þŠàþf£¢·hšâúÿ†Ù¥

Re: Using PPPD pty option and script: controlling stdin buffer size?

[walter harms]

I am not sure if i understand you correctly,
but can change the bufferhandling of stdio by setting the buffer size
using setbuf() (man 3 setbuf)

re,
 wh


Arne Lie schrieb:
> Hi,
> 
> We have a HW/SW set-up using Linux PPP over slow links, and have recently abandoned a solution with a dedicated kernel module driver for our specific hardware, and are now trying a pure user-mode solution using the pty option in pppd. Most if its performance is quite OK, but we see that we do not have complete control of the buffering along the packet signal path. From the PPPD MAN page it says that pty script starts a pseudo-tty instance, and that our script shall read data from it as it was from stdin using read(0,...). THIS WORKS. HOWEVER, since our link is slow we have implemented flow control, and in periods we do not read from stdio. In that period data builds up in stdio, but we see no sigh of data being dropped in the ppp0 interface queue (which is set to 3 packets as is the default value). 
> 
> Question: It seems that the stdin interface has a large buffer, much larger than we appreciate. Is there any way we can configure its size, making it smaller?
> 
> Best regards,
> 
> 
> Arne Lie
> SINTEF ICT, Dept. of Communication Systems
> NO-7465 Trondheim, NORWAY
> Dir. phone +47 73 59 27 03 - mobile +47 450 20 040
> e-mail: arne.lie@sintef.no 
> 
> N�����r��y���b�X��ǧv�^�)޺{.n�+����{���b��^n�r��z�\x1a��h����&��\x1e�G���h�\x03(�階�ݢj"��\x1a�^[m�����z�ޖ���f���h���~�mml=

RE: Using PPPD pty option and script: controlling stdin buffer size?

[Arne Lie]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1254", Size: 2104 bytes --]

Walter,

thanks, this could be the trick yes, we will have a look at that one!

/Arne

> -----Original Message-----
> From: walter harms [mailto:wharms@bfs.de]
> Sent: 12. oktober 2010 16:31
> To: Arne Lie
> Cc: ppp Linux
> Subject: Re: Using PPPD pty option and script: controlling stdin buffer
> size?
> 
> I am not sure if i understand you correctly,
> but can change the bufferhandling of stdio by setting the buffer size
> using setbuf() (man 3 setbuf)
> 
> re,
>  wh
> 
> 
> Arne Lie schrieb:
> > Hi,
> >
> > We have a HW/SW set-up using Linux PPP over slow links, and have
> recently abandoned a solution with a dedicated kernel module driver for
> our specific hardware, and are now trying a pure user-mode solution
> using the pty option in pppd. Most if its performance is quite OK, but
> we see that we do not have complete control of the buffering along the
> packet signal path. From the PPPD MAN page it says that pty script
> starts a pseudo-tty instance, and that our script shall read data from
> it as it was from stdin using read(0,...). THIS WORKS. HOWEVER, since
> our link is slow we have implemented flow control, and in periods we do
> not read from stdio. In that period data builds up in stdio, but we see
> no sigh of data being dropped in the ppp0 interface queue (which is set
> to 3 packets as is the default value).
> >
> > Question: It seems that the stdin interface has a large buffer, much
> larger than we appreciate. Is there any way we can configure its size,
> making it smaller?
> >
> > Best regards,
> >
> >
> > Arne Lie
> > SINTEF ICT, Dept. of Communication Systems
> > NO-7465 Trondheim, NORWAY
> > Dir. phone +47 73 59 27 03 - mobile +47 450 20 040
> > e-mail: arne.lie@sintef.no
> >
> >
> N�����r��y���b�X��ǧv�^�)޺{.n�+����{���b��^n�r��z�\x1a��h����&��\x1e�G���h�\x03(�
> 階�ݢj"��\x1a�^[m�����z�ޖ���f���h���~�mml=
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þšiþ)í…æèw*\x1fjg¬±¨\x1e¶‰šŽŠÝ¢jÿ¾\a«þG«éÿ¢¸\f¢·¦j:+v‰¨ŠwèjØm¶Ÿÿþø\x1e¯ù\x1e®w¥þŠàþf£¢·hšâúÿ†Ù¥

Re: Using PPPD pty option and script: controlling stdin buffer size?

[James Carlson]

Arne Lie wrote:
> Walter,
> 
> thanks, this could be the trick yes, we will have a look at that one!
> 
> /Arne
> 
>> -----Original Message-----
>> From: walter harms [mailto:wharms@bfs.de]
>> Sent: 12. oktober 2010 16:31
>> To: Arne Lie
>> Cc: ppp Linux
>> Subject: Re: Using PPPD pty option and script: controlling stdin buffer
>> size?
>>
>> I am not sure if i understand you correctly,
>> but can change the bufferhandling of stdio by setting the buffer size
>> using setbuf() (man 3 setbuf)

That only does something if you're reading through the stdio functions
(e.g., fread(3C)), and not through the recommended section low-level
read(2) function.  Using stdio to read/write from the master side of a
pty pair would be ... odd.

If you're seeing unwanted buffering here, I think you'll need to modify
either the kernel's pty subsystem or the PPP line discipline that it's
using.

I wouldn't recommend doing this, though.  Buffering is generally a good
thing.  If your application doesn't do sufficient congestion control to
avoid getting unacceptably stuck behind a long queue in the middle of
the network, then I'd call that a bug in the application, rather than a
problem with the network interface.

-- 
James Carlson         42.703N 71.076W         <carlsonj@workingcode.com>

RE: Using PPPD pty option and script: controlling stdin buffer size?

[Arne Lie]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1254", Size: 2468 bytes --]

> -----Original Message-----
> From: James Carlson [mailto:carlsonj@workingcode.com]
> Sent: 12. oktober 2010 17:19
> To: Arne Lie
> Cc: wharms@bfs.de; ppp Linux
> Subject: Re: Using PPPD pty option and script: controlling stdin buffer
> size?
> 
> Arne Lie wrote:
> > Walter,
> >
> > thanks, this could be the trick yes, we will have a look at that one!
> >
> > /Arne
> >
> >> -----Original Message-----
> >> From: walter harms [mailto:wharms@bfs.de]
> >> Sent: 12. oktober 2010 16:31
> >> To: Arne Lie
> >> Cc: ppp Linux
> >> Subject: Re: Using PPPD pty option and script: controlling stdin
> buffer
> >> size?
> >>
> >> I am not sure if i understand you correctly,
> >> but can change the bufferhandling of stdio by setting the buffer
> size
> >> using setbuf() (man 3 setbuf)
> 
> That only does something if you're reading through the stdio functions
> (e.g., fread(3C)), and not through the recommended section low-level
> read(2) function.  Using stdio to read/write from the master side of a
> pty pair would be ... odd.
[Arne::] "Stdio" was a mistyping from my side: I meant "stdin" and using 
low-level read all the way.
> 
> If you're seeing unwanted buffering here, I think you'll need to modify
> either the kernel's pty subsystem or the PPP line discipline that it's
> using.
> 
> I wouldn't recommend doing this, though.  Buffering is generally a good
> thing.  If your application doesn't do sufficient congestion control to
> avoid getting unacceptably stuck behind a long queue in the middle of
> the network, then I'd call that a bug in the application, rather than a
> problem with the network interface.
> 
> --
> James Carlson         42.703N 71.076W
> <carlsonj@workingcode.com>

[Arne::] With all the respect, congestion control needs normally packet loss
due to traffic overload to respond correctly, yes? With the large buffering 
of stdin this do not happen before long... We're dealing with underwater 
networks with severely slow links and
large propagation delays. We need relative short buffers to avoid seeing too
big RTTs. Default PPP interfaces are set up with txqueuelen = 3, which is fine.
However, when the *effective* queue size is this plus the stdin buffering, which
we do not know the size of in bytes, but is in order of minutes (!) in RTT for such 
slow links, then it starts to get an annoyance. 
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þšiþ)í…æèw*\x1fjg¬±¨\x1e¶‰šŽŠÝ¢jÿ¾\a«þG«éÿ¢¸\f¢·¦j:+v‰¨ŠwèjØm¶Ÿÿþø\x1e¯ù\x1e®w¥þŠàþf£¢·hšâúÿ†Ù¥

Re: Using PPPD pty option and script: controlling stdin buffer size?

[James Carlson]

Arne Lie wrote:
> [Arne::] With all the respect, congestion control needs normally packet loss
> due to traffic overload to respond correctly, yes?

No.  You've also got measurable delay, which I think is the key to
solving this problem.

> With the large buffering
> of stdin this do not happen before long... We're dealing with underwater 
> networks with severely slow links and
> large propagation delays. We need relative short buffers to avoid seeing too
> big RTTs. Default PPP interfaces are set up with txqueuelen = 3, which is fine.
> However, when the *effective* queue size is this plus the stdin buffering, which
> we do not know the size of in bytes, but is in order of minutes (!) in RTT for such 
> slow links, then it starts to get an annoyance. 

What would you do if the "excessive" buffering were in some box in the
middle of the network?  For many older systems, transmit queues of 50
packets or more are not at all uncommon.

I expect that an algorithm that tracks both round-trip latency and
throughput (using feedback from the peer) should be able to detect when
increasing the send rate only causes latency to go up and doesn't change
throughput.  That's the optimal rate.

If you feel like hacking your kernel anyway then good luck with that,
but I suspect you'll be back to this problem in the future.

-- 
James Carlson         42.703N 71.076W         <carlsonj@workingcode.com>

RE: Using PPPD pty option and script: controlling stdin buffer size?

[Arne Lie]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1254", Size: 2255 bytes --]

> -----Original Message-----
> From: James Carlson [mailto:carlsonj@workingcode.com]
> Sent: 13. oktober 2010 14:09
> To: Arne Lie
> Cc: wharms@bfs.de; ppp Linux
> Subject: Re: Using PPPD pty option and script: controlling stdin buffer
> size?
> 
> Arne Lie wrote:
> > [Arne::] With all the respect, congestion control needs normally
> packet loss
> > due to traffic overload to respond correctly, yes?
> 
> No.  You've also got measurable delay, which I think is the key to
> solving this problem.
> 
> > With the large buffering
> > of stdin this do not happen before long... We're dealing with
> underwater
> > networks with severely slow links and
> > large propagation delays. We need relative short buffers to avoid
> seeing too
> > big RTTs. Default PPP interfaces are set up with txqueuelen = 3,
> which is fine.
> > However, when the *effective* queue size is this plus the stdin
> buffering, which
> > we do not know the size of in bytes, but is in order of minutes (!)
> in RTT for such
> > slow links, then it starts to get an annoyance.
> 
> What would you do if the "excessive" buffering were in some box in the
> middle of the network?  For many older systems, transmit queues of 50
> packets or more are not at all uncommon.
> 
> I expect that an algorithm that tracks both round-trip latency and
> throughput (using feedback from the peer) should be able to detect when
> increasing the send rate only causes latency to go up and doesn't
> change
> throughput.  That's the optimal rate.
> 
> If you feel like hacking your kernel anyway then good luck with that,
> but I suspect you'll be back to this problem in the future.
[Arne::] Thanks for your input, James. Kernel hacking is out of the question, I guess... ;-) We do however already have an alternative, where we use a specially developed kernel module sw as "tty device driver", and where we have better control over the buffering. It has not proven perfectly stable though for all possible Linux platforms, but might be a better option for further refinements/debugging, than the pure user mode solution that we have discussed here.
 thanks a lot,
Arne
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þšiþ)í…æèw*\x1fjg¬±¨\x1e¶‰šŽŠÝ¢jÿ¾\a«þG«éÿ¢¸\f¢·¦j:+v‰¨ŠwèjØm¶Ÿÿþø\x1e¯ù\x1e®w¥þŠàþf£¢·hšâúÿ†Ù¥