Underruns (borken pipes)

Underruns (borken pipes)

[Bruce Paterson]

I got around my previous problems (readn) in an output/capture
application with an envy24
(ice1712) by bypassing the plugin and going straight to hw:0,0. Had to
change my code a bit
but it's now **almost** working.

I'm now running into underrun problems...occasionally.
I am outputting and capturing at 32 bits, 96000 on 4 input channels and
1 output (but of course
using hw:0,0 its actually 10 output & 12 input hard-wired). All IO is
being stored in RAM buffers
and I'm not running into swap.

Is my machine simply not fast enough ? (700MHz Pentium, 500M RAM) Help
!!!

What I don't understand is this:

If I put write & read in a loop, with a small 10ms delay, setup both in
& out to be non-blocking, I
get only occasional underruns. Underruns are unacceptable in my
application so I have to completely
abort. About every 2nd call to read or write gets an EAGAIN, as you
would expect.

If, however, I use file descriptor polling before read and write; don't
attempt a write till
POLLOUT, or a read until POLLIN or POLLPRI, but have the whole lot in a
big loop so I handle
either as soon as they require, I get failures EVERY time I run.
Somewhere about the 5th
hardware buffer iteration. Underrun is randomly on read or write. Doing
the code "properly"
seems to be far worse !
I never get EAGAIN from read/write in this case, and I haven't seen a
poll failure yet.

Output frame buffer size is 6553, and input 5461, as returned by the
hardware.

As far as I can see I'm roughly complying with the example code
/test/pcm.c with transfer method
"write and wait for room in buffer using poll"
-- 
Cheers,
Bruce
-------------------------------------------------------------------
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom
they are addressed. If you have received this email in error please
notify the system manager.

    /\\\/\\\/\\\    /   /      Bruce Paterson          
   /  \\\ \\\ \\\  /   /    Senior Design Engineer
  /   /\\\/\\\/\\\/   /   87 Peters Ave, Mulgrave, Vic, 3170
 /   /  \\\ \\\ \\\  /  PO Box 4112, Mulgrave, Vic, 3170, Australia
/   /    \\\/\\\ \\\/   Ph: +61 3 8561 4232   Fax: +61 3 9560 9055
      Tele-IP Ltd.      Email: bruce@tele-ip.com    Icq: #32015991
                        WWW:   http://www.tele-ip.com       VK3TJN
-------------------------------------------------------------------


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
PC Mods, Computing goodies, cases & more
http://thinkgeek.com/sf

Re: Underruns (borken pipes)

[Paul Davis]

>I got around my previous problems (readn) in an output/capture
>application with an envy24
>(ice1712) by bypassing the plugin and going straight to hw:0,0. Had to
>change my code a bit
>but it's now **almost** working.

i humbly suggest that you:

  (1) read the source code for JACK's ALSA driver/client
  (2) consider using JACK

the first suggestion will reveal quite a bit. you additionally will
need to know that you can't avoid underruns with small buffer sizes
unless you run with SCHED_FIFO or SCHED_RR scheduling, and not without
a kernel with Andrew Morton's low latency patches. you didn't say what
buffer sizes you were using, so its hard to know if this is an issue.

the second suggestion will avoid you having to deal with any of this
nonsense altogether, and get focused on the core part of your
application.

http://jackit.sf.net/

--p


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Gadgets, caffeine, t-shirts, fun stuff.
http://thinkgeek.com/sf

Re: Underruns (borken pipes)

[Bruce Paterson]

Paul Davis wrote:
> 
> >I got around my previous problems (readn) in an output/capture
> >application with an envy24
> >(ice1712) by bypassing the plugin and going straight to hw:0,0. Had to
> >change my code a bit
> >but it's now **almost** working.
> 
> i humbly suggest that you:
> 
>   (1) read the source code for JACK's ALSA driver/client
>   (2) consider using JACK

I did, but the Jack introduction frankly scared me off a bit. I'm sure
that it's not its
intention, but the talk about necessary mods to the kernel and
synchronisation issues between
channels (I&O) seemed a huge amount of bother to go to at the time.
Maybe (based on your comments)
it's necessary evil afterall anyway.

> the first suggestion will reveal quite a bit. you additionally will
> need to know that you can't avoid underruns with small buffer sizes
> unless you run with SCHED_FIFO or SCHED_RR scheduling, and not without

You know nothing in the alsa docs even hints at this, well certainly not
in all the
PCM bits I went through. Actually it IS working 90% of the time at
present, as long
as; 
a) I leave lots of RAM (don't run X etc)
b) I bash away at write/read regardless every 10ms in my hacky version
of code (the "poll" version
is a disaster)
Could you possibly give me some pointers as to where to look in the Jack
code ? I am totally
unfamilier with it.

> a kernel with Andrew Morton's low latency patches. you didn't say what
> buffer sizes you were using, so its hard to know if this is an issue.

I did actually:
>>Output frame buffer size is 6553, and input 5461, as returned by the
>>hardware.
Nothing bigger can be assigned.

I have my own frame buffer for tx of full channel width 10, and another
for rx of channel
width 12. Each is only the frame sizes above only.
I have on top of that my own buffers (non interleaved) for my 4 capture
channels and 1
output channel. These are typically 40s at 96000 long * 32bits....all
are preallocated.
I copy from these in/out of the tx/rx frame buffers prior/after a
sucessful write/read
call.

Also, if my plughw:0,0 interface to the envy24 doesn't work (segfault in
the plug code),
whereas the hw:0,0 does, will Jack even work ??
-- 
Cheers,
Bruce
-------------------------------------------------------------------
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom
they are addressed. If you have received this email in error please
notify the system manager.

    /\\\/\\\/\\\    /   /      Bruce Paterson          
   /  \\\ \\\ \\\  /   /    Senior Design Engineer
  /   /\\\/\\\/\\\/   /   87 Peters Ave, Mulgrave, Vic, 3170
 /   /  \\\ \\\ \\\  /  PO Box 4112, Mulgrave, Vic, 3170, Australia
/   /    \\\/\\\ \\\/   Ph: +61 3 8561 4232   Fax: +61 3 9560 9055
      Tele-IP Ltd.      Email: bruce@tele-ip.com    Icq: #32015991
                        WWW:   http://www.tele-ip.com       VK3TJN
-------------------------------------------------------------------


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Gadgets, caffeine, t-shirts, fun stuff.
http://thinkgeek.com/sf

Re: Underruns (borken pipes)

[Paul Davis]

>I did, but the Jack introduction frankly scared me off a bit. I'm
>sure that it's not its intention, but the talk about necessary mods
>to the kernel and synchronisation issues between channels (I&O)
>seemed a huge amount of bother to go to at the time.

no kernel mods are necessary for JACK. kernel mods (1 patch) are
necessary if you want really low latency, but this is true for ALSA
and OSS as well.

the channel synchronisation issue is a basic design feature. when run
in full-duplex mode, JACK wants its clients to be able to do both
input and output in the same amounts at the same time (rather than say
"you can do N frames of input now" and then later "you can do M frames
of output now" - most programs would find this very difficult to
do; the other alternative is to provide much smaller frame counts to
the process() callback (ie. the smaller of the current data/space
available for playback/capture at a given interrupt), but this is much
less efficient since it means you have to invoke the clients much more
often.)

this design is based (like a lot of JACK's API) on the way just about
every non-Unix audio API works. it also models the way all the better
hardware works. and finally, although JACK works on most
consumer-level software, it wasn't designed as a general purpose
consumer sound server. if you are using it on crappy h/w, yes, this
channel sync issue will cause problems. you have a choice: you can
either write a custom ALSA interface that handles the lack of full
duplex sync, or you can get a decent audio interface for US$60 or so
and then be able to hook into the other applications (more added every
week) that use JACK. you're in luck: you already have a decent interface.

>> the first suggestion will reveal quite a bit. you additionally will
>> need to know that you can't avoid underruns with small buffer sizes
>> unless you run with SCHED_FIFO or SCHED_RR scheduling, and not without
>
>You know nothing in the alsa docs even hints at this, well certainly
>not in all the PCM bits I went through. Actually it IS working 90% of
>the time at present, as long

precisely what would be expected. its not the average case scheduling
latency that is bad in the regular kernel, its the worst case. thats
why it works 90% of the time.

its not documented in the alsa docs because almost nothing is
documented in the alsa docs. :) :(
 
>Could you possibly give me some pointers as to where to look in the
>Jack code ? I am totally unfamilier with it.

the relevant code is all in the ALSA driver/client, which lives in
jack/drivers/alsa/alsa_driver.c. what's there is fairly complex,
because of the full duplex requirement and the use of poll/mmap
access. however, it works very well, at least on soundcards that have
their playback and capture streams pretty well in sync (i.e. the same
requirement needed for ASIO and probably EASI as well).

>> a kernel with Andrew Morton's low latency patches. you didn't say what
>> buffer sizes you were using, so its hard to know if this is an issue.
>
>I did actually:
>>>Output frame buffer size is 6553, and input 5461, as returned by the
>>>hardware.
>Nothing bigger can be assigned.

sorry, i missed that. if thats in frames, they are pretty big, and
you're just facing the inevitable poor scheduling latency of the
regular linux kernel. if its in bytes, its still fairly large. they
are really odd values however, and they would normally be power-of-2
figures no matter what the unit.

>Also, if my plughw:0,0 interface to the envy24 doesn't work (segfault
>in the plug code), whereas the hw:0,0 does, will Jack even work ??

since your application doesn't have any control over the hardware, and
since JACK is known to work on ice1712-based hardware, there should be
no problem.

--p


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: Underruns (borken pipes)

[Bruce Paterson]

Paul Davis wrote:
> 
> this design is based (like a lot of JACK's API) on the way just about
> every non-Unix audio API works. it also models the way all the better

OK. I believe I now understand the tradeoffs. Also, it was not clear to
me that the Jack
audio driver would solve my problems since most of what Jack is about is
allowing multiple
audio clients (which I don't have, and in fact don't want at all in this
case).

> >You know nothing in the alsa docs even hints at this, well certainly
> >not in all the PCM bits I went through. Actually it IS working 90% of
> >the time at present, as long
> 
> precisely what would be expected. its not the average case scheduling
> latency that is bad in the regular kernel, its the worst case. thats
> why it works 90% of the time.

Would you care to speculate that the reason it works better *without*
poll is that
on average I'm stuffing something into the buffers before they are
totally empty
(which would be the case for poll), so the impact of a long system
latency
just prior to stuffing is less since there's a bit more left in the
hardware buffer ?

I suspect writing less in each time won't help because the behaviour of
'poll' remains
unchanged only reporting when the hardware buffer is empty (too late !).

> its not documented in the alsa docs because almost nothing is
> documented in the alsa docs. :) :(

I noticed !
 
> >Could you possibly give me some pointers as to where to look in the
> >Jack code ? I am totally unfamilier with it.
> 
> the relevant code is all in the ALSA driver/client, which lives in
> jack/drivers/alsa/alsa_driver.c. what's there is fairly complex,
> because of the full duplex requirement and the use of poll/mmap
> access. however, it works very well, at least on soundcards that have
> their playback and capture streams pretty well in sync (i.e. the same
> requirement needed for ASIO and probably EASI as well).

ok, will dive in when & if necessary.

> >> a kernel with Andrew Morton's low latency patches. you didn't say what
> >> buffer sizes you were using, so its hard to know if this is an issue.
> >
> >I did actually:
> >>>Output frame buffer size is 6553, and input 5461, as returned by the
> >>>hardware.
> >Nothing bigger can be assigned.
> 
> sorry, i missed that. if thats in frames, they are pretty big, and

Yes, in frames. frames size is the number of hardware channels (10in /
12out).

> you're just facing the inevitable poor scheduling latency of the

...and this would be fixed by jack driver style code, or would require
the low
latency kernel patch as well ?  No point going to Jack at this stage if
it's not
going to help.  
The fact that it's much better without Xwindows supports the occasional
latency 
problem.

> regular linux kernel. if its in bytes, its still fairly large. they
> are really odd values however, and they would normally be power-of-2
> figures no matter what the unit.

Ah, but if you divide 65536 by 10 and 12 respectively (the frame widths)
and
round down those are the numbers you get ! (I just worked that out
then). 
 
> >Also, if my plughw:0,0 interface to the envy24 doesn't work (segfault
> >in the plug code), whereas the hw:0,0 does, will Jack even work ??
> 
> since your application doesn't have any control over the hardware, and
> since JACK is known to work on ice1712-based hardware, there should be
> no problem.

Hmmm ok. Maybe the alsa 'plug' normal read/write code is screwed but the
mmap read/write used by Jack is fine. That's the only theory I can can
up with, 
since I'm sure Jack uses non-interleaved buffers.


-- 
Cheers,
Bruce
-------------------------------------------------------------------
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom
they are addressed. If you have received this email in error please
notify the system manager.

    /\\\/\\\/\\\    /   /      Bruce Paterson          
   /  \\\ \\\ \\\  /   /    Senior Design Engineer
  /   /\\\/\\\/\\\/   /   87 Peters Ave, Mulgrave, Vic, 3170
 /   /  \\\ \\\ \\\  /  PO Box 4112, Mulgrave, Vic, 3170, Australia
/   /    \\\/\\\ \\\/   Ph: +61 3 8561 4232   Fax: +61 3 9560 9055
      Tele-IP Ltd.      Email: bruce@tele-ip.com    Icq: #32015991
                        WWW:   http://www.tele-ip.com       VK3TJN
-------------------------------------------------------------------


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: Underruns (borken pipes)

[Paul Davis]

>OK. I believe I now understand the tradeoffs. Also, it was not clear
>to me that the Jack audio driver would solve my problems since most
>of what Jack is about is allowing multiple audio clients (which I
>don't have, and in fact don't want at all in this case).

Thats not entirely true. At least 30-50% of JACK's design comes from
the desire to have a simple, highly abstracted, highly efficient model
for audio i/o. 95%+ of the time that i use JACK, its with one client
(for now, anyway).

>Would you care to speculate that the reason it works better *without*
>poll is that on average I'm stuffing something into the buffers
>before they are totally empty (which would be the case for poll), so
>the impact of a long system latency just prior to stuffing is less
>since there's a bit more left in the hardware buffer ?

well, there's that aspect of things, but just as likely is the
behaviour of the kernel is scheduling certain things, which is
erratic, and though deterministic in a strict sense, tends to feel
mostly random.

>I suspect writing less in each time won't help because the behaviour
>of 'poll' remains unchanged only reporting when the hardware buffer
>is empty (too late !).

no, poll reports back to you when the amount of data/space equals or
exceeds the "avail_min" count set by your program.

>> you're just facing the inevitable poor scheduling latency of the
>
>...and this would be fixed by jack driver style code, or would
>require the low latency kernel patch as well ?  No point going to
>Jack at this stage if it's not going to help.  The fact that it's
>much better without Xwindows supports the occasional latency problem.

the problems you have will not be solved by JACK alone. however, you
will find that JACK's design and what it forces on your application,
will provide some assistance in this area.

--p


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: Underruns (borken pipes)

[Bruce Paterson]

Paul Davis wrote:
> 
> >OK. I believe I now understand the tradeoffs. Also, it was not clear
> >to me that the Jack audio driver would solve my problems since most
> >of what Jack is about is allowing multiple audio clients (which I
> >don't have, and in fact don't want at all in this case).
> 
> Thats not entirely true. At least 30-50% of JACK's design comes from
> the desire to have a simple, highly abstracted, highly efficient model
> for audio i/o. 95%+ of the time that i use JACK, its with one client
> (for now, anyway).

ok. 
[Might be worth doing a re-visit to your intro to Jack...add a line
about
the above.] 

> no, poll reports back to you when the amount of data/space equals or
> exceeds the "avail_min" count set by your program.

AHHHHHHHHHH !!!!!  Click ! :-)

It now works fine with polling once I set the avail_min to 1/8 the max
size of 
the frame buffer. 
Since it's working fine I'll leave it for now, but if we need to support
diff.
soundcards etc. in the future Jack may be worth a visit.

-- 
Cheers,
Bruce
-------------------------------------------------------------------
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom
they are addressed. If you have received this email in error please
notify the system manager.

    /\\\/\\\/\\\    /   /      Bruce Paterson          
   /  \\\ \\\ \\\  /   /    Senior Design Engineer
  /   /\\\/\\\/\\\/   /   87 Peters Ave, Mulgrave, Vic, 3170
 /   /  \\\ \\\ \\\  /  PO Box 4112, Mulgrave, Vic, 3170, Australia
/   /    \\\/\\\ \\\/   Ph: +61 3 8561 4232   Fax: +61 3 9560 9055
      Tele-IP Ltd.      Email: bruce@tele-ip.com    Icq: #32015991
                        WWW:   http://www.tele-ip.com       VK3TJN
-------------------------------------------------------------------


-------------------------------------------------------
This sf.net email is sponsored by: Jabber - The world's fastest growing 
real-time communications platform! Don't just IM. Build it in! 
http://www.jabber.com/osdn/xim