[RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

Hi everybody.

Linux provides a USB Audio Class (UAC) driver on the USB host side, but a UAC 
gadget driver on the USB device side is currently missing.

Considering plugging in this hole, I'm looking at userspace <-> kernelspace 
APIs.

In a nutshell (let's consider recording, where the device appears to the host 
as a USB microphone), the kernel driver needs to be fed with audio data from a 
userspace application and stream those data on USB. Audio data can come from a 
real microphone (through an ALSA driver), a file, a network socket, ... the 
driver will not care.

I need an API to transfer audio data from userspace to kernelspace. I 
initially thought about ALSA, but it turns out some assumptions made by ALSA 
are not fulfilled by my system. One of the most serious problems is that the 
UAC gadget driver doesn't have any audio clock. The only hardware clock 
available is the USB device controller interrupts generated at the USB 
transfer rate, and those are much faster than the audio sample rate. This will 
cause buffer underruns that I need to handle.

I can thing of 3 solutions to solve this issue, and I'd like your opinion on 
them.

1. Get the userspace application to provide a clock to the driver. This could 
be done by writing to a /proc file, a special char device, or even by adding 
an ioctl to ALSA. Every time the driver receives a clock tick it would 
transfer one period of data.

2. Find out how much data is available in the ALSA ring buffer. I could then 
throttle USB interrupts when no data is available in the buffer and try again 
later. ALSA maintains a read and a write pointer (hw_ptr and appl_ptr) that 
can be accessed through the snd_pcm_*_avail() functions. However, I'm not sure 
how to handle proper locking, and I don't understand how runtime->boundary 
interacts with those functions. I've also heard that there's a free-running 
mode that will make those functions return bogus values. I'd appreciate help 
on this.

3. Use another API (dedicated character device, socket, ...) to transfer audio 
data from userspace to kernelspace. ALSA is way too big for my requirements (2 
layers of abstraction, architectured around assumptions that make sense for 
real sound cards but not for my hardware, ...). On the other hand, it would be 
handy to use existing ALSA applications to driver the device.

I'd appreciate comments regarding the API choice. How do you all expect to 
push data from a userspace application to a UAC driver on the device side ? If 
ALSA is *the* way, is one of the above solutions practical or is there a 
better way ?

Best regards,

Laurent Pinchart

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Hal Murray]

> I need an API to transfer audio data from userspace to kernelspace. I
> initially thought about ALSA, but it turns out some assumptions made
> by ALSA  are not fulfilled by my system. One of the most serious
> problems is that the  UAC gadget driver doesn't have any audio clock.
> The only hardware clock  available is the USB device controller
> interrupts generated at the USB  transfer rate, and those are much
> faster than the audio sample rate. This will  cause buffer underruns
> that I need to handle. 

You don't need a clock.  The data will come to you at the right rate.  All 
you need to do is pass it on when you have enough to fill up a buffer.  The 
buffer size is fixed.  It's part of the spec for the device you are emulating.

Assume that you get samples one at a time each time the source clock ticked.  
The normal state of your system would be to have a buffer that is partially 
filled.  When a new sample fills up the buffer, you would move it from the 
input side to the ready-for-USB queue and setup a new buffer for future input 
samples.

Soon the USB side will read the queued buffer, you free that buffer and you 
are now back to the normal state of collecting input data.

You are in trouble if that doesn't happen before the nest buffer is ready, 
that is the ready-for-USB queue should normally be empty.  If you can't keep 
it empty it will eventually overflow.  Short chunks of time where the queue 
builds up might be OK.  For debugging, you should probably count them.

Now consider the case where you get several samples at a time rather than 
one.  Logically, copy them over one at a time.  That turns into a burst of 
source clocks, but the average will work out right.  Your ready-for-USB queue 
should still be empty most of the time.

If that doesn't make sense, I'll try again.  Consider something like audio 
over a network.  There is no audio clock on the network.  The receiver can 
derive the source clock by watching the data stream.





-- 
These are my opinions, not necessarily my employer's.  I hate spam.



--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

Hi Hal,

first of all, thanks for your answer.

On Thursday 14 May 2009 20:18:07 Hal Murray wrote:
> > I need an API to transfer audio data from userspace to kernelspace. I
> > initially thought about ALSA, but it turns out some assumptions made
> > by ALSA  are not fulfilled by my system. One of the most serious
> > problems is that the  UAC gadget driver doesn't have any audio clock.
> > The only hardware clock  available is the USB device controller
> > interrupts generated at the USB  transfer rate, and those are much
> > faster than the audio sample rate. This will  cause buffer underruns
> > that I need to handle.
>
> You don't need a clock.  The data will come to you at the right rate.  All
> you need to do is pass it on when you have enough to fill up a buffer.  The
> buffer size is fixed.  It's part of the spec for the device you are
> emulating.

I'm not emulating any device. The buffer size is up to me, and I actually have 
a fixed number of small buffers, but that shouldn't make a difference.

> Assume that you get samples one at a time each time the source clock
> ticked. The normal state of your system would be to have a buffer that is
> partially filled.  When a new sample fills up the buffer, you would move it
> from the input side to the ready-for-USB queue and setup a new buffer for
> future input samples.
>
> Soon the USB side will read the queued buffer, you free that buffer and you
> are now back to the normal state of collecting input data.
>
> You are in trouble if that doesn't happen before the nest buffer is ready,
> that is the ready-for-USB queue should normally be empty.  If you can't
> keep it empty it will eventually overflow.  Short chunks of time where the
> queue builds up might be OK.  For debugging, you should probably count
> them.
>
> Now consider the case where you get several samples at a time rather than
> one.  Logically, copy them over one at a time.  That turns into a burst of
> source clocks, but the average will work out right.  Your ready-for-USB
> queue should still be empty most of the time.
>
> If that doesn't make sense, I'll try again.  Consider something like audio
> over a network.  There is no audio clock on the network.  The receiver can
> derive the source clock by watching the data stream.

This is more or less what I'm trying to do, with the difference that I don't 
move samples from the ALSA ring buffer when they arrive but when the USB layer 
asks for more data.

My trouble is that I don't get notified when new samples are written to the 
ALSA ring buffer, and I'm not sure to find out how many samples are present in 
the buffer. If I could get some kind of "sample written" notification, or 
better, compute the size of data present in the ring buffer, my problem would 
be (mostly) solved. From what I understand, such a notification isn't possible 
when the ALSA ring buffer is mmap'ed, as ALSA itself doesn't get notified.

This makes the second approach (reading the number of available samples) 
better, except that some kind of "free-wheeling" mode makes it impossible 
under some circumstances. From what Takashi Iwai explained in a mail to Jon 
Smirl on alsa-devel ("appl_ptr and DMA overrun at end of stream"), "we have 
also mode when appl_ptr is not updated at all (when stop_threshold == 
boundary)". This free-wheeling mode seems to be used by dmix.

Best regards,

Laurent Pinchart

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Steve Calfee]

On Thu, May 14, 2009 at 1:58 PM, Laurent Pinchart
<laurent.pinchart-AgBVmzD5pcezQB+pC5nmwQ@public.gmane.org> wrote:
> Hi Hal,
>
> first of all, thanks for your answer.
>
> On Thursday 14 May 2009 20:18:07 Hal Murray wrote:
>> > I need an API to transfer audio data from userspace to kernelspace. I
>> > initially thought about ALSA, but it turns out some assumptions made
>> > by ALSA  are not fulfilled by my system. One of the most serious
>> > problems is that the  UAC gadget driver doesn't have any audio clock.
>> > The only hardware clock  available is the USB device controller
>> > interrupts generated at the USB  transfer rate, and those are much
>> > faster than the audio sample rate. This will  cause buffer underruns
>> > that I need to handle.
>>
>> You don't need a clock.  The data will come to you at the right rate.  All
>> you need to do is pass it on when you have enough to fill up a buffer.  The
>> buffer size is fixed.  It's part of the spec for the device you are
>> emulating.
>
> I'm not emulating any device. The buffer size is up to me, and I actually have
> a fixed number of small buffers, but that shouldn't make a difference.
>
Hi Laurent,

The problems might be clearer to you if you do the output side first.
You have to define one or more alt/interfaces with isoc OUT sizes to
match the sample rate. Some rates are straight forward and don't care
much about the buffer/packet sizes like 45,000 HZ. However the 44100
hz rate (and 22050 and 11025) made popular by CDs is interesting
because it doesn't divide by 1000, the FS frame rate. So a driver must
output 9 packets of 44 samples (times the number of bytes in sample
width times stereo/mono) and 1 of 45 samples. Typically USB chips
determine the sample rate by the interface and the number of bytes
coming per frame.

This is a challenging project, compounded by the audio usb class spec.
In my opinion audio is the worst, hardest to understand spec. Second
is the HID-PID spec.

Regards, Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

Hi Steve,

On Thursday 14 May 2009 23:12:41 Steve Calfee wrote:
> On Thu, May 14, 2009 at 1:58 PM, Laurent Pinchart
> > Hi Hal,
> >
> > first of all, thanks for your answer.
> >
> > On Thursday 14 May 2009 20:18:07 Hal Murray wrote:
> > > > I need an API to transfer audio data from userspace to kernelspace. I
> > > > initially thought about ALSA, but it turns out some assumptions made
> > > > by ALSA  are not fulfilled by my system. One of the most serious
> > > > problems is that the  UAC gadget driver doesn't have any audio clock.
> > > > The only hardware clock  available is the USB device controller
> > > > interrupts generated at the USB  transfer rate, and those are much
> > > > faster than the audio sample rate. This will  cause buffer underruns
> > > > that I need to handle.
> > >
> > > You don't need a clock.  The data will come to you at the right rate.
> > >  All you need to do is pass it on when you have enough to fill up a
> > > buffer.  The buffer size is fixed.  It's part of the spec for the device
> > > you are emulating.
> >
> > I'm not emulating any device. The buffer size is up to me, and I actually
> > have a fixed number of small buffers, but that shouldn't make a
> > difference.
>
> Hi Laurent,
>
> The problems might be clearer to you if you do the output side first.
> You have to define one or more alt/interfaces with isoc OUT sizes to
> match the sample rate. Some rates are straight forward and don't care
> much about the buffer/packet sizes like 45,000 HZ. However the 44100
> hz rate (and 22050 and 11025) made popular by CDs is interesting
> because it doesn't divide by 1000, the FS frame rate. So a driver must
> output 9 packets of 44 samples (times the number of bytes in sample
> width times stereo/mono) and 1 of 45 samples. Typically USB chips
> determine the sample rate by the interface and the number of bytes
> coming per frame.

The USB side is already implemented. I need an isochronous endpoint with a 
packet size just a bit bigger than what would be required by the sample rate. 
The amount of data per packet will vary (some might even be empty), but that 
won't be a problem.

I now need to fill the packets with data. When a packet has been successfully 
transferred on USB the USB device controller will notify me with an interrupt. 
I then have to copy audio data from the ALSA ring buffer to the USB request 
buffer and queue the request for transmission (a USB request, in the USB 
gadget device driver terminology, is a data structure that describes a USB 
transfer).

To copy data from the ALSA ring buffer I need to know how much data is 
available at that particular moment, in order to avoid buffer underruns. I'm 
still not sure how to do that.

> This is a challenging project, compounded by the audio usb class spec.
> In my opinion audio is the worst, hardest to understand spec. Second
> is the HID-PID spec.

Challenges are usually interesting :-) With a little luck part of the 
complexity will be handle outside the UAC gadget driver, either on the host 
side or by the userspace application on the device side.

Best regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [alsa-devel] [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Clemens Ladisch]

Laurent Pinchart wrote:
> To copy data from the ALSA ring buffer I need to know how much data is 
> available at that particular moment, in order to avoid buffer underruns.

This is not how ALSA drivers are supposed to work.  The framework was
designed for DMA devices that run asynchronously and just take data out
of the ring buffer whenever they need them.  Your driver should work in
the same way, i.e., you queue a packet of data when you need to queue
it.


Best regards,
Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

Hi Clemens,

thanks for your answer.

On Friday 15 May 2009 09:01:48 Clemens Ladisch wrote:
> Laurent Pinchart wrote:
> > To copy data from the ALSA ring buffer I need to know how much data is
> > available at that particular moment, in order to avoid buffer underruns.
>
> This is not how ALSA drivers are supposed to work.  The framework was
> designed for DMA devices that run asynchronously and just take data out
> of the ring buffer whenever they need them.  Your driver should work in
> the same way, i.e., you queue a packet of data when you need to queue
> it.

I know that ALSA is designed for asynchronous operation. This is fine for 
devices that provide an audio clock (all sound cards do), but leads to a few 
issues for devices that don't, like this one.

When streaming audio data from a file to USB, there is simply no audio clock 
in the system. When streaming audio data from a real audio device to USB, the 
audio clock comes from the real audio device and is beyond control of the USB 
Audio Class gadget driver.

I can see two ways around this.

1. Use an asynchronous isochronous endpoint to stream data. The USB Audio 
Class gadget driver will either have to be provided with a clock from 
userspace, or stream data as fast as it can, needing a way to check the amount 
of data present in the buffer (as explained in my previous e-mail).

2. Use a synchronous isochronous endpoint to stream data. The USB Audio Class 
gadget driver will derive its clock from the USB Start Of Frame clock and the 
sample rate, and send a fixed number of audio samples per packet. In this case 
the userspace application will have to resample audio data to match the source 
and sink rates. The issue with this solution is that I'm not sure if the USB 
stack provides the necessary information. I need to fill USB packets with 
audio data and queue them for transmission, but I don't know when they will be 
transmitted, making it difficult to adjust the amount of data per packet based 
on the sample rate.

As a side note, I'm not sure how difficult audio rate matching is, I suppose 
the userspace ALSA API provides enough information to match rates, but I'd 
appreciate if someone could confirm this (and possibly provide a few pointers 
to related documentation).

Best regards,

Laurent Pinchart


--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Clemens Ladisch]

Laurent Pinchart wrote:
> I know that ALSA is designed for asynchronous operation. This is fine for
> devices that provide an audio clock (all sound cards do), but leads to a few
> issues for devices that don't, like this one.
> 
> When streaming audio data from a file to USB, there is simply no audio clock
> in the system.

In this case, you can use the USB frame clock as audio clock, i.e.,
whenever one frame has been completed, you also submit audio data for
one USB frame (one millisecond).

This would be a synchronous endpoint.

> When streaming audio data from a real audio device to USB, the audio
> clock comes from the real audio device and is beyond control of the
> USB Audio Class gadget driver.

In this case, you have _two_ clocks, the clock of the actual audio
device, and the USB SOF clock.  What you need to know is the relative
speed of these clocks, because that determines the number of samples per
USB frame.

> I can see two ways around this.
> 
> 1. Use an asynchronous isochronous endpoint to stream data. The USB Audio 
> Class gadget driver will either have to be provided with a clock from 
> userspace, or stream data as fast as it can,

The latter is not allowed by the USB specification; the data rate must
change only gradually, i.e., the difference in successive frames must be
no more than +/- one sample.

> needing a way to check the amount of data present in the buffer (as
> explained in my previous e-mail).

The _current_ amount of valid data does not help because applications
can write data irregularly.  You do not want to measure how fast the
application writes data into your buffer, but the actual speed of the
actual audio device.

I'd guess that your gadget driver does not have access to the real audio
device, so it is the application that must do the measurement.


The ALSA API has no mechanism to tell the driver how fast to play, so it
looks as if you have to write you own device driver.


Best regards,
Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

Hi Clemens,

On Friday 15 May 2009 17:07:30 Clemens Ladisch wrote:
> Laurent Pinchart wrote:
> > I know that ALSA is designed for asynchronous operation. This is fine for
> > devices that provide an audio clock (all sound cards do), but leads to a
> > few issues for devices that don't, like this one.
> >
> > When streaming audio data from a file to USB, there is simply no audio
> > clock in the system.
>
> In this case, you can use the USB frame clock as audio clock, i.e.,
> whenever one frame has been completed, you also submit audio data for
> one USB frame (one millisecond).
>
> This would be a synchronous endpoint.

Using a synchronous endpoint in this case sounds easy. I'll probably do that.

> > When streaming audio data from a real audio device to USB, the audio
> > clock comes from the real audio device and is beyond control of the
> > USB Audio Class gadget driver.
>
> In this case, you have _two_ clocks, the clock of the actual audio
> device, and the USB SOF clock.  What you need to know is the relative
> speed of these clocks, because that determines the number of samples per
> USB frame.

If I use a synchronous endpoint, isn't the number of samples per frame 
determined by the nominal sampling rate and the nominal SOF frequency ? With 
the SOF clock running at 1kHz, I expect a synchronous endpoint for a 16 bits 
mono 48kHz stream to deliver exactly 48 frames (96 bytes) per USB frame.

> > I can see two ways around this.
> >
> > 1. Use an asynchronous isochronous endpoint to stream data. The USB Audio
> > Class gadget driver will either have to be provided with a clock from
> > userspace, or stream data as fast as it can,
>
> The latter is not allowed by the USB specification; the data rate must
> change only gradually, i.e., the difference in successive frames must be
> no more than +/- one sample.

Thanks for pointing this out.

> > needing a way to check the amount of data present in the buffer (as
> > explained in my previous e-mail).
>
> The _current_ amount of valid data does not help because applications
> can write data irregularly.  You do not want to measure how fast the
> application writes data into your buffer, but the actual speed of the
> actual audio device.
>
> I'd guess that your gadget driver does not have access to the real audio
> device, so it is the application that must do the measurement.

That's right. The gadget driver doesn't care if data comes from a file or a 
real audio device.

> The ALSA API has no mechanism to tell the driver how fast to play, so it
> looks as if you have to write you own device driver.

If I use a synchronous endpoint I can solve the problem on the driver side. 
There would still be a problem on the application side, as the audio source 
clock will obviously not be synchronous with the USB SOF clock. The userspace 
application will request audio data at a nominal rate equal to the nominal 
sample frequency advertised on USB, and will need to perform rate matching 
between its source (real audio device) and sink (USB driver). Is that (at 
least relatively) easy to do with ALSA ? I suppose this is a common use case 
for all audio streaming applications such as internet radios, as in that case 
audio data streamed over the network will not be synchronous with the sound 
card audio clock and thus need to be resampled in the audio player before 
being sent to the sound card.

Best regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Clemens Ladisch]

Laurent Pinchart wrote:
> On Friday 15 May 2009 17:07:30 Clemens Ladisch wrote:
> > Laurent Pinchart wrote:
> > > When streaming audio data from a real audio device to USB, the audio
> > > clock comes from the real audio device and is beyond control of the
> > > USB Audio Class gadget driver.
> >
> > In this case, you have _two_ clocks, the clock of the actual audio
> > device, and the USB SOF clock.  What you need to know is the relative
> > speed of these clocks, because that determines the number of samples per
> > USB frame.
> 
> If I use a synchronous endpoint, isn't the number of samples per frame
> determined by the nominal sampling rate and the nominal SOF frequency ?

Yes, but in _this_ case, the second case, you cannot use a synchronous
endpoint.  Since the real audio device controls the rate, this would be
an asynchronous endpoint.

> > I'd guess that your gadget driver does not have access to the real audio
> > device, so it is the application that must do the measurement.
> 
> That's right. The gadget driver doesn't care if data comes from a file or a
> real audio device.

I'd guess you don't want to restrict your gadget to reading from a file,
so you'll have to implement an asynchronous endpoint, and reading from
a file is just an uninteresting special case.

> > The ALSA API has no mechanism to tell the driver how fast to play, so it
> > looks as if you have to write you own device driver.
> 
> If I use a synchronous endpoint I can solve the problem on the driver side.
> There would still be a problem on the application side, as the audio source
> clock will obviously not be synchronous with the USB SOF clock. The userspace
> application will request audio data at a nominal rate equal to the nominal
> sample frequency advertised on USB, and will need to perform rate matching
> between its source (real audio device) and sink (USB driver). Is that (at
> least relatively) easy to do with ALSA ?

It is possible, but not trivially easy.

To measure the rate of the real audio device, you have to continually
(or regularly) correlate the sample position (derived from
snd_pcm_delay()) and some time stamp.  Using the computer's time would
introduce a third clock, so I'd guess you should use the USB SOF
interrupts as timer ticks (so your driver should notify the application
of SOFs).

With these measurements, you have the audio device rate, measured in
(i.e., relative to) the USB SOF rate.  With this relation, you can do
the rate matching; or you could use it to adjust the packet sizes of
the asynchronous endpoint, which I think is much easier.

> I suppose this is a common use case for all audio streaming
> applications such as internet radios, as in that case audio data
> streamed over the network will not be synchronous with the sound card
> audio clock and thus need to be resampled in the audio player before
> being sent to the sound card.

Or the player uses a buffer that is big enough so that there is enough
data even if the sound card runs faster than the server.  :-)


Best regards,
Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Hal Murray]

> If I use a synchronous endpoint, isn't the number of samples per frame
> determined by the nominal sampling rate and the nominal SOF frequency
> ? With  the SOF clock running at 1kHz, I expect a synchronous endpoint
> for a 16 bits  mono 48kHz stream to deliver exactly 48 frames (96
> bytes) per USB frame. 

Beware of the "exact" in there.  In real life, crystals have a tolerance.  
The USB clock will not match the audio clock perfectly.  What should be 1 kHz 
might be 1.00003 kHz or it might be 0.99997 kHz.

If the USB clock in this example is slightly fast, you will get occasional 
times when there are only 47 samples ready.  If it's slightly slow, you will 
occasionally have 49 samples.

That's slow/fast relative to the audio clock.  If the USB clock is 0.001% 
slow but the audio clock is 0.002% slow, then the USB clock will be fast for 
this discussion.


This is a common problem in communications.  There are several techniques to 
cope with it.

The simplest is to make the transport mechanism have a bit of extra bandwidth 
so you can always keep up.  It's just a matter of how often you use that 
extra bandwidth.  In this case, you would allocate enough bandwidth for 49 
samples and only use 48 most of the time.  An occasional USB frame would have 
47 or 49.


Another approach is to use big enough FIFO so you never get in trouble.  For 
this to work, you need a limit on the length of the data stream.  For 
example, suppose the longest song you want to send is 1000 seconds and both 
clocks are accurate to 50 ppm (parts per million).  The max difference in 
clocks is 100 ppm.  48 k samples/second * 1000 seconds is 48 million samples 
total.  100 ppm means that the worst difference would be 4800 samples.  So if 
the USB side waits until it gets 4800 samples before it starts forwarding 
data, it won't run out if the audio clock is worst-case slow and the USB 
clock is worst-case fast.  You also need buffering for another 4800 samples 
in case the clock differences are reversed.  This adds a delay to get started 
while you collect the 4800 samples.

You can just drop or duplicate a sample whenever the clocks slip.

You can derive both clocks from a common source.

You can lock one clock to the other with a PLL (Phase Locked Loop).

There are probably other approaches.


-- 
These are my opinions, not necessarily my employer's.  I hate spam.



--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

On Friday 15 May 2009 19:04:10 Clemens Ladisch wrote:
> Laurent Pinchart wrote:
> > On Friday 15 May 2009 17:07:30 Clemens Ladisch wrote:
> > > Laurent Pinchart wrote:
> > > > When streaming audio data from a real audio device to USB, the audio
> > > > clock comes from the real audio device and is beyond control of the
> > > > USB Audio Class gadget driver.
> > >
> > > In this case, you have _two_ clocks, the clock of the actual audio
> > > device, and the USB SOF clock.  What you need to know is the relative
> > > speed of these clocks, because that determines the number of samples
> > > per USB frame.
> >
> > If I use a synchronous endpoint, isn't the number of samples per frame
> > determined by the nominal sampling rate and the nominal SOF frequency ?
>
> Yes, but in _this_ case, the second case, you cannot use a synchronous
> endpoint.  Since the real audio device controls the rate, this would be
> an asynchronous endpoint.

Even if I resample the audio stream using the SOF clock ? As you pointed 
before, ALSA was designed around an asynchronous model. I need an audio clock 
to control the output of audio data. If I can't use the SOF clock, and as my 
driver can't access the audio input clock, the driver will not be able to copy 
data asynchronously from the ALSA ring buffer.

> > > I'd guess that your gadget driver does not have access to the real
> > > audio device, so it is the application that must do the measurement.
> >
> > That's right. The gadget driver doesn't care if data comes from a file or
> > a real audio device.
>
> I'd guess you don't want to restrict your gadget to reading from a file,
> so you'll have to implement an asynchronous endpoint, and reading from
> a file is just an uninteresting special case.

Then I need to either provide the driver with an audio clock from the 
userspace application (ALSA doesn't have any mechanism for that, as it assumes 
the device provides an audio clock), or check periodically how much data is 
available in the ALSA ring buffer. This is made even more complex by the fact 
that packets can vary by +/- 1 byte only according to your last e-mail.

> > > The ALSA API has no mechanism to tell the driver how fast to play, so
> > > it looks as if you have to write you own device driver.
> >
> > If I use a synchronous endpoint I can solve the problem on the driver
> > side. There would still be a problem on the application side, as the
> > audio source clock will obviously not be synchronous with the USB SOF
> > clock. The userspace application will request audio data at a nominal
> > rate equal to the nominal sample frequency advertised on USB, and will
> > need to perform rate matching between its source (real audio device) and
> > sink (USB driver). Is that (at least relatively) easy to do with ALSA ?
>
> It is possible, but not trivially easy.
>
> To measure the rate of the real audio device, you have to continually
> (or regularly) correlate the sample position (derived from
> snd_pcm_delay()) and some time stamp.  Using the computer's time would
> introduce a third clock, so I'd guess you should use the USB SOF
> interrupts as timer ticks (so your driver should notify the application
> of SOFs).
>
> With these measurements, you have the audio device rate, measured in
> (i.e., relative to) the USB SOF rate.  With this relation, you can do
> the rate matching; or you could use it to adjust the packet sizes of
> the asynchronous endpoint, which I think is much easier.

How can I adjust the asynchronous endpoint packet sizes in the driver if I 
can't access the audio clock there ?

> > I suppose this is a common use case for all audio streaming
> > applications such as internet radios, as in that case audio data
> > streamed over the network will not be synchronous with the sound card
> > audio clock and thus need to be resampled in the audio player before
> > being sent to the sound card.
>
> Or the player uses a buffer that is big enough so that there is enough
> data even if the sound card runs faster than the server.  :-)

Lazy players :-)

Best regards,

Laurent Pinchart

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

Hi Hal,

On Friday 15 May 2009 22:15:18 Hal Murray wrote:
> > If I use a synchronous endpoint, isn't the number of samples per frame
> > determined by the nominal sampling rate and the nominal SOF frequency
> > ? With  the SOF clock running at 1kHz, I expect a synchronous endpoint
> > for a 16 bits  mono 48kHz stream to deliver exactly 48 frames (96
> > bytes) per USB frame.
>
> Beware of the "exact" in there.  In real life, crystals have a tolerance.
> The USB clock will not match the audio clock perfectly.  What should be 1
> kHz might be 1.00003 kHz or it might be 0.99997 kHz.
>
> If the USB clock in this example is slightly fast, you will get occasional
> times when there are only 47 samples ready.  If it's slightly slow, you
> will occasionally have 49 samples.
>
> That's slow/fast relative to the audio clock.  If the USB clock is 0.001%
> slow but the audio clock is 0.002% slow, then the USB clock will be fast
> for this discussion.

Except that the SOF clock *is* the reference clock. It will be slower or 
faster than the audio clock, and the userspace application will have to 
perform sample rate matching so that packets will be exactly 48 bytes long.

> This is a common problem in communications.  There are several techniques
> to cope with it.
>
> The simplest is to make the transport mechanism have a bit of extra
> bandwidth so you can always keep up.  It's just a matter of how often you
> use that extra bandwidth.  In this case, you would allocate enough
> bandwidth for 49 samples and only use 48 most of the time.  An occasional
> USB frame would have 47 or 49.

That's what would happen with an asynchronous endpoint. The hard bit is to 
find out when to send that occasional packet, as the driver doesn't have 
access to the audio clock.

> Another approach is to use big enough FIFO so you never get in trouble. 
> For this to work, you need a limit on the length of the data stream.  For
> example, suppose the longest song you want to send is 1000 seconds and both
> clocks are accurate to 50 ppm (parts per million).  The max difference in
> clocks is 100 ppm.  48 k samples/second * 1000 seconds is 48 million
> samples total.  100 ppm means that the worst difference would be 4800
> samples.  So if the USB side waits until it gets 4800 samples before it
> starts forwarding data, it won't run out if the audio clock is worst-case
> slow and the USB clock is worst-case fast.  You also need buffering for
> another 4800 samples in case the clock differences are reversed.  This adds
> a delay to get started while you collect the 4800 samples.

That won't work here, the stream can last forever (well, not quite, but still 
a long time).

> You can just drop or duplicate a sample whenever the clocks slip.

> You can derive both clocks from a common source.
>
> You can lock one clock to the other with a PLL (Phase Locked Loop).

The SOF clock is driven by the USB host, and the audio clock is driven by the 
audio codec. I can't lock one to another or derive both of them from a common 
source. There could even be no audio clock at all if I stream audio data from 
a file.

> There are probably other approaches.

The applicable techniques require knowledge of both the audio clock and the 
SOF clock in a common place. My driver has no access to the audio clock. All 
it knows about is the SOF clock. 

There are only two options I can think of.

The first one is to use an asynchronous endpoint and sent the occasional 
smaller or bigger packet (or duplicate/drop one sample). As the driver can't 
access the audio clock it needs to derive the information from the amount of 
data present in the ALSA ring buffer. To be honest I'm not sure if that will 
be possible at all, as the application will write data at a non-constant rate.

The second one, which sounds easier, at least on the driver side, is to use a 
synchronous endpoint with a fixed packet size. The application will perform 
rate matching (duplicating/dropping a sample or resampling the audio stream) 
using the audio clock and the SOF clock. What I'm still unsure about is how 
the application can access the audio clock and the SOF clock through ALSA, but 
I suppose that's possible.

Best regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Alan Stern]

On Sun, 17 May 2009, Laurent Pinchart wrote:

> The applicable techniques require knowledge of both the audio clock and the 
> SOF clock in a common place. My driver has no access to the audio clock. All 
> it knows about is the SOF clock. 

This whole discussion is a little puzzling.

The Gadget API doesn't provide any way to tie the buffer contents of
Isochronous usb_request's to the frame number.  Here's what I mean:
Suppose you want to transfer a new audio data buffer every frame.  You
queue some requests, let's say

	R0 for frame F0
	R1 for frame F1
	R2 for frame F2
	etc.

But what happens if a communications error prevents R0 from being 
delivered during F0?  That's the sort of thing you expect to happen 
from time to time, and Isochronous streams are supposed to handle such 
errors by simply ignoring them.  So ideally you'd like to forget about 
the missing data, and go ahead with R1 during F1 and so on.

But as far as I can see, the Gadget API doesn't provide any way to do
this!  Depending on the implementation of the device controller driver,
you might end up transferring R0 during F1, R1 during F2, and so on.  
Everything would be misaligned from then on.  I don't see any solution
to this problem.

I'd like to answer your questions about synchronizing the USB audio 
stream with an ALSA audio stream, but since I don't know anything about 
how either audio protocol is supposed to work, I can't.  Suppose you 
were trying to write a normal program that accepted data from an ALSA 
microphone and sent it to an ALSA speaker; how would such a program 
synchronize the input and output streams?

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Clemens Ladisch]

Laurent Pinchart wrote:
> The applicable techniques require knowledge of both the audio clock and the
> SOF clock in a common place. My driver has no access to the audio clock. All
> it knows about is the SOF clock.
> 
> There are only two options I can think of.
> 
> The first one is to use an asynchronous endpoint and sent the occasional
> smaller or bigger packet (or duplicate/drop one sample). As the driver can't
> access the audio clock it needs to derive the information from the amount of
> data present in the ALSA ring buffer. To be honest I'm not sure if that will
> be possible at all, as the application will write data at a non-constant rate.

The ALSA API assumes that the device controls the audio clock and that
the application derives its own speed from that, not the other way
round.

> The second one, which sounds easier, at least on the driver side, is to use a
> synchronous endpoint with a fixed packet size. The application will perform
> rate matching (duplicating/dropping a sample or resampling the audio stream)
> using the audio clock and the SOF clock. What I'm still unsure about is how
> the application can access the audio clock and the SOF clock through ALSA, but
> I suppose that's possible.

The ALSA API doesn't give you access to the SOF clock.


What you want to do is not possible (or impracticably hard) with the
ALSA API; I strongly suggest that you define your own API for your
audio gadget driver so that SOF clock information is made available to
the application, and that packet sizes can be directly controlled by the
application.


Best regards,
Clemens

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

Hi Alan,

On Sunday 17 May 2009 23:28:20 Alan Stern wrote:
> On Sun, 17 May 2009, Laurent Pinchart wrote:
> > The applicable techniques require knowledge of both the audio clock and
> > the SOF clock in a common place. My driver has no access to the audio
> > clock. All it knows about is the SOF clock.
>
> This whole discussion is a little puzzling.
>
> The Gadget API doesn't provide any way to tie the buffer contents of
> Isochronous usb_request's to the frame number.  Here's what I mean:
> Suppose you want to transfer a new audio data buffer every frame.  You
> queue some requests, let's say
>
> 	R0 for frame F0
> 	R1 for frame F1
> 	R2 for frame F2
> 	etc.
>
> But what happens if a communications error prevents R0 from being
> delivered during F0?  That's the sort of thing you expect to happen
> from time to time, and Isochronous streams are supposed to handle such
> errors by simply ignoring them.  So ideally you'd like to forget about
> the missing data, and go ahead with R1 during F1 and so on.
>
> But as far as I can see, the Gadget API doesn't provide any way to do
> this!  Depending on the implementation of the device controller driver,
> you might end up transferring R0 during F1, R1 during F2, and so on.
> Everything would be misaligned from then on.  I don't see any solution
> to this problem.

Your analysis is right, but I think I can work around the problem by checking 
the SOF counter in the USB request completion handlers. If the counter is too 
much ahead its ideal value I can assume that at least one packet failed to 
transfer and resulted in a time shift. I will then just drop one packet. 
Instead of the ideal R0/F0, --/F1, R2/F2, R3/F3, R4/F4, R5/F5, ... situation I 
will get R0/F0, --/F1, R1/F2, R2/F3, R4/F4, R5/F5, ...

> I'd like to answer your questions about synchronizing the USB audio
> stream with an ALSA audio stream, but since I don't know anything about
> how either audio protocol is supposed to work, I can't.  Suppose you
> were trying to write a normal program that accepted data from an ALSA
> microphone and sent it to an ALSA speaker; how would such a program
> synchronize the input and output streams?

That's a very good question. If an ALSA guru could answer it I would 
(hopefully) get one step closer to a solution. I can't imagine that ALSA 
wouldn't support this use case.

Best regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Laurent Pinchart]

Hi Clemens,

On Monday 18 May 2009 10:39:24 Clemens Ladisch wrote:
> Laurent Pinchart wrote:
> > The applicable techniques require knowledge of both the audio clock and
> > the SOF clock in a common place. My driver has no access to the audio
> > clock. All it knows about is the SOF clock.
> >
> > There are only two options I can think of.
> >
> > The first one is to use an asynchronous endpoint and sent the occasional
> > smaller or bigger packet (or duplicate/drop one sample). As the driver
> > can't access the audio clock it needs to derive the information from the
> > amount of data present in the ALSA ring buffer. To be honest I'm not sure
> > if that will be possible at all, as the application will write data at a
> > non-constant rate.
>
> The ALSA API assumes that the device controls the audio clock and that
> the application derives its own speed from that, not the other way
> round.

That's a sensible requirement for a sound card. I now know that my use case is 
out of ALSA's bounds :-)

> > The second one, which sounds easier, at least on the driver side, is to
> > use a synchronous endpoint with a fixed packet size. The application will
> > perform rate matching (duplicating/dropping a sample or resampling the
> > audio stream) using the audio clock and the SOF clock. What I'm still
> > unsure about is how the application can access the audio clock and the
> > SOF clock through ALSA, but I suppose that's possible.
>
> The ALSA API doesn't give you access to the SOF clock.

Does the ALSA API give applications access to the audio clock ? As Alan Stern 
mention in his e-mail, does ALSA allow an application to resample audio from 
an ALSA source (microphone on sound card 1) before sending it to an ALSA sink 
(speaker on sound card 2) ?

> What you want to do is not possible (or impracticably hard) with the
> ALSA API; I strongly suggest that you define your own API for your
> audio gadget driver so that SOF clock information is made available to
> the application, and that packet sizes can be directly controlled by the
> application.

I'll have a look at that. ALSA was appealing because of the existing user base 
and its ability to transfer audio data from/to userspace applications. I knew 
it hasn't been designed with my use case in mind, so I didn't close to door to 
using a custom API, but I still wanted to give it a try.

Best regards,

Laurent Pinchart

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver

[Clemens Ladisch]

Laurent Pinchart wrote:
> On Monday 18 May 2009 10:39:24 Clemens Ladisch wrote:
> > The ALSA API doesn't give you access to the SOF clock.
> 
> Does the ALSA API give applications access to the audio clock ?

ALSA wakes up the application (if it's blocked) whenever a sound card
interrupt happens, i.e., at period boundaries.  At all times, you can
get the current sample position by calling snd_pcm_delay().

> As Alan Stern mention in his e-mail, does ALSA allow an application to
> resample audio from an ALSA source (microphone on sound card 1) before
> sending it to an ALSA sink (speaker on sound card 2) ?

It is possible, if the application monitors the speed of both devices.


Best regards,
Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html