Re: [PATCH v30 00/30] Introduce QC USB SND audio offloading support

[Wesley Cheng <quic_wcheng@quicinc.com>]

On 12/10/2024 8:40 AM, Takashi Iwai wrote:
> On Tue, 10 Dec 2024 01:59:10 +0100,
> Wesley Cheng wrote:
>> On 12/5/2024 4:53 PM, Wesley Cheng wrote:
>>> On 12/4/2024 2:49 PM, Pierre-Louis Bossart wrote:
>>>>>>> UAOL is one of our priorities right now and some (e.g.: me) prefer to not pollute their mind with another approaches until what they have in mind is crystalized. In short, I'd vote for a approach where USB device has a ASoC representative in sound/soc/codecs/ just like it is the case for HDAudio. Either that or at least a ASoC-component representative, a dependency for UAOL-capable card to enumerate.
>>>>>> The main difference is that we don’t want the USB audio *control* part to be seen in two places. The only requirement is to stream data with an alternate optimized path, but all the volume control and whatnot is supposed to be done using the regular usb-audio card. It would be complete chaos for userspace if the same volume can be represented differently.
>>>>>> The comparison with HDaudio is not quite right either. In the case of HDaudio, it’s an all-or-nothing solution. The external device is controlled by one entity, either legacy or ASoC based. That choice is made at driver probe time. In the case of USB, the application needs to have the choice of using either the legacy path, or the optimized path that goes through a DSP. I think the last thing you want given this context is to make the USB audio device an ASoC codec.
>>>>>> I find it rather interesting that this architectural feedback comes at the v30, it’s unfair to Wesley really...
>>>>> Hi Pierre,
>>>>>
>>>>> Obviously I'm late. After scanning the history of this one, indeed it's been a while since v1 has been sent. And thus I posted no NACKs. At the same time if I am to choose between: provide feedback vs provide no-feedback, I'd rather choose the former even if I'm to be ignored/overridden by a subsystem maintainer.
>>>>>
>>>>> The subsystem maintainers also hold the last word, and I have no problem with them merging the patches if they believe its existing shape is good-enough. For example, my team could follow up this implementation next year with a patchset expanding/updating the functionality. I see this as a viable option.
>>>> That’s what we had in mind before I left Intel. The interfaces seen by userspace are PCM devices and kcontrols, it doesn’t matter too much if there is one card, two cards, and if the implementation relies on an ASoC codec, a library or something else. 
>>>> The bulk of the work is to enable the USB offload from top to bottom, by changing PipeWire/CRAS/HAL to select the new optimized path when available and deal with plug/unplug events.
>>>> Improvements at the kernel level can be done later if required. It’s hard to argue that the proposal in this series is fundamentally broken, but as usual it’s likely that some requirements are missing or not known yet. The same thing happened with compressed offload, none one thought about gapless playback until Android made it a requirement. Maybe what we’d need is a ‘protocol version’ for USB offload so that changes can be tracked and handled?
>>> Thanks for chiming in, Pierre.  So for now, with the next revision I have prepared, I'm currently adding:
>>>
>>> 1.  Some improvements to xHCI sideband to account for core sequences that need to be notified to the offload driver, ie transfer ring free
>>>
>>> 2.  Moved the USB SND offload mixer driver into the QC vendor module for now, as instructed by Takashi:
>>>
>>> https://lore.kernel.org/linux-usb/87cyiiaxpc.wl-tiwai@suse.de/
>>>
>>> 3.  Added separate kcontrols for fetching mapped PCM device and card indexes (versus one that returns a card and PCM device pair [array])
>>>
>>> 4.  Removed some jack controls (enable/disable) from soc-usb
>>>
>>> 5.  Updated documentation for #3
>>>
>>>
>>> Those are the major changes that will come in the next revision.  I'm just trying to figure out who/where the "protocol version" should be checked if we decided to add it.  (or if we need to check for it anywhere...)  From the userspace perspective, it should be agnostic to how we've implemented offloading from the kernel, and I don't see any major shifts in how userspace implements things even if we make improvements from kernel.
>>
>> Hi Takashi,
>>
>> Could you possibly help share some direction on what you think of the current design, and if you think its detrimental that we make modifications mentioned by Cezary?  I have the next revision ready for review, but I wanted to get a better sense on the likeliness of this landing upstream w/o the major modifications.
> Honestly speaking, I have no big preference about that design
> question.  The most important thing is rather what's visible change to
> users.  An advantage of the current design (sort of add-on to the
> existing USB-audio driver) is that it's merely a few card controls
> that are added and visible, and the rest is just as of now.  The
> remaining design issue (two cards or single card) is rather
> kernel-internal, and has nothing to do with users.  So I'm fine with
> the current design.
>
> OTOH, if we follow the pattern of HD-audio, at least there will be
> more preliminary changes in USB-audio driver side like we've done for
> HD-audio.  That is, make most of USB-audio code to be usable as (a
> kind of) library code.  It's more work, but certainly doable.  And if
> that can be achieved and there other similar use cases of this stuff
> not only from Qualcomm, it might make sense to go in that way, too.
> That said, it's rather a question about what's extended in future.
> If Intel will need / want to move on that direction, too, that's a
> good reason to reconsider the basic design.
>

So to clarify, what Cezary and I are proposing are actually two different concepts to achieve some sort of offloading for audio data.  In my use case, we're trying to leverage as much of the USB SND implementation as possible, and only offloading the handling of audio transfers.  Everything else is still handled by USB SND, hence the reason for it being an add-on since most of it stays the same.  Unfortunately, I don't have any details about the full HW offload design, as public material on it is fairly minimal.  So it would be difficult for me to rework my series to something I don't have a line of sight into.  Personally (and as you can probably tell :)), I would prefer if we could do the refactoring in stages (if actually required), since I've been pushing this method for awhile now, and I'm not sure if I can take up that effort to do that on my next submission.  At least from the QC perspective if we did move to the HDaudio-type implementation, I think I'd need to also
change up the ASoC design we have currently implemented as well, so it wouldn't be a trivial change.


Thanks

Wesley Cheng