* [Discussion]: Enhance virtio rpmsg bus driver buffer allocation [not found] <1af16ff8-5706-45e5-9737-05da39957c95@arm.com> @ 2023-11-17 22:24 ` Divin Raj 2023-11-20 10:14 ` Arnaud POULIQUEN 2023-11-21 5:54 ` Tanmay Shah 0 siblings, 2 replies; 10+ messages in thread From: Divin Raj @ 2023-11-17 22:24 UTC (permalink / raw) To: linux-remoteproc On 10/23/23 11:44 AM, Divin Raj wrote: > Hello all, > > I am reaching out with reference to the patch discussed here: Enhanced > virtio rpmsg bus driver buffer allocation. > <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> > > I've been keenly following the developments around enhancing buffer > allocation strategies, especially those focused on dynamic buffer sizing > and the considerations for systems under varying memory constraints.This > work is highly relevant to several projects I am involved in, and I am > quite interested in its progression. May I kindly request an update on > the current phase of these initiatives? Additionally, I am eager to know > if there would be an opportunity for me to contribute to enhancing the > patch, possibly by working on improvements or assisting in verification > processes. > > Furthermore, if there are any condensed resources, summaries, or > specific threads that encapsulate recent advancements or discussions on > this topic, I would be grateful to receive directions to them. > > I appreciate everyone's dedicated efforts and invaluable contributions > to this area of development. Looking forward to the updates. > > Regards Divin > Hello Linux Community, In one of our internal projects, we encountered a challenge with RPMSG buffer allocation. Our goal is to optimize memory allocation for an out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure support for packet sizes matching the standard MTU (Maximum Transmission Unit) size of 1500 bytes. To mitigate this issue, There are few possible solutions: 1. Configure buffer size and number through Kconfig. 2. Permit the firmware creator to determine the most suitable value from the resource table. 3. Enable independent configurations on both ends. This approach would support both dynamic and fixed buffer configurations using a generic allocator. Reference: [1]: https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ Draft Design Overview: Based on the reference patch and the discussions, we have outlined the following key points for the belw design: 1. Assure compatibility, enabling both Linux and the remote system to interchangeably transmit and receive messages, irrespective of size. 2. For systems with constrained shared memory: Systems with small, shared memory, we need to deal with a limited/optimized memory chunk. To avoid memory fragmentation, the allocator should have a pre-reserved buffer pool 3. The implementation should ensure that the remote side does not receive messages based on its allocation parameters. do you think it could make sense? High level view: +------------------+ +------------------+ | | | | | Linux | | Remote | | | | | | +----------+ | +-----------------+ | +----------+ | | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | | +----------+ | | (Dynamic/Static)| | +----------+ | | | +-----------------+ | | +------------------+ +------------------+ Detailed view: +-------------------------+ | Message Creation | | (Both Linux/Remote) | +------------+------------+ | v +-------------------------+ | Determine the allocation| | strategy | +------------+------------+ | +--------------+--------------+ | | +-------------------------------+ +-------------------------------+ | Dynamic allocation | | Static allocation | | (Buffer allocator allocates | | (Pre-reserved memory | | memory space as needed, | | space) | | based on the current | | | | message requirement ) | | | +-------------------------------+ +-------------------------------+ We would greatly appreciate any feedback, suggestions, or improvements you could provide. Thank you for your time and consideration. Regards Divin IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-11-17 22:24 ` [Discussion]: Enhance virtio rpmsg bus driver buffer allocation Divin Raj @ 2023-11-20 10:14 ` Arnaud POULIQUEN 2023-11-24 16:45 ` Divin Raj 2023-11-21 5:54 ` Tanmay Shah 1 sibling, 1 reply; 10+ messages in thread From: Arnaud POULIQUEN @ 2023-11-20 10:14 UTC (permalink / raw) To: Divin Raj, linux-remoteproc Hi Divin, On 11/17/23 23:24, Divin Raj wrote: > On 10/23/23 11:44 AM, Divin Raj wrote: >> Hello all, >> >> I am reaching out with reference to the patch discussed here: Enhanced >> virtio rpmsg bus driver buffer allocation. >> <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> >> >> I've been keenly following the developments around enhancing buffer >> allocation strategies, especially those focused on dynamic buffer sizing >> and the considerations for systems under varying memory constraints.This >> work is highly relevant to several projects I am involved in, and I am >> quite interested in its progression. May I kindly request an update on >> the current phase of these initiatives? Additionally, I am eager to know >> if there would be an opportunity for me to contribute to enhancing the >> patch, possibly by working on improvements or assisting in verification >> processes. >> >> Furthermore, if there are any condensed resources, summaries, or >> specific threads that encapsulate recent advancements or discussions on >> this topic, I would be grateful to receive directions to them. >> >> I appreciate everyone's dedicated efforts and invaluable contributions >> to this area of development. Looking forward to the updates. >> >> Regards Divin >> > Hello Linux Community, > > In one of our internal projects, we encountered a challenge with RPMSG > buffer allocation. Our goal is to optimize memory allocation for an > out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure > support for packet sizes matching the standard MTU (Maximum Transmission > Unit) size of 1500 bytes. > > To mitigate this issue, There are few possible solutions: > > 1. Configure buffer size and number through Kconfig. > 2. Permit the firmware creator to determine the most suitable value from > the resource table. > 3. Enable independent configurations on both ends. This approach would > support both dynamic and fixed buffer configurations using a generic > allocator. > > Reference: > > [1]: > https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ > [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ > > > Draft Design Overview: > > Based on the reference patch and the discussions, we have outlined the > following key points for the belw design: > > 1. Assure compatibility, enabling both Linux and the remote system to > interchangeably transmit and receive messages, irrespective of size. > 2. For systems with constrained shared memory: > Systems with small, shared memory, we need to deal with a > limited/optimized memory chunk. To avoid memory fragmentation, the > allocator should have a pre-reserved buffer pool > 3. The implementation should ensure that the remote side does not > receive messages based on its allocation parameters. > > do you think it could make sense? > > High level view: > +------------------+ +------------------+ > | | | | > | Linux | | Remote | > | | | | > | +----------+ | +-----------------+ | +----------+ | > | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | > | +----------+ | | (Dynamic/Static)| | +----------+ | > | | +-----------------+ | | > +------------------+ +------------------+ > > > Detailed view: > > +-------------------------+ > | Message Creation | > | (Both Linux/Remote) | > +------------+------------+ > | > v > +-------------------------+ > | Determine the allocation| > | strategy | > +------------+------------+ > | > +--------------+--------------+ > | | > +-------------------------------+ +-------------------------------+ > | Dynamic allocation | | Static allocation | > | (Buffer allocator allocates | | (Pre-reserved memory | > | memory space as needed, | | space) | > | based on the current | | | > | message requirement ) | | | > +-------------------------------+ +-------------------------------+ Do you have a proposal for dynamic allocation? RPMSG is based on the virtio protocol. The virtio driver in the Linux kernel is responsible for allocating buffers for the virtio device on the remote processor. In the current implementation (static allocation) the Linux kernel allocates predefined buffers for the remote processor. How would you manage the fact that the sender allocates its own buffers and references them in the vring descriptor? This would require each core to have a dual role, right? - a virtio driver role on its TX vring - a virtio device role on its RX vring." Regards, Arnaud > > We would greatly appreciate any feedback, suggestions, or improvements > you could provide. > > Thank you for your time and consideration. > > Regards > Divin > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended recipient, > please notify the sender immediately and do not disclose the contents to any > other person, use it for any purpose, or store or copy the information in any > medium. Thank you. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-11-20 10:14 ` Arnaud POULIQUEN @ 2023-11-24 16:45 ` Divin Raj 2023-11-28 8:34 ` Arnaud POULIQUEN 0 siblings, 1 reply; 10+ messages in thread From: Divin Raj @ 2023-11-24 16:45 UTC (permalink / raw) To: Arnaud POULIQUEN, linux-remoteproc; +Cc: Rahul.Singh@arm.com Hi Arnaud, Please find my comments inline. On 11/20/23 10:14 AM, Arnaud POULIQUEN wrote: > Hi Divin, > > On 11/17/23 23:24, Divin Raj wrote: >> On 10/23/23 11:44 AM, Divin Raj wrote: >>> Hello all, >>> >>> I am reaching out with reference to the patch discussed here: Enhanced >>> virtio rpmsg bus driver buffer allocation. >>> <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> >>> >>> I've been keenly following the developments around enhancing buffer >>> allocation strategies, especially those focused on dynamic buffer sizing >>> and the considerations for systems under varying memory constraints.This >>> work is highly relevant to several projects I am involved in, and I am >>> quite interested in its progression. May I kindly request an update on >>> the current phase of these initiatives? Additionally, I am eager to know >>> if there would be an opportunity for me to contribute to enhancing the >>> patch, possibly by working on improvements or assisting in verification >>> processes. >>> >>> Furthermore, if there are any condensed resources, summaries, or >>> specific threads that encapsulate recent advancements or discussions on >>> this topic, I would be grateful to receive directions to them. >>> >>> I appreciate everyone's dedicated efforts and invaluable contributions >>> to this area of development. Looking forward to the updates. >>> >>> Regards Divin >>> >> Hello Linux Community, >> >> In one of our internal projects, we encountered a challenge with RPMSG >> buffer allocation. Our goal is to optimize memory allocation for an >> out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure >> support for packet sizes matching the standard MTU (Maximum Transmission >> Unit) size of 1500 bytes. >> >> To mitigate this issue, There are few possible solutions: >> >> 1. Configure buffer size and number through Kconfig. >> 2. Permit the firmware creator to determine the most suitable value from >> the resource table. >> 3. Enable independent configurations on both ends. This approach would >> support both dynamic and fixed buffer configurations using a generic >> allocator. >> >> Reference: >> >> [1]: >> https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ >> [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ >> >> >> Draft Design Overview: >> >> Based on the reference patch and the discussions, we have outlined the >> following key points for the belw design: >> >> 1. Assure compatibility, enabling both Linux and the remote system to >> interchangeably transmit and receive messages, irrespective of size. >> 2. For systems with constrained shared memory: >> Systems with small, shared memory, we need to deal with a >> limited/optimized memory chunk. To avoid memory fragmentation, the >> allocator should have a pre-reserved buffer pool >> 3. The implementation should ensure that the remote side does not >> receive messages based on its allocation parameters. >> >> do you think it could make sense? >> >> High level view: >> +------------------+ +------------------+ >> | | | | >> | Linux | | Remote | >> | | | | >> | +----------+ | +-----------------+ | +----------+ | >> | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | >> | +----------+ | | (Dynamic/Static)| | +----------+ | >> | | +-----------------+ | | >> +------------------+ +------------------+ >> >> >> Detailed view: >> >> +-------------------------+ >> | Message Creation | >> | (Both Linux/Remote) | >> +------------+------------+ >> | >> v >> +-------------------------+ >> | Determine the allocation| >> | strategy | >> +------------+------------+ >> | >> +--------------+--------------+ >> | | >> +-------------------------------+ +-------------------------------+ >> | Dynamic allocation | | Static allocation | >> | (Buffer allocator allocates | | (Pre-reserved memory | >> | memory space as needed, | | space) | >> | based on the current | | | >> | message requirement ) | | | >> +-------------------------------+ +-------------------------------+ > > Do you have a proposal for dynamic allocation? > > RPMSG is based on the virtio protocol. The virtio driver in the Linux kernel > is responsible for allocating buffers for the virtio device on the remote processor. > > In the current implementation (static allocation) the Linux > kernel allocates predefined buffers for the remote processor. > > How would you manage the fact that the sender allocates its own buffers and > references > them in the vring descriptor? This would require each core to have > a dual role, right? > - a virtio driver role on its TX vring > - a virtio device role on its RX vring." > I'm unsure if a dual role is feasible under the Virtio specification. However, would it make sense to set the size of the outbuf based on the Maximum Transmission Unit (MTU) size that is supported? Additionally, the size of the inbuf could be set by the firmware, suggesting that it should be derived from the resource table. With this approach, I believe the sender can decide the maximum size. Regards Divin > > Regards, > Arnaud > > >> >> We would greatly appreciate any feedback, suggestions, or improvements >> you could provide. >> >> Thank you for your time and consideration. >> >> Regards >> Divin >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended recipient, >> please notify the sender immediately and do not disclose the contents to any >> other person, use it for any purpose, or store or copy the information in any >> medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-11-24 16:45 ` Divin Raj @ 2023-11-28 8:34 ` Arnaud POULIQUEN 2023-11-28 11:19 ` Divin Raj 0 siblings, 1 reply; 10+ messages in thread From: Arnaud POULIQUEN @ 2023-11-28 8:34 UTC (permalink / raw) To: Divin Raj, linux-remoteproc; +Cc: Rahul.Singh@arm.com On 11/24/23 17:45, Divin Raj wrote: > Hi Arnaud, > Please find my comments inline. > > On 11/20/23 10:14 AM, Arnaud POULIQUEN wrote: >> Hi Divin, >> >> On 11/17/23 23:24, Divin Raj wrote: >>> On 10/23/23 11:44 AM, Divin Raj wrote: >>>> Hello all, >>>> >>>> I am reaching out with reference to the patch discussed here: Enhanced >>>> virtio rpmsg bus driver buffer allocation. >>>> <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> >>>> >>>> I've been keenly following the developments around enhancing buffer >>>> allocation strategies, especially those focused on dynamic buffer sizing >>>> and the considerations for systems under varying memory constraints.This >>>> work is highly relevant to several projects I am involved in, and I am >>>> quite interested in its progression. May I kindly request an update on >>>> the current phase of these initiatives? Additionally, I am eager to know >>>> if there would be an opportunity for me to contribute to enhancing the >>>> patch, possibly by working on improvements or assisting in verification >>>> processes. >>>> >>>> Furthermore, if there are any condensed resources, summaries, or >>>> specific threads that encapsulate recent advancements or discussions on >>>> this topic, I would be grateful to receive directions to them. >>>> >>>> I appreciate everyone's dedicated efforts and invaluable contributions >>>> to this area of development. Looking forward to the updates. >>>> >>>> Regards Divin >>>> >>> Hello Linux Community, >>> >>> In one of our internal projects, we encountered a challenge with RPMSG >>> buffer allocation. Our goal is to optimize memory allocation for an >>> out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure >>> support for packet sizes matching the standard MTU (Maximum Transmission >>> Unit) size of 1500 bytes. >>> >>> To mitigate this issue, There are few possible solutions: >>> >>> 1. Configure buffer size and number through Kconfig. >>> 2. Permit the firmware creator to determine the most suitable value from >>> the resource table. >>> 3. Enable independent configurations on both ends. This approach would >>> support both dynamic and fixed buffer configurations using a generic >>> allocator. >>> >>> Reference: >>> >>> [1]: >>> https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ >>> [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ >>> >>> >>> Draft Design Overview: >>> >>> Based on the reference patch and the discussions, we have outlined the >>> following key points for the belw design: >>> >>> 1. Assure compatibility, enabling both Linux and the remote system to >>> interchangeably transmit and receive messages, irrespective of size. >>> 2. For systems with constrained shared memory: >>> Systems with small, shared memory, we need to deal with a >>> limited/optimized memory chunk. To avoid memory fragmentation, the >>> allocator should have a pre-reserved buffer pool >>> 3. The implementation should ensure that the remote side does not >>> receive messages based on its allocation parameters. >>> >>> do you think it could make sense? >>> >>> High level view: >>> +------------------+ +------------------+ >>> | | | | >>> | Linux | | Remote | >>> | | | | >>> | +----------+ | +-----------------+ | +----------+ | >>> | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | >>> | +----------+ | | (Dynamic/Static)| | +----------+ | >>> | | +-----------------+ | | >>> +------------------+ +------------------+ >>> >>> >>> Detailed view: >>> >>> +-------------------------+ >>> | Message Creation | >>> | (Both Linux/Remote) | >>> +------------+------------+ >>> | >>> v >>> +-------------------------+ >>> | Determine the allocation| >>> | strategy | >>> +------------+------------+ >>> | >>> +--------------+--------------+ >>> | | >>> +-------------------------------+ +-------------------------------+ >>> | Dynamic allocation | | Static allocation | >>> | (Buffer allocator allocates | | (Pre-reserved memory | >>> | memory space as needed, | | space) | >>> | based on the current | | | >>> | message requirement ) | | | >>> +-------------------------------+ +-------------------------------+ >> >> Do you have a proposal for dynamic allocation? >> >> RPMSG is based on the virtio protocol. The virtio driver in the Linux kernel >> is responsible for allocating buffers for the virtio device on the remote >> processor. >> >> In the current implementation (static allocation) the Linux >> kernel allocates predefined buffers for the remote processor. >> >> How would you manage the fact that the sender allocates its own buffers and >> references >> them in the vring descriptor? This would require each core to have >> a dual role, right? >> - a virtio driver role on its TX vring >> - a virtio device role on its RX vring." >> > I'm unsure if a dual role is feasible under the Virtio specification. At least, it does not seem to align with the philosophy of VirtIO. > However, would it make sense to set the size of the outbuf based on the > Maximum Transmission Unit (MTU) size that is supported? Additionally, > the size of the inbuf could be set by the firmware, suggesting that it > should be derived from the resource table. With this approach, I believe > the sender can decide the maximum size. It is not clear to me what your proposal is. Are you speaking about a pre-allocated buffers as proposed in [1], or are you speaking about dynamic allocation of the RPMsg in a pool? Regards, Arnaud > > Regards > Divin > >> >> Regards, >> Arnaud >> > >> >>> >>> We would greatly appreciate any feedback, suggestions, or improvements >>> you could provide. >>> >>> Thank you for your time and consideration. >>> >>> Regards >>> Divin >>> IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended recipient, >>> please notify the sender immediately and do not disclose the contents to any >>> other person, use it for any purpose, or store or copy the information in any >>> medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended recipient, > please notify the sender immediately and do not disclose the contents to any > other person, use it for any purpose, or store or copy the information in any > medium. Thank you. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-11-28 8:34 ` Arnaud POULIQUEN @ 2023-11-28 11:19 ` Divin Raj 2023-11-28 13:45 ` Arnaud POULIQUEN 2024-01-12 22:39 ` Mathieu Poirier 0 siblings, 2 replies; 10+ messages in thread From: Divin Raj @ 2023-11-28 11:19 UTC (permalink / raw) To: Arnaud POULIQUEN, linux-remoteproc; +Cc: Rahul.Singh@arm.com On 11/28/23 8:34 AM, Arnaud POULIQUEN wrote: > > > On 11/24/23 17:45, Divin Raj wrote: >> Hi Arnaud, >> Please find my comments inline. >> >> On 11/20/23 10:14 AM, Arnaud POULIQUEN wrote: >>> Hi Divin, >>> >>> On 11/17/23 23:24, Divin Raj wrote: >>>> On 10/23/23 11:44 AM, Divin Raj wrote: >>>>> Hello all, >>>>> >>>>> I am reaching out with reference to the patch discussed here: Enhanced >>>>> virtio rpmsg bus driver buffer allocation. >>>>> <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> >>>>> >>>>> I've been keenly following the developments around enhancing buffer >>>>> allocation strategies, especially those focused on dynamic buffer sizing >>>>> and the considerations for systems under varying memory constraints.This >>>>> work is highly relevant to several projects I am involved in, and I am >>>>> quite interested in its progression. May I kindly request an update on >>>>> the current phase of these initiatives? Additionally, I am eager to know >>>>> if there would be an opportunity for me to contribute to enhancing the >>>>> patch, possibly by working on improvements or assisting in verification >>>>> processes. >>>>> >>>>> Furthermore, if there are any condensed resources, summaries, or >>>>> specific threads that encapsulate recent advancements or discussions on >>>>> this topic, I would be grateful to receive directions to them. >>>>> >>>>> I appreciate everyone's dedicated efforts and invaluable contributions >>>>> to this area of development. Looking forward to the updates. >>>>> >>>>> Regards Divin >>>>> >>>> Hello Linux Community, >>>> >>>> In one of our internal projects, we encountered a challenge with RPMSG >>>> buffer allocation. Our goal is to optimize memory allocation for an >>>> out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure >>>> support for packet sizes matching the standard MTU (Maximum Transmission >>>> Unit) size of 1500 bytes. >>>> >>>> To mitigate this issue, There are few possible solutions: >>>> >>>> 1. Configure buffer size and number through Kconfig. >>>> 2. Permit the firmware creator to determine the most suitable value from >>>> the resource table. >>>> 3. Enable independent configurations on both ends. This approach would >>>> support both dynamic and fixed buffer configurations using a generic >>>> allocator. >>>> >>>> Reference: >>>> >>>> [1]: >>>> https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ >>>> [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ >>>> >>>> >>>> Draft Design Overview: >>>> >>>> Based on the reference patch and the discussions, we have outlined the >>>> following key points for the belw design: >>>> >>>> 1. Assure compatibility, enabling both Linux and the remote system to >>>> interchangeably transmit and receive messages, irrespective of size. >>>> 2. For systems with constrained shared memory: >>>> Systems with small, shared memory, we need to deal with a >>>> limited/optimized memory chunk. To avoid memory fragmentation, the >>>> allocator should have a pre-reserved buffer pool >>>> 3. The implementation should ensure that the remote side does not >>>> receive messages based on its allocation parameters. >>>> >>>> do you think it could make sense? >>>> >>>> High level view: >>>> +------------------+ +------------------+ >>>> | | | | >>>> | Linux | | Remote | >>>> | | | | >>>> | +----------+ | +-----------------+ | +----------+ | >>>> | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | >>>> | +----------+ | | (Dynamic/Static)| | +----------+ | >>>> | | +-----------------+ | | >>>> +------------------+ +------------------+ >>>> >>>> >>>> Detailed view: >>>> >>>> +-------------------------+ >>>> | Message Creation | >>>> | (Both Linux/Remote) | >>>> +------------+------------+ >>>> | >>>> v >>>> +-------------------------+ >>>> | Determine the allocation| >>>> | strategy | >>>> +------------+------------+ >>>> | >>>> +--------------+--------------+ >>>> | | >>>> +-------------------------------+ +-------------------------------+ >>>> | Dynamic allocation | | Static allocation | >>>> | (Buffer allocator allocates | | (Pre-reserved memory | >>>> | memory space as needed, | | space) | >>>> | based on the current | | | >>>> | message requirement ) | | | >>>> +-------------------------------+ +-------------------------------+ >>> >>> Do you have a proposal for dynamic allocation? >>> >>> RPMSG is based on the virtio protocol. The virtio driver in the Linux kernel >>> is responsible for allocating buffers for the virtio device on the remote >>> processor. >>> >>> In the current implementation (static allocation) the Linux >>> kernel allocates predefined buffers for the remote processor. >>> >>> How would you manage the fact that the sender allocates its own buffers and >>> references >>> them in the vring descriptor? This would require each core to have >>> a dual role, right? >>> - a virtio driver role on its TX vring >>> - a virtio device role on its RX vring." >>> >> I'm unsure if a dual role is feasible under the Virtio specification. > > At least, it does not seem to align with the philosophy of VirtIO. > > >> However, would it make sense to set the size of the outbuf based on the >> Maximum Transmission Unit (MTU) size that is supported? Additionally, >> the size of the inbuf could be set by the firmware, suggesting that it >> should be derived from the resource table. With this approach, I believe >> the sender can decide the maximum size. > > It is not clear to me what your proposal is. > Are you speaking about a pre-allocated buffers as proposed in [1], > or are you speaking about dynamic allocation of the RPMsg in a pool? we are at the initial phase of this investigation. As we previously discussed, option 3 is not feasible in accordance with the virtio specification.The above proposed solution aligns with [1], suggesting preallocated in_buf and out_buf, with sizes determined from the resource table and MTU. By allowing Linux to decide the out_buf size and the remote to decide the in_buf size, I believe we can avoid conflicts. If everyone agrees on a common idea, then it would be a good starting point Regards Divin > Regards, > Arnaud > >> >> Regards >> Divin >> >>> >>> Regards, >>> Arnaud >>> >> >>> >>>> >>>> We would greatly appreciate any feedback, suggestions, or improvements >>>> you could provide. >>>> >>>> Thank you for your time and consideration. >>>> >>>> Regards >>>> Divin >>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>> confidential and may also be privileged. If you are not the intended recipient, >>>> please notify the sender immediately and do not disclose the contents to any >>>> other person, use it for any purpose, or store or copy the information in any >>>> medium. Thank you. >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended recipient, >> please notify the sender immediately and do not disclose the contents to any >> other person, use it for any purpose, or store or copy the information in any >> medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-11-28 11:19 ` Divin Raj @ 2023-11-28 13:45 ` Arnaud POULIQUEN 2023-12-07 12:47 ` Divin Raj 2024-01-12 22:39 ` Mathieu Poirier 1 sibling, 1 reply; 10+ messages in thread From: Arnaud POULIQUEN @ 2023-11-28 13:45 UTC (permalink / raw) To: Divin Raj, linux-remoteproc; +Cc: Rahul.Singh@arm.com On 11/28/23 12:19, Divin Raj wrote: > On 11/28/23 8:34 AM, Arnaud POULIQUEN wrote: >> >> >> On 11/24/23 17:45, Divin Raj wrote: >>> Hi Arnaud, >>> Please find my comments inline. >>> >>> On 11/20/23 10:14 AM, Arnaud POULIQUEN wrote: >>>> Hi Divin, >>>> >>>> On 11/17/23 23:24, Divin Raj wrote: >>>>> On 10/23/23 11:44 AM, Divin Raj wrote: >>>>>> Hello all, >>>>>> >>>>>> I am reaching out with reference to the patch discussed here: Enhanced >>>>>> virtio rpmsg bus driver buffer allocation. >>>>>> <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> >>>>>> >>>>>> I've been keenly following the developments around enhancing buffer >>>>>> allocation strategies, especially those focused on dynamic buffer sizing >>>>>> and the considerations for systems under varying memory constraints.This >>>>>> work is highly relevant to several projects I am involved in, and I am >>>>>> quite interested in its progression. May I kindly request an update on >>>>>> the current phase of these initiatives? Additionally, I am eager to know >>>>>> if there would be an opportunity for me to contribute to enhancing the >>>>>> patch, possibly by working on improvements or assisting in verification >>>>>> processes. >>>>>> >>>>>> Furthermore, if there are any condensed resources, summaries, or >>>>>> specific threads that encapsulate recent advancements or discussions on >>>>>> this topic, I would be grateful to receive directions to them. >>>>>> >>>>>> I appreciate everyone's dedicated efforts and invaluable contributions >>>>>> to this area of development. Looking forward to the updates. >>>>>> >>>>>> Regards Divin >>>>>> >>>>> Hello Linux Community, >>>>> >>>>> In one of our internal projects, we encountered a challenge with RPMSG >>>>> buffer allocation. Our goal is to optimize memory allocation for an >>>>> out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure >>>>> support for packet sizes matching the standard MTU (Maximum Transmission >>>>> Unit) size of 1500 bytes. >>>>> >>>>> To mitigate this issue, There are few possible solutions: >>>>> >>>>> 1. Configure buffer size and number through Kconfig. >>>>> 2. Permit the firmware creator to determine the most suitable value from >>>>> the resource table. >>>>> 3. Enable independent configurations on both ends. This approach would >>>>> support both dynamic and fixed buffer configurations using a generic >>>>> allocator. >>>>> >>>>> Reference: >>>>> >>>>> [1]: >>>>> https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ >>>>> [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ >>>>> >>>>> >>>>> Draft Design Overview: >>>>> >>>>> Based on the reference patch and the discussions, we have outlined the >>>>> following key points for the belw design: >>>>> >>>>> 1. Assure compatibility, enabling both Linux and the remote system to >>>>> interchangeably transmit and receive messages, irrespective of size. >>>>> 2. For systems with constrained shared memory: >>>>> Systems with small, shared memory, we need to deal with a >>>>> limited/optimized memory chunk. To avoid memory fragmentation, the >>>>> allocator should have a pre-reserved buffer pool >>>>> 3. The implementation should ensure that the remote side does not >>>>> receive messages based on its allocation parameters. >>>>> >>>>> do you think it could make sense? >>>>> >>>>> High level view: >>>>> +------------------+ +------------------+ >>>>> | | | | >>>>> | Linux | | Remote | >>>>> | | | | >>>>> | +----------+ | +-----------------+ | +----------+ | >>>>> | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | >>>>> | +----------+ | | (Dynamic/Static)| | +----------+ | >>>>> | | +-----------------+ | | >>>>> +------------------+ +------------------+ >>>>> >>>>> >>>>> Detailed view: >>>>> >>>>> +-------------------------+ >>>>> | Message Creation | >>>>> | (Both Linux/Remote) | >>>>> +------------+------------+ >>>>> | >>>>> v >>>>> +-------------------------+ >>>>> | Determine the allocation| >>>>> | strategy | >>>>> +------------+------------+ >>>>> | >>>>> +--------------+--------------+ >>>>> | | >>>>> +-------------------------------+ +-------------------------------+ >>>>> | Dynamic allocation | | Static allocation | >>>>> | (Buffer allocator allocates | | (Pre-reserved memory | >>>>> | memory space as needed, | | space) | >>>>> | based on the current | | | >>>>> | message requirement ) | | | >>>>> +-------------------------------+ +-------------------------------+ >>>> >>>> Do you have a proposal for dynamic allocation? >>>> >>>> RPMSG is based on the virtio protocol. The virtio driver in the Linux kernel >>>> is responsible for allocating buffers for the virtio device on the remote >>>> processor. >>>> >>>> In the current implementation (static allocation) the Linux >>>> kernel allocates predefined buffers for the remote processor. >>>> >>>> How would you manage the fact that the sender allocates its own buffers and >>>> references >>>> them in the vring descriptor? This would require each core to have >>>> a dual role, right? >>>> - a virtio driver role on its TX vring >>>> - a virtio device role on its RX vring." >>>> >>> I'm unsure if a dual role is feasible under the Virtio specification. >> >> At least, it does not seem to align with the philosophy of VirtIO. >> >> >>> However, would it make sense to set the size of the outbuf based on the >>> Maximum Transmission Unit (MTU) size that is supported? Additionally, >>> the size of the inbuf could be set by the firmware, suggesting that it >>> should be derived from the resource table. With this approach, I believe >>> the sender can decide the maximum size. >> >> It is not clear to me what your proposal is. >> Are you speaking about a pre-allocated buffers as proposed in [1], >> or are you speaking about dynamic allocation of the RPMsg in a pool? > > we are at the initial phase of this investigation. As we previously > discussed, option 3 is not feasible in accordance with the virtio > specification.The above proposed solution aligns with [1], suggesting > preallocated in_buf and out_buf, with sizes determined from the resource > table and MTU. By allowing Linux to decide the out_buf size and the > remote to decide the in_buf size, I believe we can avoid conflicts. If > everyone agrees on a common idea, then it would be a good starting point Thanks for the clarification. It seems reasonable to me to start with a pre-allocated buffer with a fixed size specified by the remote firmware. Bjorn, Mathieu, Please, could you share you point of view on the topic? Thanks, Arnaud > > Regards > Divin > >> Regards, >> Arnaud >> >>> >>> Regards >>> Divin >>> >>>> >>>> Regards, >>>> Arnaud >>>> >>> >>>> >>>>> >>>>> We would greatly appreciate any feedback, suggestions, or improvements >>>>> you could provide. >>>>> >>>>> Thank you for your time and consideration. >>>>> >>>>> Regards >>>>> Divin >>>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>>> confidential and may also be privileged. If you are not the intended >>>>> recipient, >>>>> please notify the sender immediately and do not disclose the contents to any >>>>> other person, use it for any purpose, or store or copy the information in any >>>>> medium. Thank you. >>> >>> IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended recipient, >>> please notify the sender immediately and do not disclose the contents to any >>> other person, use it for any purpose, or store or copy the information in any >>> medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended recipient, > please notify the sender immediately and do not disclose the contents to any > other person, use it for any purpose, or store or copy the information in any > medium. Thank you. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-11-28 13:45 ` Arnaud POULIQUEN @ 2023-12-07 12:47 ` Divin Raj 2024-01-11 17:33 ` Mathieu Poirier 0 siblings, 1 reply; 10+ messages in thread From: Divin Raj @ 2023-12-07 12:47 UTC (permalink / raw) To: Arnaud POULIQUEN, linux-remoteproc, Bjorn Andersson, Mathieu Poirier Cc: Rahul.Singh@arm.com On 11/28/23 1:45 PM, Arnaud POULIQUEN wrote: > > > On 11/28/23 12:19, Divin Raj wrote: >> On 11/28/23 8:34 AM, Arnaud POULIQUEN wrote: >>> >>> >>> On 11/24/23 17:45, Divin Raj wrote: >>>> Hi Arnaud, >>>> Please find my comments inline. >>>> >>>> On 11/20/23 10:14 AM, Arnaud POULIQUEN wrote: >>>>> Hi Divin, >>>>> >>>>> On 11/17/23 23:24, Divin Raj wrote: >>>>>> On 10/23/23 11:44 AM, Divin Raj wrote: >>>>>>> Hello all, >>>>>>> >>>>>>> I am reaching out with reference to the patch discussed here: Enhanced >>>>>>> virtio rpmsg bus driver buffer allocation. >>>>>>> <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> >>>>>>> >>>>>>> I've been keenly following the developments around enhancing buffer >>>>>>> allocation strategies, especially those focused on dynamic buffer sizing >>>>>>> and the considerations for systems under varying memory constraints.This >>>>>>> work is highly relevant to several projects I am involved in, and I am >>>>>>> quite interested in its progression. May I kindly request an update on >>>>>>> the current phase of these initiatives? Additionally, I am eager to know >>>>>>> if there would be an opportunity for me to contribute to enhancing the >>>>>>> patch, possibly by working on improvements or assisting in verification >>>>>>> processes. >>>>>>> >>>>>>> Furthermore, if there are any condensed resources, summaries, or >>>>>>> specific threads that encapsulate recent advancements or discussions on >>>>>>> this topic, I would be grateful to receive directions to them. >>>>>>> >>>>>>> I appreciate everyone's dedicated efforts and invaluable contributions >>>>>>> to this area of development. Looking forward to the updates. >>>>>>> >>>>>>> Regards Divin >>>>>>> >>>>>> Hello Linux Community, >>>>>> >>>>>> In one of our internal projects, we encountered a challenge with RPMSG >>>>>> buffer allocation. Our goal is to optimize memory allocation for an >>>>>> out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure >>>>>> support for packet sizes matching the standard MTU (Maximum Transmission >>>>>> Unit) size of 1500 bytes. >>>>>> >>>>>> To mitigate this issue, There are few possible solutions: >>>>>> >>>>>> 1. Configure buffer size and number through Kconfig. >>>>>> 2. Permit the firmware creator to determine the most suitable value from >>>>>> the resource table. >>>>>> 3. Enable independent configurations on both ends. This approach would >>>>>> support both dynamic and fixed buffer configurations using a generic >>>>>> allocator. >>>>>> >>>>>> Reference: >>>>>> >>>>>> [1]: >>>>>> https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ >>>>>> [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ >>>>>> >>>>>> >>>>>> Draft Design Overview: >>>>>> >>>>>> Based on the reference patch and the discussions, we have outlined the >>>>>> following key points for the belw design: >>>>>> >>>>>> 1. Assure compatibility, enabling both Linux and the remote system to >>>>>> interchangeably transmit and receive messages, irrespective of size. >>>>>> 2. For systems with constrained shared memory: >>>>>> Systems with small, shared memory, we need to deal with a >>>>>> limited/optimized memory chunk. To avoid memory fragmentation, the >>>>>> allocator should have a pre-reserved buffer pool >>>>>> 3. The implementation should ensure that the remote side does not >>>>>> receive messages based on its allocation parameters. >>>>>> >>>>>> do you think it could make sense? >>>>>> >>>>>> High level view: >>>>>> +------------------+ +------------------+ >>>>>> | | | | >>>>>> | Linux | | Remote | >>>>>> | | | | >>>>>> | +----------+ | +-----------------+ | +----------+ | >>>>>> | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | >>>>>> | +----------+ | | (Dynamic/Static)| | +----------+ | >>>>>> | | +-----------------+ | | >>>>>> +------------------+ +------------------+ >>>>>> >>>>>> >>>>>> Detailed view: >>>>>> >>>>>> +-------------------------+ >>>>>> | Message Creation | >>>>>> | (Both Linux/Remote) | >>>>>> +------------+------------+ >>>>>> | >>>>>> v >>>>>> +-------------------------+ >>>>>> | Determine the allocation| >>>>>> | strategy | >>>>>> +------------+------------+ >>>>>> | >>>>>> +--------------+--------------+ >>>>>> | | >>>>>> +-------------------------------+ +-------------------------------+ >>>>>> | Dynamic allocation | | Static allocation | >>>>>> | (Buffer allocator allocates | | (Pre-reserved memory | >>>>>> | memory space as needed, | | space) | >>>>>> | based on the current | | | >>>>>> | message requirement ) | | | >>>>>> +-------------------------------+ +-------------------------------+ >>>>> >>>>> Do you have a proposal for dynamic allocation? >>>>> >>>>> RPMSG is based on the virtio protocol. The virtio driver in the Linux kernel >>>>> is responsible for allocating buffers for the virtio device on the remote >>>>> processor. >>>>> >>>>> In the current implementation (static allocation) the Linux >>>>> kernel allocates predefined buffers for the remote processor. >>>>> >>>>> How would you manage the fact that the sender allocates its own buffers and >>>>> references >>>>> them in the vring descriptor? This would require each core to have >>>>> a dual role, right? >>>>> - a virtio driver role on its TX vring >>>>> - a virtio device role on its RX vring." >>>>> >>>> I'm unsure if a dual role is feasible under the Virtio specification. >>> >>> At least, it does not seem to align with the philosophy of VirtIO. >>> >>> >>>> However, would it make sense to set the size of the outbuf based on the >>>> Maximum Transmission Unit (MTU) size that is supported? Additionally, >>>> the size of the inbuf could be set by the firmware, suggesting that it >>>> should be derived from the resource table. With this approach, I believe >>>> the sender can decide the maximum size. >>> >>> It is not clear to me what your proposal is. >>> Are you speaking about a pre-allocated buffers as proposed in [1], >>> or are you speaking about dynamic allocation of the RPMsg in a pool? >> >> we are at the initial phase of this investigation. As we previously >> discussed, option 3 is not feasible in accordance with the virtio >> specification.The above proposed solution aligns with [1], suggesting >> preallocated in_buf and out_buf, with sizes determined from the resource >> table and MTU. By allowing Linux to decide the out_buf size and the >> remote to decide the in_buf size, I believe we can avoid conflicts. If >> everyone agrees on a common idea, then it would be a good starting point > > Thanks for the clarification. It seems reasonable to me to start with a > pre-allocated buffer with a fixed size specified by the remote firmware. > > Bjorn, Mathieu, > Please, could you share you point of view on the topic? > + Bjorn + Mathieu > Thanks, > Arnaud > >> >> Regards >> Divin >> >>> Regards, >>> Arnaud >>> >>>> >>>> Regards >>>> Divin >>>> >>>>> >>>>> Regards, >>>>> Arnaud >>>>> >>>> >>>>> >>>>>> >>>>>> We would greatly appreciate any feedback, suggestions, or improvements >>>>>> you could provide. >>>>>> >>>>>> Thank you for your time and consideration. >>>>>> >>>>>> Regards >>>>>> Divin IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-12-07 12:47 ` Divin Raj @ 2024-01-11 17:33 ` Mathieu Poirier 0 siblings, 0 replies; 10+ messages in thread From: Mathieu Poirier @ 2024-01-11 17:33 UTC (permalink / raw) To: Divin Raj Cc: Arnaud POULIQUEN, linux-remoteproc, Bjorn Andersson, Rahul.Singh@arm.com On Thu, Dec 07, 2023 at 12:47:02PM +0000, Divin Raj wrote: > On 11/28/23 1:45 PM, Arnaud POULIQUEN wrote: > > > > > > On 11/28/23 12:19, Divin Raj wrote: > > > On 11/28/23 8:34 AM, Arnaud POULIQUEN wrote: > > > > > > > > > > > > On 11/24/23 17:45, Divin Raj wrote: > > > > > Hi Arnaud, > > > > > Please find my comments inline. > > > > > > > > > > On 11/20/23 10:14 AM, Arnaud POULIQUEN wrote: > > > > > > Hi Divin, > > > > > > > > > > > > On 11/17/23 23:24, Divin Raj wrote: > > > > > > > On 10/23/23 11:44 AM, Divin Raj wrote: > > > > > > > > Hello all, > > > > > > > > > > > > > > > > I am reaching out with reference to the patch discussed here: Enhanced > > > > > > > > virtio rpmsg bus driver buffer allocation. > > > > > > > > <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> > > > > > > > > > > > > > > > > I've been keenly following the developments around enhancing buffer > > > > > > > > allocation strategies, especially those focused on dynamic buffer sizing > > > > > > > > and the considerations for systems under varying memory constraints.This > > > > > > > > work is highly relevant to several projects I am involved in, and I am > > > > > > > > quite interested in its progression. May I kindly request an update on > > > > > > > > the current phase of these initiatives? Additionally, I am eager to know > > > > > > > > if there would be an opportunity for me to contribute to enhancing the > > > > > > > > patch, possibly by working on improvements or assisting in verification > > > > > > > > processes. > > > > > > > > > > > > > > > > Furthermore, if there are any condensed resources, summaries, or > > > > > > > > specific threads that encapsulate recent advancements or discussions on > > > > > > > > this topic, I would be grateful to receive directions to them. > > > > > > > > > > > > > > > > I appreciate everyone's dedicated efforts and invaluable contributions > > > > > > > > to this area of development. Looking forward to the updates. > > > > > > > > > > > > > > > > Regards Divin > > > > > > > > > > > > > > > Hello Linux Community, > > > > > > > > > > > > > > In one of our internal projects, we encountered a challenge with RPMSG > > > > > > > buffer allocation. Our goal is to optimize memory allocation for an > > > > > > > out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure > > > > > > > support for packet sizes matching the standard MTU (Maximum Transmission > > > > > > > Unit) size of 1500 bytes. > > > > > > > > > > > > > > To mitigate this issue, There are few possible solutions: > > > > > > > > > > > > > > 1. Configure buffer size and number through Kconfig. > > > > > > > 2. Permit the firmware creator to determine the most suitable value from > > > > > > > the resource table. > > > > > > > 3. Enable independent configurations on both ends. This approach would > > > > > > > support both dynamic and fixed buffer configurations using a generic > > > > > > > allocator. > > > > > > > > > > > > > > Reference: > > > > > > > > > > > > > > [1]: > > > > > > > https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ > > > > > > > [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ > > > > > > > > > > > > > > > > > > > > > Draft Design Overview: > > > > > > > > > > > > > > Based on the reference patch and the discussions, we have outlined the > > > > > > > following key points for the belw design: > > > > > > > > > > > > > > 1. Assure compatibility, enabling both Linux and the remote system to > > > > > > > interchangeably transmit and receive messages, irrespective of size. > > > > > > > 2. For systems with constrained shared memory: > > > > > > > Systems with small, shared memory, we need to deal with a > > > > > > > limited/optimized memory chunk. To avoid memory fragmentation, the > > > > > > > allocator should have a pre-reserved buffer pool > > > > > > > 3. The implementation should ensure that the remote side does not > > > > > > > receive messages based on its allocation parameters. > > > > > > > > > > > > > > do you think it could make sense? > > > > > > > > > > > > > > High level view: > > > > > > > +------------------+ +------------------+ > > > > > > > | | | | > > > > > > > | Linux | | Remote | > > > > > > > | | | | > > > > > > > | +----------+ | +-----------------+ | +----------+ | > > > > > > > | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | > > > > > > > | +----------+ | | (Dynamic/Static)| | +----------+ | > > > > > > > | | +-----------------+ | | > > > > > > > +------------------+ +------------------+ > > > > > > > > > > > > > > > > > > > > > Detailed view: > > > > > > > > > > > > > > +-------------------------+ > > > > > > > | Message Creation | > > > > > > > | (Both Linux/Remote) | > > > > > > > +------------+------------+ > > > > > > > | > > > > > > > v > > > > > > > +-------------------------+ > > > > > > > | Determine the allocation| > > > > > > > | strategy | > > > > > > > +------------+------------+ > > > > > > > | > > > > > > > +--------------+--------------+ > > > > > > > | | > > > > > > > +-------------------------------+ +-------------------------------+ > > > > > > > | Dynamic allocation | | Static allocation | > > > > > > > | (Buffer allocator allocates | | (Pre-reserved memory | > > > > > > > | memory space as needed, | | space) | > > > > > > > | based on the current | | | > > > > > > > | message requirement ) | | | > > > > > > > +-------------------------------+ +-------------------------------+ > > > > > > > > > > > > Do you have a proposal for dynamic allocation? > > > > > > > > > > > > RPMSG is based on the virtio protocol. The virtio driver in the Linux kernel > > > > > > is responsible for allocating buffers for the virtio device on the remote > > > > > > processor. > > > > > > > > > > > > In the current implementation (static allocation) the Linux > > > > > > kernel allocates predefined buffers for the remote processor. > > > > > > > > > > > > How would you manage the fact that the sender allocates its own buffers and > > > > > > references > > > > > > them in the vring descriptor? This would require each core to have > > > > > > a dual role, right? > > > > > > - a virtio driver role on its TX vring > > > > > > - a virtio device role on its RX vring." > > > > > > > > > > > I'm unsure if a dual role is feasible under the Virtio specification. > > > > > > > > At least, it does not seem to align with the philosophy of VirtIO. > > > > > > > > > > > > > However, would it make sense to set the size of the outbuf based on the > > > > > Maximum Transmission Unit (MTU) size that is supported? Additionally, > > > > > the size of the inbuf could be set by the firmware, suggesting that it > > > > > should be derived from the resource table. With this approach, I believe > > > > > the sender can decide the maximum size. > > > > > > > > It is not clear to me what your proposal is. > > > > Are you speaking about a pre-allocated buffers as proposed in [1], > > > > or are you speaking about dynamic allocation of the RPMsg in a pool? > > > > > > we are at the initial phase of this investigation. As we previously > > > discussed, option 3 is not feasible in accordance with the virtio > > > specification.The above proposed solution aligns with [1], suggesting > > > preallocated in_buf and out_buf, with sizes determined from the resource > > > table and MTU. By allowing Linux to decide the out_buf size and the > > > remote to decide the in_buf size, I believe we can avoid conflicts. If > > > everyone agrees on a common idea, then it would be a good starting point > > > > Thanks for the clarification. It seems reasonable to me to start with a > > pre-allocated buffer with a fixed size specified by the remote firmware. > > > > Bjorn, Mathieu, > > Please, could you share you point of view on the topic? > > > + Bjorn > + Mathieu Arnaud brought this conversation to my attention. I will address it in the coming days. > > > Thanks, > > Arnaud > > > > > > > > Regards > > > Divin > > > > > > > Regards, > > > > Arnaud > > > > > > > > > > > > > > Regards > > > > > Divin > > > > > > > > > > > > > > > > > Regards, > > > > > > Arnaud > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We would greatly appreciate any feedback, suggestions, or improvements > > > > > > > you could provide. > > > > > > > > > > > > > > Thank you for your time and consideration. > > > > > > > > > > > > > > Regards > > > > > > > Divin > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-11-28 11:19 ` Divin Raj 2023-11-28 13:45 ` Arnaud POULIQUEN @ 2024-01-12 22:39 ` Mathieu Poirier 1 sibling, 0 replies; 10+ messages in thread From: Mathieu Poirier @ 2024-01-12 22:39 UTC (permalink / raw) To: Divin Raj; +Cc: Arnaud POULIQUEN, linux-remoteproc, Rahul.Singh@arm.com On Tue, Nov 28, 2023 at 11:19:10AM +0000, Divin Raj wrote: > On 11/28/23 8:34 AM, Arnaud POULIQUEN wrote: > > > > > > On 11/24/23 17:45, Divin Raj wrote: > > > Hi Arnaud, > > > Please find my comments inline. > > > > > > On 11/20/23 10:14 AM, Arnaud POULIQUEN wrote: > > > > Hi Divin, > > > > > > > > On 11/17/23 23:24, Divin Raj wrote: > > > > > On 10/23/23 11:44 AM, Divin Raj wrote: > > > > > > Hello all, > > > > > > > > > > > > I am reaching out with reference to the patch discussed here: Enhanced > > > > > > virtio rpmsg bus driver buffer allocation. > > > > > > <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> > > > > > > > > > > > > I've been keenly following the developments around enhancing buffer > > > > > > allocation strategies, especially those focused on dynamic buffer sizing > > > > > > and the considerations for systems under varying memory constraints.This > > > > > > work is highly relevant to several projects I am involved in, and I am > > > > > > quite interested in its progression. May I kindly request an update on > > > > > > the current phase of these initiatives? Additionally, I am eager to know > > > > > > if there would be an opportunity for me to contribute to enhancing the > > > > > > patch, possibly by working on improvements or assisting in verification > > > > > > processes. > > > > > > > > > > > > Furthermore, if there are any condensed resources, summaries, or > > > > > > specific threads that encapsulate recent advancements or discussions on > > > > > > this topic, I would be grateful to receive directions to them. > > > > > > > > > > > > I appreciate everyone's dedicated efforts and invaluable contributions > > > > > > to this area of development. Looking forward to the updates. > > > > > > > > > > > > Regards Divin > > > > > > > > > > > Hello Linux Community, > > > > > > > > > > In one of our internal projects, we encountered a challenge with RPMSG > > > > > buffer allocation. Our goal is to optimize memory allocation for an > > > > > out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure > > > > > support for packet sizes matching the standard MTU (Maximum Transmission > > > > > Unit) size of 1500 bytes. > > > > > > > > > > To mitigate this issue, There are few possible solutions: > > > > > > > > > > 1. Configure buffer size and number through Kconfig. > > > > > 2. Permit the firmware creator to determine the most suitable value from > > > > > the resource table. > > > > > 3. Enable independent configurations on both ends. This approach would > > > > > support both dynamic and fixed buffer configurations using a generic > > > > > allocator. > > > > > > > > > > Reference: > > > > > > > > > > [1]: > > > > > https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ > > > > > [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ > > > > > > > > > > > > > > > Draft Design Overview: > > > > > > > > > > Based on the reference patch and the discussions, we have outlined the > > > > > following key points for the belw design: > > > > > > > > > > 1. Assure compatibility, enabling both Linux and the remote system to > > > > > interchangeably transmit and receive messages, irrespective of size. > > > > > 2. For systems with constrained shared memory: > > > > > Systems with small, shared memory, we need to deal with a > > > > > limited/optimized memory chunk. To avoid memory fragmentation, the > > > > > allocator should have a pre-reserved buffer pool > > > > > 3. The implementation should ensure that the remote side does not > > > > > receive messages based on its allocation parameters. > > > > > > > > > > do you think it could make sense? > > > > > > > > > > High level view: > > > > > +------------------+ +------------------+ > > > > > | | | | > > > > > | Linux | | Remote | > > > > > | | | | > > > > > | +----------+ | +-----------------+ | +----------+ | > > > > > | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | > > > > > | +----------+ | | (Dynamic/Static)| | +----------+ | > > > > > | | +-----------------+ | | > > > > > +------------------+ +------------------+ > > > > > > > > > > > > > > > Detailed view: > > > > > > > > > > +-------------------------+ > > > > > | Message Creation | > > > > > | (Both Linux/Remote) | > > > > > +------------+------------+ > > > > > | > > > > > v > > > > > +-------------------------+ > > > > > | Determine the allocation| > > > > > | strategy | > > > > > +------------+------------+ > > > > > | > > > > > +--------------+--------------+ > > > > > | | > > > > > +-------------------------------+ +-------------------------------+ > > > > > | Dynamic allocation | | Static allocation | > > > > > | (Buffer allocator allocates | | (Pre-reserved memory | > > > > > | memory space as needed, | | space) | > > > > > | based on the current | | | > > > > > | message requirement ) | | | > > > > > +-------------------------------+ +-------------------------------+ > > > > > > > > Do you have a proposal for dynamic allocation? > > > > > > > > RPMSG is based on the virtio protocol. The virtio driver in the Linux kernel > > > > is responsible for allocating buffers for the virtio device on the remote > > > > processor. > > > > > > > > In the current implementation (static allocation) the Linux > > > > kernel allocates predefined buffers for the remote processor. > > > > > > > > How would you manage the fact that the sender allocates its own buffers and > > > > references > > > > them in the vring descriptor? This would require each core to have > > > > a dual role, right? > > > > - a virtio driver role on its TX vring > > > > - a virtio device role on its RX vring." > > > > No matter how many times I read this thread I can't figure out what is mean when using the word "role". > > > I'm unsure if a dual role is feasible under the Virtio specification. > > > > At least, it does not seem to align with the philosophy of VirtIO. > > Dynamic memory allocation based on packet size would make the implementation quite complex, and I don't remember any virtio device that do this. > > > > > However, would it make sense to set the size of the outbuf based on the > > > Maximum Transmission Unit (MTU) size that is supported? Additionally, > > > the size of the inbuf could be set by the firmware, suggesting that it > > > should be derived from the resource table. With this approach, I believe > > > the sender can decide the maximum size. > > > > It is not clear to me what your proposal is. > > Are you speaking about a pre-allocated buffers as proposed in [1], > > or are you speaking about dynamic allocation of the RPMsg in a pool? > > we are at the initial phase of this investigation. As we previously > discussed, option 3 is not feasible in accordance with the virtio > specification.The above proposed solution aligns with [1], suggesting > preallocated in_buf and out_buf, with sizes determined from the resource > table and MTU. By allowing Linux to decide the out_buf size and the > remote to decide the in_buf size, I believe we can avoid conflicts. If > everyone agrees on a common idea, then it would be a good starting point > I think Xiao's proposal in [1] was on the right track. There were a few things to adjust like the type used in the rpmsg_config structure (_u32 rather than le32), the lack of a version number and the problem generated by the double memory allocation. The latter should likely be cobbled in a single allocation. That said it leaves no room for error about the size of the rx and tx buffers. Lastly, any implementation needs to be standard with enhancements to the rpmsg config space accepted by OASIS. I hope this is enough to move forward - let me know if it isn't the case. > Regards > Divin > > > Regards, > > Arnaud > > > > > > > > Regards > > > Divin > > > > > > > > > > > Regards, > > > > Arnaud > > > > > > > > > > > > > > > > > > > > > We would greatly appreciate any feedback, suggestions, or improvements > > > > > you could provide. > > > > > > > > > > Thank you for your time and consideration. > > > > > > > > > > Regards > > > > > Divin > > > > > IMPORTANT NOTICE: The contents of this email and any attachments are > > > > > confidential and may also be privileged. If you are not the intended recipient, > > > > > please notify the sender immediately and do not disclose the contents to any > > > > > other person, use it for any purpose, or store or copy the information in any > > > > > medium. Thank you. > > > > > > IMPORTANT NOTICE: The contents of this email and any attachments are > > > confidential and may also be privileged. If you are not the intended recipient, > > > please notify the sender immediately and do not disclose the contents to any > > > other person, use it for any purpose, or store or copy the information in any > > > medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Discussion]: Enhance virtio rpmsg bus driver buffer allocation 2023-11-17 22:24 ` [Discussion]: Enhance virtio rpmsg bus driver buffer allocation Divin Raj 2023-11-20 10:14 ` Arnaud POULIQUEN @ 2023-11-21 5:54 ` Tanmay Shah 1 sibling, 0 replies; 10+ messages in thread From: Tanmay Shah @ 2023-11-21 5:54 UTC (permalink / raw) To: Divin Raj, linux-remoteproc Hi Divin, Please find my comments inline. On 11/17/23 4:24 PM, Divin Raj wrote: > CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. > > > On 10/23/23 11:44 AM, Divin Raj wrote: > > Hello all, > > > > I am reaching out with reference to the patch discussed here: Enhanced > > virtio rpmsg bus driver buffer allocation. > > <https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/> > > > > I've been keenly following the developments around enhancing buffer > > allocation strategies, especially those focused on dynamic buffer sizing > > and the considerations for systems under varying memory constraints.This > > work is highly relevant to several projects I am involved in, and I am > > quite interested in its progression. May I kindly request an update on > > the current phase of these initiatives? Additionally, I am eager to know > > if there would be an opportunity for me to contribute to enhancing the > > patch, possibly by working on improvements or assisting in verification > > processes. > > > > Furthermore, if there are any condensed resources, summaries, or > > specific threads that encapsulate recent advancements or discussions on > > this topic, I would be grateful to receive directions to them. > > > > I appreciate everyone's dedicated efforts and invaluable contributions > > to this area of development. Looking forward to the updates. > > > > Regards Divin > > > Hello Linux Community, > > In one of our internal projects, we encountered a challenge with RPMSG > buffer allocation. Our goal is to optimize memory allocation for an > out-of-tree RPMSG Ethernet device driver using virtio. This is to ensure > support for packet sizes matching the standard MTU (Maximum Transmission > Unit) size of 1500 bytes. > > To mitigate this issue, There are few possible solutions: > > 1. Configure buffer size and number through Kconfig. This has been posted multiple times on mailing list, but really doesn't solve the problem. > 2. Permit the firmware creator to determine the most suitable value from > the resource table. This could be one of the solution, but if we can achieve dynamic allocation (below 3) we prefer that. > 3. Enable independent configurations on both ends. This approach would > support both dynamic and fixed buffer configurations using a generic > allocator. > > Reference: > > [1]: > https://lore.kernel.org/all/1548949280-31794-4-git-send-email-xiaoxiang@xiaomi.com/ > [2]: https://lore.kernel.org/all/20190701061353.GE1263@builder/ > We can use above patch to start with. I did not have chance to look at above yet, but we had discussed it before and seems like a good solution. > Draft Design Overview: > > Based on the reference patch and the discussions, we have outlined the > following key points for the belw design: > > 1. Assure compatibility, enabling both Linux and the remote system to > interchangeably transmit and receive messages, irrespective of size. > 2. For systems with constrained shared memory: > Systems with small, shared memory, we need to deal with a > limited/optimized memory chunk. To avoid memory fragmentation, the > allocator should have a pre-reserved buffer pool > 3. The implementation should ensure that the remote side does not > receive messages based on its allocation parameters. > > do you think it could make sense? > > High level view: > +------------------+ +------------------+ > | | | | > | Linux | | Remote | > | | | | > | +----------+ | +-----------------+ | +----------+ | > | | RPMSG | | <---> | Buffer Allocator|<--->| | RPMSG | | > | +----------+ | | (Dynamic/Static)| | +----------+ | > | | +-----------------+ | | > +------------------+ +------------------+ > > > Detailed view: > > +-------------------------+ > | Message Creation | > | (Both Linux/Remote) | > +------------+------------+ > | > v > +-------------------------+ > | Determine the allocation| > | strategy | > +------------+------------+ > | > +--------------+--------------+ > | | > +-------------------------------+ +-------------------------------+ > | Dynamic allocation | | Static allocation | > | (Buffer allocator allocates | | (Pre-reserved memory | > | memory space as needed, | | space) | > | based on the current | | | > | message requirement ) | | | > +-------------------------------+ +-------------------------------+ > > We would greatly appreciate any feedback, suggestions, or improvements > you could provide. > > Thank you for your time and consideration. > > Regards > Divin > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. When working with upstream list, could you try to remove above footer please ? ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-01-12 22:39 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1af16ff8-5706-45e5-9737-05da39957c95@arm.com>
2023-11-17 22:24 ` [Discussion]: Enhance virtio rpmsg bus driver buffer allocation Divin Raj
2023-11-20 10:14 ` Arnaud POULIQUEN
2023-11-24 16:45 ` Divin Raj
2023-11-28 8:34 ` Arnaud POULIQUEN
2023-11-28 11:19 ` Divin Raj
2023-11-28 13:45 ` Arnaud POULIQUEN
2023-12-07 12:47 ` Divin Raj
2024-01-11 17:33 ` Mathieu Poirier
2024-01-12 22:39 ` Mathieu Poirier
2023-11-21 5:54 ` Tanmay Shah
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox