[PATCH] usb: gadget: u_ether: remove interrupt throttling

[PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

According to Dave Miller "the networking stack has a
hard requirement that all SKBs which are transmitted
must have their completion signalled in a fininte
amount of time. This is because, until the SKB is
freed by the driver, it holds onto socket,
netfilter, and other subsystem resources."

In summary, this means that using TX IRQ throttling
for the networking gadgets is, at least, complex and
we should avoid it for the time being.

Cc: <stable@vger.kernel.org>
Reported-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Suggested-by: David Miller <davem@davemloft.net>
Signed-off-by: Felipe Balbi <felipe.balbi@linux.intel.com>
---
 drivers/usb/gadget/function/u_ether.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/drivers/usb/gadget/function/u_ether.c b/drivers/usb/gadget/function/u_ether.c
index f4a640216913..119a2e5848e8 100644
--- a/drivers/usb/gadget/function/u_ether.c
+++ b/drivers/usb/gadget/function/u_ether.c
@@ -589,14 +589,6 @@ static netdev_tx_t eth_start_xmit(struct sk_buff *skb,
 
 	req->length = length;
 
-	/* throttle high/super speed IRQ rate back slightly */
-	if (gadget_is_dualspeed(dev->gadget))
-		req->no_interrupt = (((dev->gadget->speed == USB_SPEED_HIGH ||
-				       dev->gadget->speed == USB_SPEED_SUPER)) &&
-					!list_empty(&dev->tx_reqs))
-			? ((atomic_read(&dev->tx_qlen) % dev->qmult) != 0)
-			: 0;
-
 	retval = usb_ep_queue(in, req, GFP_ATOMIC);
 	switch (retval) {
 	default:
-- 
2.10.1

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Ville Syrjälä]

On Tue, Nov 01, 2016 at 01:29:59PM +0200, Felipe Balbi wrote:
> According to Dave Miller "the networking stack has a
> hard requirement that all SKBs which are transmitted
> must have their completion signalled in a fininte
> amount of time. This is because, until the SKB is
> freed by the driver, it holds onto socket,
> netfilter, and other subsystem resources."
> 
> In summary, this means that using TX IRQ throttling
> for the networking gadgets is, at least, complex and
> we should avoid it for the time being.
> 
> Cc: <stable@vger.kernel.org>
> Reported-by: Ville Syrj�l� <ville.syrjala@linux.intel.com>
> Suggested-by: David Miller <davem@davemloft.net>
> Signed-off-by: Felipe Balbi <felipe.balbi@linux.intel.com>

Fixes laggy/hangy g_ether on my BYT FFRD8 tablet, caused by
commit 55a0237f8f47 ("usb: dwc3: gadget: use allocated/queued reqs for LST bit")

Tested-by: Ville Syrj�l� <ville.syrjala@linux.intel.com>

> ---
>  drivers/usb/gadget/function/u_ether.c | 8 --------
>  1 file changed, 8 deletions(-)
> 
> diff --git a/drivers/usb/gadget/function/u_ether.c b/drivers/usb/gadget/function/u_ether.c
> index f4a640216913..119a2e5848e8 100644
> --- a/drivers/usb/gadget/function/u_ether.c
> +++ b/drivers/usb/gadget/function/u_ether.c
> @@ -589,14 +589,6 @@ static netdev_tx_t eth_start_xmit(struct sk_buff *skb,
>  
>  	req->length = length;
>  
> -	/* throttle high/super speed IRQ rate back slightly */
> -	if (gadget_is_dualspeed(dev->gadget))
> -		req->no_interrupt = (((dev->gadget->speed == USB_SPEED_HIGH ||
> -				       dev->gadget->speed == USB_SPEED_SUPER)) &&
> -					!list_empty(&dev->tx_reqs))
> -			? ((atomic_read(&dev->tx_qlen) % dev->qmult) != 0)
> -			: 0;
> -
>  	retval = usb_ep_queue(in, req, GFP_ATOMIC);
>  	switch (retval) {
>  	default:
> -- 
> 2.10.1

-- 
Ville Syrj�l�
Intel OTC

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Tue, Nov 01, 2016 at 01:29:59PM +0200, Felipe Balbi wrote:
> According to Dave Miller "the networking stack has a
> hard requirement that all SKBs which are transmitted
> must have their completion signalled in a fininte
> amount of time. This is because, until the SKB is
> freed by the driver, it holds onto socket,
> netfilter, and other subsystem resources."
> 
> In summary, this means that using TX IRQ throttling
> for the networking gadgets is, at least, complex and
> we should avoid it for the time being.
> 
> Cc: <stable@vger.kernel.org>
> Reported-by: Ville Syrj�l� <ville.syrjala@linux.intel.com>
> Suggested-by: David Miller <davem@davemloft.net>
> Signed-off-by: Felipe Balbi <felipe.balbi@linux.intel.com>
> ---
>  drivers/usb/gadget/function/u_ether.c | 8 --------
>  1 file changed, 8 deletions(-)
> 
> diff --git a/drivers/usb/gadget/function/u_ether.c b/drivers/usb/gadget/function/u_ether.c
> index f4a640216913..119a2e5848e8 100644
> --- a/drivers/usb/gadget/function/u_ether.c
> +++ b/drivers/usb/gadget/function/u_ether.c
> @@ -589,14 +589,6 @@ static netdev_tx_t eth_start_xmit(struct sk_buff *skb,
>  
>  	req->length = length;
>  
> -	/* throttle high/super speed IRQ rate back slightly */
> -	if (gadget_is_dualspeed(dev->gadget))
> -		req->no_interrupt = (((dev->gadget->speed == USB_SPEED_HIGH ||
> -				       dev->gadget->speed == USB_SPEED_SUPER)) &&
> -					!list_empty(&dev->tx_reqs))
> -			? ((atomic_read(&dev->tx_qlen) % dev->qmult) != 0)
> -			: 0;
> -
>  	retval = usb_ep_queue(in, req, GFP_ATOMIC);
>  	switch (retval) {
>  	default:
> -- 

Felipe, it may increase cpu utilization since more interrupts will be there,
it may affect the SoC which has lower cpu frequency. This code existed
many years, why this problem has only reported at dwc3 recently?

-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 3007 bytes --]


Hi,

Peter Chen <hzpeterchen@gmail.com> writes:
> On Tue, Nov 01, 2016 at 01:29:59PM +0200, Felipe Balbi wrote:
>> According to Dave Miller "the networking stack has a
>> hard requirement that all SKBs which are transmitted
>> must have their completion signalled in a fininte
>> amount of time. This is because, until the SKB is
>> freed by the driver, it holds onto socket,
>> netfilter, and other subsystem resources."
>> 
>> In summary, this means that using TX IRQ throttling
>> for the networking gadgets is, at least, complex and
>> we should avoid it for the time being.
>> 
>> Cc: <stable@vger.kernel.org>
>> Reported-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
>> Suggested-by: David Miller <davem@davemloft.net>
>> Signed-off-by: Felipe Balbi <felipe.balbi@linux.intel.com>
>> ---
>>  drivers/usb/gadget/function/u_ether.c | 8 --------
>>  1 file changed, 8 deletions(-)
>> 
>> diff --git a/drivers/usb/gadget/function/u_ether.c b/drivers/usb/gadget/function/u_ether.c
>> index f4a640216913..119a2e5848e8 100644
>> --- a/drivers/usb/gadget/function/u_ether.c
>> +++ b/drivers/usb/gadget/function/u_ether.c
>> @@ -589,14 +589,6 @@ static netdev_tx_t eth_start_xmit(struct sk_buff *skb,
>>  
>>  	req->length = length;
>>  
>> -	/* throttle high/super speed IRQ rate back slightly */
>> -	if (gadget_is_dualspeed(dev->gadget))
>> -		req->no_interrupt = (((dev->gadget->speed == USB_SPEED_HIGH ||
>> -				       dev->gadget->speed == USB_SPEED_SUPER)) &&
>> -					!list_empty(&dev->tx_reqs))
>> -			? ((atomic_read(&dev->tx_qlen) % dev->qmult) != 0)
>> -			: 0;
>> -
>>  	retval = usb_ep_queue(in, req, GFP_ATOMIC);
>>  	switch (retval) {
>>  	default:
>> -- 
>
> Felipe, it may increase cpu utilization since more interrupts will be there,
> it may affect the SoC which has lower cpu frequency. This code existed
> many years, why this problem has only reported at dwc3 recently?

No idea, but at least for networking gadgets we shouldn't throttle. This
has been a bug since the beginning. Read Dave Miller's explanation at
[1]

moreover, dwc3 seems to be the only one actually throttling IRQ. Here's
a rundown of a few of the UDCs:

- chipidea: uses TD_IOC conditionally, but always sets TD_TERMINATE

	lastnode->ptr->next = cpu_to_le32(TD_TERMINATE);
	if (!hwreq->req.no_interrupt)
		lastnode->ptr->token |= cpu_to_le32(TD_IOC);

	I'm guessing TD_TERMINATE works similar to dwc3's LST bit. If
	it's set, it will force an interrupt.

- musb: no_interrupt only used for tracing

- atmel_usba_udc: no_interrupt only used for tracing

- mv_u3d_core: probably throttles interrupt, but probably exhibits same
  behavior. It's just that it hasn't been reported.

- fsl_udc_core: probably throttles interrupt, but probably exhibits same
  behavior. It's just that it hasn't been reported.

and there are the only uses for the no_interrupt flag.

[1] https://marc.info/?l=linux-usb&m=147793992922064&w=2

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Wed, Nov 02, 2016 at 09:55:44AM +0200, Felipe Balbi wrote:
> 
> Hi,
> 
> Peter Chen <hzpeterchen@gmail.com> writes:
> > On Tue, Nov 01, 2016 at 01:29:59PM +0200, Felipe Balbi wrote:
> >> According to Dave Miller "the networking stack has a
> >> hard requirement that all SKBs which are transmitted
> >> must have their completion signalled in a fininte
> >> amount of time. This is because, until the SKB is
> >> freed by the driver, it holds onto socket,
> >> netfilter, and other subsystem resources."
> >> 
> >> In summary, this means that using TX IRQ throttling
> >> for the networking gadgets is, at least, complex and
> >> we should avoid it for the time being.
> >> 
> >> Cc: <stable@vger.kernel.org>
> >> Reported-by: Ville Syrj�l� <ville.syrjala@linux.intel.com>
> >> Suggested-by: David Miller <davem@davemloft.net>
> >> Signed-off-by: Felipe Balbi <felipe.balbi@linux.intel.com>
> >> ---
> >>  drivers/usb/gadget/function/u_ether.c | 8 --------
> >>  1 file changed, 8 deletions(-)
> >> 
> >> diff --git a/drivers/usb/gadget/function/u_ether.c b/drivers/usb/gadget/function/u_ether.c
> >> index f4a640216913..119a2e5848e8 100644
> >> --- a/drivers/usb/gadget/function/u_ether.c
> >> +++ b/drivers/usb/gadget/function/u_ether.c
> >> @@ -589,14 +589,6 @@ static netdev_tx_t eth_start_xmit(struct sk_buff *skb,
> >>  
> >>  	req->length = length;
> >>  
> >> -	/* throttle high/super speed IRQ rate back slightly */
> >> -	if (gadget_is_dualspeed(dev->gadget))
> >> -		req->no_interrupt = (((dev->gadget->speed == USB_SPEED_HIGH ||
> >> -				       dev->gadget->speed == USB_SPEED_SUPER)) &&
> >> -					!list_empty(&dev->tx_reqs))
> >> -			? ((atomic_read(&dev->tx_qlen) % dev->qmult) != 0)
> >> -			: 0;
> >> -
> >>  	retval = usb_ep_queue(in, req, GFP_ATOMIC);
> >>  	switch (retval) {
> >>  	default:
> >> -- 
> >
> > Felipe, it may increase cpu utilization since more interrupts will be there,
> > it may affect the SoC which has lower cpu frequency. This code existed
> > many years, why this problem has only reported at dwc3 recently?
> 
> No idea, but at least for networking gadgets we shouldn't throttle. This
> has been a bug since the beginning. Read Dave Miller's explanation at
> [1]
> 
> moreover, dwc3 seems to be the only one actually throttling IRQ. Here's
> a rundown of a few of the UDCs:
> 
> - chipidea: uses TD_IOC conditionally, but always sets TD_TERMINATE
> 
> 	lastnode->ptr->next = cpu_to_le32(TD_TERMINATE);
> 	if (!hwreq->req.no_interrupt)
> 		lastnode->ptr->token |= cpu_to_le32(TD_IOC);
> 
> 	I'm guessing TD_TERMINATE works similar to dwc3's LST bit. If
> 	it's set, it will force an interrupt.

No, TD_TERMINATE just stands for it is the last TD, and this pointer will
be updated when the new request is added. The interrupt is only triggered
by IOC (Interrupt On Complete) bit at TD.

I am not sure if dwc3 supports ITC (Interrupt Threshold Control)
software control, it is an EHCI compliant register entry, and
the device mode is supported for chipidea too. It is a timeout
mechanism from controller side for pending requests.

The interrupt will be triggered either the request has completed for TD
which IOC bit is set or the ITC is fired (125us currently) and the
request has completed, so the problem David described should not exist,
at least for chipidea.

If DWC3 has similar ITC bits, would you try to tune it? The default ITC
value for chipidea is not enough, and we tuned it before.

> 
> - musb: no_interrupt only used for tracing
> 
> - atmel_usba_udc: no_interrupt only used for tracing
> 
> - mv_u3d_core: probably throttles interrupt, but probably exhibits same
>   behavior. It's just that it hasn't been reported.
> 
> - fsl_udc_core: probably throttles interrupt, but probably exhibits same
>   behavior. It's just that it hasn't been reported.

The above two uses chipidea IP core too.

-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 3291 bytes --]


Hi,

Peter Chen <hzpeterchen@gmail.com> writes:
>> >> diff --git a/drivers/usb/gadget/function/u_ether.c b/drivers/usb/gadget/function/u_ether.c
>> >> index f4a640216913..119a2e5848e8 100644
>> >> --- a/drivers/usb/gadget/function/u_ether.c
>> >> +++ b/drivers/usb/gadget/function/u_ether.c
>> >> @@ -589,14 +589,6 @@ static netdev_tx_t eth_start_xmit(struct sk_buff *skb,
>> >>  
>> >>  	req->length = length;
>> >>  
>> >> -	/* throttle high/super speed IRQ rate back slightly */
>> >> -	if (gadget_is_dualspeed(dev->gadget))
>> >> -		req->no_interrupt = (((dev->gadget->speed == USB_SPEED_HIGH ||
>> >> -				       dev->gadget->speed == USB_SPEED_SUPER)) &&
>> >> -					!list_empty(&dev->tx_reqs))
>> >> -			? ((atomic_read(&dev->tx_qlen) % dev->qmult) != 0)
>> >> -			: 0;
>> >> -
>> >>  	retval = usb_ep_queue(in, req, GFP_ATOMIC);
>> >>  	switch (retval) {
>> >>  	default:
>> >> -- 
>> >
>> > Felipe, it may increase cpu utilization since more interrupts will be there,
>> > it may affect the SoC which has lower cpu frequency. This code existed
>> > many years, why this problem has only reported at dwc3 recently?
>> 
>> No idea, but at least for networking gadgets we shouldn't throttle. This
>> has been a bug since the beginning. Read Dave Miller's explanation at
>> [1]
>> 
>> moreover, dwc3 seems to be the only one actually throttling IRQ. Here's
>> a rundown of a few of the UDCs:
>> 
>> - chipidea: uses TD_IOC conditionally, but always sets TD_TERMINATE
>> 
>> 	lastnode->ptr->next = cpu_to_le32(TD_TERMINATE);
>> 	if (!hwreq->req.no_interrupt)
>> 		lastnode->ptr->token |= cpu_to_le32(TD_IOC);
>> 
>> 	I'm guessing TD_TERMINATE works similar to dwc3's LST bit. If
>> 	it's set, it will force an interrupt.
>
> No, TD_TERMINATE just stands for it is the last TD, and this pointer will
> be updated when the new request is added. The interrupt is only triggered
> by IOC (Interrupt On Complete) bit at TD.
>
> I am not sure if dwc3 supports ITC (Interrupt Threshold Control)
> software control, it is an EHCI compliant register entry, and
> the device mode is supported for chipidea too. It is a timeout
> mechanism from controller side for pending requests.
>
> The interrupt will be triggered either the request has completed for TD
> which IOC bit is set or the ITC is fired (125us currently) and the
> request has completed, so the problem David described should not exist,
> at least for chipidea.

In other words, you don't *really* throttle interrupt as they'll fire
after the micro-frame expires :-p

> If DWC3 has similar ITC bits, would you try to tune it? The default ITC
> value for chipidea is not enough, and we tuned it before.

there's no such thing in dwc3

>> - musb: no_interrupt only used for tracing
>> 
>> - atmel_usba_udc: no_interrupt only used for tracing
>> 
>> - mv_u3d_core: probably throttles interrupt, but probably exhibits same
>>   behavior. It's just that it hasn't been reported.
>> 
>> - fsl_udc_core: probably throttles interrupt, but probably exhibits same
>>   behavior. It's just that it hasn't been reported.
>
> The above two uses chipidea IP core too.

so why do we still have these drivers in tree? Might as well remove them
already.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[David Miller]

From: Peter Chen <hzpeterchen@gmail.com>
Date: Wed, 2 Nov 2016 14:02:02 +0800

> Felipe, it may increase cpu utilization since more interrupts will be there,
> it may affect the SoC which has lower cpu frequency. This code existed
> many years, why this problem has only reported at dwc3 recently?

It's a bug, and it's going to cause TCP sockets to potentially hang.

You will need to find a valid way to decrease cpu utilization if it
in fact becomes an issue because of this revert.  But it is not an
argument for not doing this revert.

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
> From: Peter Chen <hzpeterchen@gmail.com>
> Date: Wed, 2 Nov 2016 14:02:02 +0800
> 
> > Felipe, it may increase cpu utilization since more interrupts will be there,
> > it may affect the SoC which has lower cpu frequency. This code existed
> > many years, why this problem has only reported at dwc3 recently?
> 
> It's a bug, and it's going to cause TCP sockets to potentially hang.
> 

For some controllers, it is, so we need to add parameter for user
to see if interrupt migration is supported or not.

But just like some ethernet controllers, some USB controllers support
hardware timeout mechanism which interrupt will be triggered after
some uFrame occurs if the transaction has completed but not required
to interrupt, it is used to support interrupt migration like ethernet.

-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Wed, Nov 02, 2016 at 01:02:59PM +0200, Felipe Balbi wrote:
> 
> Hi,
> 
> Peter Chen <hzpeterchen@gmail.com> writes:
> >> >> diff --git a/drivers/usb/gadget/function/u_ether.c b/drivers/usb/gadget/function/u_ether.c
> >> >> index f4a640216913..119a2e5848e8 100644
> >> >> --- a/drivers/usb/gadget/function/u_ether.c
> >> >> +++ b/drivers/usb/gadget/function/u_ether.c
> >> >> @@ -589,14 +589,6 @@ static netdev_tx_t eth_start_xmit(struct sk_buff *skb,
> >> >>  
> >> >>  	req->length = length;
> >> >>  
> >> >> -	/* throttle high/super speed IRQ rate back slightly */
> >> >> -	if (gadget_is_dualspeed(dev->gadget))
> >> >> -		req->no_interrupt = (((dev->gadget->speed == USB_SPEED_HIGH ||
> >> >> -				       dev->gadget->speed == USB_SPEED_SUPER)) &&
> >> >> -					!list_empty(&dev->tx_reqs))
> >> >> -			? ((atomic_read(&dev->tx_qlen) % dev->qmult) != 0)
> >> >> -			: 0;
> >> >> -
> >> >>  	retval = usb_ep_queue(in, req, GFP_ATOMIC);
> >> >>  	switch (retval) {
> >> >>  	default:
> >> >> -- 
> >> >
> >> > Felipe, it may increase cpu utilization since more interrupts will be there,
> >> > it may affect the SoC which has lower cpu frequency. This code existed
> >> > many years, why this problem has only reported at dwc3 recently?
> >> 
> >> No idea, but at least for networking gadgets we shouldn't throttle. This
> >> has been a bug since the beginning. Read Dave Miller's explanation at
> >> [1]
> >> 
> >> moreover, dwc3 seems to be the only one actually throttling IRQ. Here's
> >> a rundown of a few of the UDCs:
> >> 
> >> - chipidea: uses TD_IOC conditionally, but always sets TD_TERMINATE
> >> 
> >> 	lastnode->ptr->next = cpu_to_le32(TD_TERMINATE);
> >> 	if (!hwreq->req.no_interrupt)
> >> 		lastnode->ptr->token |= cpu_to_le32(TD_IOC);
> >> 
> >> 	I'm guessing TD_TERMINATE works similar to dwc3's LST bit. If
> >> 	it's set, it will force an interrupt.
> >
> > No, TD_TERMINATE just stands for it is the last TD, and this pointer will
> > be updated when the new request is added. The interrupt is only triggered
> > by IOC (Interrupt On Complete) bit at TD.
> >
> > I am not sure if dwc3 supports ITC (Interrupt Threshold Control)
> > software control, it is an EHCI compliant register entry, and
> > the device mode is supported for chipidea too. It is a timeout
> > mechanism from controller side for pending requests.
> >
> > The interrupt will be triggered either the request has completed for TD
> > which IOC bit is set or the ITC is fired (125us currently) and the
> > request has completed, so the problem David described should not exist,
> > at least for chipidea.
> 
> In other words, you don't *really* throttle interrupt as they'll fire
> after the micro-frame expires :-p

No, even in one uFrame, there are at most ~10 packets for bulk at USB2.
At least, you can throttle interrupt within SoF, it is useful for 
high throughout use case.

> 
> > If DWC3 has similar ITC bits, would you try to tune it? The default ITC
> > value for chipidea is not enough, and we tuned it before.
> 
> there's no such thing in dwc3
> 

So, how about add another parameter to support throttling interrupt
separately. Current parameter 'mult' combined user request number
and throttle interrupt together.

-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

Hi,

Peter Chen <hzpeterchen@gmail.com> writes:
> On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
>> From: Peter Chen <hzpeterchen@gmail.com>
>> Date: Wed, 2 Nov 2016 14:02:02 +0800
>> 
>> > Felipe, it may increase cpu utilization since more interrupts will be there,
>> > it may affect the SoC which has lower cpu frequency. This code existed
>> > many years, why this problem has only reported at dwc3 recently?
>> 
>> It's a bug, and it's going to cause TCP sockets to potentially hang.
>> 
>
> For some controllers, it is, so we need to add parameter for user
> to see if interrupt migration is supported or not.

not for some controller, for ALL networking drivers.

> But just like some ethernet controllers, some USB controllers support
> hardware timeout mechanism which interrupt will be triggered after
> some uFrame occurs if the transaction has completed but not required
> to interrupt, it is used to support interrupt migration like ethernet.

you're missing the point. What Dave Miller is saying is that it's ALWAYS
a bug to delay completion of SKBs. The only thing you're doing with
chipidea is delaying interrupt by up to 125us; which is still a bug from
the point of view of the networking layer, but it's more difficult to
perceive any problems because of the short time where interrupt is
delayed.

-- 
balbi

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 3475 bytes --]


Hi,

Peter Chen <hzpeterchen@gmail.com> writes:
>> Peter Chen <hzpeterchen@gmail.com> writes:
>> >> >> diff --git a/drivers/usb/gadget/function/u_ether.c b/drivers/usb/gadget/function/u_ether.c
>> >> >> index f4a640216913..119a2e5848e8 100644
>> >> >> --- a/drivers/usb/gadget/function/u_ether.c
>> >> >> +++ b/drivers/usb/gadget/function/u_ether.c
>> >> >> @@ -589,14 +589,6 @@ static netdev_tx_t eth_start_xmit(struct sk_buff *skb,
>> >> >>  
>> >> >>  	req->length = length;
>> >> >>  
>> >> >> -	/* throttle high/super speed IRQ rate back slightly */
>> >> >> -	if (gadget_is_dualspeed(dev->gadget))
>> >> >> -		req->no_interrupt = (((dev->gadget->speed == USB_SPEED_HIGH ||
>> >> >> -				       dev->gadget->speed == USB_SPEED_SUPER)) &&
>> >> >> -					!list_empty(&dev->tx_reqs))
>> >> >> -			? ((atomic_read(&dev->tx_qlen) % dev->qmult) != 0)
>> >> >> -			: 0;
>> >> >> -
>> >> >>  	retval = usb_ep_queue(in, req, GFP_ATOMIC);
>> >> >>  	switch (retval) {
>> >> >>  	default:
>> >> >> -- 
>> >> >
>> >> > Felipe, it may increase cpu utilization since more interrupts will be there,
>> >> > it may affect the SoC which has lower cpu frequency. This code existed
>> >> > many years, why this problem has only reported at dwc3 recently?
>> >> 
>> >> No idea, but at least for networking gadgets we shouldn't throttle. This
>> >> has been a bug since the beginning. Read Dave Miller's explanation at
>> >> [1]
>> >> 
>> >> moreover, dwc3 seems to be the only one actually throttling IRQ. Here's
>> >> a rundown of a few of the UDCs:
>> >> 
>> >> - chipidea: uses TD_IOC conditionally, but always sets TD_TERMINATE
>> >> 
>> >> 	lastnode->ptr->next = cpu_to_le32(TD_TERMINATE);
>> >> 	if (!hwreq->req.no_interrupt)
>> >> 		lastnode->ptr->token |= cpu_to_le32(TD_IOC);
>> >> 
>> >> 	I'm guessing TD_TERMINATE works similar to dwc3's LST bit. If
>> >> 	it's set, it will force an interrupt.
>> >
>> > No, TD_TERMINATE just stands for it is the last TD, and this pointer will
>> > be updated when the new request is added. The interrupt is only triggered
>> > by IOC (Interrupt On Complete) bit at TD.
>> >
>> > I am not sure if dwc3 supports ITC (Interrupt Threshold Control)
>> > software control, it is an EHCI compliant register entry, and
>> > the device mode is supported for chipidea too. It is a timeout
>> > mechanism from controller side for pending requests.
>> >
>> > The interrupt will be triggered either the request has completed for TD
>> > which IOC bit is set or the ITC is fired (125us currently) and the
>> > request has completed, so the problem David described should not exist,
>> > at least for chipidea.
>> 
>> In other words, you don't *really* throttle interrupt as they'll fire
>> after the micro-frame expires :-p
>
> No, even in one uFrame, there are at most ~10 packets for bulk at USB2.

13

> At least, you can throttle interrupt within SoF, it is useful for 
> high throughout use case.

And that's still a bug for Networking drivers, that's what Dave Miller
is saying.

>> > If DWC3 has similar ITC bits, would you try to tune it? The default ITC
>> > value for chipidea is not enough, and we tuned it before.
>> 
>> there's no such thing in dwc3
>> 
>
> So, how about add another parameter to support throttling interrupt
> separately. Current parameter 'mult' combined user request number
> and throttle interrupt together.

sorry, but no.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Thu, Nov 03, 2016 at 09:04:54AM +0200, Felipe Balbi wrote:
> 
> Hi,
> 
> Peter Chen <hzpeterchen@gmail.com> writes:
> > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
> >> From: Peter Chen <hzpeterchen@gmail.com>
> >> Date: Wed, 2 Nov 2016 14:02:02 +0800
> >> 
> >> > Felipe, it may increase cpu utilization since more interrupts will be there,
> >> > it may affect the SoC which has lower cpu frequency. This code existed
> >> > many years, why this problem has only reported at dwc3 recently?
> >> 
> >> It's a bug, and it's going to cause TCP sockets to potentially hang.
> >> 
> >
> > For some controllers, it is, so we need to add parameter for user
> > to see if interrupt migration is supported or not.
> 
> not for some controller, for ALL networking drivers.
> 
> > But just like some ethernet controllers, some USB controllers support
> > hardware timeout mechanism which interrupt will be triggered after
> > some uFrame occurs if the transaction has completed but not required
> > to interrupt, it is used to support interrupt migration like ethernet.
> 
> you're missing the point. What Dave Miller is saying is that it's ALWAYS
> a bug to delay completion of SKBs. The only thing you're doing with
> chipidea is delaying interrupt by up to 125us; which is still a bug from
> the point of view of the networking layer, but it's more difficult to
> perceive any problems because of the short time where interrupt is
> delayed.
> 

If it is ALWAYS a bug to delay completion of SKBs, how the local
ethernet driver designs interrupt migration?

-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Thu, Nov 03, 2016 at 05:03:10PM +0800, Peter Chen wrote:
> On Thu, Nov 03, 2016 at 09:04:54AM +0200, Felipe Balbi wrote:
> > 
> > Hi,
> > 
> > Peter Chen <hzpeterchen@gmail.com> writes:
> > > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
> > >> From: Peter Chen <hzpeterchen@gmail.com>
> > >> Date: Wed, 2 Nov 2016 14:02:02 +0800
> > >> 
> > >> > Felipe, it may increase cpu utilization since more interrupts will be there,
> > >> > it may affect the SoC which has lower cpu frequency. This code existed
> > >> > many years, why this problem has only reported at dwc3 recently?
> > >> 
> > >> It's a bug, and it's going to cause TCP sockets to potentially hang.
> > >> 
> > >
> > > For some controllers, it is, so we need to add parameter for user
> > > to see if interrupt migration is supported or not.
> > 
> > not for some controller, for ALL networking drivers.
> > 
> > > But just like some ethernet controllers, some USB controllers support
> > > hardware timeout mechanism which interrupt will be triggered after
> > > some uFrame occurs if the transaction has completed but not required
> > > to interrupt, it is used to support interrupt migration like ethernet.
> > 
> > you're missing the point. What Dave Miller is saying is that it's ALWAYS
> > a bug to delay completion of SKBs. The only thing you're doing with
> > chipidea is delaying interrupt by up to 125us; which is still a bug from
> > the point of view of the networking layer, but it's more difficult to
> > perceive any problems because of the short time where interrupt is
> > delayed.
> > 
> 
> If it is ALWAYS a bug to delay completion of SKBs, how the local
> ethernet driver designs interrupt migration?
> 

Just a quick test, I delete dev_kfree_skb_any at tx_complete, not find
any problems by using simple "ping test", just free memory is less and
less. David, do you really mean free tx skb buffer with limited time,
but not return NETDEV_TX_OK by ->ndo_start_xmit with limited time?

-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 1756 bytes --]


Hi,

Peter Chen <hzpeterchen@gmail.com> writes:
>> Peter Chen <hzpeterchen@gmail.com> writes:
>> > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
>> >> From: Peter Chen <hzpeterchen@gmail.com>
>> >> Date: Wed, 2 Nov 2016 14:02:02 +0800
>> >> 
>> >> > Felipe, it may increase cpu utilization since more interrupts will be there,
>> >> > it may affect the SoC which has lower cpu frequency. This code existed
>> >> > many years, why this problem has only reported at dwc3 recently?
>> >> 
>> >> It's a bug, and it's going to cause TCP sockets to potentially hang.
>> >> 
>> >
>> > For some controllers, it is, so we need to add parameter for user
>> > to see if interrupt migration is supported or not.
>> 
>> not for some controller, for ALL networking drivers.
>> 
>> > But just like some ethernet controllers, some USB controllers support
>> > hardware timeout mechanism which interrupt will be triggered after
>> > some uFrame occurs if the transaction has completed but not required
>> > to interrupt, it is used to support interrupt migration like ethernet.
>> 
>> you're missing the point. What Dave Miller is saying is that it's ALWAYS
>> a bug to delay completion of SKBs. The only thing you're doing with
>> chipidea is delaying interrupt by up to 125us; which is still a bug from
>> the point of view of the networking layer, but it's more difficult to
>> perceive any problems because of the short time where interrupt is
>> delayed.
>> 
>
> If it is ALWAYS a bug to delay completion of SKBs, how the local
> ethernet driver designs interrupt migration?

what driver are you talking about? Which "local" ethernet driver?

Also, do you mean 'moderation' instead of 'migration?

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 2223 bytes --]


Hi,

Peter Chen <hzpeterchen@gmail.com> writes:
>> > Peter Chen <hzpeterchen@gmail.com> writes:
>> > > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
>> > >> From: Peter Chen <hzpeterchen@gmail.com>
>> > >> Date: Wed, 2 Nov 2016 14:02:02 +0800
>> > >> 
>> > >> > Felipe, it may increase cpu utilization since more interrupts will be there,
>> > >> > it may affect the SoC which has lower cpu frequency. This code existed
>> > >> > many years, why this problem has only reported at dwc3 recently?
>> > >> 
>> > >> It's a bug, and it's going to cause TCP sockets to potentially hang.
>> > >> 
>> > >
>> > > For some controllers, it is, so we need to add parameter for user
>> > > to see if interrupt migration is supported or not.
>> > 
>> > not for some controller, for ALL networking drivers.
>> > 
>> > > But just like some ethernet controllers, some USB controllers support
>> > > hardware timeout mechanism which interrupt will be triggered after
>> > > some uFrame occurs if the transaction has completed but not required
>> > > to interrupt, it is used to support interrupt migration like ethernet.
>> > 
>> > you're missing the point. What Dave Miller is saying is that it's ALWAYS
>> > a bug to delay completion of SKBs. The only thing you're doing with
>> > chipidea is delaying interrupt by up to 125us; which is still a bug from
>> > the point of view of the networking layer, but it's more difficult to
>> > perceive any problems because of the short time where interrupt is
>> > delayed.
>> > 
>> 
>> If it is ALWAYS a bug to delay completion of SKBs, how the local
>> ethernet driver designs interrupt migration?
>> 
>
> Just a quick test, I delete dev_kfree_skb_any at tx_complete, not find
> any problems by using simple "ping test", just free memory is less and
> less. David, do you really mean free tx skb buffer with limited time,
> but not return NETDEV_TX_OK by ->ndo_start_xmit with limited time?

ping *will* work just fine. One easy test to *see* the problem is to SSH
to a machine using g_ether, then run:

$ while true; do dmesg; done

you will notice it is rather laggy. Now remove throttling and the lags
are gone.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[David Miller]

From: Felipe Balbi <felipe.balbi@linux.intel.com>
Date: Thu, 03 Nov 2016 09:04:54 +0200

> What Dave Miller is saying is that it's ALWAYS a bug to delay
> completion of SKBs. The only thing you're doing with chipidea is
> delaying interrupt by up to 125us; which is still a bug from the
> point of view of the networking layer, but it's more difficult to
> perceive any problems because of the short time where interrupt is
> delayed.

I didn't say delaying was illegal.

I said that the SKB free must occur in a reasonable, finite, amount of
time.

And if these timeout events ensure that, then it's fine.

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Thu, Nov 03, 2016 at 12:42:11PM +0200, Felipe Balbi wrote:
> 
> Hi,
> 
> Peter Chen <hzpeterchen@gmail.com> writes:
> >> Peter Chen <hzpeterchen@gmail.com> writes:
> >> > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
> >> >> From: Peter Chen <hzpeterchen@gmail.com>
> >> >> Date: Wed, 2 Nov 2016 14:02:02 +0800
> >> >> 
> >> >> > Felipe, it may increase cpu utilization since more interrupts will be there,
> >> >> > it may affect the SoC which has lower cpu frequency. This code existed
> >> >> > many years, why this problem has only reported at dwc3 recently?
> >> >> 
> >> >> It's a bug, and it's going to cause TCP sockets to potentially hang.
> >> >> 
> >> >
> >> > For some controllers, it is, so we need to add parameter for user
> >> > to see if interrupt migration is supported or not.
> >> 
> >> not for some controller, for ALL networking drivers.
> >> 
> >> > But just like some ethernet controllers, some USB controllers support
> >> > hardware timeout mechanism which interrupt will be triggered after
> >> > some uFrame occurs if the transaction has completed but not required
> >> > to interrupt, it is used to support interrupt migration like ethernet.
> >> 
> >> you're missing the point. What Dave Miller is saying is that it's ALWAYS
> >> a bug to delay completion of SKBs. The only thing you're doing with
> >> chipidea is delaying interrupt by up to 125us; which is still a bug from
> >> the point of view of the networking layer, but it's more difficult to
> >> perceive any problems because of the short time where interrupt is
> >> delayed.
> >> 
> >
> > If it is ALWAYS a bug to delay completion of SKBs, how the local
> > ethernet driver designs interrupt migration?
> 
> what driver are you talking about? Which "local" ethernet driver?
> 

The ethernet driver located at drivers/net/ethernet, its device uses RJ45 connector
to communicate.

-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Thu, Nov 03, 2016 at 12:42:11PM +0200, Felipe Balbi wrote:
> 
> Hi,
> 
> Peter Chen <hzpeterchen@gmail.com> writes:
> >> Peter Chen <hzpeterchen@gmail.com> writes:
> >> > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
> >> >> From: Peter Chen <hzpeterchen@gmail.com>
> >> >> Date: Wed, 2 Nov 2016 14:02:02 +0800
> >> >> 
> >> >> > Felipe, it may increase cpu utilization since more interrupts will be there,
> >> >> > it may affect the SoC which has lower cpu frequency. This code existed
> >> >> > many years, why this problem has only reported at dwc3 recently?
> >> >> 
> >> >> It's a bug, and it's going to cause TCP sockets to potentially hang.
> >> >> 
> >> >
> >> > For some controllers, it is, so we need to add parameter for user
> >> > to see if interrupt migration is supported or not.
> >> 
> >> not for some controller, for ALL networking drivers.
> >> 
> >> > But just like some ethernet controllers, some USB controllers support
> >> > hardware timeout mechanism which interrupt will be triggered after
> >> > some uFrame occurs if the transaction has completed but not required
> >> > to interrupt, it is used to support interrupt migration like ethernet.
> >> 
> >> you're missing the point. What Dave Miller is saying is that it's ALWAYS
> >> a bug to delay completion of SKBs. The only thing you're doing with
> >> chipidea is delaying interrupt by up to 125us; which is still a bug from
> >> the point of view of the networking layer, but it's more difficult to
> >> perceive any problems because of the short time where interrupt is
> >> delayed.
> >> 
> >
> > If it is ALWAYS a bug to delay completion of SKBs, how the local
> > ethernet driver designs interrupt migration?
> 
> what driver are you talking about? Which "local" ethernet driver?
> 
> Also, do you mean 'moderation' instead of 'migration?
> 

migration interrupt means throttling interrupt, sorry, I may use 'wrong'
words.

-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Thu, Nov 03, 2016 at 12:48:08PM +0200, Felipe Balbi wrote:
> 
> Hi,
> 
> Peter Chen <hzpeterchen@gmail.com> writes:
> >> > Peter Chen <hzpeterchen@gmail.com> writes:
> >> > > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
> >> > >> From: Peter Chen <hzpeterchen@gmail.com>
> >> > >> Date: Wed, 2 Nov 2016 14:02:02 +0800
> >> > >> 
> >> > >> > Felipe, it may increase cpu utilization since more interrupts will be there,
> >> > >> > it may affect the SoC which has lower cpu frequency. This code existed
> >> > >> > many years, why this problem has only reported at dwc3 recently?
> >> > >> 
> >> > >> It's a bug, and it's going to cause TCP sockets to potentially hang.
> >> > >> 
> >> > >
> >> > > For some controllers, it is, so we need to add parameter for user
> >> > > to see if interrupt migration is supported or not.
> >> > 
> >> > not for some controller, for ALL networking drivers.
> >> > 
> >> > > But just like some ethernet controllers, some USB controllers support
> >> > > hardware timeout mechanism which interrupt will be triggered after
> >> > > some uFrame occurs if the transaction has completed but not required
> >> > > to interrupt, it is used to support interrupt migration like ethernet.
> >> > 
> >> > you're missing the point. What Dave Miller is saying is that it's ALWAYS
> >> > a bug to delay completion of SKBs. The only thing you're doing with
> >> > chipidea is delaying interrupt by up to 125us; which is still a bug from
> >> > the point of view of the networking layer, but it's more difficult to
> >> > perceive any problems because of the short time where interrupt is
> >> > delayed.
> >> > 
> >> 
> >> If it is ALWAYS a bug to delay completion of SKBs, how the local
> >> ethernet driver designs interrupt migration?
> >> 
> >
> > Just a quick test, I delete dev_kfree_skb_any at tx_complete, not find
> > any problems by using simple "ping test", just free memory is less and
> > less. David, do you really mean free tx skb buffer with limited time,
> > but not return NETDEV_TX_OK by ->ndo_start_xmit with limited time?
> 
> ping *will* work just fine. One easy test to *see* the problem is to SSH
> to a machine using g_ether, then run:
> 
> $ while true; do dmesg; done
> 
> you will notice it is rather laggy. Now remove throttling and the lags
> are gone.
> 

I have ran the test using ncm, the qmult is 10, but not find the laggy 
at the screen, just the pipe will be broken after several minutes.
During this test, the data at rx is much more than tx, but throttling
interrupt we are discussing is at tx.


[    1.314390] fec 21b4000.ethernet (unnamed net_device) (uninitialized): Invalid MAC addresspacket_write_wait: Connection to 192.168.1.1: Broken pipe
root@imx6qdlsolo:~# ifconfig usb0        
usb0      Link encap:Ethernet  HWaddr d6:c9:9a:18:4b:b1  
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::d4c9:9aff:fe18:4bb1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5011 errors:0 dropped:0 overruns:0 frame:0
          TX packets:677 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3015832 (2.8 MiB)  TX bytes:94532 (92.3 KiB)
-- 

Best Regards,
Peter Chen

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 2808 bytes --]


Hi,

Peter Chen <hzpeterchen@gmail.com> writes:
>> Peter Chen <hzpeterchen@gmail.com> writes:
>> >> > Peter Chen <hzpeterchen@gmail.com> writes:
>> >> > > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
>> >> > >> From: Peter Chen <hzpeterchen@gmail.com>
>> >> > >> Date: Wed, 2 Nov 2016 14:02:02 +0800
>> >> > >> 
>> >> > >> > Felipe, it may increase cpu utilization since more interrupts will be there,
>> >> > >> > it may affect the SoC which has lower cpu frequency. This code existed
>> >> > >> > many years, why this problem has only reported at dwc3 recently?
>> >> > >> 
>> >> > >> It's a bug, and it's going to cause TCP sockets to potentially hang.
>> >> > >> 
>> >> > >
>> >> > > For some controllers, it is, so we need to add parameter for user
>> >> > > to see if interrupt migration is supported or not.
>> >> > 
>> >> > not for some controller, for ALL networking drivers.
>> >> > 
>> >> > > But just like some ethernet controllers, some USB controllers support
>> >> > > hardware timeout mechanism which interrupt will be triggered after
>> >> > > some uFrame occurs if the transaction has completed but not required
>> >> > > to interrupt, it is used to support interrupt migration like ethernet.
>> >> > 
>> >> > you're missing the point. What Dave Miller is saying is that it's ALWAYS
>> >> > a bug to delay completion of SKBs. The only thing you're doing with
>> >> > chipidea is delaying interrupt by up to 125us; which is still a bug from
>> >> > the point of view of the networking layer, but it's more difficult to
>> >> > perceive any problems because of the short time where interrupt is
>> >> > delayed.
>> >> > 
>> >> 
>> >> If it is ALWAYS a bug to delay completion of SKBs, how the local
>> >> ethernet driver designs interrupt migration?
>> >> 
>> >
>> > Just a quick test, I delete dev_kfree_skb_any at tx_complete, not find
>> > any problems by using simple "ping test", just free memory is less and
>> > less. David, do you really mean free tx skb buffer with limited time,
>> > but not return NETDEV_TX_OK by ->ndo_start_xmit with limited time?
>> 
>> ping *will* work just fine. One easy test to *see* the problem is to SSH
>> to a machine using g_ether, then run:
>> 
>> $ while true; do dmesg; done
>> 
>> you will notice it is rather laggy. Now remove throttling and the lags
>> are gone.
>> 
>
> I have ran the test using ncm, the qmult is 10, but not find the laggy 
> at the screen, just the pipe will be broken after several minutes.

the reason you don't see it is because of your forced per-uFrame
interrupt. If you remove that, then you're likely going to have issues.

Also, the fact that we're running Super-speed while you're running
High-speed can also change things.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]


Hi,

David Miller <davem@davemloft.net> writes:
> From: Felipe Balbi <felipe.balbi@linux.intel.com>
> Date: Thu, 03 Nov 2016 09:04:54 +0200
>
>> What Dave Miller is saying is that it's ALWAYS a bug to delay
>> completion of SKBs. The only thing you're doing with chipidea is
>> delaying interrupt by up to 125us; which is still a bug from the
>> point of view of the networking layer, but it's more difficult to
>> perceive any problems because of the short time where interrupt is
>> delayed.
>
> I didn't say delaying was illegal.
>
> I said that the SKB free must occur in a reasonable, finite, amount of
> time.

"reasonable" is rather subjective. Completions are, of course,
finite. It just means that once a transfer completes *and* has interrupt
enabled, then several other SKBs will be completed along with it.

That doesn't mean *all* SKBs completed at the same time, it means once
we get an interrupt, we give back all previous ones, but we don't know
exactly when they completed.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[David Miller]

From: Felipe Balbi <felipe.balbi@linux.intel.com>
Date: Mon, 07 Nov 2016 14:39:21 +0200

> 
> Hi,
> 
> David Miller <davem@davemloft.net> writes:
>> From: Felipe Balbi <felipe.balbi@linux.intel.com>
>> Date: Thu, 03 Nov 2016 09:04:54 +0200
>>
>>> What Dave Miller is saying is that it's ALWAYS a bug to delay
>>> completion of SKBs. The only thing you're doing with chipidea is
>>> delaying interrupt by up to 125us; which is still a bug from the
>>> point of view of the networking layer, but it's more difficult to
>>> perceive any problems because of the short time where interrupt is
>>> delayed.
>>
>> I didn't say delaying was illegal.
>>
>> I said that the SKB free must occur in a reasonable, finite, amount of
>> time.
> 
> "reasonable" is rather subjective. Completions are, of course,
> finite. It just means that once a transfer completes *and* has interrupt
> enabled, then several other SKBs will be completed along with it.
> 
> That doesn't mean *all* SKBs completed at the same time, it means once
> we get an interrupt, we give back all previous ones, but we don't know
> exactly when they completed.

I think you know what I was trying to say which is that
some event is guaranteed to release the SKBs on some order
of magnitude less than say half a second.

Re: [PATCH] usb: gadget: u_ether: remove interrupt throttling

[Peter Chen]

On Mon, Nov 07, 2016 at 02:36:51PM +0200, Felipe Balbi wrote:
> 
> Hi,
> 
> Peter Chen <hzpeterchen@gmail.com> writes:
> >> Peter Chen <hzpeterchen@gmail.com> writes:
> >> >> > Peter Chen <hzpeterchen@gmail.com> writes:
> >> >> > > On Wed, Nov 02, 2016 at 11:22:54AM -0400, David Miller wrote:
> >> >> > >> From: Peter Chen <hzpeterchen@gmail.com>
> >> >> > >> Date: Wed, 2 Nov 2016 14:02:02 +0800
> >> >> > >> 
> >> >> > >> > Felipe, it may increase cpu utilization since more interrupts will be there,
> >> >> > >> > it may affect the SoC which has lower cpu frequency. This code existed
> >> >> > >> > many years, why this problem has only reported at dwc3 recently?
> >> >> > >> 
> >> >> > >> It's a bug, and it's going to cause TCP sockets to potentially hang.
> >> >> > >> 
> >> >> > >
> >> >> > > For some controllers, it is, so we need to add parameter for user
> >> >> > > to see if interrupt migration is supported or not.
> >> >> > 
> >> >> > not for some controller, for ALL networking drivers.
> >> >> > 
> >> >> > > But just like some ethernet controllers, some USB controllers support
> >> >> > > hardware timeout mechanism which interrupt will be triggered after
> >> >> > > some uFrame occurs if the transaction has completed but not required
> >> >> > > to interrupt, it is used to support interrupt migration like ethernet.
> >> >> > 
> >> >> > you're missing the point. What Dave Miller is saying is that it's ALWAYS
> >> >> > a bug to delay completion of SKBs. The only thing you're doing with
> >> >> > chipidea is delaying interrupt by up to 125us; which is still a bug from
> >> >> > the point of view of the networking layer, but it's more difficult to
> >> >> > perceive any problems because of the short time where interrupt is
> >> >> > delayed.
> >> >> > 
> >> >> 
> >> >> If it is ALWAYS a bug to delay completion of SKBs, how the local
> >> >> ethernet driver designs interrupt migration?
> >> >> 
> >> >
> >> > Just a quick test, I delete dev_kfree_skb_any at tx_complete, not find
> >> > any problems by using simple "ping test", just free memory is less and
> >> > less. David, do you really mean free tx skb buffer with limited time,
> >> > but not return NETDEV_TX_OK by ->ndo_start_xmit with limited time?
> >> 
> >> ping *will* work just fine. One easy test to *see* the problem is to SSH
> >> to a machine using g_ether, then run:
> >> 
> >> $ while true; do dmesg; done
> >> 
> >> you will notice it is rather laggy. Now remove throttling and the lags
> >> are gone.
> >> 
> >
> > I have ran the test using ncm, the qmult is 10, but not find the laggy 
> > at the screen, just the pipe will be broken after several minutes.
> 
> the reason you don't see it is because of your forced per-uFrame
> interrupt. If you remove that, then you're likely going to have issues.
> 

I admit you are right, but if we can force complete interrupt occur within
limited time, it should be OK. For chipidea, there are no option to
disable forcing interrupt, you can only tune the threshold for forcing
interrupt, the maximum is 8ms.

> Also, the fact that we're running Super-speed while you're running
> High-speed can also change things.
> 

I don't think it is the reason, you can downgrade your speed and try
again, I think you still will meet issue.

-- 

Best Regards,
Peter Chen