future developments of usbnet

future developments of usbnet

[Oliver Neukum]

Hi,

I'd like to get a feeling what people are working out there regarding usbnet.
So please, if you do something, or think something ought to be done, please
speak up now.

IMHO usbnet needs better support for

- batching protocols
- double buffering on the rx path

with the latter having higher priority.

Coments?

	Regards
		Oliver

Re: future developments of usbnet

[Ming Lei]

Hi,

2011/5/7 Oliver Neukum <oliver-GvhC2dPhHPQdnm+yROfE0A@public.gmane.org>:
> Hi,
>
> I'd like to get a feeling what people are working out there regarding usbnet.
> So please, if you do something, or think something ought to be done, please
> speak up now.
>
> IMHO usbnet needs better support for
>
> - batching protocols
> - double buffering on the rx path
>
> with the latter having higher priority.
>
> Coments?

Maybe another thing about rx should be considered too: we should introduce
one flow control mechanism into rx path so that too much coming packets can
not consume many of memory and cause out of memory for atomic allocation.

Current implementation has such kind of risk certainly, as i explained in the
patch with title below:

       usbnet: runtime pm: fix out of memory

thanks,
-- 
Ming Lei
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: future developments of usbnet

[Oliver Neukum]

Am Montag, 9. Mai 2011, 05:26:04 schrieb Ming Lei:

Hi,

> 2011/5/7 Oliver Neukum <oliver@neukum.org>:
> > Hi,
> >
> > I'd like to get a feeling what people are working out there regarding usbnet.
> > So please, if you do something, or think something ought to be done, please
> > speak up now.
> >
> > IMHO usbnet needs better support for
> >
> > - batching protocols
> > - double buffering on the rx path
> >
> > with the latter having higher priority.
> >
> > Coments?
> 
> Maybe another thing about rx should be considered too: we should introduce
> one flow control mechanism into rx path so that too much coming packets can
> not consume many of memory and cause out of memory for atomic allocation.

Do we really need to avoid it, or do we just need to recover?
If avoidance is needed, should we use NAPI?

	Regards
		Oliver

Re: future developments of usbnet

[Ming Lei]

Hi,

2011/5/9 Oliver Neukum <oliver-GvhC2dPhHPQdnm+yROfE0A@public.gmane.org>:

> Do we really need to avoid it, or do we just need to recover?
> If avoidance is needed, should we use NAPI?

IMO, OOM can recover it certainly but with much cost, so we should
avoid the case. I don't think NAPI can avoid it, because NAPI will
cause skb to be allocated without any limit if there are packets
coming, still no chance left for usbnet_bh to handle and free these
SKBs.


thanks,
-- 
Ming Lei
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: future developments of usbnet

[Alan Stern]

On Mon, 9 May 2011, Ming Lei wrote:

> Hi,
> 
> 2011/5/9 Oliver Neukum <oliver@neukum.org>:
> 
> > Do we really need to avoid it, or do we just need to recover?
> > If avoidance is needed, should we use NAPI?
> 
> IMO, OOM can recover it certainly but with much cost, so we should
> avoid the case. I don't think NAPI can avoid it, because NAPI will
> cause skb to be allocated without any limit if there are packets
> coming, still no chance left for usbnet_bh to handle and free these
> SKBs.

How do other network drivers handle this problem?  Can the same 
strategy be used?

Alan Stern

Re: future developments of usbnet

[Stephen Hemminger]

On Mon, 9 May 2011 11:31:16 -0400 (EDT)
Alan Stern <stern@rowland.harvard.edu> wrote:

> On Mon, 9 May 2011, Ming Lei wrote:
> 
> > Hi,
> > 
> > 2011/5/9 Oliver Neukum <oliver@neukum.org>:
> > 
> > > Do we really need to avoid it, or do we just need to recover?
> > > If avoidance is needed, should we use NAPI?
> > 
> > IMO, OOM can recover it certainly but with much cost, so we should
> > avoid the case. I don't think NAPI can avoid it, because NAPI will
> > cause skb to be allocated without any limit if there are packets
> > coming, still no chance left for usbnet_bh to handle and free these
> > SKBs.
> 
> How do other network drivers handle this problem?  Can the same 
> strategy be used?
> 
> Alan Stern

Most Ethernet drivers have a fixed size receive ring and pass preallocated
memory (skb's or pages) for the hardware to fill in. When NAPI poll is run
it refills the ring and passes the data up to netif_receive_skb. NAPI allows
the poll routine to process a limited number of packets (weight) and after
that the poll loop exits and gets rerun by soft interrupt.  If data is
arriving faster than the kernel can process, eventually the receive ring
passed to hardware gets exhausted and the hardware drops packets.

There is no explicit memory bound limit, instead the flow control happens
when the receive ring gets full.

Re: future developments of usbnet

[Oliver Neukum]

Am Montag, 9. Mai 2011, 17:46:49 schrieb Stephen Hemminger:
> On Mon, 9 May 2011 11:31:16 -0400 (EDT)
> Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org> wrote:

> > How do other network drivers handle this problem?  Can the same 
> > strategy be used?
> > 
> > Alan Stern
> 
> Most Ethernet drivers have a fixed size receive ring and pass preallocated
> memory (skb's or pages) for the hardware to fill in. When NAPI poll is run
> it refills the ring and passes the data up to netif_receive_skb. NAPI allows
> the poll routine to process a limited number of packets (weight) and after
> that the poll loop exits and gets rerun by soft interrupt.  If data is
> arriving faster than the kernel can process, eventually the receive ring
> passed to hardware gets exhausted and the hardware drops packets.

How is the frequency NAPI uses to poll determined? We could abuse
this and resubmit the rx URBs only at poll time, but this feels dirty,
because we would still leave interrupts enabled.

Is there no other driver which faces this problem?

	Regards
		Oliver
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: future developments of usbnet

[David Miller]

From: Oliver Neukum <oliver-GvhC2dPhHPQdnm+yROfE0A@public.gmane.org>
Date: Wed, 11 May 2011 19:37:47 +0200

> How is the frequency NAPI uses to poll determined? We could abuse
> this and resubmit the rx URBs only at poll time, but this feels dirty,
> because we would still leave interrupts enabled.

It's not a frequency determined internally by the networking.

It is purely event based (meaning triggered by the device's interrupt).

Control flow is:

IRQ --> irq_handler()
	my_netdevice_disable_device_irq();
	napi_schedule();
		--> schedule POLL soft irq
SOFTIRQ --> net_rx_action()
	budget = netdev_budget;
	for_each_net_device_needing_polling() {
		...
		weight = napi_state->weight;
		...
		work = n->poll(n, weight);
		...
		budget -= work;
		...
	}

That's the general idea.

Basically once you take you interrupt, and disable device interrupts,
the generic net device layer calls your ->poll() routing with a "weight"
You should not process more RX packets than this value.

If you have less than "weight" work to do, you should do a napi_complete(),
which takes you out of the polling group, and re-enable device interrupts.

So the idea is that you keep getting ->poll()'d until there is no more
RX work to do.

The "weight" argument implements fairness amongst competing, actively
polling, devices on the same CPU.
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: future developments of usbnet

[Oliver Neukum]

Am Mittwoch, 11. Mai 2011, 19:47:27 schrieb David Miller:

> Basically once you take you interrupt, and disable device interrupts,
> the generic net device layer calls your ->poll() routing with a "weight"
> You should not process more RX packets than this value.
> 
> If you have less than "weight" work to do, you should do a napi_complete(),
> which takes you out of the polling group, and re-enable device interrupts.
> 
> So the idea is that you keep getting ->poll()'d until there is no more
> RX work to do.
> 
> The "weight" argument implements fairness amongst competing, actively
> polling, devices on the same CPU.
> 

Thank you, this is very informative. Our problem here is that USB doesn't work
sanely without interrupts. We can stop IO regarding rx, but we cannot stop
interrupts if we want to do rx.

	Regards
		Oliver

Re: future developments of usbnet

[Alan Stern]

On Thu, 12 May 2011, Oliver Neukum wrote:

> Am Mittwoch, 11. Mai 2011, 19:47:27 schrieb David Miller:
> 
> > Basically once you take you interrupt, and disable device interrupts,
> > the generic net device layer calls your ->poll() routing with a "weight"
> > You should not process more RX packets than this value.
> > 
> > If you have less than "weight" work to do, you should do a napi_complete(),
> > which takes you out of the polling group, and re-enable device interrupts.
> > 
> > So the idea is that you keep getting ->poll()'d until there is no more
> > RX work to do.
> > 
> > The "weight" argument implements fairness amongst competing, actively
> > polling, devices on the same CPU.
> > 
> 
> Thank you, this is very informative. Our problem here is that USB doesn't work
> sanely without interrupts. We can stop IO regarding rx, but we cannot stop
> interrupts if we want to do rx.

The idea behind NAPI can be adapted to the usbnet context.  The basic
idea is that the driver is allowed to accept only a limited number of 
packets, driven by polling from the NAPI core.

Therefore usbnet's poll routine should take the "weight" argument as an
indication of how many outstanding rx URBs are allowed.  Each time the
poll routine is called, it should check to see if any rx URBs have
completed since the previous poll.  If not then there is no network
traffic, so usbnet can take itself out of the poll loop.  Otherwise,
the number of outstanding URBs should be adjusted (by unlinking some or
submitting more -- subject to some fixed maximum limit) to match the
new "weight".

Does that make sense?

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: future developments of usbnet

[Oliver Neukum]

Am Donnerstag, 12. Mai 2011, 16:37:28 schrieb Alan Stern:
> Therefore usbnet's poll routine should take the "weight" argument as an
> indication of how many outstanding rx URBs are allowed.  Each time the
> poll routine is called, it should check to see if any rx URBs have
> completed since the previous poll.  If not then there is no network
> traffic, so usbnet can take itself out of the poll loop.  Otherwise,
> the number of outstanding URBs should be adjusted (by unlinking some or
> submitting more -- subject to some fixed maximum limit) to match the
> new "weight".
> 
> Does that make sense?

What does happen if we reach the weight 0 ?

	Regards
		Oliver

Re: future developments of usbnet

[Michał Mirosław]

2011/5/14 Oliver Neukum <oliver-GvhC2dPhHPQdnm+yROfE0A@public.gmane.org>:
> Am Donnerstag, 12. Mai 2011, 16:37:28 schrieb Alan Stern:
>> Therefore usbnet's poll routine should take the "weight" argument as an
>> indication of how many outstanding rx URBs are allowed.  Each time the
>> poll routine is called, it should check to see if any rx URBs have
>> completed since the previous poll.  If not then there is no network
>> traffic, so usbnet can take itself out of the poll loop.  Otherwise,
>> the number of outstanding URBs should be adjusted (by unlinking some or
>> submitting more -- subject to some fixed maximum limit) to match the
>> new "weight".
> What does happen if we reach the weight 0 ?

Poll callback won't be called with weight 0. If there are more RX
packets after processing 'weight' of them, then the callback will be
called again later.

Best Regards,
Michał Mirosław
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: future developments of usbnet

[David Miller]

From: Oliver Neukum <oliver-GvhC2dPhHPQdnm+yROfE0A@public.gmane.org>
Date: Thu, 12 May 2011 09:59:28 +0200

> Our problem here is that USB doesn't work sanely without
> interrupts. We can stop IO regarding rx, but we cannot stop
> interrupts if we want to do rx.

Understood.

However, I was wondering if you could do this in software.  When URBs
trigger, just queue that pending work up in a software queue when NAPI
is scheduled, then when you exit NAPI polling you check that queue to
see if there are interrupt events to process.

Similarly to how we handle software interrupts.

Anyways, just an idea.
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: future developments of usbnet

[Richard Cochran]

On Fri, May 06, 2011 at 08:45:37PM +0200, Oliver Neukum wrote:
> Hi,
> 
> I'd like to get a feeling what people are working out there regarding usbnet.
> So please, if you do something, or think something ought to be done, please
> speak up now.
> 
> IMHO usbnet needs better support for
> 
> - batching protocols
> - double buffering on the rx path

I would like see phylib and (if possible) NAPI, although for different
reasons than the problems mentioned in this thread. I am willing and
able to do the work, but I obviously don't have all of the usbnet
hardware available for testing.

Thanks,

Richard
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html