USB device debugging

USB device debugging

[Narasimha M]

Hi All,
Could you please help in checking receive packet flow from USB driver.

USB device debugging

[Carlo Caione]

On Tue, May 24, 2016 at 9:39 AM, Narasimha M <mnarasimha786@gmail.com> wrote:
> Hi All,
> Could you please help in checking receive packet flow from USB driver.

https://wiki.wireshark.org/CaptureSetup/USB

-- 
Carlo Caione

USB device debugging

[Narasimha M]

Thanks for the information. But i want to know the flow of receive
packet from usb driver to linux stack. I am facing an issue that
corrupted data is coming to usbnet_bh function. So i want to know
about the place where we generate the data in usb driver and sent it
to usbnet. Could you please help in finding out this. I am using
Gobinet driver as usb driver.

On Tue, May 24, 2016 at 1:29 PM, Carlo Caione <carlo@caione.org> wrote:
> On Tue, May 24, 2016 at 9:39 AM, Narasimha M <mnarasimha786@gmail.com> wrote:
>> Hi All,
>> Could you please help in checking receive packet flow from USB driver.
>
> https://wiki.wireshark.org/CaptureSetup/USB
>
> --
> Carlo Caione



-- 
Narasimha

USB device debugging

[Greg KH]

On Tue, May 24, 2016 at 01:48:16PM +0530, Narasimha M wrote:
> Thanks for the information. But i want to know the flow of receive
> packet from usb driver to linux stack. I am facing an issue that
> corrupted data is coming to usbnet_bh function. So i want to know
> about the place where we generate the data in usb driver and sent it
> to usbnet. Could you please help in finding out this. I am using
> Gobinet driver as usb driver.

Look in the usbnet driver itself, it is the one that receives the data
from the USB core.

If you suspect the USB core isn't getting the data properly, use usbmon
to look at the "raw" USB data for the device, instructions for how to
use that is in the kernel documentation directory.

greg k-h

USB device debugging

[Bjørn Mork]

Greg KH <greg@kroah.com> writes:
> On Tue, May 24, 2016 at 01:48:16PM +0530, Narasimha M wrote:
>> Thanks for the information. But i want to know the flow of receive
>> packet from usb driver to linux stack. I am facing an issue that
>> corrupted data is coming to usbnet_bh function. So i want to know
>> about the place where we generate the data in usb driver and sent it
>> to usbnet. Could you please help in finding out this. I am using
>> Gobinet driver as usb driver.
>
> Look in the usbnet driver itself, it is the one that receives the data
> from the USB core.
>
> If you suspect the USB core isn't getting the data properly, use usbmon
> to look at the "raw" USB data for the device, instructions for how to
> use that is in the kernel documentation directory.

I would have started with usbmon in this case. Some Gobi firmwares are
known to corrupt ethernet headers in various ways, if that is the
problem.  There are numerous workarounds for these issues in the
qmi_wwan driver, and AFAIK also in the GobiNet driver.  But the worst
mess is unfixable, where the header is overwritten by arbitrary data and
you don't even know if there is a header there or not.  The only
possible workaround for those devices is using "raw-ip" mode, where you
can be certain that there is no ethernet header (and therefore no mess)

Nothing can be ruled out of course, but I say that there is little
chance of any data corruption in the usbnet code.  It doesn't really
touch the packet buffers between USB controller and network stack.  The
corruption is most likely to happen in firmware, or possibly in GobiNet
although I haven't yet seen any proof of that.


Bj?rn

USB device debugging

[Narasimha M]

Thanks for explanation. Here data is getting corrupted before it comes
to the usbnet itself, so after it reaches to usbnet and go through
network stack, it is failing in tcp checksum and packet is getting
dropped. Same driver is working with linux-3.10.20 but not working
with linux-2.6.32.Not able to find the exact function in driver which
sends receive packet to usbnet, not able to proceed further. Please
suggest some pointers to proceed further.

On Tue, May 24, 2016 at 10:30 PM, Bj?rn Mork <bjorn@mork.no> wrote:
> Greg KH <greg@kroah.com> writes:
>> On Tue, May 24, 2016 at 01:48:16PM +0530, Narasimha M wrote:
>>> Thanks for the information. But i want to know the flow of receive
>>> packet from usb driver to linux stack. I am facing an issue that
>>> corrupted data is coming to usbnet_bh function. So i want to know
>>> about the place where we generate the data in usb driver and sent it
>>> to usbnet. Could you please help in finding out this. I am using
>>> Gobinet driver as usb driver.
>>
>> Look in the usbnet driver itself, it is the one that receives the data
>> from the USB core.
>>
>> If you suspect the USB core isn't getting the data properly, use usbmon
>> to look at the "raw" USB data for the device, instructions for how to
>> use that is in the kernel documentation directory.
>
> I would have started with usbmon in this case. Some Gobi firmwares are
> known to corrupt ethernet headers in various ways, if that is the
> problem.  There are numerous workarounds for these issues in the
> qmi_wwan driver, and AFAIK also in the GobiNet driver.  But the worst
> mess is unfixable, where the header is overwritten by arbitrary data and
> you don't even know if there is a header there or not.  The only
> possible workaround for those devices is using "raw-ip" mode, where you
> can be certain that there is no ethernet header (and therefore no mess)
>
> Nothing can be ruled out of course, but I say that there is little
> chance of any data corruption in the usbnet code.  It doesn't really
> touch the packet buffers between USB controller and network stack.  The
> corruption is most likely to happen in firmware, or possibly in GobiNet
> although I haven't yet seen any proof of that.
>
>
> Bj?rn



-- 
Narasimha

USB device debugging

[Bjørn Mork]

Narasimha M <mnarasimha786@gmail.com> writes:

> Thanks for explanation. Here data is getting corrupted before it comes
> to the usbnet itself, so after it reaches to usbnet and go through
> network stack, it is failing in tcp checksum and packet is getting
> dropped. Same driver is working with linux-3.10.20 but not working
> with linux-2.6.32.Not able to find the exact function in driver which
> sends receive packet to usbnet, not able to proceed further. Please
> suggest some pointers to proceed further.

This is no surprise.  There is no "send receive packet to usbnet".

usbnet allocates a receive buffer and hands it to the USB host
controller. This happens in rx_submit().  The host controller calls the
rx_complete() callback when it gets data from the device.  This callback
will trigger further handling in the usbnet_bh() tasklet, calling
rx_process().  This again calls the minidriver specific rx_fixup()
callback if there is any, which is GobiNet's only chance of inspecting
and possibly modifying the buffer.  But normally it will not touch the
buffer, since there is no fixup necessary for Gobi devices in 802.3 mode
(they transmit plain ethernet packets). rx_process() ends up calling
usbnet_skb_return() which hands the buffer over the the networking stack
using netif_rx().

So, if we ignore the possible firmware bug workarounds in rx_fixup(),
then nothing ever touches the receive buffer in usbnet. It's just a
handle being passed around.

Note that I don't remember, or care, what 2.6.32 might have done. It's too
outdated to be relevant.  But the usbnet design is much older and
haven't changed drastically, so I assume most of the above is valid
there too.


Bj?rn

USB device debugging

[Narasimha M]

I am able to see the corrupted data in rx_complete itself. What are
the possibilities for the data corruption in rx_complete. Any fixes to
resolve this. And from where the data in rx_submit generates

On Wed, May 25, 2016 at 3:41 PM, Bj?rn Mork <bjorn@mork.no> wrote:
> Narasimha M <mnarasimha786@gmail.com> writes:
>
>> Thanks for explanation. Here data is getting corrupted before it comes
>> to the usbnet itself, so after it reaches to usbnet and go through
>> network stack, it is failing in tcp checksum and packet is getting
>> dropped. Same driver is working with linux-3.10.20 but not working
>> with linux-2.6.32.Not able to find the exact function in driver which
>> sends receive packet to usbnet, not able to proceed further. Please
>> suggest some pointers to proceed further.
>
> This is no surprise.  There is no "send receive packet to usbnet".
>
> usbnet allocates a receive buffer and hands it to the USB host
> controller. This happens in rx_submit().  The host controller calls the
> rx_complete() callback when it gets data from the device.  This callback
> will trigger further handling in the usbnet_bh() tasklet, calling
> rx_process().  This again calls the minidriver specific rx_fixup()
> callback if there is any, which is GobiNet's only chance of inspecting
> and possibly modifying the buffer.  But normally it will not touch the
> buffer, since there is no fixup necessary for Gobi devices in 802.3 mode
> (they transmit plain ethernet packets). rx_process() ends up calling
> usbnet_skb_return() which hands the buffer over the the networking stack
> using netif_rx().
>
> So, if we ignore the possible firmware bug workarounds in rx_fixup(),
> then nothing ever touches the receive buffer in usbnet. It's just a
> handle being passed around.
>
> Note that I don't remember, or care, what 2.6.32 might have done. It's too
> outdated to be relevant.  But the usbnet design is much older and
> haven't changed drastically, so I assume most of the above is valid
> there too.
>
>
> Bj?rn



-- 
Narasimha

USB device debugging

[Bjørn Mork]

Narasimha M <mnarasimha786@gmail.com> writes:

> I am able to see the corrupted data in rx_complete itself.

Then the only likely source is the device.  In theory it could be the
host controller, but that is very unliekely unless you use some
out-of-tree driver there too.

You can easily verify that usbnet can be ruled out by follwing the
earlier usbmon advice.

> What are
> the possibilities for the data corruption in rx_complete. Any fixes to
> resolve this. And from where the data in rx_submit generates

rx_submit allocates new buffers using __netdev_alloc_skb_ip_align() like
many other network drivers.



Bj?rn

USB device debugging

[Narasimha M]

Thanks for the info. Sorry to ask you again, where does the usb driver
(GobiNet in my case) comes into picture in receive packet flow. I
suspect that the driver has to send some data (may be with some
interrupts) to rx_complete and then it will go to rx_submit. Is my
understanding is correct ? or if rx_submit is the first function to
generate the data, then where does Gobinet driver comes into place.
Could you please explain. I don't know how some usb driver works. This
is the first time i am working on it.

On Wed, May 25, 2016 at 4:36 PM, Bj?rn Mork <bjorn@mork.no> wrote:
> Narasimha M <mnarasimha786@gmail.com> writes:
>
>> I am able to see the corrupted data in rx_complete itself.
>
> Then the only likely source is the device.  In theory it could be the
> host controller, but that is very unliekely unless you use some
> out-of-tree driver there too.
>
> You can easily verify that usbnet can be ruled out by follwing the
> earlier usbmon advice.
>
>> What are
>> the possibilities for the data corruption in rx_complete. Any fixes to
>> resolve this. And from where the data in rx_submit generates
>
> rx_submit allocates new buffers using __netdev_alloc_skb_ip_align() like
> many other network drivers.
>
>
>
> Bj?rn



-- 
Narasimha

USB device debugging

[Bjørn Mork]

Narasimha M <mnarasimha786@gmail.com> writes:

> Thanks for the info. Sorry to ask you again, where does the usb driver
> (GobiNet in my case) comes into picture in receive packet flow. I
> suspect that the driver has to send some data (may be with some
> interrupts) to rx_complete and then it will go to rx_submit. Is my
> understanding is correct ? or if rx_submit is the first function to
> generate the data, then where does Gobinet driver comes into place.
> Could you please explain. I don't know how some usb driver works. This
> is the first time i am working on it.

GobiNet isn't part of the receive packet flow at all, if we assume a
non-buggy Gobi device operating in 802.3 mode.  The rx_fixup callback is
called, but it does nothing.  There isn't anything to do. The buffer is
filled with an ethernet packet by the device + host controller.

The usbnet_bh() tasklet is responsible for calling rx_submit (indirectly
in newer kernels).  This is triggered by setting the EVENT_DEV_OPEN
flag, which is done by usbnet_open().  GobiNet obfuscates this quite a
bit, but it doen't do anything extra-ordinary here - it simply calls
usbnet_open() when you open the netdev.



Bj?rn

USB device debugging

[Narasimha M]

Thanks for clear explanation. Could you please let me know where does
host_controller will be present is it in USB device or USB core part
of kernel.

Actually my issue is that we ported Gobinet driver from open source to
support one dongle. After porting, it is working fine on little endian
host (linux-2.6.31 kernel). We have added some changes in driver code
o support it for big endian hosts. After adding the changes with the
help of le16_to_cpu () function dongle is working in one of our big
endian hosts (which has linux-3.10.20), but facing corrupted packet
issue in other bigendian supported host (linux-2.6.32). Could you
please help in providing some pointers here

On Wed, May 25, 2016 at 5:31 PM, Bj?rn Mork <bjorn@mork.no> wrote:
> Narasimha M <mnarasimha786@gmail.com> writes:
>
>> Thanks for the info. Sorry to ask you again, where does the usb driver
>> (GobiNet in my case) comes into picture in receive packet flow. I
>> suspect that the driver has to send some data (may be with some
>> interrupts) to rx_complete and then it will go to rx_submit. Is my
>> understanding is correct ? or if rx_submit is the first function to
>> generate the data, then where does Gobinet driver comes into place.
>> Could you please explain. I don't know how some usb driver works. This
>> is the first time i am working on it.
>
> GobiNet isn't part of the receive packet flow at all, if we assume a
> non-buggy Gobi device operating in 802.3 mode.  The rx_fixup callback is
> called, but it does nothing.  There isn't anything to do. The buffer is
> filled with an ethernet packet by the device + host controller.
>
> The usbnet_bh() tasklet is responsible for calling rx_submit (indirectly
> in newer kernels).  This is triggered by setting the EVENT_DEV_OPEN
> flag, which is done by usbnet_open().  GobiNet obfuscates this quite a
> bit, but it doen't do anything extra-ordinary here - it simply calls
> usbnet_open() when you open the netdev.
>
>
>
> Bj?rn



-- 
Narasimha

USB device debugging

[Bjørn Mork]

Narasimha M <mnarasimha786@gmail.com> writes:

> Thanks for clear explanation. Could you please let me know where does
> host_controller will be present is it in USB device or USB core part
> of kernel.
>
> Actually my issue is that we ported Gobinet driver from open source to
> support one dongle. After porting, it is working fine on little endian
> host (linux-2.6.31 kernel). We have added some changes in driver code
> o support it for big endian hosts. After adding the changes with the
> help of le16_to_cpu () function dongle is working in one of our big
> endian hosts (which has linux-3.10.20), but facing corrupted packet
> issue in other bigendian supported host (linux-2.6.32). Could you
> please help in providing some pointers here

Sounds like you have a bug in your code.  Did you expect me to make a
more precise guess?


Bj?rn

USB device debugging

[Narasimha M]

If possible yes, i need a help. I am working on it for the first time
on usb subsystem. I just tried using usbmon also, i got some raw data
in text file. Could you please let me know how it will help to debug
further?.

One more observation is that if i decrease the mtu size to around 480,
then i am able to run the traffic. Which means the data is getting
corrupted after 480 bytes length or so. Is there any possible chance
for this case?


On Wed, May 25, 2016 at 6:26 PM, Bj?rn Mork <bjorn@mork.no> wrote:
> Narasimha M <mnarasimha786@gmail.com> writes:
>
>> Thanks for clear explanation. Could you please let me know where does
>> host_controller will be present is it in USB device or USB core part
>> of kernel.
>>
>> Actually my issue is that we ported Gobinet driver from open source to
>> support one dongle. After porting, it is working fine on little endian
>> host (linux-2.6.31 kernel). We have added some changes in driver code
>> o support it for big endian hosts. After adding the changes with the
>> help of le16_to_cpu () function dongle is working in one of our big
>> endian hosts (which has linux-3.10.20), but facing corrupted packet
>> issue in other bigendian supported host (linux-2.6.32). Could you
>> please help in providing some pointers here
>
> Sounds like you have a bug in your code.  Did you expect me to make a
> more precise guess?
>
>
> Bj?rn



-- 
Narasimha

USB device debugging

[Narasimha M]

I am attaching the raw data in two cases (working - 2.mon.out) and not
working (1.mon.out) files.

working one is with the device having linux 3.10.20 and not working
one is with linux 2.6.32. Here i have run same traffic in both the
cases.

Could you please suggest some pointers based on these.

On Wed, May 25, 2016 at 8:54 PM, Narasimha M <mnarasimha786@gmail.com> wrote:
> If possible yes, i need a help. I am working on it for the first time
> on usb subsystem. I just tried using usbmon also, i got some raw data
> in text file. Could you please let me know how it will help to debug
> further?.
>
> One more observation is that if i decrease the mtu size to around 480,
> then i am able to run the traffic. Which means the data is getting
> corrupted after 480 bytes length or so. Is there any possible chance
> for this case?
>
>
> On Wed, May 25, 2016 at 6:26 PM, Bj?rn Mork <bjorn@mork.no> wrote:
>> Narasimha M <mnarasimha786@gmail.com> writes:
>>
>>> Thanks for clear explanation. Could you please let me know where does
>>> host_controller will be present is it in USB device or USB core part
>>> of kernel.
>>>
>>> Actually my issue is that we ported Gobinet driver from open source to
>>> support one dongle. After porting, it is working fine on little endian
>>> host (linux-2.6.31 kernel). We have added some changes in driver code
>>> o support it for big endian hosts. After adding the changes with the
>>> help of le16_to_cpu () function dongle is working in one of our big
>>> endian hosts (which has linux-3.10.20), but facing corrupted packet
>>> issue in other bigendian supported host (linux-2.6.32). Could you
>>> please help in providing some pointers here
>>
>> Sounds like you have a bug in your code.  Did you expect me to make a
>> more precise guess?
>>
>>
>> Bj?rn
>
>
>
> --
> Narasimha



-- 
Narasimha
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1.mon.out
Type: application/octet-stream
Size: 58948 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160525/a254517c/attachment-0002.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2.mon.out
Type: application/octet-stream
Size: 175901 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160525/a254517c/attachment-0003.obj 

USB device debugging

[Greg KH]

On Wed, May 25, 2016 at 09:11:59PM +0530, Narasimha M wrote:
> I am attaching the raw data in two cases (working - 2.mon.out) and not
> working (1.mon.out) files.
> 
> working one is with the device having linux 3.10.20 and not working
> one is with linux 2.6.32. Here i have run same traffic in both the
> cases.

You really are on your own with 2.6.32, sorry.  If you have issues with
that kernel, please contact the vendor that is forcing you to stick with
it, as you are paying them to do this work.

If that vendor is yourself, well, sorry, it's all your fault, nothing
the community can do with such an obsolete and broken kernel release.

good luck!

greg k-h

USB device debugging

[Narasimha M]

Hi, Thank you so much for your help. One last query in this. From you
explanation USB host controller will receive the packet from device. So
could you please tell me, what is the first function being called when
device gets the data and if I am not working this first function will be in
USB host controller code.
On Wed, May 25, 2016 at 09:11:59PM +0530, Narasimha M wrote:
> I am attaching the raw data in two cases (working - 2.mon.out) and not
> working (1.mon.out) files.
>
> working one is with the device having linux 3.10.20 and not working
> one is with linux 2.6.32. Here i have run same traffic in both the
> cases.

You really are on your own with 2.6.32, sorry.  If you have issues with
that kernel, please contact the vendor that is forcing you to stick with
it, as you are paying them to do this work.

If that vendor is yourself, well, sorry, it's all your fault, nothing
the community can do with such an obsolete and broken kernel release.

good luck!

greg k-h
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160526/ec1bcc7e/attachment.html 

USB device debugging

[Greg KH]

On Thu, May 26, 2016 at 11:01:31AM +0530, Narasimha M wrote:
> Hi, Thank you so much for your help. One last query in this. From you
> explanation USB host controller will receive the packet from device. So could
> you please tell me, what is the first function being called when device gets
> the data

Which device?  The host controller?  Or the driver for the USB device
itself?

> and if I am not working this first function will be in USB host
> controller code.

The host controller gets the data first and then passes it to the
specific USB device driver.

Again, usbmon is going to be the best thing to use here, good luck!

greg k-h

USB device debugging

[Narasimha M]

On Wed, May 25, 2016 at 3:41 PM, Bj?rn Mork <bjorn@mork.no> wrote:
> Narasimha M <mnarasimha786@gmail.com> writes:
>
>> Thanks for explanation. Here data is getting corrupted before it comes
>> to the usbnet itself, so after it reaches to usbnet and go through
>> network stack, it is failing in tcp checksum and packet is getting
>> dropped. Same driver is working with linux-3.10.20 but not working
>> with linux-2.6.32.Not able to find the exact function in driver which
>> sends receive packet to usbnet, not able to proceed further. Please
>> suggest some pointers to proceed further.
>
> This is no surprise.  There is no "send receive packet to usbnet".
>
> usbnet allocates a receive buffer and hands it to the USB host
> controller. This happens in rx_submit().
Understood this.

>The host controller calls the
> rx_complete() callback when it gets data from the device.

Here i want the help that where will be this host controller ? and how
to it calls the rx_complete () callback ? Whet is the function in host
controller does this. ? Ehat is the exact meaning of when device gets
the data ? These things i am not able to understand. BTW iam using
some 4g dongle as usb and usb driver is gobinet to support this and i
am checking this on one router which has 2.6.32 kernel. I tried to use
usbmon also, i got the raw data. I am trying to process it. Meanwhile
i am not clear about the above explanation. Please try to explain me
if you are not irritated.


 >This callback
> will trigger further handling in the usbnet_bh() tasklet, calling
> rx_process().  This again calls the minidriver specific rx_fixup()
> callback if there is any, which is GobiNet's only chance of inspecting
> and possibly modifying the buffer.  But normally it will not touch the
> buffer, since there is no fixup necessary for Gobi devices in 802.3 mode
> (they transmit plain ethernet packets). rx_process() ends up calling
> usbnet_skb_return() which hands the buffer over the the networking stack
> using netif_rx().
>

Understood

> So, if we ignore the possible firmware bug workarounds in rx_fixup(),
> then nothing ever touches the receive buffer in usbnet. It's just a
> handle being passed around.
>
Understood.
> Note that I don't remember, or care, what 2.6.32 might have done. It's too
> outdated to be relevant.  But the usbnet design is much older and
> haven't changed drastically, so I assume most of the above is valid
> there too.
>
>
> Bj?rn



-- 
Narasimha