Re: STMMAC driver with TSO enabled issue

Re: STMMAC driver with TSO enabled issue

[Jose Abreu]

++net-dev

Hi Bhadram,

On 09-05-2018 12:03, Bhadram Varka wrote:
> Hi,
>
> Thanks for responding.
>
> Tried below suggested way. Still observing the issue -

It seems stmmac has a bug in the RX side when using TSO which is
causing all the RX descriptors to be consumed. The stmmac_rx()
function will need to be refactored. I will send a fix ASAP.

Thanks and Best Regards,
Jose Miguel Abreu

>
> [root@alarm ~]# iperf3 -c 10.19.65.141
> Connecting to host 10.19.65.141, port 5201
> [  5] local 10.19.65.210 port 57630 connected to 10.19.65.141
> port 5201
> [   65.408268] stmmac_tso_xmit(): line = 2842
> [   65.412362] stmmac_tso_xmit: tcphdrlen 32, hdr_len 66,
> pay_len 0, mss 1448
> [   65.419224]  skb->len 8754, skb->data_len 8688
> [   65.423672] stmmac_tso_xmit: curr=20 dirty=17 f=18, e=20,
> f_p=00000000178e52e1, nfrags 1
> [   65.431747] TX descriptor ring:
> [   65.434881] 000 [0x82005000]: 0x0 0x0 0x0 0x0
> [   65.439230] 001 [0x82005010]: 0x0 0x0 0x0 0x0
> [   65.443578] 002 [0x82005020]: 0x0 0x0 0x0 0x0
> [   65.447927] 003 [0x82005030]: 0x0 0x0 0x0 0x0
> [   65.452275] 004 [0x82005040]: 0x0 0x0 0x0 0x0
> [   65.456622] 005 [0x82005050]: 0x0 0x0 0x0 0x0
> [   65.460970] 006 [0x82005060]: 0x0 0x0 0x0 0x0
> [   65.465316] 007 [0x82005070]: 0x0 0x0 0x0 0x0
> [   65.469664] 008 [0x82005080]: 0x0 0x0 0x0 0x0
> [   65.474010] 009 [0x82005090]: 0x0 0x0 0x0 0x0
> [   65.478357] 010 [0x820050a0]: 0x0 0x0 0x0 0x0
> [   65.482706] 011 [0x820050b0]: 0x0 0x0 0x0 0x0
> [   65.487053] 012 [0x820050c0]: 0x0 0x0 0x0 0x0
> [   65.491400] 013 [0x820050d0]: 0x0 0x0 0x0 0x0
> [   65.495746] 014 [0x820050e0]: 0x0 0x0 0x0 0x0
> [   65.500092] 015 [0x820050f0]: 0x0 0x0 0x0 0x0
> [   65.504438] 016 [0x82005100]: 0x0 0x0 0x0 0x0
> [   65.508784] 017 [0x82005110]: 0x0 0x0 0x5a8 0xc4000000
> [   65.513910] 018 [0x82005120]: 0xfb297000 0x0 0x42 0xa04421f0
> [   65.519557] 019 [0x82005130]: 0xfb298000 0x0 0x21f0 0x90000000
> [   65.525376] 020 [0x82005140]: 0x0 0x0 0x0 0x0
> [   65.529722] 021 [0x82005150]: 0x0 0x0 0x0 0x0
> [   65.534069] 022 [0x82005160]: 0x0 0x0 0x0 0x0
> [   65.538414] 023 [0x82005170]: 0x0 0x0 0x0 0x0
> [   65.542761] 024 [0x82005180]: 0x0 0x0 0x0 0x0
> [   65.547107] 025 [0x82005190]: 0x0 0x0 0x0 0x0
> [   65.551454] 026 [0x820051a0]: 0x0 0x0 0x0 0x0
> [   65.555802] 027 [0x820051b0]: 0x0 0x0 0x0 0x0
> [   65.560147] 028 [0x820051c0]: 0x0 0x0 0x0 0x0
> [   65.564493] 029 [0x820051d0]: 0x0 0x0 0x0 0x0
> [   65.568840] 030 [0x820051e0]: 0x0 0x0 0x0 0x0
> [   65.573187] 031 [0x820051f0]: 0x0 0x0 0x0 0x0
> [   65.577533] 032 [0x82005200]: 0x0 0x0 0x0 0x0
> [   65.581879] 033 [0x82005210]: 0x0 0x0 0x0 0x0
> [   65.586225] 034 [0x82005220]: 0x0 0x0 0x0 0x0
> [   65.590571] 035 [0x82005230]: 0x0 0x0 0x0 0x0
> [   65.594917] 036 [0x82005240]: 0x0 0x0 0x0 0x0
> [   65.599262] 037 [0x82005250]: 0x0 0x0 0x0 0x0
> [   65.603607] 038 [0x82005260]: 0x0 0x0 0x0 0x0
> [   65.607952] 039 [0x82005270]: 0x0 0x0 0x0 0x0
> [   65.612297] 040 [0x82005280]: 0x0 0x0 0x0 0x0
> [   65.616643] 041 [0x82005290]: 0x0 0x0 0x0 0x0
> [   65.620989] 042 [0x820052a0]: 0x0 0x0 0x0 0x0
> [   65.625336] 043 [0x820052b0]: 0x0 0x0 0x0 0x0
> [   65.629681] 044 [0x820052c0]: 0x0 0x0 0x0 0x0
> [   65.634027] 045 [0x820052d0]: 0x0 0x0 0x0 0x0
> [   65.638372] 046 [0x820052e0]: 0x0 0x0 0x0 0x0
> [   65.642718] 047 [0x820052f0]: 0x0 0x0 0x0 0x0
> [   65.647063] 048 [0x82005300]: 0x0 0x0 0x0 0x0
> [   65.651408] 049 [0x82005310]: 0x0 0x0 0x0 0x0
> [   65.655754] 050 [0x82005320]: 0x0 0x0 0x0 0x0
> [   65.660099] 051 [0x82005330]: 0x0 0x0 0x0 0x0
> [   65.664444] 052 [0x82005340]: 0x0 0x0 0x0 0x0
> [   65.668790] 053 [0x82005350]: 0x0 0x0 0x0 0x0
> [   65.673134] 054 [0x82005360]: 0x0 0x0 0x0 0x0
> [   65.677480] 055 [0x82005370]: 0x0 0x0 0x0 0x0
> [   65.681825] 056 [0x82005380]: 0x0 0x0 0x0 0x0
> [   65.686170] 057 [0x82005390]: 0x0 0x0 0x0 0x0
> [   65.690515] 058 [0x820053a0]: 0x0 0x0 0x0 0x0
> [   65.694861] 059 [0x820053b0]: 0x0 0x0 0x0 0x0
> [   65.699206] 060 [0x820053c0]: 0x0 0x0 0x0 0x0
> [   65.703552] 061 [0x820053d0]: 0x0 0x0 0x0 0x0
> [   65.707898] 062 [0x820053e0]: 0x0 0x0 0x0 0x0
> [   65.712243] 063 [0x820053f0]: 0x0 0x0 0x0 0x0
> [   65.716706] stmmac_tso_xmit(): line = 2842
> [   65.720802] stmmac_tso_xmit: tcphdrlen 32, hdr_len 66,
> pay_len 0, mss 1448
> [   65.727669]  skb->len 4410, skb->data_len 4344
> [   65.732114] stmmac_tso_xmit: curr=22 dirty=19 f=20, e=22,
> f_p=00000000b1247b41, nfrags 1
> [   65.740190] TX descriptor ring:
> [   65.743327] 000 [0x82005000]: 0x0 0x0 0x0 0x0
> [   65.747678] 001 [0x82005010]: 0x0 0x0 0x0 0x0
> [   65.752029] 002 [0x82005020]: 0x0 0x0 0x0 0x0
> [   65.756378] 003 [0x82005030]: 0x0 0x0 0x0 0x0
> [   65.760727] 004 [0x82005040]: 0x0 0x0 0x0 0x0
> [   65.765077] 005 [0x82005050]: 0x0 0x0 0x0 0x0
> [   65.769427] 006 [0x82005060]: 0x0 0x0 0x0 0x0
> [   65.773776] 007 [0x82005070]: 0x0 0x0 0x0 0x0
> [   65.778126] 008 [0x82005080]: 0x0 0x0 0x0 0x0
> [   65.782476] 009 [0x82005090]: 0x0 0x0 0x0 0x0
> [   65.786826] 010 [0x820050a0]: 0x0 0x0 0x0 0x0
> [   65.791176] 011 [0x820050b0]: 0x0 0x0 0x0 0x0
> [   65.795526] 012 [0x820050c0]: 0x0 0x0 0x0 0x0
> [   65.799875] 013 [0x820050d0]: 0x0 0x0 0x0 0x0
> [   65.804225] 014 [0x820050e0]: 0x0 0x0 0x0 0x0
> [   65.808575] 015 [0x820050f0]: 0x0 0x0 0x0 0x0
> [   65.812925] 016 [0x82005100]: 0x0 0x0 0x0 0x0
> [   65.817274] 017 [0x82005110]: 0x0 0x0 0x0 0x0
> [   65.821625] 018 [0x82005120]: 0x0 0x0 0x0 0x0
> [   65.825976] 019 [0x82005130]: 0xfb298000 0x0 0x21f0 0x90000000
> [   65.831800] 020 [0x82005140]: 0xfb2a1000 0x0 0x42 0xa04410f8
> [   65.837450] 021 [0x82005150]: 0xfb2a2000 0x0 0x10f8 0x90000000
> [   65.843273] 022 [0x82005160]: 0x0 0x0 0x0 0x0
> [   65.847622] 023 [0x82005170]: 0x0 0x0 0x0 0x0
> [   65.851971] 024 [0x82005180]: 0x0 0x0 0x0 0x0
> [   65.856319] 025 [0x82005190]: 0x0 0x0 0x0 0x0
> [   65.860670] 026 [0x820051a0]: 0x0 0x0 0x0 0x0
> [   65.865020] 027 [0x820051b0]: 0x0 0x0 0x0 0x0
> [   65.869369] 028 [0x820051c0]: 0x0 0x0 0x0 0x0
> [   65.873719] 029 [0x820051d0]: 0x0 0x0 0x0 0x0
> [   65.878068] 030 [0x820051e0]: 0x0 0x0 0x0 0x0
> [   65.882418] 031 [0x820051f0]: 0x0 0x0 0x0 0x0
> [   65.886767] 032 [0x82005200]: 0x0 0x0 0x0 0x0
> [   65.891118] 033 [0x82005210]: 0x0 0x0 0x0 0x0
> [   65.895467] 034 [0x82005220]: 0x0 0x0 0x0 0x0
> [   65.899816] 035 [0x82005230]: 0x0 0x0 0x0 0x0
> [   65.904165] 036 [0x82005240]: 0x0 0x0 0x0 0x0
> [   65.908515] 037 [0x82005250]: 0x0 0x0 0x0 0x0
> [   65.912865] 038 [0x82005260]: 0x0 0x0 0x0 0x0
> [   65.917215] 039 [0x82005270]: 0x0 0x0 0x0 0x0
> [   65.921564] 040 [0x82005280]: 0x0 0x0 0x0 0x0
> [   65.925915] 041 [0x82005290]: 0x0 0x0 0x0 0x0
> [   65.930264] 042 [0x820052a0]: 0x0 0x0 0x0 0x0
> [   65.934615] 043 [0x820052b0]: 0x0 0x0 0x0 0x0
> [   65.938964] 044 [0x820052c0]: 0x0 0x0 0x0 0x0
> [   65.943313] 045 [0x820052d0]: 0x0 0x0 0x0 0x0
> [   65.947664] 046 [0x820052e0]: 0x0 0x0 0x0 0x0
> [   65.952012] 047 [0x820052f0]: 0x0 0x0 0x0 0x0
> [   65.956363] 048 [0x82005300]: 0x0 0x0 0x0 0x0
> [   65.960712] 049 [0x82005310]: 0x0 0x0 0x0 0x0
> [   65.965061] 050 [0x82005320]: 0x0 0x0 0x0 0x0
> [   65.969410] 051 [0x82005330]: 0x0 0x0 0x0 0x0
> [   65.973760] 052 [0x82005340]: 0x0 0x0 0x0 0x0
> [   65.978110] 053 [0x82005350]: 0x0 0x0 0x0 0x0
> [   65.982460] 054 [0x82005360]: 0x0 0x0 0x0 0x0
> [   65.986812] 055 [0x82005370]: 0x0 0x0 0x0 0x0
> [   65.991161] 056 [0x82005380]: 0x0 0x0 0x0 0x0
> [   65.995510] 057 [0x82005390]: 0x0 0x0 0x0 0x0
> [   65.999860] 058 [0x820053a0]: 0x0 0x0 0x0 0x0
> [   66.004210] 059 [0x820053b0]: 0x0 0x0 0x0 0x0
> [   66.008559] 060 [0x820053c0]: 0x0 0x0 0x0 0x0
> [   66.012908] 061 [0x820053d0]: 0x0 0x0 0x0 0x0
> [   66.017257] 062 [0x820053e0]: 0x0 0x0 0x0 0x0
> [   66.021607] 063 [0x820053f0]: 0x0 0x0 0x0 0x0
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec   184 KBytes  1.50 Mbits/sec    0  
> 1.41 KBytes
> [  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
> KBytes
> [  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
> KBytes
> [  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
> KBytes
> [  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
> KBytes
> [  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
> KBytes
> [  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
> KBytes
> [  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
> KBytes
> [  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
> KBytes
>
>
> On 5/9/2018 3:35 PM, Jose Abreu wrote:
>> Hi Bhadram,
>>
>> On 09-05-2018 08:18, Bhadram Varka wrote:
>>>
>>> +               queue0 {
>>> +                       snps,weight = <0x10>;
>>
>>> +               queue1 {
>>> +                       snps,weight = <0x10>;
>>>
>>
>>> +               queue2 {
>>> +                       snps,weight = <0x10>;
>>>
>>
>>> +               queue3 {
>>> +                       snps,weight = <0x10>;
>>>
>>
>> This is wrong. You can't use the same weight for all queues.
>> Please try with different weights (for example: 0x10, 0x11, 0x12,
>> 0x13).
>>
>> Thanks and Best Regards,
>> Jose Miguel Abreu
>>
>>
>

Re: STMMAC driver with TSO enabled issue

[Jose Abreu]

Hi Bhadram,

On 10-05-2018 09:55, Jose Abreu wrote:
> ++net-dev
>
> Hi Bhadram,
>
> On 09-05-2018 12:03, Bhadram Varka wrote:
>> Hi,
>>
>> Thanks for responding.
>>
>> Tried below suggested way. Still observing the issue -
> It seems stmmac has a bug in the RX side when using TSO which is
> causing all the RX descriptors to be consumed. The stmmac_rx()
> function will need to be refactored. I will send a fix ASAP.

Are you using this patch [1] ? Because there is a problem with
the patch. By adding the previously removed call to
stmmac_init_rx_desc() TSO works okay in my setup.

[1] https://patchwork.ozlabs.org/patch/910224/

>
> Thanks and Best Regards,
> Jose Miguel Abreu
>
>> [root@alarm ~]# iperf3 -c 10.19.65.141
>> Connecting to host 10.19.65.141, port 5201
>> [  5] local 10.19.65.210 port 57630 connected to 10.19.65.141
>> port 5201
>> [   65.408268] stmmac_tso_xmit(): line = 2842
>> [   65.412362] stmmac_tso_xmit: tcphdrlen 32, hdr_len 66,
>> pay_len 0, mss 1448
>> [   65.419224]  skb->len 8754, skb->data_len 8688
>> [   65.423672] stmmac_tso_xmit: curr=20 dirty=17 f=18, e=20,
>> f_p=00000000178e52e1, nfrags 1
>> [   65.431747] TX descriptor ring:
>> [   65.434881] 000 [0x82005000]: 0x0 0x0 0x0 0x0
>> [   65.439230] 001 [0x82005010]: 0x0 0x0 0x0 0x0
>> [   65.443578] 002 [0x82005020]: 0x0 0x0 0x0 0x0
>> [   65.447927] 003 [0x82005030]: 0x0 0x0 0x0 0x0
>> [   65.452275] 004 [0x82005040]: 0x0 0x0 0x0 0x0
>> [   65.456622] 005 [0x82005050]: 0x0 0x0 0x0 0x0
>> [   65.460970] 006 [0x82005060]: 0x0 0x0 0x0 0x0
>> [   65.465316] 007 [0x82005070]: 0x0 0x0 0x0 0x0
>> [   65.469664] 008 [0x82005080]: 0x0 0x0 0x0 0x0
>> [   65.474010] 009 [0x82005090]: 0x0 0x0 0x0 0x0
>> [   65.478357] 010 [0x820050a0]: 0x0 0x0 0x0 0x0
>> [   65.482706] 011 [0x820050b0]: 0x0 0x0 0x0 0x0
>> [   65.487053] 012 [0x820050c0]: 0x0 0x0 0x0 0x0
>> [   65.491400] 013 [0x820050d0]: 0x0 0x0 0x0 0x0
>> [   65.495746] 014 [0x820050e0]: 0x0 0x0 0x0 0x0
>> [   65.500092] 015 [0x820050f0]: 0x0 0x0 0x0 0x0
>> [   65.504438] 016 [0x82005100]: 0x0 0x0 0x0 0x0
>> [   65.508784] 017 [0x82005110]: 0x0 0x0 0x5a8 0xc4000000
>> [   65.513910] 018 [0x82005120]: 0xfb297000 0x0 0x42 0xa04421f0
>> [   65.519557] 019 [0x82005130]: 0xfb298000 0x0 0x21f0 0x90000000
>> [   65.525376] 020 [0x82005140]: 0x0 0x0 0x0 0x0
>> [   65.529722] 021 [0x82005150]: 0x0 0x0 0x0 0x0
>> [   65.534069] 022 [0x82005160]: 0x0 0x0 0x0 0x0
>> [   65.538414] 023 [0x82005170]: 0x0 0x0 0x0 0x0
>> [   65.542761] 024 [0x82005180]: 0x0 0x0 0x0 0x0
>> [   65.547107] 025 [0x82005190]: 0x0 0x0 0x0 0x0
>> [   65.551454] 026 [0x820051a0]: 0x0 0x0 0x0 0x0
>> [   65.555802] 027 [0x820051b0]: 0x0 0x0 0x0 0x0
>> [   65.560147] 028 [0x820051c0]: 0x0 0x0 0x0 0x0
>> [   65.564493] 029 [0x820051d0]: 0x0 0x0 0x0 0x0
>> [   65.568840] 030 [0x820051e0]: 0x0 0x0 0x0 0x0
>> [   65.573187] 031 [0x820051f0]: 0x0 0x0 0x0 0x0
>> [   65.577533] 032 [0x82005200]: 0x0 0x0 0x0 0x0
>> [   65.581879] 033 [0x82005210]: 0x0 0x0 0x0 0x0
>> [   65.586225] 034 [0x82005220]: 0x0 0x0 0x0 0x0
>> [   65.590571] 035 [0x82005230]: 0x0 0x0 0x0 0x0
>> [   65.594917] 036 [0x82005240]: 0x0 0x0 0x0 0x0
>> [   65.599262] 037 [0x82005250]: 0x0 0x0 0x0 0x0
>> [   65.603607] 038 [0x82005260]: 0x0 0x0 0x0 0x0
>> [   65.607952] 039 [0x82005270]: 0x0 0x0 0x0 0x0
>> [   65.612297] 040 [0x82005280]: 0x0 0x0 0x0 0x0
>> [   65.616643] 041 [0x82005290]: 0x0 0x0 0x0 0x0
>> [   65.620989] 042 [0x820052a0]: 0x0 0x0 0x0 0x0
>> [   65.625336] 043 [0x820052b0]: 0x0 0x0 0x0 0x0
>> [   65.629681] 044 [0x820052c0]: 0x0 0x0 0x0 0x0
>> [   65.634027] 045 [0x820052d0]: 0x0 0x0 0x0 0x0
>> [   65.638372] 046 [0x820052e0]: 0x0 0x0 0x0 0x0
>> [   65.642718] 047 [0x820052f0]: 0x0 0x0 0x0 0x0
>> [   65.647063] 048 [0x82005300]: 0x0 0x0 0x0 0x0
>> [   65.651408] 049 [0x82005310]: 0x0 0x0 0x0 0x0
>> [   65.655754] 050 [0x82005320]: 0x0 0x0 0x0 0x0
>> [   65.660099] 051 [0x82005330]: 0x0 0x0 0x0 0x0
>> [   65.664444] 052 [0x82005340]: 0x0 0x0 0x0 0x0
>> [   65.668790] 053 [0x82005350]: 0x0 0x0 0x0 0x0
>> [   65.673134] 054 [0x82005360]: 0x0 0x0 0x0 0x0
>> [   65.677480] 055 [0x82005370]: 0x0 0x0 0x0 0x0
>> [   65.681825] 056 [0x82005380]: 0x0 0x0 0x0 0x0
>> [   65.686170] 057 [0x82005390]: 0x0 0x0 0x0 0x0
>> [   65.690515] 058 [0x820053a0]: 0x0 0x0 0x0 0x0
>> [   65.694861] 059 [0x820053b0]: 0x0 0x0 0x0 0x0
>> [   65.699206] 060 [0x820053c0]: 0x0 0x0 0x0 0x0
>> [   65.703552] 061 [0x820053d0]: 0x0 0x0 0x0 0x0
>> [   65.707898] 062 [0x820053e0]: 0x0 0x0 0x0 0x0
>> [   65.712243] 063 [0x820053f0]: 0x0 0x0 0x0 0x0
>> [   65.716706] stmmac_tso_xmit(): line = 2842
>> [   65.720802] stmmac_tso_xmit: tcphdrlen 32, hdr_len 66,
>> pay_len 0, mss 1448
>> [   65.727669]  skb->len 4410, skb->data_len 4344
>> [   65.732114] stmmac_tso_xmit: curr=22 dirty=19 f=20, e=22,
>> f_p=00000000b1247b41, nfrags 1
>> [   65.740190] TX descriptor ring:
>> [   65.743327] 000 [0x82005000]: 0x0 0x0 0x0 0x0
>> [   65.747678] 001 [0x82005010]: 0x0 0x0 0x0 0x0
>> [   65.752029] 002 [0x82005020]: 0x0 0x0 0x0 0x0
>> [   65.756378] 003 [0x82005030]: 0x0 0x0 0x0 0x0
>> [   65.760727] 004 [0x82005040]: 0x0 0x0 0x0 0x0
>> [   65.765077] 005 [0x82005050]: 0x0 0x0 0x0 0x0
>> [   65.769427] 006 [0x82005060]: 0x0 0x0 0x0 0x0
>> [   65.773776] 007 [0x82005070]: 0x0 0x0 0x0 0x0
>> [   65.778126] 008 [0x82005080]: 0x0 0x0 0x0 0x0
>> [   65.782476] 009 [0x82005090]: 0x0 0x0 0x0 0x0
>> [   65.786826] 010 [0x820050a0]: 0x0 0x0 0x0 0x0
>> [   65.791176] 011 [0x820050b0]: 0x0 0x0 0x0 0x0
>> [   65.795526] 012 [0x820050c0]: 0x0 0x0 0x0 0x0
>> [   65.799875] 013 [0x820050d0]: 0x0 0x0 0x0 0x0
>> [   65.804225] 014 [0x820050e0]: 0x0 0x0 0x0 0x0
>> [   65.808575] 015 [0x820050f0]: 0x0 0x0 0x0 0x0
>> [   65.812925] 016 [0x82005100]: 0x0 0x0 0x0 0x0
>> [   65.817274] 017 [0x82005110]: 0x0 0x0 0x0 0x0
>> [   65.821625] 018 [0x82005120]: 0x0 0x0 0x0 0x0
>> [   65.825976] 019 [0x82005130]: 0xfb298000 0x0 0x21f0 0x90000000
>> [   65.831800] 020 [0x82005140]: 0xfb2a1000 0x0 0x42 0xa04410f8
>> [   65.837450] 021 [0x82005150]: 0xfb2a2000 0x0 0x10f8 0x90000000
>> [   65.843273] 022 [0x82005160]: 0x0 0x0 0x0 0x0
>> [   65.847622] 023 [0x82005170]: 0x0 0x0 0x0 0x0
>> [   65.851971] 024 [0x82005180]: 0x0 0x0 0x0 0x0
>> [   65.856319] 025 [0x82005190]: 0x0 0x0 0x0 0x0
>> [   65.860670] 026 [0x820051a0]: 0x0 0x0 0x0 0x0
>> [   65.865020] 027 [0x820051b0]: 0x0 0x0 0x0 0x0
>> [   65.869369] 028 [0x820051c0]: 0x0 0x0 0x0 0x0
>> [   65.873719] 029 [0x820051d0]: 0x0 0x0 0x0 0x0
>> [   65.878068] 030 [0x820051e0]: 0x0 0x0 0x0 0x0
>> [   65.882418] 031 [0x820051f0]: 0x0 0x0 0x0 0x0
>> [   65.886767] 032 [0x82005200]: 0x0 0x0 0x0 0x0
>> [   65.891118] 033 [0x82005210]: 0x0 0x0 0x0 0x0
>> [   65.895467] 034 [0x82005220]: 0x0 0x0 0x0 0x0
>> [   65.899816] 035 [0x82005230]: 0x0 0x0 0x0 0x0
>> [   65.904165] 036 [0x82005240]: 0x0 0x0 0x0 0x0
>> [   65.908515] 037 [0x82005250]: 0x0 0x0 0x0 0x0
>> [   65.912865] 038 [0x82005260]: 0x0 0x0 0x0 0x0
>> [   65.917215] 039 [0x82005270]: 0x0 0x0 0x0 0x0
>> [   65.921564] 040 [0x82005280]: 0x0 0x0 0x0 0x0
>> [   65.925915] 041 [0x82005290]: 0x0 0x0 0x0 0x0
>> [   65.930264] 042 [0x820052a0]: 0x0 0x0 0x0 0x0
>> [   65.934615] 043 [0x820052b0]: 0x0 0x0 0x0 0x0
>> [   65.938964] 044 [0x820052c0]: 0x0 0x0 0x0 0x0
>> [   65.943313] 045 [0x820052d0]: 0x0 0x0 0x0 0x0
>> [   65.947664] 046 [0x820052e0]: 0x0 0x0 0x0 0x0
>> [   65.952012] 047 [0x820052f0]: 0x0 0x0 0x0 0x0
>> [   65.956363] 048 [0x82005300]: 0x0 0x0 0x0 0x0
>> [   65.960712] 049 [0x82005310]: 0x0 0x0 0x0 0x0
>> [   65.965061] 050 [0x82005320]: 0x0 0x0 0x0 0x0
>> [   65.969410] 051 [0x82005330]: 0x0 0x0 0x0 0x0
>> [   65.973760] 052 [0x82005340]: 0x0 0x0 0x0 0x0
>> [   65.978110] 053 [0x82005350]: 0x0 0x0 0x0 0x0
>> [   65.982460] 054 [0x82005360]: 0x0 0x0 0x0 0x0
>> [   65.986812] 055 [0x82005370]: 0x0 0x0 0x0 0x0
>> [   65.991161] 056 [0x82005380]: 0x0 0x0 0x0 0x0
>> [   65.995510] 057 [0x82005390]: 0x0 0x0 0x0 0x0
>> [   65.999860] 058 [0x820053a0]: 0x0 0x0 0x0 0x0
>> [   66.004210] 059 [0x820053b0]: 0x0 0x0 0x0 0x0
>> [   66.008559] 060 [0x820053c0]: 0x0 0x0 0x0 0x0
>> [   66.012908] 061 [0x820053d0]: 0x0 0x0 0x0 0x0
>> [   66.017257] 062 [0x820053e0]: 0x0 0x0 0x0 0x0
>> [   66.021607] 063 [0x820053f0]: 0x0 0x0 0x0 0x0
>> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
>> [  5]   0.00-1.00   sec   184 KBytes  1.50 Mbits/sec    0  
>> 1.41 KBytes
>> [  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
>> KBytes
>> [  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
>> KBytes
>> [  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
>> KBytes
>> [  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
>> KBytes
>> [  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
>> KBytes
>> [  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
>> KBytes
>> [  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
>> KBytes
>> [  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41
>> KBytes
>>
>>
>> On 5/9/2018 3:35 PM, Jose Abreu wrote:
>>> Hi Bhadram,
>>>
>>> On 09-05-2018 08:18, Bhadram Varka wrote:
>>>> +               queue0 {
>>>> +                       snps,weight = <0x10>;
>>>> +               queue1 {
>>>> +                       snps,weight = <0x10>;
>>>>
>>>> +               queue2 {
>>>> +                       snps,weight = <0x10>;
>>>>
>>>> +               queue3 {
>>>> +                       snps,weight = <0x10>;
>>>>
>>> This is wrong. You can't use the same weight for all queues.
>>> Please try with different weights (for example: 0x10, 0x11, 0x12,
>>> 0x13).
>>>
>>> Thanks and Best Regards,
>>> Jose Miguel Abreu
>>>
>>>

Re: STMMAC driver with TSO enabled issue

[Bhadram Varka]

Hi Jose,

On 5/10/2018 7:59 PM, Jose Abreu wrote:
> Hi Bhadram,
> 
> On 10-05-2018 09:55, Jose Abreu wrote:
>> ++net-dev
>>
>> Hi Bhadram,
>>
>> On 09-05-2018 12:03, Bhadram Varka wrote:
>>> Hi,
>>>
>>> Thanks for responding.
>>>
>>> Tried below suggested way. Still observing the issue -
>> It seems stmmac has a bug in the RX side when using TSO which is
>> causing all the RX descriptors to be consumed. The stmmac_rx()
>> function will need to be refactored. I will send a fix ASAP.
> 
> Are you using this patch [1] ? Because there is a problem with
> the patch. By adding the previously removed call to
> stmmac_init_rx_desc() TSO works okay in my setup.
> 

No. I don't have this change in my code base. I am using net-next tree.

Can you please post the change for which TSO works ? I can help you with 
the testing.

-- 
Thanks,
Bhadram.

Re: STMMAC driver with TSO enabled issue

[Jose Abreu]

On 10-05-2018 16:08, Bhadram Varka wrote:
> Hi Jose,
>
> On 5/10/2018 7:59 PM, Jose Abreu wrote:
>> Hi Bhadram,
>>
>> On 10-05-2018 09:55, Jose Abreu wrote:
>>> ++net-dev
>>>
>>> Hi Bhadram,
>>>
>>> On 09-05-2018 12:03, Bhadram Varka wrote:
>>>> Hi,
>>>>
>>>> Thanks for responding.
>>>>
>>>> Tried below suggested way. Still observing the issue -
>>> It seems stmmac has a bug in the RX side when using TSO which is
>>> causing all the RX descriptors to be consumed. The stmmac_rx()
>>> function will need to be refactored. I will send a fix ASAP.
>>
>> Are you using this patch [1] ? Because there is a problem with
>> the patch. By adding the previously removed call to
>> stmmac_init_rx_desc() TSO works okay in my setup.
>>
>
> No. I don't have this change in my code base. I am using
> net-next tree.
>
> Can you please post the change for which TSO works ? I can help
> you with the testing.

It should work with net-next because patch was not merged yet ...
Please send me the output of "dmesg | grep -i stmmac", since boot
and your full register values (from 0x0 to 0x12E4).

Re: STMMAC driver with TSO enabled issue

[Bhadram Varka]

Hi Jose,

On 5/10/2018 9:15 PM, Jose Abreu wrote:
> 
> 
> On 10-05-2018 16:08, Bhadram Varka wrote:
>> Hi Jose,
>>
>> On 5/10/2018 7:59 PM, Jose Abreu wrote:
>>> Hi Bhadram,
>>>
>>> On 10-05-2018 09:55, Jose Abreu wrote:
>>>> ++net-dev
>>>>
>>>> Hi Bhadram,
>>>>
>>>> On 09-05-2018 12:03, Bhadram Varka wrote:
>>>>> Hi,
>>>>>
>>>>> Thanks for responding.
>>>>>
>>>>> Tried below suggested way. Still observing the issue -
>>>> It seems stmmac has a bug in the RX side when using TSO which is
>>>> causing all the RX descriptors to be consumed. The stmmac_rx()
>>>> function will need to be refactored. I will send a fix ASAP.
>>>
>>> Are you using this patch [1] ? Because there is a problem with
>>> the patch. By adding the previously removed call to
>>> stmmac_init_rx_desc() TSO works okay in my setup.
>>>
>>
>> No. I don't have this change in my code base. I am using
>> net-next tree.
>>
>> Can you please post the change for which TSO works ? I can help
>> you with the testing.
> 
> It should work with net-next because patch was not merged yet ...
> Please send me the output of "dmesg | grep -i stmmac", since boot
> and your full register values (from 0x0 to 0x12E4).
> 

[root@alarm ~]# dmesg | grep -i dwc
[    6.925005] dwc-eth-dwmac 2490000.ethernet: Cannot get CSR clock
[    6.933657] dwc-eth-dwmac 2490000.ethernet: no reset control found
[    6.955325] dwc-eth-dwmac 2490000.ethernet: User ID: 0x10, Synopsys 
ID: 0x41
[    6.962379] dwc-eth-dwmac 2490000.ethernet:  DWMAC4/5
[    6.967434] dwc-eth-dwmac 2490000.ethernet: DMA HW capability 
register supported
[    6.974827] dwc-eth-dwmac 2490000.ethernet: RX Checksum Offload 
Engine supported
[    6.982915] dwc-eth-dwmac 2490000.ethernet: TX Checksum insertion 
supported
[    6.991235] dwc-eth-dwmac 2490000.ethernet: Wake-Up On Lan supported
[    6.998974] dwc-eth-dwmac 2490000.ethernet: TSO supported
[    7.006422] dwc-eth-dwmac 2490000.ethernet: TSO feature enabled
[    7.012581] dwc-eth-dwmac 2490000.ethernet: Enable RX Mitigation via 
HW Watchdog Timer
[    7.236391] dwc-eth-dwmac 2490000.ethernet eth0: device MAC address 
4a:d1:e3:58:cb:7a
[    7.333414] dwc-eth-dwmac 2490000.ethernet eth0: IEEE 1588-2008 
Advanced Timestamp supported
[    7.342441] dwc-eth-dwmac 2490000.ethernet eth0: registered PTP clock
[   10.157066] dwc-eth-dwmac 2490000.ethernet eth0: Link is Up - 
1Gbps/Full - flow control off
[root@alarm ~]# dmesg | grep -i stmma
[    7.020567] libphy: stmmac: probed
[    7.316295] Broadcom BCM89610 stmmac-0:00: attached PHY driver 
[Broadcom BCM89610] (mii_bus:phy_addr=stmmac-0:00, irq=64)

I will get the register details -

FYI - TSO works fine with single channel. I see the issue only if multi 
channel enabled (supports 4 Tx/Rx channels).

-- 
Thanks,
Bhadram.

Re: STMMAC driver with TSO enabled issue

[Jose Abreu]

Hi Bhadram,

On 15-05-2018 07:44, Bhadram Varka wrote:
> Hi Jose,
>
> On 5/10/2018 9:15 PM, Jose Abreu wrote:
>>
>>
>> On 10-05-2018 16:08, Bhadram Varka wrote:
>>> Hi Jose,
>>>
>>> On 5/10/2018 7:59 PM, Jose Abreu wrote:
>>>> Hi Bhadram,
>>>>
>>>> On 10-05-2018 09:55, Jose Abreu wrote:
>>>>> ++net-dev
>>>>>
>>>>> Hi Bhadram,
>>>>>
>>>>> On 09-05-2018 12:03, Bhadram Varka wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Thanks for responding.
>>>>>>
>>>>>> Tried below suggested way. Still observing the issue -
>>>>> It seems stmmac has a bug in the RX side when using TSO
>>>>> which is
>>>>> causing all the RX descriptors to be consumed. The stmmac_rx()
>>>>> function will need to be refactored. I will send a fix ASAP.
>>>>
>>>> Are you using this patch [1] ? Because there is a problem with
>>>> the patch. By adding the previously removed call to
>>>> stmmac_init_rx_desc() TSO works okay in my setup.
>>>>
>>>
>>> No. I don't have this change in my code base. I am using
>>> net-next tree.
>>>
>>> Can you please post the change for which TSO works ? I can help
>>> you with the testing.
>>
>> It should work with net-next because patch was not merged yet ...
>> Please send me the output of "dmesg | grep -i stmmac", since boot
>> and your full register values (from 0x0 to 0x12E4).
>>
>
> [root@alarm ~]# dmesg | grep -i dwc
> [    6.925005] dwc-eth-dwmac 2490000.ethernet: Cannot get CSR
> clock
> [    6.933657] dwc-eth-dwmac 2490000.ethernet: no reset control
> found
> [    6.955325] dwc-eth-dwmac 2490000.ethernet: User ID: 0x10,
> Synopsys ID: 0x41
> [    6.962379] dwc-eth-dwmac 2490000.ethernet:  DWMAC4/5
> [    6.967434] dwc-eth-dwmac 2490000.ethernet: DMA HW
> capability register supported
> [    6.974827] dwc-eth-dwmac 2490000.ethernet: RX Checksum
> Offload Engine supported
> [    6.982915] dwc-eth-dwmac 2490000.ethernet: TX Checksum
> insertion supported
> [    6.991235] dwc-eth-dwmac 2490000.ethernet: Wake-Up On Lan
> supported
> [    6.998974] dwc-eth-dwmac 2490000.ethernet: TSO supported
> [    7.006422] dwc-eth-dwmac 2490000.ethernet: TSO feature enabled
> [    7.012581] dwc-eth-dwmac 2490000.ethernet: Enable RX
> Mitigation via HW Watchdog Timer
> [    7.236391] dwc-eth-dwmac 2490000.ethernet eth0: device MAC
> address 4a:d1:e3:58:cb:7a
> [    7.333414] dwc-eth-dwmac 2490000.ethernet eth0: IEEE
> 1588-2008 Advanced Timestamp supported
> [    7.342441] dwc-eth-dwmac 2490000.ethernet eth0: registered
> PTP clock
> [   10.157066] dwc-eth-dwmac 2490000.ethernet eth0: Link is Up
> - 1Gbps/Full - flow control off
> [root@alarm ~]# dmesg | grep -i stmma
> [    7.020567] libphy: stmmac: probed
> [    7.316295] Broadcom BCM89610 stmmac-0:00: attached PHY
> driver [Broadcom BCM89610] (mii_bus:phy_addr=stmmac-0:00, irq=64)
>
> I will get the register details -
>
> FYI - TSO works fine with single channel. I see the issue only
> if multi channel enabled (supports 4 Tx/Rx channels).
>

And normal data transfer works okay with multi channel, right? I
will need the register details to proceed ... You could also try
git bisect ...

Thanks and Best Regards,
Jose Miguel Abreu

Re: STMMAC driver with TSO enabled issue

[Bhadram Varka]

Hi Jose,

On 5/17/2018 7:43 PM, Jose Abreu wrote:
> Hi Bhadram,
> 
> On 15-05-2018 07:44, Bhadram Varka wrote:
>> Hi Jose,
>>
>> On 5/10/2018 9:15 PM, Jose Abreu wrote:
>>>
>>>
>>> On 10-05-2018 16:08, Bhadram Varka wrote:
>>>> Hi Jose,
>>>>
>>>> On 5/10/2018 7:59 PM, Jose Abreu wrote:
>>>>> Hi Bhadram,
>>>>>
>>>>> On 10-05-2018 09:55, Jose Abreu wrote:
>>>>>> ++net-dev
>>>>>>
>>>>>> Hi Bhadram,
>>>>>>
>>>>>> On 09-05-2018 12:03, Bhadram Varka wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks for responding.
>>>>>>>
>>>>>>> Tried below suggested way. Still observing the issue -
>>>>>> It seems stmmac has a bug in the RX side when using TSO
>>>>>> which is
>>>>>> causing all the RX descriptors to be consumed. The stmmac_rx()
>>>>>> function will need to be refactored. I will send a fix ASAP.
>>>>>
>>>>> Are you using this patch [1] ? Because there is a problem with
>>>>> the patch. By adding the previously removed call to
>>>>> stmmac_init_rx_desc() TSO works okay in my setup.
>>>>>
>>>>
>>>> No. I don't have this change in my code base. I am using
>>>> net-next tree.
>>>>
>>>> Can you please post the change for which TSO works ? I can help
>>>> you with the testing.
>>>
>>> It should work with net-next because patch was not merged yet ...
>>> Please send me the output of "dmesg | grep -i stmmac", since boot
>>> and your full register values (from 0x0 to 0x12E4).
>>>
>>
>> [root@alarm ~]# dmesg | grep -i dwc
>> [    6.925005] dwc-eth-dwmac 2490000.ethernet: Cannot get CSR
>> clock
>> [    6.933657] dwc-eth-dwmac 2490000.ethernet: no reset control
>> found
>> [    6.955325] dwc-eth-dwmac 2490000.ethernet: User ID: 0x10,
>> Synopsys ID: 0x41
>> [    6.962379] dwc-eth-dwmac 2490000.ethernet:  DWMAC4/5
>> [    6.967434] dwc-eth-dwmac 2490000.ethernet: DMA HW
>> capability register supported
>> [    6.974827] dwc-eth-dwmac 2490000.ethernet: RX Checksum
>> Offload Engine supported
>> [    6.982915] dwc-eth-dwmac 2490000.ethernet: TX Checksum
>> insertion supported
>> [    6.991235] dwc-eth-dwmac 2490000.ethernet: Wake-Up On Lan
>> supported
>> [    6.998974] dwc-eth-dwmac 2490000.ethernet: TSO supported
>> [    7.006422] dwc-eth-dwmac 2490000.ethernet: TSO feature enabled
>> [    7.012581] dwc-eth-dwmac 2490000.ethernet: Enable RX
>> Mitigation via HW Watchdog Timer
>> [    7.236391] dwc-eth-dwmac 2490000.ethernet eth0: device MAC
>> address 4a:d1:e3:58:cb:7a
>> [    7.333414] dwc-eth-dwmac 2490000.ethernet eth0: IEEE
>> 1588-2008 Advanced Timestamp supported
>> [    7.342441] dwc-eth-dwmac 2490000.ethernet eth0: registered
>> PTP clock
>> [   10.157066] dwc-eth-dwmac 2490000.ethernet eth0: Link is Up
>> - 1Gbps/Full - flow control off
>> [root@alarm ~]# dmesg | grep -i stmma
>> [    7.020567] libphy: stmmac: probed
>> [    7.316295] Broadcom BCM89610 stmmac-0:00: attached PHY
>> driver [Broadcom BCM89610] (mii_bus:phy_addr=stmmac-0:00, irq=64)
>>
>> I will get the register details -
>>
>> FYI - TSO works fine with single channel. I see the issue only
>> if multi channel enabled (supports 4 Tx/Rx channels).
>>
> 
> And normal data transfer works okay with multi channel, right? I
> will need the register details to proceed ... You could also try
> git bisect ...
> 

Yes - normal data transfers works fine. Issue observed only driver gets 
TSO packet. Looks like TX DMA channel hang.

After adding few debug logs - observed that while processing second or 
third descriptor TX DMA hangs.

[85788.137498] stmmac_tso_xmit: tcphdrlen 32, hdr_len 66, pay_len 1392, 
mss 1448
[85788.144634]  skb->len 7306, skb->data_len 5848
[..]
[85788.274876] 025 [0x82795190]: 0x0 0x0 0x5a8 0xc4000000
[85788.280020] 026 [0x827951a0]: 0xf854e000 0xf854e042 0x5700042 0xa0441c48
[85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8 0x90000000
[...]


After some time if check Tx descriptor status - then I see only below

[..]
[85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8 0x90000000

index 025 and 026 descriptors processed but not index 027.

At this stage Tx DMA is always in below state -

■ 3'b011: Running (Reading Data from system memory
buffer and queuing it to the Tx buffer (Tx FIFO))

Thanks,
Bhadram.

Re: STMMAC driver with TSO enabled issue

[Jose Abreu]

Hi Bhadram,

On 24-05-2018 06:58, Bhadram Varka wrote:
>
> After some time if check Tx descriptor status - then I see only
> below
>
> [..]
> [85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8 0x90000000
>
> index 025 and 026 descriptors processed but not index 027.
>
> At this stage Tx DMA is always in below state -
>
> ■ 3'b011: Running (Reading Data from system memory
> buffer and queuing it to the Tx buffer (Tx FIFO))

Thats strange, I think the descriptors look okay though. I will
need the registers values (before the lock) and, if possible, the
git bisect output.

Thanks and Best Regards,
Jose Miguel Abreu

>
> Thanks,
> Bhadram.

Re: STMMAC driver with TSO enabled issue

[Bhadram Varka]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

Hi Jose,

On 5/24/2018 3:01 PM, Jose Abreu wrote:
> Hi Bhadram,
> 
> On 24-05-2018 06:58, Bhadram Varka wrote:
>>
>> After some time if check Tx descriptor status - then I see only
>> below
>>
>> [..]
>> [85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8 0x90000000
>>
>> index 025 and 026 descriptors processed but not index 027.
>>
>> At this stage Tx DMA is always in below state -
>>
>> ■ 3'b011: Running (Reading Data from system memory
>> buffer and queuing it to the Tx buffer (Tx FIFO))
> 
> Thats strange, I think the descriptors look okay though. I will
> need the registers values (before the lock) and, if possible, the
> git bisect output.

Attaching the register dump file after the issue observed. Please check 
once.

-- 
Thanks,
Bhadram.

[-- Attachment #2: regdump.txt --]
[-- Type: text/plain, Size: 2966 bytes --]

0x0  = 0x08062203
0x4  = 0x00000000
0x8  = 0x00000004
0xc  = 0x00000000
0x10 = 0x00004002
0x14 = 0x00020001
0x18 = 0x00000000
0x1c = 0x00000000
0x20 = 0x00000000
0x24 = 0x00000000
0x28 = 0x00000000
0x2c = 0x00000000
0x50 = 0x00000000
0x54 = 0x00000000
0x58 = 0x00000000
0x60 = 0x00000000
0x64 = 0x00000000
0x70 = 0x00000000
0x74 = 0x00000000
0x78 = 0x00000000
0x7c = 0x00000000
0x90 = 0x00000000
0x94 = 0x00000000
0x98 = 0x00000000
0x9c = 0x00000000
0xa0 = 0x000000AA
0xa4 = 0x00000000
0xa8 = 0x03020100
0xac = 0x00000000
0xb0 = 0x00000000
0xb4 = 0x00000030
0xb8 = 0x00000000
0xc0 = 0x00000000
0xc4 = 0x00000000
0xd0 = 0x00000000
0xd4 = 0x03E80000
0xd8 = 0x00000000
0xdc = 0x00000063
0xe0 = 0x00000000
0xe4 = 0x00000000
0xe8 = 0x00000000
0xec = 0x00000000
0xf0 = 0x00000000
0xf4 = 0x00000000
0xf8 = 0x00000000
0x110 = 0x00001041
0x114 = 0x00000000
0x11c = 0x1BFD73F7
0x120 = 0x429E79C7
0x124 = 0x100C30C3
0x128 = 0x00000000
0x140 = 0x00000000
0x144 = 0x00000000
0x148 = 0x00000000
0x14c = 0x00000000
0x150 = 0x00000000
0x200 = 0x00100104
0x204 = 0x00000000
0x208 = 0x00000000
0x20c = 0x00000000
0x210 = 0x00000000
0x230 = 0x00000000
0x234 = 0x00000000
0x238 = 0x00000000
0x240 = 0x00000000
0x244 = 0x00000000
0x300 = 0x80005CE1
0x304 = 0xCAA296FE
0xc00 = 0x00000000
0xc08 = 0x00000000
0xc0c = 0x00800018
0xc10 = 0x00000000
0xc20 = 0x00000000
0xc30 = 0x02020100
0xc34 = 0x00000000
0xd00 = 0x000F000A
0xd04 = 0x00000000
0xd08 = 0x00000000
0xd0c = 0x00000000
0xd14 = 0x00000000
0xd18 = 0x00000010
0xd2c = 0x01000000
0xd30 = 0x00F0C1A0
0xd34 = 0x00000000
0xd38 = 0x00000000
0xd3c = 0x00000000
0xd40 = 0x000F000A
0xd80 = 0x000F000A
0xdc0 = 0x000F000A
0xd44 = 0x00000000
0xd84 = 0x00000000
0xdc4 = 0x00000000
0xd48 = 0x00000000
0xd88 = 0x00000000
0xdc8 = 0x00000000
0x1000 = 0x00000000
0x1004 = 0x0002100E
0x1008 = 0x00000000
0x100c = 0x33636300
0x1010 = 0x00000063
0x1014 = 0x00000000
0x1020 = 0x00000000
0x1024 = 0x00000000
0x1028 = 0x00000000
0x1100 = 0x00010000
0x1180 = 0x00010000
0x1200 = 0x00010000
0x1280 = 0x00010000
0x1104 = 0x00201001
0x1184 = 0x00201001
0x1204 = 0x00201001
0x1284 = 0x00201001
0x1108 = 0x00080001
0x1188 = 0x00080001
0x1208 = 0x00080001
0x1288 = 0x00080001
0x1110 = 0x00000000
0x1190 = 0x00000000
0x1210 = 0x00000000
0x1290 = 0x00000000
0x1114 = 0xFC044000
0x1194 = 0xFC045000
0x1214 = 0xFC046000
0x1294 = 0xFC047000
0x1118 = 0x00000000
0x1198 = 0x00000000
0x1218 = 0x00000000
0x1298 = 0x00000000
0x111C = 0xFC040000
0x119c = 0xFC041000
0x121c = 0xFC042000
0x129c = 0xFC043000
0x1120 = 0xFC044400
0x11A0 = 0xFC045400
0x1220 = 0xFC046400
0x12A0 = 0xFC047400
0x1128 = 0xFC040400
0x11A8 = 0xFC041400
0x1228 = 0xFC042400
0x12A8 = 0xFC043400
0x112c = 0x0000003F
0x11ac = 0x0000003F
0x122c = 0x0000003F
0x12ac = 0x0000003F
0x1130 = 0x0000003F
0x11b0 = 0x0000003F
0x1230 = 0x0000003F
0x12b0 = 0x0000003F

Re: STMMAC driver with TSO enabled issue

[Jose Abreu]

Hi Bhadram,

On 25-05-2018 05:41, Bhadram Varka wrote:
> Hi Jose,
>
> On 5/24/2018 3:01 PM, Jose Abreu wrote:
>> Hi Bhadram,
>>
>> On 24-05-2018 06:58, Bhadram Varka wrote:
>>>
>>> After some time if check Tx descriptor status - then I see only
>>> below
>>>
>>> [..]
>>> [85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8
>>> 0x90000000
>>>
>>> index 025 and 026 descriptors processed but not index 027.
>>>
>>> At this stage Tx DMA is always in below state -
>>>
>>> ■ 3'b011: Running (Reading Data from system memory
>>> buffer and queuing it to the Tx buffer (Tx FIFO))
>>
>> Thats strange, I think the descriptors look okay though. I will
>> need the registers values (before the lock) and, if possible, the
>> git bisect output.
>
> Attaching the register dump file after the issue observed.
> Please check once.
>

----->8-----
0x112c = 0x0000003F
0x11ac = 0x0000003F
0x122c = 0x0000003F
0x12ac = 0x0000003F

0x1130 = 0x0000003F
0x11b0 = 0x0000003F
0x1230 = 0x0000003F
0x12b0 = 0x0000003F
----->8-----

This can't be right, it should be DMA_{RX/TX}_SIZE - 1 = 511. Did
you change these values in the code?

Thanks and Best Regards,
Jose Miguel Abreu

Re: STMMAC driver with TSO enabled issue

[Bhadram Varka]

Hi Jose,

On 5/25/2018 7:35 PM, Jose Abreu wrote:
> Hi Bhadram,
> 
> On 25-05-2018 05:41, Bhadram Varka wrote:
>> Hi Jose,
>>
>> On 5/24/2018 3:01 PM, Jose Abreu wrote:
>>> Hi Bhadram,
>>>
>>> On 24-05-2018 06:58, Bhadram Varka wrote:
>>>>
>>>> After some time if check Tx descriptor status - then I see only
>>>> below
>>>>
>>>> [..]
>>>> [85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8
>>>> 0x90000000
>>>>
>>>> index 025 and 026 descriptors processed but not index 027.
>>>>
>>>> At this stage Tx DMA is always in below state -
>>>>
>>>> ■ 3'b011: Running (Reading Data from system memory
>>>> buffer and queuing it to the Tx buffer (Tx FIFO))
>>>
>>> Thats strange, I think the descriptors look okay though. I will
>>> need the registers values (before the lock) and, if possible, the
>>> git bisect output.
>>
>> Attaching the register dump file after the issue observed.
>> Please check once.
>>
> 
> ----->8-----
> 0x112c = 0x0000003F
> 0x11ac = 0x0000003F
> 0x122c = 0x0000003F
> 0x12ac = 0x0000003F
> 
> 0x1130 = 0x0000003F
> 0x11b0 = 0x0000003F
> 0x1230 = 0x0000003F
> 0x12b0 = 0x0000003F
> ----->8-----
> 
> This can't be right, it should be DMA_{RX/TX}_SIZE - 1 = 511. Did
> you change these values in the code?
> 

Yes. I have changed the descriptor length to 64 - so that searching for 
the current descriptor status would be easy.

-- 
Thanks,
Bhadram.

Re: STMMAC driver with TSO enabled issue

[Jose Abreu]

On 25-05-2018 15:25, Bhadram Varka wrote:
> Hi Jose,
>
> On 5/25/2018 7:35 PM, Jose Abreu wrote:
>> Hi Bhadram,
>>
>> On 25-05-2018 05:41, Bhadram Varka wrote:
>>> Hi Jose,
>>>
>>> On 5/24/2018 3:01 PM, Jose Abreu wrote:
>>>> Hi Bhadram,
>>>>
>>>> On 24-05-2018 06:58, Bhadram Varka wrote:
>>>>>
>>>>> After some time if check Tx descriptor status - then I see
>>>>> only
>>>>> below
>>>>>
>>>>> [..]
>>>>> [85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8
>>>>> 0x90000000
>>>>>
>>>>> index 025 and 026 descriptors processed but not index 027.
>>>>>
>>>>> At this stage Tx DMA is always in below state -
>>>>>
>>>>> ■ 3'b011: Running (Reading Data from system memory
>>>>> buffer and queuing it to the Tx buffer (Tx FIFO))
>>>>
>>>> Thats strange, I think the descriptors look okay though. I will
>>>> need the registers values (before the lock) and, if
>>>> possible, the
>>>> git bisect output.
>>>
>>> Attaching the register dump file after the issue observed.
>>> Please check once.
>>>
>>
>> ----->8-----
>> 0x112c = 0x0000003F
>> 0x11ac = 0x0000003F
>> 0x122c = 0x0000003F
>> 0x12ac = 0x0000003F
>>
>> 0x1130 = 0x0000003F
>> 0x11b0 = 0x0000003F
>> 0x1230 = 0x0000003F
>> 0x12b0 = 0x0000003F
>> ----->8-----
>>
>> This can't be right, it should be DMA_{RX/TX}_SIZE - 1 = 511. Did
>> you change these values in the code?
>>
>
> Yes. I have changed the descriptor length to 64 - so that
> searching for the current descriptor status would be easy.

Ok, it shouldn't impact anything. The only thing I'm remembering
now is that you can have TSO not enabled in all DMA channels (HW
configuration allows this). Please check if TSO in single-queue
works.

Thanks and Best Regards,
Jose Miguel Abreu

Re: STMMAC driver with TSO enabled issue

[Bhadram Varka]

Hi Jose,

On 5/25/2018 8:02 PM, Jose Abreu wrote:
> On 25-05-2018 15:25, Bhadram Varka wrote:
>> Hi Jose,
>>
>> On 5/25/2018 7:35 PM, Jose Abreu wrote:
>>> Hi Bhadram,
>>>
>>> On 25-05-2018 05:41, Bhadram Varka wrote:
>>>> Hi Jose,
>>>>
>>>> On 5/24/2018 3:01 PM, Jose Abreu wrote:
>>>>> Hi Bhadram,
>>>>>
>>>>> On 24-05-2018 06:58, Bhadram Varka wrote:
>>>>>>
>>>>>> After some time if check Tx descriptor status - then I see
>>>>>> only
>>>>>> below
>>>>>>
>>>>>> [..]
>>>>>> [85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8
>>>>>> 0x90000000
>>>>>>
>>>>>> index 025 and 026 descriptors processed but not index 027.
>>>>>>
>>>>>> At this stage Tx DMA is always in below state -
>>>>>>
>>>>>> ■ 3'b011: Running (Reading Data from system memory
>>>>>> buffer and queuing it to the Tx buffer (Tx FIFO))
>>>>>
>>>>> Thats strange, I think the descriptors look okay though. I will
>>>>> need the registers values (before the lock) and, if
>>>>> possible, the
>>>>> git bisect output.
>>>>
>>>> Attaching the register dump file after the issue observed.
>>>> Please check once.
>>>>
>>>
>>> ----->8-----
>>> 0x112c = 0x0000003F
>>> 0x11ac = 0x0000003F
>>> 0x122c = 0x0000003F
>>> 0x12ac = 0x0000003F
>>>
>>> 0x1130 = 0x0000003F
>>> 0x11b0 = 0x0000003F
>>> 0x1230 = 0x0000003F
>>> 0x12b0 = 0x0000003F
>>> ----->8-----
>>>
>>> This can't be right, it should be DMA_{RX/TX}_SIZE - 1 = 511. Did
>>> you change these values in the code?
>>>
>>
>> Yes. I have changed the descriptor length to 64 - so that
>> searching for the current descriptor status would be easy.
> 
> Ok, it shouldn't impact anything. The only thing I'm remembering
> now is that you can have TSO not enabled in all DMA channels (HW
> configuration allows this). Please check if TSO in single-queue
> works.
TSO works fine if only single queue enabled. I don't see any limitation 
from HW side because TSO works fine with other driver which we received 
from Synopsys with IP drop.

Thanks,
Bhadram.

Re: STMMAC driver with TSO enabled issue

[Jose Abreu]

Hi Bhadram,

On 28-05-2018 10:15, Bhadram Varka wrote:
> Hi Jose,
>
> On 5/25/2018 8:02 PM, Jose Abreu wrote:
>> On 25-05-2018 15:25, Bhadram Varka wrote:
>>> Hi Jose,
>>>
>>> On 5/25/2018 7:35 PM, Jose Abreu wrote:
>>>> Hi Bhadram,
>>>>
>>>> On 25-05-2018 05:41, Bhadram Varka wrote:
>>>>> Hi Jose,
>>>>>
>>>>> On 5/24/2018 3:01 PM, Jose Abreu wrote:
>>>>>> Hi Bhadram,
>>>>>>
>>>>>> On 24-05-2018 06:58, Bhadram Varka wrote:
>>>>>>>
>>>>>>> After some time if check Tx descriptor status - then I see
>>>>>>> only
>>>>>>> below
>>>>>>>
>>>>>>> [..]
>>>>>>> [85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8
>>>>>>> 0x90000000
>>>>>>>
>>>>>>> index 025 and 026 descriptors processed but not index 027.
>>>>>>>
>>>>>>> At this stage Tx DMA is always in below state -
>>>>>>>
>>>>>>> ■ 3'b011: Running (Reading Data from system memory
>>>>>>> buffer and queuing it to the Tx buffer (Tx FIFO))
>>>>>>
>>>>>> Thats strange, I think the descriptors look okay though. I
>>>>>> will
>>>>>> need the registers values (before the lock) and, if
>>>>>> possible, the
>>>>>> git bisect output.
>>>>>
>>>>> Attaching the register dump file after the issue observed.
>>>>> Please check once.
>>>>>
>>>>
>>>> ----->8-----
>>>> 0x112c = 0x0000003F
>>>> 0x11ac = 0x0000003F
>>>> 0x122c = 0x0000003F
>>>> 0x12ac = 0x0000003F
>>>>
>>>> 0x1130 = 0x0000003F
>>>> 0x11b0 = 0x0000003F
>>>> 0x1230 = 0x0000003F
>>>> 0x12b0 = 0x0000003F
>>>> ----->8-----
>>>>
>>>> This can't be right, it should be DMA_{RX/TX}_SIZE - 1 =
>>>> 511. Did
>>>> you change these values in the code?
>>>>
>>>
>>> Yes. I have changed the descriptor length to 64 - so that
>>> searching for the current descriptor status would be easy.
>>
>> Ok, it shouldn't impact anything. The only thing I'm remembering
>> now is that you can have TSO not enabled in all DMA channels (HW
>> configuration allows this). Please check if TSO in single-queue
>> works.
> TSO works fine if only single queue enabled. I don't see any
> limitation from HW side because TSO works fine with other
> driver which we received from Synopsys with IP drop.

You need to check with HW team if TSO is enabled for all channels
because you can have TSO channels < DMA channels and there is no
way to confirm this in the registers. Also check if received
driver is routing packets to queue != 0.

Thanks and Best Regards,
Jose Miguel Abreu

>
>
> Thanks,
> Bhadram.

Re: STMMAC driver with TSO enabled issue

[Bhadram Varka]

Hi Jose,

On 5/28/2018 4:26 PM, Jose Abreu wrote:
> Hi Bhadram,
> 
> On 28-05-2018 10:15, Bhadram Varka wrote:
>> Hi Jose,
>>
>> On 5/25/2018 8:02 PM, Jose Abreu wrote:
>>> On 25-05-2018 15:25, Bhadram Varka wrote:
>>>> Hi Jose,
>>>>
>>>> On 5/25/2018 7:35 PM, Jose Abreu wrote:
>>>>> Hi Bhadram,
>>>>>
>>>>> On 25-05-2018 05:41, Bhadram Varka wrote:
>>>>>> Hi Jose,
>>>>>>
>>>>>> On 5/24/2018 3:01 PM, Jose Abreu wrote:
>>>>>>> Hi Bhadram,
>>>>>>>
>>>>>>> On 24-05-2018 06:58, Bhadram Varka wrote:
>>>>>>>>
>>>>>>>> After some time if check Tx descriptor status - then I see
>>>>>>>> only
>>>>>>>> below
>>>>>>>>
>>>>>>>> [..]
>>>>>>>> [85788.286730] 027 [0x827951b0]: 0xf854f000 0x0 0x16d8
>>>>>>>> 0x90000000
>>>>>>>>
>>>>>>>> index 025 and 026 descriptors processed but not index 027.
>>>>>>>>
>>>>>>>> At this stage Tx DMA is always in below state -
>>>>>>>>
>>>>>>>> ■ 3'b011: Running (Reading Data from system memory
>>>>>>>> buffer and queuing it to the Tx buffer (Tx FIFO))
>>>>>>>
>>>>>>> Thats strange, I think the descriptors look okay though. I
>>>>>>> will
>>>>>>> need the registers values (before the lock) and, if
>>>>>>> possible, the
>>>>>>> git bisect output.
>>>>>>
>>>>>> Attaching the register dump file after the issue observed.
>>>>>> Please check once.
>>>>>>
>>>>>
>>>>> ----->8-----
>>>>> 0x112c = 0x0000003F
>>>>> 0x11ac = 0x0000003F
>>>>> 0x122c = 0x0000003F
>>>>> 0x12ac = 0x0000003F
>>>>>
>>>>> 0x1130 = 0x0000003F
>>>>> 0x11b0 = 0x0000003F
>>>>> 0x1230 = 0x0000003F
>>>>> 0x12b0 = 0x0000003F
>>>>> ----->8-----
>>>>>
>>>>> This can't be right, it should be DMA_{RX/TX}_SIZE - 1 =
>>>>> 511. Did
>>>>> you change these values in the code?
>>>>>
>>>>
>>>> Yes. I have changed the descriptor length to 64 - so that
>>>> searching for the current descriptor status would be easy.
>>>
>>> Ok, it shouldn't impact anything. The only thing I'm remembering
>>> now is that you can have TSO not enabled in all DMA channels (HW
>>> configuration allows this). Please check if TSO in single-queue
>>> works.
>> TSO works fine if only single queue enabled. I don't see any
>> limitation from HW side because TSO works fine with other
>> driver which we received from Synopsys with IP drop.
> 
> You need to check with HW team if TSO is enabled for all channels
> because you can have TSO channels < DMA channels and there is no
> way to confirm this in the registers. Also check if received
> driver is routing packets to queue != 0.
> 

Root caused the issue to TxPBL settings. In current configuration driver 
using the TxPBL = 32 which will be fine for single channel but its not 
recommended settings for multi-queue scenario. Recommended setting for 
TxPBL is half of the queue size.

o Total MTL Tx queue size - 16KB
o For multi-queue - total size divided by number of queues -
(16KB/4) = 4KB for each queue.
o So we need to set the TxPBL value so that we can place memory request 
for 2KB from system memory. For this to achieve we need to set TxPBL=16.

Thanks for the help in debugging.

-- 
Thanks,
Bhadram.