Re: [SPDK] Content for v18.10.x

[Sasha Kotchubievsky <sashakot at dev.mellanox.co.il>]

[-- Attachment #1: Type: text/plain, Size: 11339 bytes --]


Do you see the same pattern in both branches in "master" and in "18.10.1"?

>> Seth found a similar failure on the master branch, but it was way back on
>> November 5th. That's more than a month ago, and with the number of patches
>> that
>> are run daily it sure seems like the problem must have been fixed there. I did
>> a
>> diff of lib/nvmf/rdma.c on 18.10.1 and master and there doesn't seem to be a
>> huge number of differences. There is some improved SGL handling on master, but
>> this test is using the Linux kernel initiator which never sends more than 1
>> SGL
>> element.
>>
>> Still investigating.
> This is still an open issue, but Seth did figure out why we don't see the
> problem on master - the test changed. The plan right now is to move forward with
> 18.10.1 without addressing this problem. 18.10.1 is far better than 18.10 in
> this area already, and because this problem likely exists on master in the same
> form (and may not even ultimately be an SPDK problem), we'd rather not wait to
> get 18.10.1 out any longer.

Dose it mean, the patch fixing "ref count" will be merged into 18.10.1?


>
> Any assistance with investigating the root cause of the issue on the master
> branch would be greatly appreciated. It replicates using the 18.10 version of
> the test/nvmf/lvol/nvmf_lvol.sh test, but not the version on master. The problem
> appears to be that the connection just drops, and both the initiator and the
> target reactor to that correctly. But why the connection is dropping (it's
> loopback) is the unknown.

Which SW stack is used in the testing: SoftROCE/MOFED/OFED? OS?

As I understand, for reproduction, I run test/nvmf/lvol/nvmf_lvol.sh  ( 
from 18.10 ) on source coming from "master". Is that correct?



>
>>> Best regards
>>>
>>> Sasha
>>>
>>> On 12/17/2018 5:36 PM, Harris, James R wrote:
>>>> Hi Sasha,
>>>>
>>>> 437348 failed the first time in the Chandler test pool.  It had a
>>>> signature
>>>> that neither Ben nor me had seen before, and we were concerned that fixing
>>>> the refcnt was exposing some other issue.  I personally haven't looked
>>>> through all of the test failures on master since that patch was
>>>> merged.  It's possible there are other fixes on master that aren't in
>>>> 18.10.x yet.  So for now, I'm leaning against merging the refcnt patch for
>>>> 18.10.1.
>>>>
>>>> -Jim
>>>>
>>>>
>>>> On 12/16/18, 4:31 AM, "SPDK on behalf of Sasha Kotchubievsky" <
>>>> spdk-bounces(a)lists.01.org on behalf of sashakot(a)dev.mellanox.co.il> wrote:
>>>>
>>>>       On 12/14/2018 6:21 PM, Harris, James R wrote:
>>>>       >
>>>>       > On 12/13/18, 12:46 AM, "SPDK on behalf of Sasha Kotchubievsky" <
>>>> spdk-bounces(a)lists.01.org on behalf of sashakot(a)dev.mellanox.co.il> wrote:
>>>>       >
>>>>       >      Hi,
>>>>       >
>>>>       >      Thanks, for information.
>>>>       >
>>>>       >      Patch https://review.gerrithub.io/c/spdk/spdk/+/430802 is
>>>> merged
>>>> today
>>>>       >      to "master". It will be nice, if you get it to 18.10.1 too.
>>>>       >
>>>>       >      Regarding to:
>>>>       >
>>>>       >      https://review.gerrithub.io/c/spdk/spdk/+/436974
>>>>       >
>>>>       >      https://review.gerrithub.io/c/spdk/spdk/+/436887
>>>>       >
>>>>       > Hi Sasha,
>>>>       >
>>>>       > The two patches mentioned have been merged to master and are in the
>>>> build queue for 18.10.1 (URLs below):
>>>>       > https://review.gerrithub.io/#/c/spdk/spdk/+/437348/
>>>>       > https://review.gerrithub.io/#/c/spdk/spdk/+/437349/
>>>>       
>>>>       Hi Jim,
>>>>       
>>>>       Patch https://review.gerrithub.io/c/spdk/spdk/+/437348/ is abandoned
>>>>       from 18.10.1
>>>>       
>>>>       The patch fixes ref counters issue introduced by
>>>>       
>>>>
> https://github.com/spdk/spdk/commit/567b1006a9d4533e5ae03d78b11040f5da0fe278#diff-28f9c2af26bafb4a7268c9aa7b9b65b7L590
>>>>       and merged already into 18.10.1
>>>>       
>>>>       Sasha
>>>>       
>>>>       
>>>>       >
>>>>       > Regards,
>>>>       >
>>>>       > -Jim
>>>>       >
>>>>       >
>>>>       >      I think, they fix/improve NVMF-OF (RDMA) solution and should
>>>> be
>>>> considered for merging into release version.
>>>>       >      Ben, Jim  could you review those patches and merge them to
>>>> "master" if they are OK with you.
>>>>       >
>>>>       >      Best regards
>>>>       >      Sasha
>>>>       >
>>>>       >      On 12/13/2018 7:11 AM, Zawadzki, Tomasz wrote:
>>>>       >      > Hi Sasha,
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > The patch 'env_dpdk: tell DPDK to not free dynamically
>>>> allocated memory' is merged to 18.10.x.
>>>>       >      >
>>>>       >      > Please see this list<
>>>> https://review.gerrithub.io/#/q/project:spdk/spdk+branch:v18.10.x>; on
>>>> Gerrithub for up to date patches going into that branch.
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > For the rest of the patches listed, we didn’t merge every
>>>> fix
>>>> between 18.10 and current master. Only focusing on the critical ones that
>>>> got merged to master.
>>>>       >      >
>>>>       >      > We will look into those and see which could be integrated
>>>> into
>>>> 18.10.x , assuming they get merged to master first.
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > Thanks,
>>>>       >      >
>>>>       >      > Tomek
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > -----Original Message-----
>>>>       >      > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
>>>> Sasha Kotchubievsky
>>>>       >      > Sent: Wednesday, December 12, 2018 9:29 PM
>>>>       >      > To: spdk(a)lists.01.org
>>>>       >      > Subject: Re: [SPDK] Content for v18.10.x
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > Hi Tomek,
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > Great news
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > Can you confirm that 2M page issue in NVME-OF RDMA is fixed
>>>> in
>>>> 18.10.1?
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > It was fixed by https://github.com/spdk/spdk/commit/7a58660
>>>> ,
>>>> which is not in the 18.10.X .  But, maybe it's fixed by other patches that
>>>> are in.
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > If you need some help to reproduce and retest the issue, let
>>>> me know.
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > I'd consider to merge following patches  into 18.10.1:
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      >    -  https://review.gerrithub.io/c/spdk/spdk/+/4369747474
>>>> -
>>>> This patch fixes resource leakage
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      >   -  https://review.gerrithub.io/c/spdk/spdk/+/436887
>>>> improves
>>>> error handling
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      >   - https://review.gerrithub.io/c/spdk/spdk/+/4308020202  -
>>>> This
>>>> patch fixes tests compilation on ARM platform. It's pretty annoying issue.
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > Thanks
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > Sasha
>>>>       >      >
>>>>       >      >
>>>>       >      >
>>>>       >      > On 12/12/2018 10:17 PM, Zawadzki, Tomasz wrote:
>>>>       >      >
>>>>       >      >> Hi Sasha,
>>>>       >      >> Yes, there is an update to SPDK 18.10.1 coming that focuses
>>>> on adding packaging for SPDK, along with important fixes that were merged
>>>> since last release.
>>>>       >      >> Most of the patches were cherry-picked and merged into
>>>> 18.10.x branch. Currently finishing up couple of the last ones.
>>>>       >      >> The plan is to release it by end of this week.
>>>>       >      >> Thanks,
>>>>       >      >> Tomek
>>>>       >      >> -----Original Message-----
>>>>       >      >> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
>>>> Sasha
>>>>       >      >> Kotchubievsky
>>>>       >      >> Sent: Wednesday, December 12, 2018 11:02 AM
>>>>       >      >> To: Storage Performance Development Kit <spdk(a)lists.01.org<
>>>> ma
>>>> ilto:spdk(a)lists.01.org>>
>>>>       >      >> Subject: [SPDK] Content for v18.10.x
>>>>       >      >> Hi,
>>>>       >      >> As I see, some patches from "master" are merged into
>>>> v18.10.x
>>>> branch.
>>>>       >      >> https://github.com/spdk/spdk/tree/v18.10.x
>>>>       >      >> Are there plans to update v18.10 released in October?
>>>>       >      >> Best regards
>>>>       >      >> Sasha
>>>>       >      >> _______________________________________________
>>>>       >      >> SPDK mailing list
>>>>       >      >> SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
>>>>       >      >> https://lists.01.org/mailman/listinfo/spdk
>>>>       >      >> _______________________________________________
>>>>       >      >> SPDK mailing list
>>>>       >      >> SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
>>>>       >      >> https://lists.01.org/mailman/listinfo/spdk
>>>>       >      > _______________________________________________
>>>>       >      >
>>>>       >      > SPDK mailing list
>>>>       >      >
>>>>       >      > SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
>>>>       >      >
>>>>       >      > https://lists.01.org/mailman/listinfo/spdk
>>>>       >      > _______________________________________________
>>>>       >      > SPDK mailing list
>>>>       >      > SPDK(a)lists.01.org
>>>>       >      > https://lists.01.org/mailman/listinfo/spdk
>>>>       >      _______________________________________________
>>>>       >      SPDK mailing list
>>>>       >      SPDK(a)lists.01.org
>>>>       >      https://lists.01.org/mailman/listinfo/spdk
>>>>       >
>>>>       >
>>>>       > _______________________________________________
>>>>       > SPDK mailing list
>>>>       > SPDK(a)lists.01.org
>>>>       > https://lists.01.org/mailman/listinfo/spdk
>>>>       _______________________________________________
>>>>       SPDK mailing list
>>>>       SPDK(a)lists.01.org
>>>>       https://lists.01.org/mailman/listinfo/spdk
>>>>       
>>>>
>>>> _______________________________________________
>>>> SPDK mailing list
>>>> SPDK(a)lists.01.org
>>>> https://lists.01.org/mailman/listinfo/spdk
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk