From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2164785277105598735==" MIME-Version: 1.0 From: Sasha Kotchubievsky Subject: Re: [SPDK] Content for v18.10.x Date: Wed, 19 Dec 2018 20:11:21 +0200 Message-ID: <435e9a40-f904-e517-d0ef-6dedcd7ea724@dev.mellanox.co.il> In-Reply-To: ce220aad7e88e46108af759f3d3a607259686e13.camel@intel.com List-ID: To: spdk@lists.01.org --===============2164785277105598735== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Do you see the same pattern in both branches in "master" and in "18.10.1"? >> Seth found a similar failure on the master branch, but it was way back on >> November 5th. That's more than a month ago, and with the number of patch= es >> that >> are run daily it sure seems like the problem must have been fixed there.= I did >> a >> diff of lib/nvmf/rdma.c on 18.10.1 and master and there doesn't seem to = be a >> huge number of differences. There is some improved SGL handling on maste= r, but >> this test is using the Linux kernel initiator which never sends more tha= n 1 >> SGL >> element. >> >> Still investigating. > This is still an open issue, but Seth did figure out why we don't see the > problem on master - the test changed. The plan right now is to move forwa= rd with > 18.10.1 without addressing this problem. 18.10.1 is far better than 18.10= in > this area already, and because this problem likely exists on master in th= e same > form (and may not even ultimately be an SPDK problem), we'd rather not wa= it to > get 18.10.1 out any longer. Dose it mean, the patch fixing "ref count" will be merged into 18.10.1? > > Any assistance with investigating the root cause of the issue on the mast= er > branch would be greatly appreciated. It replicates using the 18.10 versio= n of > the test/nvmf/lvol/nvmf_lvol.sh test, but not the version on master. The = problem > appears to be that the connection just drops, and both the initiator and = the > target reactor to that correctly. But why the connection is dropping (it's > loopback) is the unknown. Which SW stack is used in the testing: SoftROCE/MOFED/OFED? OS? As I understand, for reproduction, I run test/nvmf/lvol/nvmf_lvol.sh=C2=A0 = ( = from 18.10 ) on source coming from "master". Is that correct? > >>> Best regards >>> >>> Sasha >>> >>> On 12/17/2018 5:36 PM, Harris, James R wrote: >>>> Hi Sasha, >>>> >>>> 437348 failed the first time in the Chandler test pool. It had a >>>> signature >>>> that neither Ben nor me had seen before, and we were concerned that fi= xing >>>> the refcnt was exposing some other issue. I personally haven't looked >>>> through all of the test failures on master since that patch was >>>> merged. It's possible there are other fixes on master that aren't in >>>> 18.10.x yet. So for now, I'm leaning against merging the refcnt patch= for >>>> 18.10.1. >>>> >>>> -Jim >>>> >>>> >>>> =EF=BB=BFOn 12/16/18, 4:31 AM, "SPDK on behalf of Sasha Kotchubievsky"= < >>>> spdk-bounces(a)lists.01.org on behalf of sashakot(a)dev.mellanox.co.il= > wrote: >>>> >>>> On 12/14/2018 6:21 PM, Harris, James R wrote: >>>> > >>>> > On 12/13/18, 12:46 AM, "SPDK on behalf of Sasha Kotchubievsky"= < >>>> spdk-bounces(a)lists.01.org on behalf of sashakot(a)dev.mellanox.co.il= > wrote: >>>> > >>>> > Hi, >>>> > >>>> > Thanks, for information. >>>> > >>>> > Patch https://review.gerrithub.io/c/spdk/spdk/+/430802 is >>>> merged >>>> today >>>> > to "master". It will be nice, if you get it to 18.10.1 to= o. >>>> > >>>> > Regarding to: >>>> > >>>> > https://review.gerrithub.io/c/spdk/spdk/+/436974 >>>> > >>>> > https://review.gerrithub.io/c/spdk/spdk/+/436887 >>>> > >>>> > Hi Sasha, >>>> > >>>> > The two patches mentioned have been merged to master and are i= n the >>>> build queue for 18.10.1 (URLs below): >>>> > https://review.gerrithub.io/#/c/spdk/spdk/+/437348/ >>>> > https://review.gerrithub.io/#/c/spdk/spdk/+/437349/ >>>> = >>>> Hi Jim, >>>> = >>>> Patch https://review.gerrithub.io/c/spdk/spdk/+/437348/ is aband= oned >>>> from 18.10.1 >>>> = >>>> The patch fixes ref counters issue introduced by >>>> = >>>> > https://github.com/spdk/spdk/commit/567b1006a9d4533e5ae03d78b11040f5da0fe= 278#diff-28f9c2af26bafb4a7268c9aa7b9b65b7L590 >>>> and merged already into 18.10.1 >>>> = >>>> Sasha >>>> = >>>> = >>>> > >>>> > Regards, >>>> > >>>> > -Jim >>>> > >>>> > >>>> > I think, they fix/improve NVMF-OF (RDMA) solution and sho= uld >>>> be >>>> considered for merging into release version. >>>> > Ben, Jim could you review those patches and merge them to >>>> "master" if they are OK with you. >>>> > >>>> > Best regards >>>> > Sasha >>>> > >>>> > On 12/13/2018 7:11 AM, Zawadzki, Tomasz wrote: >>>> > > Hi Sasha, >>>> > > >>>> > > >>>> > > >>>> > > The patch 'env_dpdk: tell DPDK to not free dynamically >>>> allocated memory' is merged to 18.10.x. >>>> > > >>>> > > Please see this list< >>>> https://review.gerrithub.io/#/q/project:spdk/spdk+branch:v18.10.x>; on >>>> Gerrithub for up to date patches going into that branch. >>>> > > >>>> > > >>>> > > >>>> > > For the rest of the patches listed, we didn=E2=80=99t m= erge every >>>> fix >>>> between 18.10 and current master. Only focusing on the critical ones t= hat >>>> got merged to master. >>>> > > >>>> > > We will look into those and see which could be integrat= ed >>>> into >>>> 18.10.x , assuming they get merged to master first. >>>> > > >>>> > > >>>> > > >>>> > > Thanks, >>>> > > >>>> > > Tomek >>>> > > >>>> > > >>>> > > >>>> > > -----Original Message----- >>>> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Beha= lf Of >>>> Sasha Kotchubievsky >>>> > > Sent: Wednesday, December 12, 2018 9:29 PM >>>> > > To: spdk(a)lists.01.org >>>> > > Subject: Re: [SPDK] Content for v18.10.x >>>> > > >>>> > > >>>> > > >>>> > > Hi Tomek, >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > Great news >>>> > > >>>> > > >>>> > > >>>> > > Can you confirm that 2M page issue in NVME-OF RDMA is f= ixed >>>> in >>>> 18.10.1? >>>> > > >>>> > > >>>> > > >>>> > > It was fixed by https://github.com/spdk/spdk/commit/7a5= 8660 >>>> , >>>> which is not in the 18.10.X . But, maybe it's fixed by other patches = that >>>> are in. >>>> > > >>>> > > >>>> > > >>>> > > If you need some help to reproduce and retest the issue= , let >>>> me know. >>>> > > >>>> > > >>>> > > >>>> > > I'd consider to merge following patches into 18.10.1: >>>> > > >>>> > > >>>> > > >>>> > > - https://review.gerrithub.io/c/spdk/spdk/+/4369747= 474 >>>> - >>>> This patch fixes resource leakage >>>> > > >>>> > > >>>> > > >>>> > > - https://review.gerrithub.io/c/spdk/spdk/+/436887 >>>> improves >>>> error handling >>>> > > >>>> > > >>>> > > >>>> > > - https://review.gerrithub.io/c/spdk/spdk/+/430802020= 2 - >>>> This >>>> patch fixes tests compilation on ARM platform. It's pretty annoying is= sue. >>>> > > >>>> > > >>>> > > >>>> > > Thanks >>>> > > >>>> > > >>>> > > >>>> > > Sasha >>>> > > >>>> > > >>>> > > >>>> > > On 12/12/2018 10:17 PM, Zawadzki, Tomasz wrote: >>>> > > >>>> > >> Hi Sasha, >>>> > >> Yes, there is an update to SPDK 18.10.1 coming that fo= cuses >>>> on adding packaging for SPDK, along with important fixes that were mer= ged >>>> since last release. >>>> > >> Most of the patches were cherry-picked and merged into >>>> 18.10.x branch. Currently finishing up couple of the last ones. >>>> > >> The plan is to release it by end of this week. >>>> > >> Thanks, >>>> > >> Tomek >>>> > >> -----Original Message----- >>>> > >> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Beh= alf Of >>>> Sasha >>>> > >> Kotchubievsky >>>> > >> Sent: Wednesday, December 12, 2018 11:02 AM >>>> > >> To: Storage Performance Development Kit >>> ma >>>> ilto:spdk(a)lists.01.org>> >>>> > >> Subject: [SPDK] Content for v18.10.x >>>> > >> Hi, >>>> > >> As I see, some patches from "master" are merged into >>>> v18.10.x >>>> branch. >>>> > >> https://github.com/spdk/spdk/tree/v18.10.x >>>> > >> Are there plans to update v18.10 released in October? >>>> > >> Best regards >>>> > >> Sasha >>>> > >> _______________________________________________ >>>> > >> SPDK mailing list >>>> > >> SPDK(a)lists.01.org >>>> > >> https://lists.01.org/mailman/listinfo/spdk >>>> > >> _______________________________________________ >>>> > >> SPDK mailing list >>>> > >> SPDK(a)lists.01.org >>>> > >> https://lists.01.org/mailman/listinfo/spdk >>>> > > _______________________________________________ >>>> > > >>>> > > SPDK mailing list >>>> > > >>>> > > SPDK(a)lists.01.org >>>> > > >>>> > > https://lists.01.org/mailman/listinfo/spdk >>>> > > _______________________________________________ >>>> > > SPDK mailing list >>>> > > SPDK(a)lists.01.org >>>> > > https://lists.01.org/mailman/listinfo/spdk >>>> > _______________________________________________ >>>> > SPDK mailing list >>>> > SPDK(a)lists.01.org >>>> > https://lists.01.org/mailman/listinfo/spdk >>>> > >>>> > >>>> > _______________________________________________ >>>> > SPDK mailing list >>>> > SPDK(a)lists.01.org >>>> > https://lists.01.org/mailman/listinfo/spdk >>>> _______________________________________________ >>>> SPDK mailing list >>>> SPDK(a)lists.01.org >>>> https://lists.01.org/mailman/listinfo/spdk >>>> = >>>> >>>> _______________________________________________ >>>> SPDK mailing list >>>> SPDK(a)lists.01.org >>>> https://lists.01.org/mailman/listinfo/spdk >>> _______________________________________________ >>> SPDK mailing list >>> SPDK(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/spdk >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============2164785277105598735==--