Re: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 10214 bytes --]



On 2/15/19, 2:19 PM, "SPDK on behalf of Lance Hartmann ORACLE" <spdk-bounces(a)lists.01.org on behalf of lance.hartmann(a)oracle.com> wrote:

    
<trim>
    
    So the risk on nightly-testing for #2 is the scenario that a submitted patch passed against the DPDK submodule, was quickly reviewed and then merged before we would find out later -- overnight -- that the same patch against the DPDK last stable did not.   I don't know how often the case exists where a newly submitted patch runs through the test-pool, gets two maintainer's reviews and merged all in a single day.   That said, do you fear that by always holding off one day before doing a merge, we could inspect the overnight test against DPDK stable first?

[Jim] Well...the nightly testing would be done against tip of the SPDK master branch - meaning only against patches that have been merged.  So by definition, what you're describing is exactly how we would find problems with DPDK last stable that aren't there with the DPDK submodule.  As a project, we'll need to treat these failures with the utmost priority - in most cases immediately reverting while it can be investigated.
    
    I'd also like to pose my question again with respect to "DPDK last stable".   Right now as I compose this per http://core.dpdk.org/download/, they identify "Latest Stable" as 18.05.1 (which I find surprising and wonder if that's correct).   They identify "Latest Major" as 19.02 (released 2019, February 1st), and the release prior to that was DPDK 18.11.0 (LTS).    So, given those three (3) versions of the upstream DPDK, which of them aligns with our use of the term "DPDK last stable release"?  

[Jim] Good point, we need to be more precise on this.  I had been thinking "latest major" release, since that's what the DPDK submodule is based on.
    
    
    --
    Lance Hartmann
    
    
    
    > 
    > -Jim
    > 
    > 
    > On 2/15/19, 11:25 AM, "Lance Hartmann ORACLE" <lance.hartmann(a)oracle.com> wrote:
    > 
    > 
    >    Responses interlined below:
    > 
    > 
    > 
    >> On Feb 15, 2019, at 8:26 AM, Latecki, Karol <karol.latecki(a)intel.com> wrote:
    >> 
    >> Hi everyone,
    >> We had a talk about this here and we have following proposal about testing vs DPDK.
    >> We'd like to start with 3 nightly job configurations, with e-mail notifications enabled:
    > 
    > 
    >    Just nightly?   I think Jim had initially expressed an interest in doing a run per-patch I believe with the hope of identifying and addressing any issues more quickly.   I'll be eager to hear his (and others') feedback on this.
    > 
    > 
    >> * SPDK master vs DPDK master
    > 
    >    When you write "DPDK master" above -- you mean the SPDK's DPDK submodule?   If so, maybe we might refer to that as "sDPDK".
    > 
    >> * SPDK master vs DPDK last stable release
    > 
    >    Again, clarification:  By "DPDK last stable release", you certainly mean the *upstream* DPDK's most recently cut stable release (suggested notation, "uDPDK").    However, a new uDPDK stable release is cut one month after each SPDK release (based on the current respective release schedules).   So would someone (or some automated task?) poll for a newly cut uDPDK stable and then change the Jenkins job so it started building against that? 
    > 
    > 
    > 
    >> * SPDK last stable release vs DPDK master
    > 
    > 
    >    By "DPDK master", you mean the current master for the SPDK's DPDK submodule (sDPDK)?
    > 
    >> That way we'd know fast enough about new issues between SPDK and DPDK. 
    >> 
    >> I think that your point Jim still applies for these configurations - we'd have to configure them not to test things that we know are failing. That also means that we would have to re-evaluate the configs from time to time.
    > 
    > 
    >    --
    >    Lance Hartmann
    > 
    > 
    > 
    >> 
    >> Karol
    >> 
    >> -----Original Message-----
    >> From: Harris, James R 
    >> Sent: Thursday, February 14, 2019 3:32 PM
    >> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>; Latecki, Karol <karol.latecki(a)intel.com>
    >> Subject: Re: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
    >> 
    >> We could also just switch some of our existing jobs to build/run against the latest stable DPDK release.  We can be careful about picking jobs that don't have dependencies on patches from our DPDK submodule.
    >> 
    >> On 2/14/19, 6:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
    >> 
    >>   So we talked a lot about this in last night's  community meeting and everyone agreed as a first step we need to:
    >> 
    >>   * create new Jenkins subjob for per-patch testing that tests against last stable DPDK release
    >>   * configure the job to not test things that we know will fail (like crypto due to a patch we pushed upstream that won't land until 19.05).  There may be others like this well (some secondary process tests, etc) so it may take a little experimentation and investigation to get the job right.
    >>   * We'll need to review job settings at each release to enable/disable features to test based on the contents of both the SPDK and DPDK releases at the time.
    >> 
    >>   Darek/Karol, can you guys get started on this and keep everyone posted?
    >> 
    >>   Thanks! 
    >>   Paul
    >> 
    >>   PS: Thanks Lance for driving this discussion
    >> 
    >>   -----Original Message-----
    >>   From: Luse, Paul E 
    >>   Sent: Tuesday, February 12, 2019 5:10 AM
    >>   To: Storage Performance Development Kit <spdk(a)lists.01.org>
    >>   Subject: RE: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
    >> 
    >>   I agree that would be a great addition! For some reason I thought that was already being done somewhere by someone, no matter though, setting up a Jenkins job is the right way to do it now. Karol, can you take a look at this and let everyone know your thoughts?
    >> 
    >>   Thanks
    >>   Paul
    >> 
    >>   -----Original Message-----
    >>   From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lance Hartmann ORACLE
    >>   Sent: Monday, February 11, 2019 4:44 PM
    >>   To: spdk(a)lists.01.org
    >>   Subject: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
    >> 
    >> 
    >>   Group,
    >> 
    >>   I recently had a sideband chat with Jim regarding our SPDK testing with the DPDK, and he urged I open this up to the email list for more discussion.
    >> 
    >>   When a Linux distro produces SPDK package rpms, they satisfy the DPDK dependency via a specified stable DPDK release from upstream that has been packaged. However, on the run up to each SPDK quarterly release, we are constantly building and testing against the SPDK's fork'd copy of the DPDK (as a git-submodule) on which we often (virtually always?) have our own patches.   While this works fine for a development perspective as we work toward the next SPDK release, it's problematic when the time arrives to produce new SPDK packages because they will be built against a pristine DPDK stable release which very likely is not identical with what we have been testing against.   When I described this with Jim, he agreed that it would be an excellent idea to augment our SPDK Jenkins setup with some tests which would run against the latest "pristine" (i.e. upstream stable) DPDK release.   With that in place, when the time arrives to produce new SPDK packages, we will have gained the testing c
    >>    onfidence of having validated running with the exact version of the DPDK against which the SPDK packages will have been built (and will run alongside at runtime).
    >> 
    >>   Having said all that, I did float another idea on how we could, if we really needed to, produce SPDK package rpms based on the combination of a stable DPDK upstream release and our own collection of patches to it.   The rpm packaging system natively supports such a build, but as you might quickly deduce it comes at the cost of additional maintenance efforts which can become very unpalatable, esp. when/if some of our patches are never accepted/merged into the official upstream DPDK tree.
    >> 
    >> 
    >>   --
    >>   Lance Hartmann
    >> 
    >> 
    >> 
    >>   _______________________________________________
    >>   SPDK mailing list
    >>   SPDK(a)lists.01.org
    >>   https://lists.01.org/mailman/listinfo/spdk
    >>   _______________________________________________
    >>   SPDK mailing list
    >>   SPDK(a)lists.01.org
    >>   https://lists.01.org/mailman/listinfo/spdk
    >> 
    >> 
    >> --------------------------------------------------------------------
    >> 
    >> Intel Technology Poland sp. z o.o.
    >> ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
    >> 
    >> Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
    >> przegladanie lub rozpowszechnianie jest zabronione.
    >> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
    >> others is strictly prohibited.
    >> _______________________________________________
    >> SPDK mailing list
    >> SPDK(a)lists.01.org
    >> https://lists.01.org/mailman/listinfo/spdk
    > 
    > 
    > 
    
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk