Re: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 9570 bytes --]

Hi Lance,

In this case, "DPDK master" means the tip of DPDK.  So to summarize, we have three cases here:

1) SPDK master v. DPDK master
2) SPDK master v. DPDK last stable release
3) SPDK last stable release v. DPDK master

Since #3 is stable from an SPDK point of view, nightly tests here are fine.

I think #1 has to be nightly as well.  Since DPDK master is constantly in flux, we don't want our SPDK patches to get blocked due to DPDK bugs that get checked in between releases.

It's #2 that's the question.  I talked with Darek about it this morning.  Here are the pros and cons as I see them:

Per-patch testing:
Pros:
* immediately find incompatibilities between DPDK last stable release and DPDK submodule
Cons:
* only a subset of SPDK tests run against DPDK last stable release - running all SPDK tests (or at least that don't have known current incompatibilities) against both DPDK last stable release and DPDK submodule would require doubling the CI systems

Nightly testing:
Pros:
* we run all SPDK tests against DPDK last stable release
Cons:
* patches could potentially get merged to master which expose an incompatibility between DPDK stable and the DPDK submodule

I think to start, we get the nightly testing in place ASAP.  If a nightly test fails it's a critical issue that likely leads to an immediate revert.  We can still add per-patch testing on top of it - maybe on just a very limited number of SPDK tests - but we'll still need to rely on the nightly testing for full coverage.  I'd be interested in hearing other opinions on this.

-Jim


On 2/15/19, 11:25 AM, "Lance Hartmann ORACLE" <lance.hartmann(a)oracle.com> wrote:

    
    Responses interlined below:
    
    
    
    > On Feb 15, 2019, at 8:26 AM, Latecki, Karol <karol.latecki(a)intel.com> wrote:
    > 
    > Hi everyone,
    > We had a talk about this here and we have following proposal about testing vs DPDK.
    > We'd like to start with 3 nightly job configurations, with e-mail notifications enabled:
    
    
    Just nightly?   I think Jim had initially expressed an interest in doing a run per-patch I believe with the hope of identifying and addressing any issues more quickly.   I'll be eager to hear his (and others') feedback on this.
    
    
    > * SPDK master vs DPDK master
    
    When you write "DPDK master" above -- you mean the SPDK's DPDK submodule?   If so, maybe we might refer to that as "sDPDK".
    
    > * SPDK master vs DPDK last stable release
    
    Again, clarification:  By "DPDK last stable release", you certainly mean the *upstream* DPDK's most recently cut stable release (suggested notation, "uDPDK").    However, a new uDPDK stable release is cut one month after each SPDK release (based on the current respective release schedules).   So would someone (or some automated task?) poll for a newly cut uDPDK stable and then change the Jenkins job so it started building against that? 
    
    
    
    > * SPDK last stable release vs DPDK master
    
    
    By "DPDK master", you mean the current master for the SPDK's DPDK submodule (sDPDK)?
    
    > That way we'd know fast enough about new issues between SPDK and DPDK. 
    > 
    > I think that your point Jim still applies for these configurations - we'd have to configure them not to test things that we know are failing. That also means that we would have to re-evaluate the configs from time to time.
    
    
    --
    Lance Hartmann
    
    
    
    > 
    > Karol
    > 
    > -----Original Message-----
    > From: Harris, James R 
    > Sent: Thursday, February 14, 2019 3:32 PM
    > To: Storage Performance Development Kit <spdk(a)lists.01.org>; Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>; Latecki, Karol <karol.latecki(a)intel.com>
    > Subject: Re: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
    > 
    > We could also just switch some of our existing jobs to build/run against the latest stable DPDK release.  We can be careful about picking jobs that don't have dependencies on patches from our DPDK submodule.
    > 
    > On 2/14/19, 6:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
    > 
    >    So we talked a lot about this in last night's  community meeting and everyone agreed as a first step we need to:
    > 
    >    * create new Jenkins subjob for per-patch testing that tests against last stable DPDK release
    >    * configure the job to not test things that we know will fail (like crypto due to a patch we pushed upstream that won't land until 19.05).  There may be others like this well (some secondary process tests, etc) so it may take a little experimentation and investigation to get the job right.
    >    * We'll need to review job settings at each release to enable/disable features to test based on the contents of both the SPDK and DPDK releases at the time.
    > 
    >    Darek/Karol, can you guys get started on this and keep everyone posted?
    > 
    >    Thanks! 
    >    Paul
    > 
    >    PS: Thanks Lance for driving this discussion
    > 
    >    -----Original Message-----
    >    From: Luse, Paul E 
    >    Sent: Tuesday, February 12, 2019 5:10 AM
    >    To: Storage Performance Development Kit <spdk(a)lists.01.org>
    >    Subject: RE: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
    > 
    >    I agree that would be a great addition! For some reason I thought that was already being done somewhere by someone, no matter though, setting up a Jenkins job is the right way to do it now. Karol, can you take a look at this and let everyone know your thoughts?
    > 
    >    Thanks
    >    Paul
    > 
    >    -----Original Message-----
    >    From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lance Hartmann ORACLE
    >    Sent: Monday, February 11, 2019 4:44 PM
    >    To: spdk(a)lists.01.org
    >    Subject: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
    > 
    > 
    >    Group,
    > 
    >    I recently had a sideband chat with Jim regarding our SPDK testing with the DPDK, and he urged I open this up to the email list for more discussion.
    > 
    >    When a Linux distro produces SPDK package rpms, they satisfy the DPDK dependency via a specified stable DPDK release from upstream that has been packaged. However, on the run up to each SPDK quarterly release, we are constantly building and testing against the SPDK's fork'd copy of the DPDK (as a git-submodule) on which we often (virtually always?) have our own patches.   While this works fine for a development perspective as we work toward the next SPDK release, it's problematic when the time arrives to produce new SPDK packages because they will be built against a pristine DPDK stable release which very likely is not identical with what we have been testing against.   When I described this with Jim, he agreed that it would be an excellent idea to augment our SPDK Jenkins setup with some tests which would run against the latest "pristine" (i.e. upstream stable) DPDK release.   With that in place, when the time arrives to produce new SPDK packages, we will have gained the testing c
    >     onfidence of having validated running with the exact version of the DPDK against which the SPDK packages will have been built (and will run alongside at runtime).
    > 
    >    Having said all that, I did float another idea on how we could, if we really needed to, produce SPDK package rpms based on the combination of a stable DPDK upstream release and our own collection of patches to it.   The rpm packaging system natively supports such a build, but as you might quickly deduce it comes at the cost of additional maintenance efforts which can become very unpalatable, esp. when/if some of our patches are never accepted/merged into the official upstream DPDK tree.
    > 
    > 
    >    --
    >    Lance Hartmann
    > 
    > 
    > 
    >    _______________________________________________
    >    SPDK mailing list
    >    SPDK(a)lists.01.org
    >    https://lists.01.org/mailman/listinfo/spdk
    >    _______________________________________________
    >    SPDK mailing list
    >    SPDK(a)lists.01.org
    >    https://lists.01.org/mailman/listinfo/spdk
    > 
    > 
    > --------------------------------------------------------------------
    > 
    > Intel Technology Poland sp. z o.o.
    > ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
    > 
    > Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
    > przegladanie lub rozpowszechnianie jest zabronione.
    > This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
    > others is strictly prohibited.
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk