[SPDK] Re: Preparation for SPDK 19.10 release

[Luse, Paul E <paul.e.luse at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 10234 bytes --]

I think discussion on a call is a great idea, I've got lots of lots of input ☺ I put it on the agenda already, we can do something sooner if you'd like Sasha, we can always fire up a freeconference call at any time as long as get those who have the most interest in participating to agree on a time....

Thx
Paul

On 10/16/19, 10:13 AM, "Harris, James R" <james.r.harris(a)intel.com> wrote:

    Hi Sasha,
    
    See inline [Jim] below.
    
    -Jim
    
    
    On 10/16/19, 10:01 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:
    
        Hi Jim,
        
        PSB [SK]
        
        -----Original Message-----
        From: Harris, James R <james.r.harris(a)intel.com> 
        Sent: Wednesday, October 16, 2019 6:22 PM
        To: Storage Performance Development Kit <spdk(a)lists.01.org>
        Subject: [SPDK] Re: Preparation for SPDK 19.10 release
        
        Hi Sasha,
        
        The CHANGELOG is built during the months leading up to the release, and Tomek typically does some cleanups just before the release.  If you'd like to help with finalizing the CHANGELOG before the release, please push some patches!  Documentation is always an area where the project could use more help.
        
        [SK] Changelog is good. I'm talking about "Release notes"
        https://github.com/spdk/spdk/releases/tag/v19.07
        I think, that's first point to see that's new in the release
    
    [Jim]  OK.  It sounded like there were also some concerns that some things were getting missed in the CHANGELOG.  So any help there would still be appreciated.
    
    [Jim]  Regarding copying CHANGELOG contents to the GitHub release notes:  looking at CHANGELOG.md, the v19.07 section was almost 300 lines of text.  My personal preference would be to not copy that verbatim into the GitHub release notes.  If we could keep the Release Notes high level, and provide a link in the release notes to the relevant section of the CHANGELOG, would that be sufficient?
        
        For the RDMA testing, I have some of the same questions as Paul.  If there are gaps in upstream per-patch or nightly tests, please help us get them closed.  Can you describe the tests you're running internally, how often they take, and how often you are running them?
        
        [SK] Correct me if I'm wrong, but most of existing tests in SPDK use SoftRoce, or single host without actual transferring data over network.  Runtime behavior on real HW can be different. 
        Our verification system today tests solutions which include SPDK as building block. We work with stable version of SPDK and don't update it frequently. It mostly focus on Mellanox's parts and less on SPDK.
        From functional perspective, we test complex scenarios like OS install/boot to/from distributed storage.  We test reboots and reconnects. In addition,  we test on non-x86 platform too.
        For upstream SPDK (reacting on each patch) we have limited number of sanity tests.
        From my point of view, for network part the verification system should test:
        - Matrix of HW, SW stacks (OS,  OFED)
        - Matrix of protocols: RDMA, TCP 
        - Corner cases like link failure, network congestion , SW/HW failures during real traffic
        - Stress testing
        - Long tests 
    
    [Jim]  Thanks for this level of detail Sasha.  I think this would be a great agenda topic for next community meeting.   Could we plan on discussing this further at the October 29th meeting?
    
    -Jim
    
        
        
        Thanks,
        
        -Jim
        
        On 10/16/19, 7:47 AM, "Luse, Paul E" <paul.e.luse(a)intel.com> wrote:
        
            Hi Sasha,
            
            I'm sure you'll get some other opinions as well but as the changelog now pretty much only contains interface and parameter changes sorted by module, are you just suggesting re-organizing it to make it clearer per module where interface changes have occurred vs parameter changes?
            
            Wrt testing, I love that you guys are testing this so thoroughly! I have to ask though, what additional testing are you proposing that isn't already in either the main CI or your own CI? Is there a possibility to get these tests unstreamed? We really want to continue with a model of continuous integration and test without any additional testing nearing a release.  The freeze window is to provide time to prepare the release, make sure we've got everything in it that was intended, go through some Intel security checklist stuff, etc.  Ben said it best once at a summit, our releases are really just a bookkeeping activity.
            
            You can most definitely use that freeze window to do testing on your own and submit a bug that could potentially halt the release, but I'd wouldn't support making that part of the process.
            
            Thx
            Paul
            
            On 10/16/19, 4:55 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:
            
                Hi Tomek,
                
                That's great news. 
                
                I have couple questions:
                
                - Some of latest SPDK releases (18.10, 19.04, 19.07) were released with
                serious bugs in NVME-OF RDMA. Especially latest 19.07. Is it possible to
                wait with formal release till our group in Mellanox will verify release
                candidate in internal verification system? I think, couple of days will be
                enough for us provide a feedback to the community.
                -  Can we add couple special sections to SPDK release notes
                (https://github.com/spdk/spdk/releases/tag/v19.07) ? 
                 	- Changes in interfaces
                	- Changes in configuration parameters (including defaults)
                	- Known bugs/issues
                For customers building solution on SPDK and considering upgrade, it's
                important to see changes in interfaces (including configuration parameters).
                
                
                Example of changes in an interface in coming 19.10 is 
                https://github.com/spdk/spdk/commit/9522ed36f8474c9314b76b2618e8d2d9de56afee
                #diff-2c205ae4170cbd2bbf219969f03c9c8d
                This commit adds a new  event to bdev interface.
                
                One of commits changed defaults was
                https://github.com/spdk/spdk/commit/b6b0a0ba59a21faf1a320dd3ea7079b282f9fc31
                It changes default IOUnit size from 4K to 8K and affected existing
                configurations.
                
                From my experience, having a table/list in release notes with such changes
                (interfaces, parameters) can help to users to understand side-effects of the
                upgrade.
                I would suggest to maintain the table in Changelog and ask for contributors
                to update it is a part of patch. At release, this table can be just copied
                to release notes.  
                
                Best regards
                Sasha
                
                -----Original Message-----
                From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
                Sent: Wednesday, October 9, 2019 5:41 PM
                To: Storage Performance Development Kit <spdk(a)lists.01.org>
                Subject: [SPDK] Preparation for SPDK 19.10 release
                
                Hello all,
                
                The merge window for SPDK 19.10 release will close by October 24th.
                Please ensure all patches you believe should be included in the release are
                merged to master branch by this date.
                You can do it by adding a hashtag '19.10' in GerritHub on those patches.
                
                The current set of patches that are tagged and need to be reviewed can be
                seen here:
                https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
                
                Starting with this release, process used so far for Maintenance releases
                (ex. 19.04.1) will be applied to main releases. On October 24th new branch
                'v19.10.x' will be created, and a patch on it will be tagged as release
                candidate.
                Then, on October 31th, a formal release will take place tagging the last
                patch on the branch as SPDK 19.10.
                
                Between release candidate and formal release, only critical fixes shall be
                backported to the 'v19.10.x' branch.
                
                Development can continue without disruptions on 'master' branch.
                
                Thanks,
                Tomek
                _______________________________________________
                SPDK mailing list -- spdk(a)lists.01.org
                To unsubscribe send an email to spdk-leave(a)lists.01.org
                _______________________________________________
                SPDK mailing list -- spdk(a)lists.01.org
                To unsubscribe send an email to spdk-leave(a)lists.01.org
                
            
            _______________________________________________
            SPDK mailing list -- spdk(a)lists.01.org
            To unsubscribe send an email to spdk-leave(a)lists.01.org
            
        
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        
    
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org