giant and hammer dates

giant and hammer dates

[Sage Weil]

We've talked a bit about moving to a ~4 month (instead of 3 month) 
cadence.  I'm still inclined in this direction because it means fewer 
stable releases that we will be maintaining and a longer and (hopefully) 
more productive interval to do real work in between.

The other key point is that we don't want a repeat of the firefly delay.  
I think we should stay as close to a train model as we can.  If something 
isn't ready by freeze, let it wait for the next cycle.  We shouldn't be 
cramming things in at the end, especially big things.  As a general rule, 
big things should be merged early in the cycle so that we have lots of 
time to shake out the issues that only come out of lots of testing and 
aren't obvious from code review.

Anyway, how about:

          Freeze         Approx Release
  Giant   Mon Sep  1     Mon Sep 29
  Hammer  Mon Jan  4     Mon Feb  2

That gives us another month for Giant, then September to shake out 
anything issues.  And then three full months before the Hammer freeze.

What say ye?
sage

Re: giant and hammer dates

[Mark Nelson]

On 07/29/2014 06:11 PM, Sage Weil wrote:
> We've talked a bit about moving to a ~4 month (instead of 3 month)
> cadence.  I'm still inclined in this direction because it means fewer
> stable releases that we will be maintaining and a longer and (hopefully)
> more productive interval to do real work in between.
>
> The other key point is that we don't want a repeat of the firefly delay.
> I think we should stay as close to a train model as we can.  If something
> isn't ready by freeze, let it wait for the next cycle.  We shouldn't be
> cramming things in at the end, especially big things.  As a general rule,
> big things should be merged early in the cycle so that we have lots of
> time to shake out the issues that only come out of lots of testing and
> aren't obvious from code review.
>
> Anyway, how about:
>
>            Freeze         Approx Release
>    Giant   Mon Sep  1     Mon Sep 29
>    Hammer  Mon Jan  4     Mon Feb  2
>
> That gives us another month for Giant, then September to shake out
> anything issues.  And then three full months before the Hammer freeze.
>
> What say ye?

6 months. ;)

> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: giant and hammer dates

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 3362 bytes --]

Hi Sage,

From my (biased) point of view, the upside is that it will give me more time to complete the locally repairable code for Giant ;-). The downside is that it puts a little less pressure to improve the tools and methods that make a rapid release cycles possible (i.e. unit tests, bug tracking, patch acceptance workflow, package building/gitbuilder, teuthology, pulpito, upgrades testing, ...). In a perfect world Ceph could sustain a three month release cycle without inconveniencing anyone. A longer release cycle (five or six months) would encourage even more complex / bigger changes within a release cycle. It would also probably encourage Ceph developers to forget about the release process tools during two or three months and not improve them as they should be.

IMHO the test cycle is significantly slowing down the release process and a faster, more comprehensive test cycle would help a lot. 

Each commit should be unit / functional tested within seconds, locally (see https://github.com/ceph/ceph/blob/master/src/test/osd/types.cc#L1295 for instance). It is usually more difficult to diagnose / fix a border case when it is discovered during integration tests (i.e. teuthology) rather than with a unit / functional test designed for it. Creating unit tests is often problematic because some of the code base cannot be easily isolated. With a continuous effort to re-arrange parts of the code to be more test friendly, this can eventually be resolved.

Every commit proposed to master should be run against the relevant teuthology suite to help the reviewer. The problem here is that it requires more resources than what Ceph currently has. Harvesting more machines, making it possible for people and organizations amicable to Ceph to easily donate virtual machines could probably help.

This deserves a separate discussions but I wanted to expand on what I meant by "test cycle" and its impact on the release cycle. 

Cheers

On 30/07/2014 05:11, Sage Weil wrote:
> We've talked a bit about moving to a ~4 month (instead of 3 month) 
> cadence.  I'm still inclined in this direction because it means fewer 
> stable releases that we will be maintaining and a longer and (hopefully) 
> more productive interval to do real work in between.
> 
> The other key point is that we don't want a repeat of the firefly delay.  
> I think we should stay as close to a train model as we can.  If something 
> isn't ready by freeze, let it wait for the next cycle.  We shouldn't be 
> cramming things in at the end, especially big things.  As a general rule, 
> big things should be merged early in the cycle so that we have lots of 
> time to shake out the issues that only come out of lots of testing and 
> aren't obvious from code review.
> 
> Anyway, how about:
> 
>           Freeze         Approx Release
>   Giant   Mon Sep  1     Mon Sep 29
>   Hammer  Mon Jan  4     Mon Feb  2
> 
> That gives us another month for Giant, then September to shake out 
> anything issues.  And then three full months before the Hammer freeze.
> 
> What say ye?
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: giant and hammer dates

[Sage Weil]

On Wed, 30 Jul 2014, Loic Dachary wrote:
> Hi Sage,
> 
> From my (biased) point of view, the upside is that it will give me more 
> time to complete the locally repairable code for Giant ;-). The downside 
> is that it puts a little less pressure to improve the tools and methods 
> that make a rapid release cycles possible (i.e. unit tests, bug 
> tracking, patch acceptance workflow, package building/gitbuilder, 
> teuthology, pulpito, upgrades testing, ...). In a perfect world Ceph 
> could sustain a three month release cycle without inconveniencing 
> anyone. A longer release cycle (five or six months) would encourage even 
> more complex / bigger changes within a release cycle. It would also 
> probably encourage Ceph developers to forget about the release process 
> tools during two or three months and not improve them as they should be.
> 
> IMHO the test cycle is significantly slowing down the release process 
> and a faster, more comprehensive test cycle would help a lot.

No argument here. :)

I should clarify that this is the "stable release cycle" for the named 
released.  I still think we should maintain a ~2 week "development release 
cycle" where we are continuously integrating changes and regularly putting 
out a usable release.  The 'next' or 'last' branches should be recent and 
stable starting points for doing any new work so that the integration 
tests, when run, will reflect bugs in your code and not stuff that was 
already there.  We've slipped a bit here (0.82 to 0.83 was 5 weeks); this 
is partly because the release process itself is still pretty expensive in 
terms of effort and we don't want to eat up more of Alfredo's and Sandon's 
time than we need to, but it is getting better.

In any case, the real point of a longer "stable release cycle" is just 
that there are fewer stable releases in flight that we are backporting 
fixes too.  In practice, having all of dumpling, emperor, and firefly 
outstanding hasn't worked particularly well (IMO).  We backport to 
dumpling and firefly and urge people away from emperor to avoid the 
cognitive overhead of keeping track of another release.  Going from 3 to 4 
months means only 3 stable releases per year, which I think is enough...?

> Each commit should be unit / functional tested within seconds, locally 
> (see 
> https://github.com/ceph/ceph/blob/master/src/test/osd/types.cc#L1295 for 
> instance). It is usually more difficult to diagnose / fix a border case 
> when it is discovered during integration tests (i.e. teuthology) rather 
> than with a unit / functional test designed for it. Creating unit tests 
> is often problematic because some of the code base cannot be easily 
> isolated. With a continuous effort to re-arrange parts of the code to be 
> more test friendly, this can eventually be resolved.
> 
> Every commit proposed to master should be run against the relevant 
> teuthology suite to help the reviewer. The problem here is that it 
> requires more resources than what Ceph currently has. Harvesting more 
> machines, making it possible for people and organizations amicable to 
> Ceph to easily donate virtual machines could probably help.

Zack is making good progress on rejiggering the way that teuthology 
separates the core task locking and task runners from the tasks themselves 
(which get versioned along with the test suite for firefly, dumpling, 
etc.).  This is all groundwork to enable the important bits, like pulling 
machine locking into a single, easy to deploy process, and plugging in 
different providers (in addition to bare metal and downburst) like 
OpenStack.  The end goal is to make teuthology much easier to deploy in 
other environments.  I'm hoping we can get to a place similar to openstack 
where organizations can hang their CI deployment off the 'upstream' 
build/CI infrastructure and supplement by running the same suites on 
different hardware or by adding their own test suites...

> This deserves a separate discussions but I wanted to expand on what I 
> meant by "test cycle" and its impact on the release cycle.

We had a discussion during the G/H CDS about doing an ephemeral 
'integration' branch to group things together for full testing by the 
teuthology test suites that you probably caught.  There was a follow-on 
internal discussion while you were gone on how to get this rolling and Sam 
is currently working on a tool to easily build an integration branch 
merging pending work on a nightly so that it can go through the tests 
before getting merged into master.  I think this will help.

We also have our first batch of new hardware ordered inside Red Hat 
(another ~130 machines) that will expand our testing throughput, and 
Sandon is working on reclaiming a lot of existing machines that aren't 
getting put to good use (burnupi) so that we can expand the size of the 
existing test pool.

Alfredo recently did some background research on what other projects are 
doing for CI and releases, and he and Sandon have some work in flight to 
move some of the bursty release builds into openstack VMs.  Unfortunately 
nobody has their full bandwidth allocated to improving the state of 
things, but I think we're making some slow progress.

sage


> 
> Cheers
> 
> On 30/07/2014 05:11, Sage Weil wrote:
> > We've talked a bit about moving to a ~4 month (instead of 3 month) 
> > cadence.  I'm still inclined in this direction because it means fewer 
> > stable releases that we will be maintaining and a longer and (hopefully) 
> > more productive interval to do real work in between.
> > 
> > The other key point is that we don't want a repeat of the firefly delay.  
> > I think we should stay as close to a train model as we can.  If something 
> > isn't ready by freeze, let it wait for the next cycle.  We shouldn't be 
> > cramming things in at the end, especially big things.  As a general rule, 
> > big things should be merged early in the cycle so that we have lots of 
> > time to shake out the issues that only come out of lots of testing and 
> > aren't obvious from code review.
> > 
> > Anyway, how about:
> > 
> >           Freeze         Approx Release
> >   Giant   Mon Sep  1     Mon Sep 29
> >   Hammer  Mon Jan  4     Mon Feb  2
> > 
> > That gives us another month for Giant, then September to shake out 
> > anything issues.  And then three full months before the Hammer freeze.
> > 
> > What say ye?
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> -- 
> Lo?c Dachary, Artisan Logiciel Libre
> 
> 

Re: giant and hammer dates

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 7223 bytes --]

Hi Sage,

Thanks for taking the time to write this overview of the release cycle tools and their evolutions : I did not realize so much work was going on :-)

Cheers

On 30/07/2014 20:22, Sage Weil wrote:
> On Wed, 30 Jul 2014, Loic Dachary wrote:
>> Hi Sage,
>>
>> From my (biased) point of view, the upside is that it will give me more 
>> time to complete the locally repairable code for Giant ;-). The downside 
>> is that it puts a little less pressure to improve the tools and methods 
>> that make a rapid release cycles possible (i.e. unit tests, bug 
>> tracking, patch acceptance workflow, package building/gitbuilder, 
>> teuthology, pulpito, upgrades testing, ...). In a perfect world Ceph 
>> could sustain a three month release cycle without inconveniencing 
>> anyone. A longer release cycle (five or six months) would encourage even 
>> more complex / bigger changes within a release cycle. It would also 
>> probably encourage Ceph developers to forget about the release process 
>> tools during two or three months and not improve them as they should be.
>>
>> IMHO the test cycle is significantly slowing down the release process 
>> and a faster, more comprehensive test cycle would help a lot.
> 
> No argument here. :)
> 
> I should clarify that this is the "stable release cycle" for the named 
> released.  I still think we should maintain a ~2 week "development release 
> cycle" where we are continuously integrating changes and regularly putting 
> out a usable release.  The 'next' or 'last' branches should be recent and 
> stable starting points for doing any new work so that the integration 
> tests, when run, will reflect bugs in your code and not stuff that was 
> already there.  We've slipped a bit here (0.82 to 0.83 was 5 weeks); this 
> is partly because the release process itself is still pretty expensive in 
> terms of effort and we don't want to eat up more of Alfredo's and Sandon's 
> time than we need to, but it is getting better.
> 
> In any case, the real point of a longer "stable release cycle" is just 
> that there are fewer stable releases in flight that we are backporting 
> fixes too.  In practice, having all of dumpling, emperor, and firefly 
> outstanding hasn't worked particularly well (IMO).  We backport to 
> dumpling and firefly and urge people away from emperor to avoid the 
> cognitive overhead of keeping track of another release.  Going from 3 to 4 
> months means only 3 stable releases per year, which I think is enough...?
> 
>> Each commit should be unit / functional tested within seconds, locally 
>> (see 
>> https://github.com/ceph/ceph/blob/master/src/test/osd/types.cc#L1295 for 
>> instance). It is usually more difficult to diagnose / fix a border case 
>> when it is discovered during integration tests (i.e. teuthology) rather 
>> than with a unit / functional test designed for it. Creating unit tests 
>> is often problematic because some of the code base cannot be easily 
>> isolated. With a continuous effort to re-arrange parts of the code to be 
>> more test friendly, this can eventually be resolved.
>>
>> Every commit proposed to master should be run against the relevant 
>> teuthology suite to help the reviewer. The problem here is that it 
>> requires more resources than what Ceph currently has. Harvesting more 
>> machines, making it possible for people and organizations amicable to 
>> Ceph to easily donate virtual machines could probably help.
> 
> Zack is making good progress on rejiggering the way that teuthology 
> separates the core task locking and task runners from the tasks themselves 
> (which get versioned along with the test suite for firefly, dumpling, 
> etc.).  This is all groundwork to enable the important bits, like pulling 
> machine locking into a single, easy to deploy process, and plugging in 
> different providers (in addition to bare metal and downburst) like 
> OpenStack.  The end goal is to make teuthology much easier to deploy in 
> other environments.  I'm hoping we can get to a place similar to openstack 
> where organizations can hang their CI deployment off the 'upstream' 
> build/CI infrastructure and supplement by running the same suites on 
> different hardware or by adding their own test suites...
> 
>> This deserves a separate discussions but I wanted to expand on what I 
>> meant by "test cycle" and its impact on the release cycle.
> 
> We had a discussion during the G/H CDS about doing an ephemeral 
> 'integration' branch to group things together for full testing by the 
> teuthology test suites that you probably caught.  There was a follow-on 
> internal discussion while you were gone on how to get this rolling and Sam 
> is currently working on a tool to easily build an integration branch 
> merging pending work on a nightly so that it can go through the tests 
> before getting merged into master.  I think this will help.
> 
> We also have our first batch of new hardware ordered inside Red Hat 
> (another ~130 machines) that will expand our testing throughput, and 
> Sandon is working on reclaiming a lot of existing machines that aren't 
> getting put to good use (burnupi) so that we can expand the size of the 
> existing test pool.
> 
> Alfredo recently did some background research on what other projects are 
> doing for CI and releases, and he and Sandon have some work in flight to 
> move some of the bursty release builds into openstack VMs.  Unfortunately 
> nobody has their full bandwidth allocated to improving the state of 
> things, but I think we're making some slow progress.
> 
> sage
> 
> 
>>
>> Cheers
>>
>> On 30/07/2014 05:11, Sage Weil wrote:
>>> We've talked a bit about moving to a ~4 month (instead of 3 month) 
>>> cadence.  I'm still inclined in this direction because it means fewer 
>>> stable releases that we will be maintaining and a longer and (hopefully) 
>>> more productive interval to do real work in between.
>>>
>>> The other key point is that we don't want a repeat of the firefly delay.  
>>> I think we should stay as close to a train model as we can.  If something 
>>> isn't ready by freeze, let it wait for the next cycle.  We shouldn't be 
>>> cramming things in at the end, especially big things.  As a general rule, 
>>> big things should be merged early in the cycle so that we have lots of 
>>> time to shake out the issues that only come out of lots of testing and 
>>> aren't obvious from code review.
>>>
>>> Anyway, how about:
>>>
>>>           Freeze         Approx Release
>>>   Giant   Mon Sep  1     Mon Sep 29
>>>   Hammer  Mon Jan  4     Mon Feb  2
>>>
>>> That gives us another month for Giant, then September to shake out 
>>> anything issues.  And then three full months before the Hammer freeze.
>>>
>>> What say ye?
>>> sage
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> -- 
>> Lo?c Dachary, Artisan Logiciel Libre
>>
>>

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: giant and hammer dates

[Gregory Farnum]

On Tue, Jul 29, 2014 at 7:11 PM, Sage Weil <sweil@redhat.com> wrote:
> We've talked a bit about moving to a ~4 month (instead of 3 month)
> cadence.  I'm still inclined in this direction because it means fewer
> stable releases that we will be maintaining and a longer and (hopefully)
> more productive interval to do real work in between.
>
> The other key point is that we don't want a repeat of the firefly delay.
> I think we should stay as close to a train model as we can.  If something
> isn't ready by freeze, let it wait for the next cycle.  We shouldn't be
> cramming things in at the end, especially big things.  As a general rule,
> big things should be merged early in the cycle so that we have lots of
> time to shake out the issues that only come out of lots of testing and
> aren't obvious from code review.

These two points are sort of opposing. In particular, extending the
release cycle just makes the release seem more important, and
increases the pressure to merge features in one cycle instead of
waiting for the next one. (*Especially* if we continue to maintain
every other named release.)
I continue to prefer a 3-month cycle where we maintain enough merging
discipline that the follow-on one-month shakeout period is something
that the development team largely doesn't have to worry about, because
the code is already working *before* we start it and we're just
uncovering rare and longer-term bugs during that period.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

Re: giant and hammer dates

[Sage Weil]

On Wed, 30 Jul 2014, Gregory Farnum wrote:
> On Tue, Jul 29, 2014 at 7:11 PM, Sage Weil <sweil@redhat.com> wrote:
> > We've talked a bit about moving to a ~4 month (instead of 3 month)
> > cadence.  I'm still inclined in this direction because it means fewer
> > stable releases that we will be maintaining and a longer and (hopefully)
> > more productive interval to do real work in between.
> >
> > The other key point is that we don't want a repeat of the firefly delay.
> > I think we should stay as close to a train model as we can.  If something
> > isn't ready by freeze, let it wait for the next cycle.  We shouldn't be
> > cramming things in at the end, especially big things.  As a general rule,
> > big things should be merged early in the cycle so that we have lots of
> > time to shake out the issues that only come out of lots of testing and
> > aren't obvious from code review.
> 
> These two points are sort of opposing. In particular, extending the
> release cycle just makes the release seem more important, and
> increases the pressure to merge features in one cycle instead of
> waiting for the next one. (*Especially* if we continue to maintain
> every other named release.)

That's a really good point.  However, I think it's not the frequency of 
stabilization periods but the frequency of stable releases that's in 
conflict with the pressure to merge.  If we maintain every other release, 
then every other release will have that pressure--even greater, because 
it'll be 6 months before the next one.  And if we aren't maintaining the 
odd releases for any significant period, I'm not sure its worth naming 
them and confusing users about which release they should be running.

Basically, I think the trade-off is merge pressure vs backporting and 
upgrade testing work...

> I continue to prefer a 3-month cycle where we maintain enough merging
> discipline that the follow-on one-month shakeout period is something
> that the development team largely doesn't have to worry about, because
> the code is already working *before* we start it and we're just
> uncovering rare and longer-term bugs during that period.

I like the idea of doing more frequent stabilization sprints to keep 
things in good order instead of deferring that until the end of the 
release cycle.

sage

Re: giant and hammer dates

[Justin Erenkrantz]

On Wed, Jul 30, 2014 at 12:15 PM, Sage Weil <sweil@redhat.com> wrote:
> Basically, I think the trade-off is merge pressure vs backporting and
> upgrade testing work...

FWIW, over in Subversion (which is about as paranoid as Ceph regarding
data corruption issues and compatibility), we have a 4-week
"stabilization" period after a release candidate for a "minor" release
(we have never issued a new major API release; so API compatibility is
almost always kept). This stabilization period is when the tree
switches over to enforcing prior review on all merges/backports in our
STATUS file.  This is documented at:

http://subversion.apache.org/docs/community-guide/releasing.html#release-stabilization

One particular point is that we will restart the soak (stabilization
period) if something critical is found.  (Alpha/betas are usually
issued about 2 weeks prior to the first release candidate.)

As a downstream user, I'd advocate the same for Ceph - while it's
great to be on a strict release frequency train, I'd strongly prefer
not to have a rehash of the firefly xfs corruption bug.  I would also
advocate that new features should not be backported to anything but
the most recent "named" release - now that firefly is out, dumpling
only gets critical bug fixes - no new features.

Cheers.  -- justin

Re: giant and hammer dates

[Mark Nelson]

On 07/30/2014 09:22 AM, Gregory Farnum wrote:
> On Tue, Jul 29, 2014 at 7:11 PM, Sage Weil <sweil@redhat.com> wrote:
>> We've talked a bit about moving to a ~4 month (instead of 3 month)
>> cadence.  I'm still inclined in this direction because it means fewer
>> stable releases that we will be maintaining and a longer and (hopefully)
>> more productive interval to do real work in between.
>>
>> The other key point is that we don't want a repeat of the firefly delay.
>> I think we should stay as close to a train model as we can.  If something
>> isn't ready by freeze, let it wait for the next cycle.  We shouldn't be
>> cramming things in at the end, especially big things.  As a general rule,
>> big things should be merged early in the cycle so that we have lots of
>> time to shake out the issues that only come out of lots of testing and
>> aren't obvious from code review.
>
> These two points are sort of opposing. In particular, extending the
> release cycle just makes the release seem more important, and
> increases the pressure to merge features in one cycle instead of
> waiting for the next one. (*Especially* if we continue to maintain
> every other named release.)
> I continue to prefer a 3-month cycle where we maintain enough merging
> discipline that the follow-on one-month shakeout period is something
> that the development team largely doesn't have to worry about, because
> the code is already working *before* we start it and we're just
> uncovering rare and longer-term bugs during that period.

Personally I'd prefer a longer testing and bugfix period for named 
releases.  4 months of development plus 2 months of testing. 
Development can easily go several weeks or a month over schedule 
(Personally I believe even good teams can't consistently stop this from 
happening, especially with something as complicated as next generation 
distributed storage!).  Our bugfix/test period gets too crunched as it 
is and our initial named releases feel more like release candidates. 
Soon after release we tend to make a bunch of point releases to fix 
semi-major bugs.  If we are doing that anyway, I think it would be 
better to just extend the test time and really make sure we've devoted a 
solid chunk of time for RC or Beta releases before the named release 
goes out.

I think the short point release cycle does help mitigate the merge 
pressure problem, but only if we are disciplined.  If the point releases 
are getting behind, stuff has to get cut.  6 months isn't that much 
longer to wait than 4 months, especially if people are already waiting 
until the 2nd or 3rd point release before they deploy for production 
(not sure if this is happening, but I suspect it is).

Mark

> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: giant and hammer dates

[Eric Eastman]

> Personally I'd prefer a longer testing and bugfix period for named

> releases.  4 months of development plus 2 months of testing.



I would also like to see a 6 month release scheduled for named 

releases. As someone trying to build a product with the Ceph software, 

I would like to see less, and better QAed, named releases.  I am 

currently skipping every other named release, so I am on a 6 month 

cycle anyways.  I play with the non-named development releases, and 

would like to see them continue, as it allows me to test out the new 

features in my lab before they get into a named release.



Eric