Avocado notes from KVM forum 2019

Avocado notes from KVM forum 2019

[Philippe Mathieu-Daudé]

Hi Cleber,

Here are my notes from talking about Avocado with various people during 
the KVM forum in Lyon last month.

All comments are QEMU oriented.


1) Working offline

Various people complained Avocado requires online access, and they would 
like to use it offline.

   Maintainer workflow example is:

   - run avocado
   - hack QEMU, build
   - git pull
   - build
   - hack QEMU
   (go offline)
   - hack QEMU
   - build
   - run avocado <- FAILS

Failure is because mainstream added new tests, which got pulled in, and 
the user only notice when running avocado again, but offline.
Test example is boot_linux_console.py, which added various tests from 
other subsystems, so the maintainer has to disable the new tests 
manually to be able to run his previous tests.

Expected solution: skip tests when artifact is not available, eventually 
when the --offline option is used


2) Add artifacts manually to the cache

Not all artifacts can be easily downloadable, some are public but 
require the user to accept an End User License Agreement.
Users would like to share their tests with the documentation about 
where/how to download the requisite files (accepting the EULA) to run 
the tests.


2b) Add reference to artifact to the cache

Group of users might share group of files (like NFS storage) and would 
like to use directly their remote read-only files, instead of copying it 
to their home directory.


3) Provide qemu/avocado-qemu Python packages

The mainstream project uses Avocado to test QEMU. Others projects use 
QEMU to test their code, and would like to automatize that using 
Avocado. They usually not rebuild QEMU but use a stable binary from 
distributions. The python classes are not available, so they have to 
clone QEMU to use Avocado (I guess they only need 5 python files).
When running on Continuous Integration, this is overkill, because when 
you clone QEMU you also clone various other submodules.


4) Warn the user when Avocado is too old for the tests

Some users tried Avocado following the examples on the mailing list and 
the one in some commit's descriptions where we simply show "avocado run 
...". They installed the distribution Avocado package and tried and it 
fails for few of them with no obvious reason (the .log file is hard to 
read when you are not custom to). IIUC their distribution provides a 
older Avocado (69?) while we use recent features (72).

We never noticed it because we use 'make check-venv' and do not test the 
distrib Avocado. While we can not test all distribs, we could add a 
version test if the Avocado version is too old, display a friendly 
message to the console (not the logfile).


That it for my notes.

Eduardo/Wainer, are there other topics I forgot?


Regards,

Phil.

Re: Avocado notes from KVM forum 2019

[Eduardo Habkost]

Thank you, Philippe, those are great ideas.  I have copied them
to the Avocado+QEMU Trello board so we don't forget about them:
https://trello.com/b/6Qi1pxVn/avocado-qemu

Additional comments below:

On Mon, Nov 25, 2019 at 01:35:13PM +0100, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> Here are my notes from talking about Avocado with various people during the
> KVM forum in Lyon last month.
> 
> All comments are QEMU oriented.
> 
> 
> 1) Working offline
> 
> Various people complained Avocado requires online access, and they would
> like to use it offline.
> 
>   Maintainer workflow example is:
> 
>   - run avocado
>   - hack QEMU, build
>   - git pull
>   - build
>   - hack QEMU
>   (go offline)
>   - hack QEMU
>   - build
>   - run avocado <- FAILS
> 

Ouch.  This shouldn't happen even with no explicit --offline
option.  Failure to download artifacts shouldn't make tests
report failure.


> Failure is because mainstream added new tests, which got pulled in, and the
> user only notice when running avocado again, but offline.
> Test example is boot_linux_console.py, which added various tests from other
> subsystems, so the maintainer has to disable the new tests manually to be
> able to run his previous tests.
> 
> Expected solution: skip tests when artifact is not available, eventually
> when the --offline option is used
> 
> 
> 2) Add artifacts manually to the cache
> 
> Not all artifacts can be easily downloadable, some are public but require
> the user to accept an End User License Agreement.
> Users would like to share their tests with the documentation about where/how
> to download the requisite files (accepting the EULA) to run the tests.
> 
> 
> 2b) Add reference to artifact to the cache
> 
> Group of users might share group of files (like NFS storage) and would like
> to use directly their remote read-only files, instead of copying it to their
> home directory.

This sounds nice and useful, but I don't know how to make the
interface for this usable.


> 
> 
> 3) Provide qemu/avocado-qemu Python packages
> 
> The mainstream project uses Avocado to test QEMU. Others projects use QEMU
> to test their code, and would like to automatize that using Avocado. They
> usually not rebuild QEMU but use a stable binary from distributions. The
> python classes are not available, so they have to clone QEMU to use Avocado
> (I guess they only need 5 python files).
> When running on Continuous Integration, this is overkill, because when you
> clone QEMU you also clone various other submodules.

I only have one concern, here: I don't think we have the
bandwidth to start maintaining a stable external Python API.
Users of those packages will need to be aware that future
versions of the modules might have incompatible APIs.

> 
> 
> 4) Warn the user when Avocado is too old for the tests
> 
> Some users tried Avocado following the examples on the mailing list and the
> one in some commit's descriptions where we simply show "avocado run ...".

Oops.

> They installed the distribution Avocado package and tried and it fails for
> few of them with no obvious reason (the .log file is hard to read when you
> are not custom to). IIUC their distribution provides a older Avocado (69?)
> while we use recent features (72).
> 
> We never noticed it because we use 'make check-venv' and do not test the
> distrib Avocado. While we can not test all distribs, we could add a version
> test if the Avocado version is too old, display a friendly message to the
> console (not the logfile).

Sounds like a good idea.

> 
> 
> That it for my notes.
> 
> Eduardo/Wainer, are there other topics I forgot?

I don't remember anything specific right now.  Thanks again!

-- 
Eduardo

Re: Avocado notes from KVM forum 2019

[Gerd Hoffmann]

> > 1) Working offline
> > 
> > Various people complained Avocado requires online access, and they would
> > like to use it offline.
> > 
> >   Maintainer workflow example is:
> > 
> >   - run avocado
> >   - hack QEMU, build
> >   - git pull
> >   - build
> >   - hack QEMU
> >   (go offline)
> >   - hack QEMU
> >   - build
> >   - run avocado <- FAILS
> > 
> 
> Ouch.  This shouldn't happen even with no explicit --offline
> option.  Failure to download artifacts shouldn't make tests
> report failure.

Related (and already discussed in the past):  There should be a separate
"downloads artifacts", especially for larger ones which easily fail to
download on slower internet links due to hitting the test timeout while
downloading ...

cheers,
  Gerd

Re: Avocado notes from KVM forum 2019

[Cleber Rosa]

----- Original Message -----
> From: "Gerd Hoffmann" <kraxel@redhat.com>
> To: "Eduardo Habkost" <ehabkost@redhat.com>
> Cc: avocado-devel@redhat.com, "Philippe Mathieu-Daudé" <philmd@redhat.com>, "qemu-devel" <qemu-devel@nongnu.org>,
> "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Cleber Rosa" <crosa@redhat.com>
> Sent: Monday, November 25, 2019 9:15:53 AM
> Subject: Re: Avocado notes from KVM forum 2019
> 
> > > 1) Working offline
> > > 
> > > Various people complained Avocado requires online access, and they would
> > > like to use it offline.
> > > 
> > >   Maintainer workflow example is:
> > > 
> > >   - run avocado
> > >   - hack QEMU, build
> > >   - git pull
> > >   - build
> > >   - hack QEMU
> > >   (go offline)
> > >   - hack QEMU
> > >   - build
> > >   - run avocado <- FAILS
> > > 
> > 
> > Ouch.  This shouldn't happen even with no explicit --offline
> > option.  Failure to download artifacts shouldn't make tests
> > report failure.
> 
> Related (and already discussed in the past):  There should be a separate
> "downloads artifacts", especially for larger ones which easily fail to
> download on slower internet links due to hitting the test timeout while
> downloading ...
> 

Hi Gerd,

We listened to you :) and that's an Avocado 73.0 feature (released Friday,
very first bullet point item):

  https://avocado-framework.readthedocs.io/en/73.0/releases/73_0.html

It doesn't cover all use cases, but works as expected for QEMU.  I'm just
waiting the end of the code freeze to bump the Avocado version and have
that in by default.

Thanks!
- Cleber.

> cheers,
>   Gerd
> 
> 
> 

Re: Avocado notes from KVM forum 2019

[Cleber Rosa]

On Mon, Nov 25, 2019 at 10:58:02AM -0300, Eduardo Habkost wrote:
> Thank you, Philippe, those are great ideas.  I have copied them
> to the Avocado+QEMU Trello board so we don't forget about them:
> https://trello.com/b/6Qi1pxVn/avocado-qemu
> 
> Additional comments below:
> 
> On Mon, Nov 25, 2019 at 01:35:13PM +0100, Philippe Mathieu-Daudé wrote:
> > Hi Cleber,
> > 
> > Here are my notes from talking about Avocado with various people during the
> > KVM forum in Lyon last month.
> > 
> > All comments are QEMU oriented.
> > 
> > 
> > 1) Working offline
> > 
> > Various people complained Avocado requires online access, and they would
> > like to use it offline.
> > 
> >   Maintainer workflow example is:
> > 
> >   - run avocado
> >   - hack QEMU, build
> >   - git pull
> >   - build
> >   - hack QEMU
> >   (go offline)
> >   - hack QEMU
> >   - build
> >   - run avocado <- FAILS
> > 
> 
> Ouch.  This shouldn't happen even with no explicit --offline
> option.  Failure to download artifacts shouldn't make tests
> report failure.
> 
>

Agreed.  There are a number of work items already to cover this.  One
is a more generic test metadata collection system:

   https://trello.com/c/lumR8u8Y/1526-rfc-nrunner-extended-metadata

We already have code that can find the required assets, and with that,
we can let the job (not the test) attempt to fulfill those
requirements, skipping the tests if they can not be fulfilled.

Until this is available, we can wrap the "fetch_asset()" calls and
cancel the test if it fails.

> > Failure is because mainstream added new tests, which got pulled in, and the
> > user only notice when running avocado again, but offline.
> > Test example is boot_linux_console.py, which added various tests from other
> > subsystems, so the maintainer has to disable the new tests manually to be
> > able to run his previous tests.
> > 
> > Expected solution: skip tests when artifact is not available, eventually
> > when the --offline option is used
> > 
> > 
> > 2) Add artifacts manually to the cache
> > 
> > Not all artifacts can be easily downloadable, some are public but require
> > the user to accept an End User License Agreement.
> > Users would like to share their tests with the documentation about where/how
> > to download the requisite files (accepting the EULA) to run the tests.
> > 
> > 
> > 2b) Add reference to artifact to the cache
> > 
> > Group of users might share group of files (like NFS storage) and would like
> > to use directly their remote read-only files, instead of copying it to their
> > home directory.
> 
> This sounds nice and useful, but I don't know how to make the
> interface for this usable.
> 
>

I guess this would require an Avocado installation-wide configuration
entry for the available cache directories.  IMO given that
configuration is applied, the tests should transparently find assets
in the configured locations.

> > 
> > 
> > 3) Provide qemu/avocado-qemu Python packages
> > 
> > The mainstream project uses Avocado to test QEMU. Others projects use QEMU
> > to test their code, and would like to automatize that using Avocado. They
> > usually not rebuild QEMU but use a stable binary from distributions. The
> > python classes are not available, so they have to clone QEMU to use Avocado
> > (I guess they only need 5 python files).
> > When running on Continuous Integration, this is overkill, because when you
> > clone QEMU you also clone various other submodules.
> 
> I only have one concern, here: I don't think we have the
> bandwidth to start maintaining a stable external Python API.
> Users of those packages will need to be aware that future
> versions of the modules might have incompatible APIs.
>

My understanding is that we would publish those files as a Python
module with versions matching QEMU.  No stability would be promised.
Users can always require a specific version of the Python module that
matches the QEMU version they expect/want to use.

> > 
> > 
> > 4) Warn the user when Avocado is too old for the tests
> > 
> > Some users tried Avocado following the examples on the mailing list and the
> > one in some commit's descriptions where we simply show "avocado run ...".
> 
> Oops.
>
> > They installed the distribution Avocado package and tried and it fails for
> > few of them with no obvious reason (the .log file is hard to read when you
> > are not custom to). IIUC their distribution provides a older Avocado (69?)
> > while we use recent features (72).
> > 
> > We never noticed it because we use 'make check-venv' and do not test the
> > distrib Avocado. While we can not test all distribs, we could add a version
> > test if the Avocado version is too old, display a friendly message to the
> > console (not the logfile).
> 
> Sounds like a good idea.
>

A simpler (complementary?) solution, or maybe just a good practice, is
to try to use in the examples:

  "./tests/venv/bin/avocado run ..."

Do you think this would be enough?  It would of course not cover the
examples in previous commit messages.

Thanks!
- Cleber.

> > 
> > 
> > That it for my notes.
> > 
> > Eduardo/Wainer, are there other topics I forgot?
> 
> I don't remember anything specific right now.  Thanks again!
> 
> -- 
> Eduardo

Re: Avocado notes from KVM forum 2019

[Willian Rampazzo]

On Mon, Nov 25, 2019 at 3:10 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> On Mon, Nov 25, 2019 at 10:58:02AM -0300, Eduardo Habkost wrote:
> > Thank you, Philippe, those are great ideas.  I have copied them
> > to the Avocado+QEMU Trello board so we don't forget about them:
> > https://trello.com/b/6Qi1pxVn/avocado-qemu
> >
> > Additional comments below:
> >
> > On Mon, Nov 25, 2019 at 01:35:13PM +0100, Philippe Mathieu-Daudé wrote:
> > > Hi Cleber,
> > >
> > > Here are my notes from talking about Avocado with various people during the
> > > KVM forum in Lyon last month.
> > >
> > > All comments are QEMU oriented.
> > >
> > >
> > > 1) Working offline
> > >
> > > Various people complained Avocado requires online access, and they would
> > > like to use it offline.
> > >
> > >   Maintainer workflow example is:
> > >
> > >   - run avocado
> > >   - hack QEMU, build
> > >   - git pull
> > >   - build
> > >   - hack QEMU
> > >   (go offline)
> > >   - hack QEMU
> > >   - build
> > >   - run avocado <- FAILS
> > >
> >
> > Ouch.  This shouldn't happen even with no explicit --offline
> > option.  Failure to download artifacts shouldn't make tests
> > report failure.
> >
> >
>
> Agreed.  There are a number of work items already to cover this.  One
> is a more generic test metadata collection system:
>
>    https://trello.com/c/lumR8u8Y/1526-rfc-nrunner-extended-metadata
>
> We already have code that can find the required assets, and with that,
> we can let the job (not the test) attempt to fulfill those
> requirements, skipping the tests if they can not be fulfilled.
>
> Until this is available, we can wrap the "fetch_asset()" calls and
> cancel the test if it fails.
>
> > > Failure is because mainstream added new tests, which got pulled in, and the
> > > user only notice when running avocado again, but offline.
> > > Test example is boot_linux_console.py, which added various tests from other
> > > subsystems, so the maintainer has to disable the new tests manually to be
> > > able to run his previous tests.
> > >
> > > Expected solution: skip tests when artifact is not available, eventually
> > > when the --offline option is used
> > >
> > >
> > > 2) Add artifacts manually to the cache
> > >
> > > Not all artifacts can be easily downloadable, some are public but require
> > > the user to accept an End User License Agreement.
> > > Users would like to share their tests with the documentation about where/how
> > > to download the requisite files (accepting the EULA) to run the tests.
> > >
> > >
> > > 2b) Add reference to artifact to the cache
> > >
> > > Group of users might share group of files (like NFS storage) and would like
> > > to use directly their remote read-only files, instead of copying it to their
> > > home directory.
> >
> > This sounds nice and useful, but I don't know how to make the
> > interface for this usable.
> >
> >
>
> I guess this would require an Avocado installation-wide configuration
> entry for the available cache directories.  IMO given that
> configuration is applied, the tests should transparently find assets
> in the configured locations.
>
> > >
> > >
> > > 3) Provide qemu/avocado-qemu Python packages
> > >
> > > The mainstream project uses Avocado to test QEMU. Others projects use QEMU
> > > to test their code, and would like to automatize that using Avocado. They
> > > usually not rebuild QEMU but use a stable binary from distributions. The
> > > python classes are not available, so they have to clone QEMU to use Avocado
> > > (I guess they only need 5 python files).
> > > When running on Continuous Integration, this is overkill, because when you
> > > clone QEMU you also clone various other submodules.
> >
> > I only have one concern, here: I don't think we have the
> > bandwidth to start maintaining a stable external Python API.
> > Users of those packages will need to be aware that future
> > versions of the modules might have incompatible APIs.
> >
>
> My understanding is that we would publish those files as a Python
> module with versions matching QEMU.  No stability would be promised.
> Users can always require a specific version of the Python module that
> matches the QEMU version they expect/want to use.
>
> > >
> > >
> > > 4) Warn the user when Avocado is too old for the tests
> > >
> > > Some users tried Avocado following the examples on the mailing list and the
> > > one in some commit's descriptions where we simply show "avocado run ...".
> >
> > Oops.
> >
> > > They installed the distribution Avocado package and tried and it fails for
> > > few of them with no obvious reason (the .log file is hard to read when you
> > > are not custom to). IIUC their distribution provides a older Avocado (69?)
> > > while we use recent features (72).
> > >
> > > We never noticed it because we use 'make check-venv' and do not test the
> > > distrib Avocado. While we can not test all distribs, we could add a version
> > > test if the Avocado version is too old, display a friendly message to the
> > > console (not the logfile).
> >
> > Sounds like a good idea.
> >
>
> A simpler (complementary?) solution, or maybe just a good practice, is
> to try to use in the examples:
>
>   "./tests/venv/bin/avocado run ..."
>
> Do you think this would be enough?  It would of course not cover the
> examples in previous commit messages.
>
> Thanks!
> - Cleber.
>
> > >
> > >
> > > That it for my notes.
> > >
> > > Eduardo/Wainer, are there other topics I forgot?
> >
> > I don't remember anything specific right now.  Thanks again!
> >
> > --
> > Eduardo
>
>

Following up with this discussion, I'm gathering requirements for the
asset management architecture on Avocado.

So far, these are the use cases I could gather from e-mail
discussions, Avocado meetings, and individual talks followed by the
Trello board cards I found that may be related:

- Ability to skip a test when an asset is not available [4];
- Download the assets previous to the test start (covered on version
73 of Avocado):
    - Using the command line [5];
    - Or transparently to the test execution, during the pre-job
execution phase [6];
- Manually add an asset to the cache [8]:
    - By copying the asset to the cache;
    - By adding a reference to the cache, or maybe using a location
other than the cache;
- Reference an asset globally, like a module object accessible by
different tests [9];
- Collect assets from a test [1,3]

Following are the Trello Board cards I found on Avocado and Avocado +
QEMU boards related to assets:

Avocado Trello Board:

- Backlog:

[1] Add support to specify assets in test docstring -
https://trello.com/c/WPd4FrIy/1479-add-support-to-specify-assets-in-test-docstring
[2] Asset Fetcher: avoid recalculation of cached hash and locking
errors - https://trello.com/c/WdnjY4n7/1528-asset-fetcher-avoid-recalculation-of-cached-hash-and-locking-errors
[3] RFC: NRunner: extended metadata -
https://trello.com/c/lumR8u8Y/1526-rfc-nrunner-extended-metadata

- Work in progress:

[4] On cache check for asset fetcher -
https://trello.com/c/CKP7YS6G/1481-on-cache-check-for-asset-fetcher

- Completed:

[5] Implement fetch-assets command line -
https://trello.com/c/T3SC1sZs/1521-implement-fetch-assets-command-line
[6] Extend Assets plugin to run the fetch during JobPreTests -
https://trello.com/c/jcqChIqN/1531-extend-assets-plugin-to-run-the-fetch-during-jobpretests


Avocado + QEMU Trello Board:

- Backlog:

[7] Mirror fedora/debian images on
https://avocado-project.org/data/assets/ -
https://trello.com/c/vY8eg0eZ/102-mirror-fedora-debian-images-on-https-avocado-projectorg-data-assets
[8] ability to add artifacts manually -
https://trello.com/c/czLc1Ouc/114-ability-to-add-artifacts-manually
[9] asset_library - https://trello.com/c/gJsXJz57/120-assetlibrary
[10] CI jobs shouldn't fail if image download fails -
https://trello.com/c/gJyVHCz2/122-ci-jobs-shouldnt-fail-if-image-download-fails
[11] offline mode - https://trello.com/c/DRnMsEEZ/113-offline-mode

My objective is to create a parent tracking card on the Avocado Trello
board listing individual task-related cards mapped to the use case
requirements. As a start point, I will create the card tree with the
use cases I gathered and, later, add any other missing use cases that
we cover in this thread.

Any confirmation of use cases or proposals of other use cases is
appreciated to improve Avocado's ability to run your tests.

Re: Avocado notes from KVM forum 2019

[Cleber Rosa]

On Mon, Nov 25, 2019 at 01:35:13PM +0100, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> Here are my notes from talking about Avocado with various people during the
> KVM forum in Lyon last month.
> 
> All comments are QEMU oriented.
> 
> 
> 1) Working offline
> 
> Various people complained Avocado requires online access, and they would
> like to use it offline.
>

Just to be extra clear, Avocado itself doesn't require online access, but:

 * "make check-venv" will download packages from PyPI if the same
   Avocado version is not previously installed in the system

 * "make check-acceptance" depends on "make check-venv"

 * many tests depend on downlodable assets

>   Maintainer workflow example is:
> 
>   - run avocado
>   - hack QEMU, build
>   - git pull
>   - build
>   - hack QEMU
>   (go offline)
>   - hack QEMU
>   - build
>   - run avocado <- FAILS
> 
> Failure is because mainstream added new tests, which got pulled in, and the
> user only notice when running avocado again, but offline.
> Test example is boot_linux_console.py, which added various tests from other
> subsystems, so the maintainer has to disable the new tests manually to be
> able to run his previous tests.
> 
> Expected solution: skip tests when artifact is not available, eventually
> when the --offline option is used
>

Understood and agreed.  I've already adopted this approach in the
"boot_linux.py" series I'm working (about to send v8).  If the
download/preparation of images fail, we cancel the tests.  I'll look
into making that the default across all tests or in the base
avocado_qemu.Test class.

We could also have a "make check-acceptance-prepare" kind of target,
that won't run the tests, but will download all needed assets.

> 
> 2) Add artifacts manually to the cache
> 
> Not all artifacts can be easily downloadable, some are public but require
> the user to accept an End User License Agreement.
> Users would like to share their tests with the documentation about where/how
> to download the requisite files (accepting the EULA) to run the tests.
> 
>

OK, RFE taken.

> 2b) Add reference to artifact to the cache
> 
> Group of users might share group of files (like NFS storage) and would like
> to use directly their remote read-only files, instead of copying it to their
> home directory.
> 
>

The underlying "asset fetcher" utility code supports multiple locations
and multiple cache dirs (and one could be a read-only NFS-like storage):

  https://avocado-framework.readthedocs.io/en/73.0/api/utils/avocado.utils.html#avocado.utils.asset.Asset

But we need to make that easily accessible/configurable to users
of the fetch_asset() Test method.  RFE taken.

> 3) Provide qemu/avocado-qemu Python packages
> 
> The mainstream project uses Avocado to test QEMU. Others projects use QEMU
> to test their code, and would like to automatize that using Avocado. They
> usually not rebuild QEMU but use a stable binary from distributions. The
> python classes are not available, so they have to clone QEMU to use Avocado
> (I guess they only need 5 python files).
> When running on Continuous Integration, this is overkill, because when you
> clone QEMU you also clone various other submodules.
>

Agreed, and I already have a prototype.  I'll send the RFC/patches to
the list ASAP.

> 
> 4) Warn the user when Avocado is too old for the tests
> 
> Some users tried Avocado following the examples on the mailing list and the
> one in some commit's descriptions where we simply show "avocado run ...".
> They installed the distribution Avocado package and tried and it fails for
> few of them with no obvious reason (the .log file is hard to read when you
> are not custom to). IIUC their distribution provides a older Avocado (69?)
> while we use recent features (72).
> 
> We never noticed it because we use 'make check-venv' and do not test the
> distrib Avocado. While we can not test all distribs, we could add a version
> test if the Avocado version is too old, display a friendly message to the
> console (not the logfile).
>

OK, agreed.  RFE taken.

> 
> That it for my notes.
> 
> Eduardo/Wainer, are there other topics I forgot?
> 
> 
> Regards,
> 
> Phil.
> 

Thanks *so much* for this feedback!  I'll provide individual feedback as
each of those items progresses.

- Cleber.