ceph-disk and 'make'

ceph-disk and 'make'

[Sage Weil]

I noticed that with the new cmake when I do a 'make' the ceph-disk target 
isn't built.  And ctest will fail most of its unit tests.  It looks like 
it is part of the 'tests' target, though.  Is it deliberate that a simple 
'make' doesn't build everything?

Thanks!
sage

Re: ceph-disk and 'make'

[Loic Dachary]

On 19/09/2016 02:40, Sage Weil wrote:
> I noticed that with the new cmake when I do a 'make' the ceph-disk target 
> isn't built.  And ctest will fail most of its unit tests.  It looks like 
> it is part of the 'tests' target, though.  Is it deliberate that a simple 
> 'make' doesn't build everything?

I think make should build it. I'll look into it.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: ceph-disk and 'make'

[Loic Dachary]

After running

./install-deps.sh
mkdir build
cd build
cmake ../
make

I have

loic@a64e7ff3b216:~/ceph-ubuntu-14.04-loic/build$ ls -l src/ceph-disk
ls -l src/ceph-disk
total 28
drwxr-xr-x 4 loic loic 4096 Sep 19 05:57 CMakeFiles
-rw-r--r-- 1 loic loic  313 Sep 19 05:57 CTestTestfile.cmake
-rw-r--r-- 1 loic loic 7240 Sep 19 05:57 Makefile
lrwxrwxrwx 1 loic loic   58 Sep 19 06:00 README.rst -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/README.rst
lrwxrwxrwx 1 loic loic   57 Sep 19 06:00 ceph_disk -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/ceph_disk
-rw-r--r-- 1 loic loic 1878 Sep 19 05:57 cmake_install.cmake
lrwxrwxrwx 1 loic loic   64 Sep 19 06:00 requirements.txt -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/requirements.txt
lrwxrwxrwx 1 loic loic   56 Sep 19 06:00 setup.py -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/setup.py
lrwxrwxrwx 1 loic loic   69 Sep 19 06:00 test-requirements.txt -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/test-requirements.txt

which I interpret as make building ceph-disk. Did you expect something else ?

Cheers

On 19/09/2016 07:54, Loic Dachary wrote:
> 
> 
> On 19/09/2016 02:40, Sage Weil wrote:
>> I noticed that with the new cmake when I do a 'make' the ceph-disk target 
>> isn't built.  And ctest will fail most of its unit tests.  It looks like 
>> it is part of the 'tests' target, though.  Is it deliberate that a simple 
>> 'make' doesn't build everything?
> 
> I think make should build it. I'll look into it.
> 
> Cheers
> 

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: ceph-disk and 'make'

[Gregory Farnum]

On Sun, Sep 18, 2016 at 5:40 PM, Sage Weil <sweil@redhat.com> wrote:
> I noticed that with the new cmake when I do a 'make' the ceph-disk target
> isn't built.  And ctest will fail most of its unit tests.  It looks like
> it is part of the 'tests' target, though.  Is it deliberate that a simple
> 'make' doesn't build everything?

Dunno about this specific case, but in general we don't want
*everything* to be built with a simple "make", since it slows down
compile (and, especially, linking). I don't think ceph-disk is used
during development for anything except the tests so this would make
some sense, right?

On Mon, Sep 19, 2016 at 12:35 AM, Loic Dachary <loic@dachary.org> wrote:
> After running
>
> ./install-deps.sh
> mkdir build
> cd build
> cmake ../
> make
>
> I have
>
> loic@a64e7ff3b216:~/ceph-ubuntu-14.04-loic/build$ ls -l src/ceph-disk
> ls -l src/ceph-disk
> total 28
> drwxr-xr-x 4 loic loic 4096 Sep 19 05:57 CMakeFiles
> -rw-r--r-- 1 loic loic  313 Sep 19 05:57 CTestTestfile.cmake
> -rw-r--r-- 1 loic loic 7240 Sep 19 05:57 Makefile
> lrwxrwxrwx 1 loic loic   58 Sep 19 06:00 README.rst -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/README.rst
> lrwxrwxrwx 1 loic loic   57 Sep 19 06:00 ceph_disk -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/ceph_disk
> -rw-r--r-- 1 loic loic 1878 Sep 19 05:57 cmake_install.cmake
> lrwxrwxrwx 1 loic loic   64 Sep 19 06:00 requirements.txt -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/requirements.txt
> lrwxrwxrwx 1 loic loic   56 Sep 19 06:00 setup.py -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/setup.py
> lrwxrwxrwx 1 loic loic   69 Sep 19 06:00 test-requirements.txt -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/test-requirements.txt
>
> which I interpret as make building ceph-disk. Did you expect something else ?

Shouldn't any "built" stuff in cmake be going in the build directory
rather than the ceph/src one?
-Greg

Re: ceph-disk and 'make'

[kefu chai]

On Tue, Sep 20, 2016 at 2:30 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Sun, Sep 18, 2016 at 5:40 PM, Sage Weil <sweil@redhat.com> wrote:
>> I noticed that with the new cmake when I do a 'make' the ceph-disk target
>> isn't built.  And ctest will fail most of its unit tests.  It looks like
>> it is part of the 'tests' target, though.  Is it deliberate that a simple
>> 'make' doesn't build everything?

Sage, it is on purpose. "make" or "make all" should only build the
artifacts supposed
to be installed. this was pointed out by Marcus. and i am inclined to agree with
him on this.

>
> Dunno about this specific case, but in general we don't want
> *everything* to be built with a simple "make", since it slows down
> compile (and, especially, linking). I don't think ceph-disk is used
> during development for anything except the tests so this would make
> some sense, right?

agreed, actually, a test-ready ceph-disk needs venv which pip to install the
dependencies, but debian pbuilder uses a cleanroom for building packages
in which no network access is allowed by default. so if we have builded
ceph-disk while "make", pbuilder will have to enable the network access
from within the cleanroom. there is a bug filed against jewel on this
[1] because
we always have ceph-disk venv as a dependency of "make all".

--
[1] http://tracker.ceph.com/issues/16169

>
> On Mon, Sep 19, 2016 at 12:35 AM, Loic Dachary <loic@dachary.org> wrote:
>> After running
>>
>> ./install-deps.sh
>> mkdir build
>> cd build
>> cmake ../
>> make
>>
>> I have
>>
>> loic@a64e7ff3b216:~/ceph-ubuntu-14.04-loic/build$ ls -l src/ceph-disk
>> ls -l src/ceph-disk
>> total 28
>> drwxr-xr-x 4 loic loic 4096 Sep 19 05:57 CMakeFiles
>> -rw-r--r-- 1 loic loic  313 Sep 19 05:57 CTestTestfile.cmake
>> -rw-r--r-- 1 loic loic 7240 Sep 19 05:57 Makefile
>> lrwxrwxrwx 1 loic loic   58 Sep 19 06:00 README.rst -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/README.rst
>> lrwxrwxrwx 1 loic loic   57 Sep 19 06:00 ceph_disk -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/ceph_disk
>> -rw-r--r-- 1 loic loic 1878 Sep 19 05:57 cmake_install.cmake
>> lrwxrwxrwx 1 loic loic   64 Sep 19 06:00 requirements.txt -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/requirements.txt
>> lrwxrwxrwx 1 loic loic   56 Sep 19 06:00 setup.py -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/setup.py
>> lrwxrwxrwx 1 loic loic   69 Sep 19 06:00 test-requirements.txt -> /home/loic/ceph-ubuntu-14.04-loic/src/ceph-disk/test-requirements.txt
>>
>> which I interpret as make building ceph-disk. Did you expect something else ?
>
> Shouldn't any "built" stuff in cmake be going in the build directory
> rather than the ceph/src one?
> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Regards
Kefu Chai

Re: ceph-disk and 'make'

[Ken Dreyer]

On Tue, Sep 20, 2016 at 9:28 AM, kefu chai <tchaikov@gmail.com> wrote:
> agreed, actually, a test-ready ceph-disk needs venv which pip to install the
> dependencies, but debian pbuilder uses a cleanroom for building packages
> in which no network access is allowed by default.

I'm curious why we need a virtualenv at all (or why we "pip install" anything) ?

The distros should have all the packaged python-* and python3-*
dependencies already, right?

If we could switch to using the packaged python dependencies for
everything, we'd fix the pbuilder USENETWORK issue, and avoid the
indeterminate nature of depending on whatever happens to be going on
with pypi. Maybe I'm missing something, though?

- Ken

Re: ceph-disk and 'make'

[Dan Mick]

On 09/19/2016 11:30 AM, Gregory Farnum wrote:
> On Sun, Sep 18, 2016 at 5:40 PM, Sage Weil <sweil@redhat.com> wrote:
>> I noticed that with the new cmake when I do a 'make' the ceph-disk target
>> isn't built.  And ctest will fail most of its unit tests.  It looks like
>> it is part of the 'tests' target, though.  Is it deliberate that a simple
>> 'make' doesn't build everything?
> 
> Dunno about this specific case, but in general we don't want
> *everything* to be built with a simple "make", since it slows down
> compile (and, especially, linking). I don't think ceph-disk is used
> during development for anything except the tests so this would make
> some sense, right?

I think I disagree; make -> make all-> make everything is certainly my
assumption.  There's no reason not to have other targets that build
less, but I expect "make" to default to "build everything that's
configured".  (That doesn't include tests, and doesn't install anything,
but if it *would be* installed by make install, I expect it to be
constructed.)

Re: ceph-disk and 'make'

[kefu chai]

On Wed, Sep 21, 2016 at 2:04 AM, Ken Dreyer <kdreyer@redhat.com> wrote:
> On Tue, Sep 20, 2016 at 9:28 AM, kefu chai <tchaikov@gmail.com> wrote:
>> agreed, actually, a test-ready ceph-disk needs venv which pip to install the
>> dependencies, but debian pbuilder uses a cleanroom for building packages
>> in which no network access is allowed by default.
>
> I'm curious why we need a virtualenv at all (or why we "pip install" anything) ?
>
> The distros should have all the packaged python-* and python3-*
> dependencies already, right?
>
> If we could switch to using the packaged python dependencies for
> everything, we'd fix the pbuilder USENETWORK issue, and avoid the
> indeterminate nature of depending on whatever happens to be going on
> with pypi. Maybe I'm missing something, though?

we could. the python modules installed by pip are only used for
testing purpose, like tox, pytest, flake8, testtools, fixtures, i
think we can add them to the Build-Depends list. and let pbuilder to
take care of them.

@loic what do you think?


>
> - Ken



-- 
Regards
Kefu Chai

Re: ceph-disk and 'make'

[Loic Dachary]

On 21/09/2016 04:39, kefu chai wrote:
> On Wed, Sep 21, 2016 at 2:04 AM, Ken Dreyer <kdreyer@redhat.com> wrote:
>> On Tue, Sep 20, 2016 at 9:28 AM, kefu chai <tchaikov@gmail.com> wrote:
>>> agreed, actually, a test-ready ceph-disk needs venv which pip to install the
>>> dependencies, but debian pbuilder uses a cleanroom for building packages
>>> in which no network access is allowed by default.
>>
>> I'm curious why we need a virtualenv at all (or why we "pip install" anything) ?
>>
>> The distros should have all the packaged python-* and python3-*
>> dependencies already, right?
>>
>> If we could switch to using the packaged python dependencies for
>> everything, we'd fix the pbuilder USENETWORK issue, and avoid the
>> indeterminate nature of depending on whatever happens to be going on
>> with pypi. Maybe I'm missing something, though?
> 
> we could. the python modules installed by pip are only used for
> testing purpose, like tox, pytest, flake8, testtools, fixtures, i
> think we can add them to the Build-Depends list. and let pbuilder to
> take care of them.
> 
> @loic what do you think?

It's worth a shot.

The primary reason for encapsulating this into install-deps.sh was to cope with all supported operating systems with just two package file / directories (I.e. one debian directory with no #if/#else capability and one spec file). More often than not satisfying python requirements with packages is challenging, to say the least (even pip).

That being said, the older operating systems are no longer supported and this may not be an issue right now. And when/if it becomes an issue once again in the future we can probably deal with it in a better way.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre