cscope: issue with symlinks in tools/testing/selftests/powerpc/copyloops/

cscope: issue with symlinks in tools/testing/selftests/powerpc/copyloops/

[Yann Droneaud]

Hi,

I'm using cscope to browse kernel sources, but I'm facing warnings from
the tool since following commit:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=22d651dcef536c75f75537290bf3da5038e68b6b

    commit 22d651dcef536c75f75537290bf3da5038e68b6b
    Author: Michael Ellerman <mpe@ellerman.id.au>
    Date:   Tue Jan 21 15:22:17 2014 +1100

    selftests/powerpc: Import Anton's memcpy / copy_tofrom_user tests
    
    Turn Anton's memcpy / copy_tofrom_user test into something that can
    live in tools/testing/selftests.
    
    It requires one turd in arch/powerpc/lib/memcpy_64.S, but it's 
    pretty harmless IMHO.
    
    We are sailing very close to the wind with the feature macros. We 
    define them to nothing, which currently means we get a few extra 
    nops and include the unaligned calls.
    
    Signed-off-by: Anton Blanchard <anton@samba.org>
    Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
    Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>


cscope reports error when generating the cross-reference database:

    $ make ALLSOURCE_ARCHS=all O=./obj-cscope/ cscope
      GEN     cscope
    cscope: cannot find
file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
    cscope: cannot find
file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_64.S
    cscope: cannot find
file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
    cscope: cannot find
file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_64.S

And when calling cscope from ./obj-cscope/ directory, it reports errors
too.

Hopefully it doesn't stop it from working, so I'm still able to use
cscope to browse kernel sources.

It's a rather uncommon side effect of having (for the first time ?)
sources files as symlinks: looking for symlinks in the kernel sources
returns only:

    $ find . -type l
    ./arch/mips/boot/dts/include/dt-bindings
    ./arch/microblaze/boot/dts/system.dts
    ./arch/powerpc/boot/dts/include/dt-bindings
    ./arch/metag/boot/dts/include/dt-bindings
    ./arch/arm/boot/dts/include/dt-bindings
    ./tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
    ./tools/testing/selftests/powerpc/copyloops/memcpy_64.S
    ./tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
    ./tools/testing/selftests/powerpc/copyloops/copyuser_64.S
    ./obj-cscope/source
    ./Documentation/DocBook/vidioc-g-sliced-vbi-cap.xml
    ./Documentation/DocBook/vidioc-decoder-cmd.xml
...
    ./Documentation/DocBook/media-func-ioctl.xml
    ./Documentation/DocBook/vidioc-enumoutput.xml


So one can wonder if having symlinked sources files is an expected
supported feature for kbuild and all the various kernel
tools/infrastructure ?

Regarding cscope specifically, it does not support symlink, and it's the
expected behavior according to the bug reports I was able to find:

#214 cscope ignores symlinks to files 
http://sourceforge.net/p/cscope/bugs/214/

#229 -I options doesn't handle symbolic link
http://sourceforge.net/p/cscope/bugs/229/

#247 cscope: cannot find file 
http://sourceforge.net/p/cscope/bugs/247/

#252 cscope: cannot find file *** 
http://sourceforge.net/p/cscope/bugs/252/

#261 Regression - version 15.7a does not follow symbolic links 
http://sourceforge.net/p/cscope/bugs/261/


Regards.

-- 
Yann Droneaud
OPTEYA

Re: cscope: issue with symlinks in tools/testing/selftests/powerpc/copyloops/

[Neil Horman]

On Thu, Apr 03, 2014 at 03:16:15PM +0200, Yann Droneaud wrote:
> Hi,
> 
> I'm using cscope to browse kernel sources, but I'm facing warnings from
> the tool since following commit:
> 
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=22d651dcef536c75f75537290bf3da5038e68b6b
> 
>     commit 22d651dcef536c75f75537290bf3da5038e68b6b
>     Author: Michael Ellerman <mpe@ellerman.id.au>
>     Date:   Tue Jan 21 15:22:17 2014 +1100
> 
>     selftests/powerpc: Import Anton's memcpy / copy_tofrom_user tests
>     
>     Turn Anton's memcpy / copy_tofrom_user test into something that can
>     live in tools/testing/selftests.
>     
>     It requires one turd in arch/powerpc/lib/memcpy_64.S, but it's 
>     pretty harmless IMHO.
>     
>     We are sailing very close to the wind with the feature macros. We 
>     define them to nothing, which currently means we get a few extra 
>     nops and include the unaligned calls.
>     
>     Signed-off-by: Anton Blanchard <anton@samba.org>
>     Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
>     Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> 
> 
> cscope reports error when generating the cross-reference database:
> 
>     $ make ALLSOURCE_ARCHS=all O=./obj-cscope/ cscope
>       GEN     cscope
>     cscope: cannot find
> file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
>     cscope: cannot find
> file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_64.S
>     cscope: cannot find
> file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
>     cscope: cannot find
> file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_64.S
> 
> And when calling cscope from ./obj-cscope/ directory, it reports errors
> too.
> 
> Hopefully it doesn't stop it from working, so I'm still able to use
> cscope to browse kernel sources.
> 
No, it won't stop it from working, it just won't search those files.  I don't
recall exactly the reason, but IIRC there was a big discussion long ago about
symlinks and our ability to support them (around version 1.94 I think).  We
decided to not handle symlinks, as they would either point outside our search
tree, which we didn't want to include, or would point to another file in the
search tree, which made loading them pointless (as we would cover the search in
the pointed file).

Neil

> It's a rather uncommon side effect of having (for the first time ?)
> sources files as symlinks: looking for symlinks in the kernel sources
> returns only:
> 
>     $ find . -type l
>     ./arch/mips/boot/dts/include/dt-bindings
>     ./arch/microblaze/boot/dts/system.dts
>     ./arch/powerpc/boot/dts/include/dt-bindings
>     ./arch/metag/boot/dts/include/dt-bindings
>     ./arch/arm/boot/dts/include/dt-bindings
>     ./tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
>     ./tools/testing/selftests/powerpc/copyloops/memcpy_64.S
>     ./tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
>     ./tools/testing/selftests/powerpc/copyloops/copyuser_64.S
>     ./obj-cscope/source
>     ./Documentation/DocBook/vidioc-g-sliced-vbi-cap.xml
>     ./Documentation/DocBook/vidioc-decoder-cmd.xml
> ...
>     ./Documentation/DocBook/media-func-ioctl.xml
>     ./Documentation/DocBook/vidioc-enumoutput.xml
> 
> 
> So one can wonder if having symlinked sources files is an expected
> supported feature for kbuild and all the various kernel
> tools/infrastructure ?
> 
> Regarding cscope specifically, it does not support symlink, and it's the
> expected behavior according to the bug reports I was able to find:
> 
> #214 cscope ignores symlinks to files 
> http://sourceforge.net/p/cscope/bugs/214/
> 
> #229 -I options doesn't handle symbolic link
> http://sourceforge.net/p/cscope/bugs/229/
> 
> #247 cscope: cannot find file 
> http://sourceforge.net/p/cscope/bugs/247/
> 
> #252 cscope: cannot find file *** 
> http://sourceforge.net/p/cscope/bugs/252/
> 
> #261 Regression - version 15.7a does not follow symbolic links 
> http://sourceforge.net/p/cscope/bugs/261/
> 
> 
> Regards.
> 
> -- 
> Yann Droneaud
> OPTEYA
> 
> 
> 

Re: cscope: issue with symlinks in tools/testing/selftests/powerpc/copyloops/

[Gerhard Sittig]

On Mon, 2014-04-07 at 06:42 -0400, Neil Horman wrote:
> 
> On Thu, Apr 03, 2014 at 03:16:15PM +0200, Yann Droneaud wrote:
> > 
> > [ ... ]
> > 
> > cscope reports error when generating the cross-reference database:
> > 
> >     $ make ALLSOURCE_ARCHS=all O=./obj-cscope/ cscope
> >       GEN     cscope
> >     cscope: cannot find
> > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
> >     cscope: cannot find
> > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_64.S
> >     cscope: cannot find
> > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
> >     cscope: cannot find
> > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_64.S
> > 
> > And when calling cscope from ./obj-cscope/ directory, it reports errors
> > too.
> > 
> > Hopefully it doesn't stop it from working, so I'm still able to use
> > cscope to browse kernel sources.
> > 
> No, it won't stop it from working, it just won't search those files.  I don't
> recall exactly the reason, but IIRC there was a big discussion long ago about
> symlinks and our ability to support them (around version 1.94 I think).  We
> decided to not handle symlinks, as they would either point outside our search
> tree, which we didn't want to include, or would point to another file in the
> search tree, which made loading them pointless (as we would cover the search in
> the pointed file).

So there are valid reasons to not process those filesystem
entries.  Would it be useful to not emit the warnings then?  Or
to silent those warnings when the user knows it's perfectly legal
to skip those filesytem entries?  Like what you can do with the
ctags(1) command and its --links option.


virtually yours
Gerhard Sittig
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr. 5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office@denx.de

Re: cscope: issue with symlinks in tools/testing/selftests/powerpc/copyloops/

[Neil Horman]

On Mon, Apr 07, 2014 at 02:42:59PM +0200, Gerhard Sittig wrote:
> On Mon, 2014-04-07 at 06:42 -0400, Neil Horman wrote:
> > 
> > On Thu, Apr 03, 2014 at 03:16:15PM +0200, Yann Droneaud wrote:
> > > 
> > > [ ... ]
> > > 
> > > cscope reports error when generating the cross-reference database:
> > > 
> > >     $ make ALLSOURCE_ARCHS=all O=./obj-cscope/ cscope
> > >       GEN     cscope
> > >     cscope: cannot find
> > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
> > >     cscope: cannot find
> > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_64.S
> > >     cscope: cannot find
> > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
> > >     cscope: cannot find
> > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_64.S
> > > 
> > > And when calling cscope from ./obj-cscope/ directory, it reports errors
> > > too.
> > > 
> > > Hopefully it doesn't stop it from working, so I'm still able to use
> > > cscope to browse kernel sources.
> > > 
> > No, it won't stop it from working, it just won't search those files.  I don't
> > recall exactly the reason, but IIRC there was a big discussion long ago about
> > symlinks and our ability to support them (around version 1.94 I think).  We
> > decided to not handle symlinks, as they would either point outside our search
> > tree, which we didn't want to include, or would point to another file in the
> > search tree, which made loading them pointless (as we would cover the search in
> > the pointed file).
> 
> So there are valid reasons to not process those filesystem
> entries.  Would it be useful to not emit the warnings then?  Or
> to silent those warnings when the user knows it's perfectly legal
> to skip those filesytem entries?  Like what you can do with the
> ctags(1) command and its --links option.
> 
I would see no problem with an option to do that.  I'd like to make it opt-in,
so that people who want to know about symlink issues will still see them, but
I'd be supportive of an option to quiet them
Neil

> 
> virtually yours
> Gerhard Sittig
> -- 
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich, Office: Kirchenstr. 5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office@denx.de
> 

Re: cscope: issue with symlinks in tools/testing/selftests/powerpc/copyloops/

[Gerhard Sittig]

[ removed cscope-devel from Cc:, non-subscriber mails get blocked anyway ]

On Mon, 2014-04-07 at 14:42 +0200, Gerhard Sittig wrote:
> 
> On Mon, 2014-04-07 at 06:42 -0400, Neil Horman wrote:
> > 
> > On Thu, Apr 03, 2014 at 03:16:15PM +0200, Yann Droneaud wrote:
> > > 
> > > [ ... ]
> > > 
> > > cscope reports error when generating the cross-reference database:
> > > 
> > >     $ make ALLSOURCE_ARCHS=all O=./obj-cscope/ cscope
> > >       GEN     cscope
> > >     cscope: cannot find
> > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
> > >     cscope: cannot find
> > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_64.S
> > >     cscope: cannot find
> > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
> > >     cscope: cannot find
> > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_64.S
> > > 
> > > And when calling cscope from ./obj-cscope/ directory, it reports errors
> > > too.
> > > 
> > > Hopefully it doesn't stop it from working, so I'm still able to use
> > > cscope to browse kernel sources.
> > > 
> > No, it won't stop it from working, it just won't search those files.  I don't
> > recall exactly the reason, but IIRC there was a big discussion long ago about
> > symlinks and our ability to support them (around version 1.94 I think).  We
> > decided to not handle symlinks, as they would either point outside our search
> > tree, which we didn't want to include, or would point to another file in the
> > search tree, which made loading them pointless (as we would cover the search in
> > the pointed file).
> 
> So there are valid reasons to not process those filesystem
> entries.  Would it be useful to not emit the warnings then?  Or
> to silent those warnings when the user knows it's perfectly legal
> to skip those filesytem entries?  Like what you can do with the
> ctags(1) command and its --links option.

For the record:  I got a response "off list" (actually to the
cscope list only which is closed for non-subscribers, while the
Linux related recipients were removed despite the fact that the
issue appears to be in Linux), see
http://article.gmane.org/gmane.comp.programming.tools.cscope.devel/105

This reponse suggests that the issue is not in cscope(1) itself,
but instead is in how the cscope(1) command got invoked.  Which
translates into "the Linux infrastructure does something wrong".

A quick search identifies ./scripts/tags.sh:all_target_sources()
as the spot where symlinks should get filtered out.  Where both
paths of all_target_sources() end up calling all_sources().


virtually yours
Gerhard Sittig
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr. 5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office@denx.de

Re: cscope: issue with symlinks in tools/testing/selftests/powerpc/copyloops/

[Neil Horman]

On Tue, Apr 08, 2014 at 09:56:10AM +0200, Gerhard Sittig wrote:
> [ removed cscope-devel from Cc:, non-subscriber mails get blocked anyway ]
> 
> On Mon, 2014-04-07 at 14:42 +0200, Gerhard Sittig wrote:
> > 
> > On Mon, 2014-04-07 at 06:42 -0400, Neil Horman wrote:
> > > 
> > > On Thu, Apr 03, 2014 at 03:16:15PM +0200, Yann Droneaud wrote:
> > > > 
> > > > [ ... ]
> > > > 
> > > > cscope reports error when generating the cross-reference database:
> > > > 
> > > >     $ make ALLSOURCE_ARCHS=all O=./obj-cscope/ cscope
> > > >       GEN     cscope
> > > >     cscope: cannot find
> > > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
> > > >     cscope: cannot find
> > > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_64.S
> > > >     cscope: cannot find
> > > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
> > > >     cscope: cannot find
> > > > file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_64.S
> > > > 
> > > > And when calling cscope from ./obj-cscope/ directory, it reports errors
> > > > too.
> > > > 
> > > > Hopefully it doesn't stop it from working, so I'm still able to use
> > > > cscope to browse kernel sources.
> > > > 
> > > No, it won't stop it from working, it just won't search those files.  I don't
> > > recall exactly the reason, but IIRC there was a big discussion long ago about
> > > symlinks and our ability to support them (around version 1.94 I think).  We
> > > decided to not handle symlinks, as they would either point outside our search
> > > tree, which we didn't want to include, or would point to another file in the
> > > search tree, which made loading them pointless (as we would cover the search in
> > > the pointed file).
> > 
> > So there are valid reasons to not process those filesystem
> > entries.  Would it be useful to not emit the warnings then?  Or
> > to silent those warnings when the user knows it's perfectly legal
> > to skip those filesytem entries?  Like what you can do with the
> > ctags(1) command and its --links option.
> 
> For the record:  I got a response "off list" (actually to the
> cscope list only which is closed for non-subscribers, while the
> Linux related recipients were removed despite the fact that the
> issue appears to be in Linux), see
> http://article.gmane.org/gmane.comp.programming.tools.cscope.devel/105
> 
I don't agree with Hans here. He's right in that the Linux makefiles could
exclude the symlinks from the index file that it builds, but if cscope were left
to its own devices it would also exclude the assembly files and other code that
it doesn't officially parse, but does more or less ok with.  We can argue all
day weather its ok for Linux to do that, but the facts of the matter is that
people use cscope  to search the linux source tree asm files and all, and while
we could propose that the cscope makefile target filter out symlinks, it seems
IMHO to be more difficult than its worth.  Adding code to silence db build
warnings would be good.

> This reponse suggests that the issue is not in cscope(1) itself,
> but instead is in how the cscope(1) command got invoked.  Which
> translates into "the Linux infrastructure does something wrong".
> 
> A quick search identifies ./scripts/tags.sh:all_target_sources()
> as the spot where symlinks should get filtered out.  Where both
> paths of all_target_sources() end up calling all_sources().
> 
> 
> virtually yours
> Gerhard Sittig
> -- 
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich, Office: Kirchenstr. 5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office@denx.de
> 

Re: cscope: issue with symlinks in tools/testing/selftests/powerpc/copyloops/

[Michael Ellerman]

On Thu, 2014-04-03 at 15:16 +0200, Yann Droneaud wrote:
> Hi,
> 
> I'm using cscope to browse kernel sources, but I'm facing warnings from
> the tool since following commit:
> 
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=22d651dcef536c75f75537290bf3da5038e68b6b
> 
>     commit 22d651dcef536c75f75537290bf3da5038e68b6b
>     Author: Michael Ellerman <mpe@ellerman.id.au>
>     Date:   Tue Jan 21 15:22:17 2014 +1100
> 
>     selftests/powerpc: Import Anton's memcpy / copy_tofrom_user tests

Ooops, sorry.

> cscope reports error when generating the cross-reference database:
> 
>     $ make ALLSOURCE_ARCHS=all O=./obj-cscope/ cscope
>       GEN     cscope
>     cscope: cannot find
> file /home/ydroneaud/src/linux/tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
> 
> It's a rather uncommon side effect of having (for the first time ?)
> sources files as symlinks: looking for symlinks in the kernel sources
> returns only:
> 
>     $ find . -type l
>     ./arch/mips/boot/dts/include/dt-bindings
>     ./arch/microblaze/boot/dts/system.dts
>     ./arch/powerpc/boot/dts/include/dt-bindings
>     ./arch/metag/boot/dts/include/dt-bindings
>     ./arch/arm/boot/dts/include/dt-bindings
>     ./tools/testing/selftests/powerpc/copyloops/copyuser_power7.S
>     ./tools/testing/selftests/powerpc/copyloops/memcpy_64.S
>     ./tools/testing/selftests/powerpc/copyloops/memcpy_power7.S
>     ./tools/testing/selftests/powerpc/copyloops/copyuser_64.S


Right.

I did check that there were other symlinks already in the tree, but you're
correct I seem to be the first clever person to add symlinks to source files.

> So one can wonder if having symlinked sources files is an expected
> supported feature for kbuild and all the various kernel
> tools/infrastructure ?

Kbuild is not involved really, these files are just built with plain Makefiles,
at least from the symlink side.

FWIW ctags seems to cope, that's what I use.

But I didn't think of cscope.


Given you're the only person who's noticed I suspect most other things haven't
broken, fingers crossed :)

cheers