[cip-dev] [RFC] Script to find used sources in the kernel

[cip-dev] [RFC] Script to find used sources in the kernel

[Ben Hutchings]

I've been promising for some time to provide a way to identify which
changes on Linux stable branches are relevant to CIP members, and
conversely which are irrelevant and don't need to be reviewed.

This is a proof-of-concept of the major part of that, which is to
identify which source files are used based on the submitted
configurations.  Once this is working to everyone's satisfaction, it
should be fairly easy to automate the remaining parts, which are (1)
merge the source lists for each branch (4.4, 4.19) and (2) check a set
of patches or commits against a list.

I implemented:

* A script to select the appropriate value of CROSS_COMPILE for a given
  configuration (cross-compile-prefix).
* A script to call the assembler or compiler and collect dependency
  information from the preprocessor into a single file
  (get_used_sources_wrapper).
* A script that builds a kernel from a given configuration file with
  get_used_sources_wrapper?wrapped around the asssembler and compilers.
  This also filters out references to missing files (like firmware)
  from the configuration file.

I tied these together with a Makefile rule that generates a source list
for each of the submitted configurations.  I was then able to generate
source lists for each of them (although it took quite a few hours).

You can find the scripts and lists on the "benhutchings/get-used-
sources" branch of
https://gitlab.com/cip-project/cip-kernel/cip-kernel-config.git

There are a couple of open questions, on which I would like to hear
other's opinions:

* Should the source lists be added to the repository or not?  If they
  are added, then they should not be changed by the standard "all" and
  "clean" targets.
  I tend to think that they should be added, because they take a long
  time to generate and require cross-compilers etc. to be installed.

* If the answer to the above is "yes", should the source lists be
  generated (a) by members, and sent along with the config files, or
  (b) by the kernel maintainers?  Option (a) makes a bit more work for
  members, but option (b) might result in missing build tools due to
  the config filtering mentioned above.

Please feel free to comment on anything else.

Ben.

-- 
Ben Hutchings, Software Developer                ?        Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom

[cip-dev] [RFC] Script to find used sources in the kernel

[Pavel Machek]

Hi!

> There are a couple of open questions, on which I would like to hear
> other's opinions:
> 
> * Should the source lists be added to the repository or not?  If they
>   are added, then they should not be changed by the standard "all" and
>   "clean" targets.
>   I tend to think that they should be added, because they take a long
>   time to generate and require cross-compilers etc. to be installed.

I believe they should go to the repository, because we may want to
manually adjust the lists.

I don't believe "example" configurations we have necessarily have
enabled all the options "final products" may need. If some common
option (vfat?) is not enabled in our configurations, because it is not
hardware specific and not needed for testing, we may still want to
review it, because it is common enough that someone will need it...

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.cip-project.org/pipermail/cip-dev/attachments/20190622/b2c00791/attachment.sig>

[cip-dev] [RFC] Script to find used sources in the kernel

[Ben Hutchings]

On Sat, 2019-06-22 at 22:07 +0200, Pavel Machek wrote:
> Hi!
> 
> > There are a couple of open questions, on which I would like to hear
> > other's opinions:
> > 
> > * Should the source lists be added to the repository or not?  If they
> >   are added, then they should not be changed by the standard "all" and
> >   "clean" targets.
> >   I tend to think that they should be added, because they take a long
> >   time to generate and require cross-compilers etc. to be installed.
> 
> I believe they should go to the repository, because we may want to
> manually adjust the lists.

I've implemented that option.

> I don't believe "example" configurations we have necessarily have
> enabled all the options "final products" may need. If some common
> option (vfat?) is not enabled in our configurations, because it is not
> hardware specific and not needed for testing, we may still want to
> review it, because it is common enough that someone will need it...

I hadn't thought of that, but it does seem possible.  However, I think
that rather than editing a generated list it would be better to have a
separate manually maintained list that is merged into the generated
list.

Ben.

-- 
Ben Hutchings, Software Developer                         Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom

[cip-dev] [RFC] Script to find used sources in the kernel

[Pavel Machek]

Hi!

(Ressurecting old discussion).

On Wed 2019-07-10 18:45:38, Ben Hutchings wrote:
> On Sat, 2019-06-22 at 22:07 +0200, Pavel Machek wrote:
> > Hi!
> > 
> > > There are a couple of open questions, on which I would like to hear
> > > other's opinions:
> > > 
> > > * Should the source lists be added to the repository or not?  If they
> > >   are added, then they should not be changed by the standard "all" and
> > >   "clean" targets.
> > >   I tend to think that they should be added, because they take a long
> > >   time to generate and require cross-compilers etc. to be installed.
> > 
> > I believe they should go to the repository, because we may want to
> > manually adjust the lists.
> 
> I've implemented that option.

Are the lists stored somewhere? I see the scripts in "cip-kernel-config"
but I guess it needs non-trivial setup. I'd eventually like filtered version
in lts-commit-list repository.

> > I don't believe "example" configurations we have necessarily have
> > enabled all the options "final products" may need. If some common
> > option (vfat?) is not enabled in our configurations, because it is not
> > hardware specific and not needed for testing, we may still want to
> > review it, because it is common enough that someone will need it...
> 
> I hadn't thought of that, but it does seem possible.  However, I think
> that rather than editing a generated list it would be better to have a
> separate manually maintained list that is merged into the generated
> list.

I guess first step would be to select few subsystems with large
ammounts of -stable patches that we don't need (s390), get the approval
and then create automated tools to mark patches as "ignored" when
they are not interesting.

Best regards,
								Pavel
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[cip-dev] [RFC] Script to find used sources in the kernel

[Ben Hutchings]

On Tue, 2019-11-26 at 20:31 +0100, Pavel Machek wrote:
> Hi!
> 
> (Ressurecting old discussion).
> 
> On Wed 2019-07-10 18:45:38, Ben Hutchings wrote:
> > On Sat, 2019-06-22 at 22:07 +0200, Pavel Machek wrote:
> > > Hi!
> > > 
> > > > There are a couple of open questions, on which I would like to hear
> > > > other's opinions:
> > > > 
> > > > * Should the source lists be added to the repository or not?  If they
> > > >   are added, then they should not be changed by the standard "all" and
> > > >   "clean" targets.
> > > >   I tend to think that they should be added, because they take a long
> > > >   time to generate and require cross-compilers etc. to be installed.
> > > 
> > > I believe they should go to the repository, because we may want to
> > > manually adjust the lists.
> > 
> > I've implemented that option.
> 
> Are the lists stored somewhere?

Yes they are the *.sources files.

> I see the scripts in "cip-kernel-config"
> but I guess it needs non-trivial setup. I'd eventually like filtered version
> in lts-commit-list repository.
[...]

They require a kernel repository with the CIP branches, and suitable
compilers installed.

Ben.
								Pavel
-- 
Ben Hutchings, Software Developer                         Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom