* [PATCH 1/3] iommu: Use pr_crit() instead of long fancy messages
From: Geert Uytterhoeven @ 2021-03-31 9:31 UTC (permalink / raw)
To: Joerg Roedel, Will Deacon, Steven Rostedt, Ingo Molnar,
Petr Mladek, Sergey Senozhatsky
Cc: Linus Torvalds, Andrew Morton, Andy Shevchenko, Rasmus Villemoes,
John Ogness, Gary R Hook, Marco Elver, Randy Dunlap,
Vlastimil Babka, iommu, linux-kernel, linux-embedded,
Geert Uytterhoeven
In-Reply-To: <20210331093104.383705-1-geert+renesas@glider.be>
While long fancy messages have a higher probability of being seen than
small messages, they may scroll of the screen fast, if visible at all,
and may still be missed. In addition, they increase boot time and
kernel size.
The correct mechanism to increase importance of a kernel message is not
to draw fancy boxes with more text, but to shout louder, i.e. increase
the message's reporting level. Making sure the administrator of the
system is aware of such a message is a system policy, and is the
responsability of a user-space log daemon.
Fix this by increasing the reporting level from KERN_WARNING to
KERN_CRIT, and removing irrelevant text and graphics.
This reduces kernel size by ca. 0.5 KiB.
Fixes: bad614b24293ae46 ("iommu: Enable debugfs exposure of IOMMU driver internals")
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
drivers/iommu/iommu-debugfs.c | 19 ++++---------------
1 file changed, 4 insertions(+), 15 deletions(-)
diff --git a/drivers/iommu/iommu-debugfs.c b/drivers/iommu/iommu-debugfs.c
index f0354894209648fd..c3306998de0687bd 100644
--- a/drivers/iommu/iommu-debugfs.c
+++ b/drivers/iommu/iommu-debugfs.c
@@ -32,20 +32,9 @@ void iommu_debugfs_setup(void)
{
if (!iommu_debugfs_dir) {
iommu_debugfs_dir = debugfs_create_dir("iommu", NULL);
- pr_warn("\n");
- pr_warn("*************************************************************\n");
- pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n");
- pr_warn("** **\n");
- pr_warn("** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL **\n");
- pr_warn("** **\n");
- pr_warn("** This means that this kernel is built to expose internal **\n");
- pr_warn("** IOMMU data structures, which may compromise security on **\n");
- pr_warn("** your system. **\n");
- pr_warn("** **\n");
- pr_warn("** If you see this message and you are not debugging the **\n");
- pr_warn("** kernel, report this immediately to your vendor! **\n");
- pr_warn("** **\n");
- pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n");
- pr_warn("*************************************************************\n");
+ pr_crit("IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL\n");
+ pr_crit("This means that this kernel is built to expose internal\n");
+ pr_crit("IOMMU data structures, which may compromise security on\n");
+ pr_crit("your system.\n");
}
}
--
2.25.1
^ permalink raw reply related
* [PATCH 0/3] Use pr_crit() instead of long fancy messages
From: Geert Uytterhoeven @ 2021-03-31 9:31 UTC (permalink / raw)
To: Joerg Roedel, Will Deacon, Steven Rostedt, Ingo Molnar,
Petr Mladek, Sergey Senozhatsky
Cc: Linus Torvalds, Andrew Morton, Andy Shevchenko, Rasmus Villemoes,
John Ogness, Gary R Hook, Marco Elver, Randy Dunlap,
Vlastimil Babka, iommu, linux-kernel, linux-embedded,
Geert Uytterhoeven
Hi all,
While long fancy messages have a higher probability of being seen than
small messages, they may scroll of the screen fast, if visible at all,
and may still be missed. In addition, they increase boot time and
kernel size.
The correct mechanism to increase importance of a kernel message is not
to draw fancy boxes with more text, but to shout louder, i.e. increase
the message's reporting level. Making sure the administrator of the
system is aware of such a message is a system policy, and is the
responsability of a user-space log daemon.
This series increases the reporting level of such messages from
KERN_WARNING to KERN_CRIT, and removes irrelevant text and graphics.
It was made against linux-next, but applies to v5.12-rc5 with an offset
for the last patch.
Each of these patches reduces kernel size by ca. 0.5 KiB.
Thanks for your comments!
Geert Uytterhoeven (3):
iommu: Use pr_crit() instead of long fancy messages
tracing: Use pr_crit() instead of long fancy messages
lib/vsprintf: Use pr_crit() instead of long fancy messages
drivers/iommu/iommu-debugfs.c | 19 ++++---------------
kernel/trace/trace.c | 17 +++--------------
lib/vsprintf.c | 17 +++--------------
3 files changed, 10 insertions(+), 43 deletions(-)
--
2.25.1
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Yocto Project Virtual Summit 2021
From: Trevor Woerner @ 2021-03-30 16:18 UTC (permalink / raw)
To: yocto-gJxINTE1zeQw8ufyyF7U+SST3g8Odh+X,
openembedded-architecture-ZwoEplunGu3dfDuKDZ/zN51Ccm5ICvs9,
openembedded-core-ZwoEplunGu3dfDuKDZ/zN51Ccm5ICvs9,
automated-testing-EtnWKYl6rD/WsZ/bQMPhNw,
yocto-advocacy-EtnWKYl6rD/WsZ/bQMPhNw,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
[-- Attachment #1: Type: text/plain, Size: 874 bytes --]
The Yocto Project Summit Planning Committee is happy to announce the
upcoming 3rd Yocto Project Summit to take place Tuesday and Wednesday
May 25-26 2021, virtually.
The 2-day event will run in 2 tracks including a virtual developers meeting,
beginner tutorial sessions, hands-on intermediate instruction, lightning
talks, regular talks, and social events.
The cost for all attendees will be $40USD. The event will run both days from
noon until 8pm GMT. Registration is not yet open but will be shortly, please
watch for further announcements.
The call for papers is now open and will close at 11:59 PM PST on Sunday,
April 25, 2021. To submit a proposal please visit:
https://pretalx.com/yocto-project-summit-2021/cfp
For more information please visit:
https://www.yoctoproject.org/yocto-project-virtual-summit-2021/
We look forward to seeing you at the conference!
[-- Attachment #2: Type: text/plain, Size: 421 bytes --]
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#52970): https://lists.yoctoproject.org/g/yocto/message/52970
Mute This Topic: https://lists.yoctoproject.org/mt/81726304/4382963
Group Owner: yocto+owner-gJxINTE1zeQw8ufyyF7U+SST3g8Odh+X@public.gmane.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [gley-yocto@m.gmane-mx.org]
-=-=-=-=-=-=-=-=-=-=-=-
^ permalink raw reply
* Re: [PATCH v2 2/2] init/Kconfig: Increase default log buffer size from 128 KB to 512 KB
From: Petr Mladek @ 2020-10-30 15:28 UTC (permalink / raw)
To: Paul Menzel
Cc: Sergey Senozhatsky, linux-kernel, John Ogness, Linus Torvalds,
Steven Rostedt, Andy Shevchenko, Greg Kroah-Hartman,
linux-embedded
In-Reply-To: <92a13465-d133-0b39-f64b-7074dbbb3fcc@molgen.mpg.de>
On Thu 2020-10-29 23:16:01, Paul Menzel wrote:
> Dear Petr,
>
>
> Am 11.08.20 um 12:53 schrieb Petr Mladek:
> > On Tue 2020-08-11 11:29:24, Paul Menzel wrote:
> > > Commit f17a32e97e (let LOG_BUF_SHIFT default to 17) from 2008 was the
> > > last time, the the default log buffer size bump was increased.
> > >
> > > Machines have evolved, and on current hardware, enough memory is
> > > present, and some devices have over 200 PCI devices, like a two socket
> > > Skylake-E server, resulting a lot of lines.
> > >
> > > Therefore, increase the default from 128 KB to 512 KB. Anyone, with
> > > limited memory, can still lower it.
> > >
> > > --- a/init/Kconfig
> > > +++ b/init/Kconfig
> > > @@ -681,9 +681,9 @@ config IKHEADERS
> > > kheaders.ko is built which can be loaded on-demand to get access to headers.
> > > config LOG_BUF_SHIFT
> > > - int "Kernel log buffer size (16 => 64KB, 17 => 128KB)"
> > > + int "Kernel log buffer size (17 => 128KB, 19 => 512KB)"
> > > range 12 25
> > > - default 17
> > > + default 19
> > > depends on PRINTK
> > > help
> > > Select the minimal kernel log buffer size as a power of 2.
> >
> > Honestly, I do not have experience with changing the defaults. People
> > hacking small devices might complain. Well, this can be solved
> > by increasing the default only when BASE_FULL is set.
> >
> > I am personally fine with increasing the default when BASE_FULL
> > is set. The amount of messages is growing over time because of
> > increasing complexity of both the hardware and software.
> > Fortunately also the amount of available memory is growing.
> >
> > Well, this should get discussed in wider audience. Adding some
> > people into CC.
> >
> > JFYI, it started with report of lost messages, see
> > https://lore.kernel.org/lkml/264bfbae-122d-9c41-59ea-6413f91bd866@molgen.mpg.de/
>
> As there was no objection, is it possible to apply the two patches, and
> maybe even get them into Linux 5.10?
Thanks for reminding me. I am sorry but it is too late for
5.10. Such a change should be added during the merge window.
Well, the size of the ring buffer has effectively increased in 5.10.
The lockless implementation stores strings and metadata separately.
It basically doubled the memory needs and people around embedded
devices were not happy, see
https://lore.kernel.org/r/CAMuHMdXHFFUrjRMEHnXXU8QQkgD9x_S6R3N0Q7Q4H2RSfy2GGw@mail.gmail.com
Please update the patch so that the default stays the same for
BASE_SMALL. Please, add Rasmus Villemoes
<linux@rasmusvillemoes.dk> and Geert Uytterhoeven
<geert@linux-m68k.org> into CC when you send it.
Best Regards,
Petr
^ permalink raw reply
* Re: [PATCH v2 2/2] init/Kconfig: Increase default log buffer size from 128 KB to 512 KB
From: Geert Uytterhoeven @ 2020-10-30 8:01 UTC (permalink / raw)
To: Petr Mladek
Cc: Paul Menzel, Sergey Senozhatsky, Linux Kernel Mailing List,
John Ogness, Linus Torvalds, Steven Rostedt, Andy Shevchenko,
Greg Kroah-Hartman, Linux Embedded
In-Reply-To: <20200811105352.GG6215@alley>
On Tue, Aug 11, 2020 at 12:55 PM Petr Mladek <pmladek@suse.com> wrote:
> On Tue 2020-08-11 11:29:24, Paul Menzel wrote:
> > Commit f17a32e97e (let LOG_BUF_SHIFT default to 17) from 2008 was the
> > last time, the the default log buffer size bump was increased.
> >
> > Machines have evolved, and on current hardware, enough memory is
> > present, and some devices have over 200 PCI devices, like a two socket
> > Skylake-E server, resulting a lot of lines.
> >
> > Therefore, increase the default from 128 KB to 512 KB. Anyone, with
> > limited memory, can still lower it.
> >
> > --- a/init/Kconfig
> > +++ b/init/Kconfig
> > @@ -681,9 +681,9 @@ config IKHEADERS
> > kheaders.ko is built which can be loaded on-demand to get access to headers.
> >
> > config LOG_BUF_SHIFT
> > - int "Kernel log buffer size (16 => 64KB, 17 => 128KB)"
> > + int "Kernel log buffer size (17 => 128KB, 19 => 512KB)"
> > range 12 25
> > - default 17
> > + default 19
> > depends on PRINTK
> > help
> > Select the minimal kernel log buffer size as a power of 2.
>
> Honestly, I do not have experience with changing the defaults. People
> hacking small devices might complain. Well, this can be solved
> by increasing the default only when BASE_FULL is set.
>
> I am personally fine with increasing the default when BASE_FULL
> is set. The amount of messages is growing over time because of
> increasing complexity of both the hardware and software.
> Fortunately also the amount of available memory is growing.
Note that making this change means that some of the embedded
defconfigs may need to gain a CONFIG_LOG_BUF_SHIFT=17
line...
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Re: [PATCH v2 2/2] init/Kconfig: Increase default log buffer size from 128 KB to 512 KB
From: Paul Menzel @ 2020-10-29 22:16 UTC (permalink / raw)
To: Petr Mladek
Cc: Sergey Senozhatsky, linux-kernel, John Ogness, Linus Torvalds,
Steven Rostedt, Andy Shevchenko, Greg Kroah-Hartman,
linux-embedded
In-Reply-To: <20200811105352.GG6215@alley>
Dear Petr,
Am 11.08.20 um 12:53 schrieb Petr Mladek:
> On Tue 2020-08-11 11:29:24, Paul Menzel wrote:
>> Commit f17a32e97e (let LOG_BUF_SHIFT default to 17) from 2008 was the
>> last time, the the default log buffer size bump was increased.
>>
>> Machines have evolved, and on current hardware, enough memory is
>> present, and some devices have over 200 PCI devices, like a two socket
>> Skylake-E server, resulting a lot of lines.
>>
>> Therefore, increase the default from 128 KB to 512 KB. Anyone, with
>> limited memory, can still lower it.
>>
>> --- a/init/Kconfig
>> +++ b/init/Kconfig
>> @@ -681,9 +681,9 @@ config IKHEADERS
>> kheaders.ko is built which can be loaded on-demand to get access to headers.
>>
>> config LOG_BUF_SHIFT
>> - int "Kernel log buffer size (16 => 64KB, 17 => 128KB)"
>> + int "Kernel log buffer size (17 => 128KB, 19 => 512KB)"
>> range 12 25
>> - default 17
>> + default 19
>> depends on PRINTK
>> help
>> Select the minimal kernel log buffer size as a power of 2.
>
> Honestly, I do not have experience with changing the defaults. People
> hacking small devices might complain. Well, this can be solved
> by increasing the default only when BASE_FULL is set.
>
> I am personally fine with increasing the default when BASE_FULL
> is set. The amount of messages is growing over time because of
> increasing complexity of both the hardware and software.
> Fortunately also the amount of available memory is growing.
>
> Well, this should get discussed in wider audience. Adding some
> people into CC.
>
> JFYI, it started with report of lost messages, see
> https://lore.kernel.org/lkml/264bfbae-122d-9c41-59ea-6413f91bd866@molgen.mpg.de/
As there was no objection, is it possible to apply the two patches, and
maybe even get them into Linux 5.10?
Kind regards,
Paul
^ permalink raw reply
* Re: [PATCH v2 2/2] init/Kconfig: Increase default log buffer size from 128 KB to 512 KB
From: Petr Mladek @ 2020-08-11 10:53 UTC (permalink / raw)
To: Paul Menzel
Cc: Sergey Senozhatsky, linux-kernel, John Ogness, Linus Torvalds,
Steven Rostedt, Andy Shevchenko, Greg Kroah-Hartman,
linux-embedded
In-Reply-To: <20200811092924.6256-2-pmenzel@molgen.mpg.de>
On Tue 2020-08-11 11:29:24, Paul Menzel wrote:
> Commit f17a32e97e (let LOG_BUF_SHIFT default to 17) from 2008 was the
> last time, the the default log buffer size bump was increased.
>
> Machines have evolved, and on current hardware, enough memory is
> present, and some devices have over 200 PCI devices, like a two socket
> Skylake-E server, resulting a lot of lines.
>
> Therefore, increase the default from 128 KB to 512 KB. Anyone, with
> limited memory, can still lower it.
>
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -681,9 +681,9 @@ config IKHEADERS
> kheaders.ko is built which can be loaded on-demand to get access to headers.
>
> config LOG_BUF_SHIFT
> - int "Kernel log buffer size (16 => 64KB, 17 => 128KB)"
> + int "Kernel log buffer size (17 => 128KB, 19 => 512KB)"
> range 12 25
> - default 17
> + default 19
> depends on PRINTK
> help
> Select the minimal kernel log buffer size as a power of 2.
Honestly, I do not have experience with changing the defaults. People
hacking small devices might complain. Well, this can be solved
by increasing the default only when BASE_FULL is set.
I am personally fine with increasing the default when BASE_FULL
is set. The amount of messages is growing over time because of
increasing complexity of both the hardware and software.
Fortunately also the amount of available memory is growing.
Well, this should get discussed in wider audience. Adding some
people into CC.
JFYI, it started with report of lost messages, see
https://lore.kernel.org/lkml/264bfbae-122d-9c41-59ea-6413f91bd866@molgen.mpg.de/
Best Regards,
Petr
^ permalink raw reply
* Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores
From: Phillip Lougher @ 2019-08-02 6:50 UTC (permalink / raw)
To: Theodore Y. Ts'o, Phillip Lougher, 921146, hartmans,
debian-ctte, László Böszörményi (GCS),
Alexander Couzens, linux-fsdevel, linux-embedded, LKML
In-Reply-To: <20190802003457.GD17372@mit.edu>
On Fri, Aug 2, 2019 at 1:35 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> Phillip,
>
Hi Theodore (Ted),
Thank-you for your kind and well intentioned email. It has
de-escalated tensions on my side.
It is very late here, and so I can only be brief, but, I want this
said as soon as possible.
> Peace.
Yes, I would very much like that to happen.
>
> You may not like the fact that David Oberhollenzer (GitHub username
> AgentD) started an effort to implement a new set of tools to generate
> squashfs images on April 30th, 2019, and called it squashfs-tools-ng.
>
> However, it's really not fair to complain that there is a "violation
> of copyright" given that all of squashfs-tools was released is under
> the GPL. Using some text from squashfs-tools in the package
> description or documentation of squashfs-tools-ng is totally allowed
> under the GPL. You could complain that they didn't include an
> acknowledgement that text was taken from your program. But then give
> them time to fix up the acknowledgements. Assuming good faith is
> always a good default.
>
> The other thing that you've complained about is that some folks have
> (inaccurately, in your view) described squashfs-tools as not being
> maintained. I'd encourage you to take a step back, and consider how
> this might be quite understandable how they might have gotten that
> impression.
>
> First of all, let's look on the documentation in kernel source tree,
> located at Documentation/filesystems/squashfs.txt. It states that
> squashfs-tools's web site is www.squashfs.org, and the git tree is at
> git://git.kernel.org/pub/scm/fs/squashfs/squashfs-tools.git.
>
> The web site www.squashfs.org is not currently responding, but
> according to the Internet Archive, it was redirecting to
> http://squashfs.sourceforge.net/. This web site describes the latest
> version of squashfs-tools is 4.2, released in February 2011, It
> apparently wasn't updated when squashfs-tools 4.3 was released in May
> 2014.
>
I gave up on Sourceforge many years ago when it came unusable (in my
opinion) due to too many adverts.
If I knew how to remove Squashfs from Sourceforge to save confusion I
would have done so.
> The git.kernel.org tree is identical to the sourceforge.net's git
> tree. That tree's most recent commit is from August 2017, e38956b9
> ("squashfs-tools: Add zstd support"). Now, the fascinating thing is
> that the github tree has a completely different commit-id for the same
> commit is 61133613 ("squashfs-tools: Add zstd support"). The git
> commit that the two trees have in common is 9c1db6d1 from September
> 2014.
>
> Reconstructing the git history, you didn't make any commits between
> September 2014 and March 2017. At that time, you merged a number of
> github pull requests between 2014 and 2017, but then exported them as
> patches and applied them on the kernel.org/sourceforge git trees.
> Why, I'm not sure.
>
I clicked the web merge button on GitHub, and then ended up with the
patches in the GitHUb repository (which I didn't use at the time), and
synced manually with kernel.org/sourceforge.
Blame lack of knowledge of GitHub. on my behalf. I tend to prefer
command-interfaces which can be scripted.
> In August 2017, you stopped updating the kernel.org and sourceforge
> git trees, and abandoned them. After that for the rest of 2017, you
> merged one more pull request, and applied one commit to add the -nold
> option. In 2018, there were only two commits, in February and June.
> And then nothing until April 2019 (about the time that squash-tools-ng
> was started/announced), there has been a flurry of activity, including
> merging github pull requests from 2017 and 2018, antd you've done a
> lot of work since then.
>
The start of development in April and the co-incidence with the
squashfs-tools-ng project is purely coincidental. I didn't know
anything about squashfs-tools-ng until Matt Turner of Gentoo mentioned
it in an issue on GitHub
(https://github.com/plougher/squashfs-tools/issues/25) nine days ago.
I didn't know about this co-incidence until you pointed it out. This
in fact may explain some of the mis-understanding.
I meant to do some development on Squashfs-tools over the last
Christmas vacation. I don't want to go into private family details.
but two deaths and a stroke in the family meant I had more important
things to deal with. April was then the first opportunity I got to do
some development.
I try to keep people up to date with my intentions and commitments,
and I mentioned all this in another issue on GitHub
(https://github.com/plougher/squashfs-tools/issues/54).
> I say this not to criticize the amount of attention you've paid to
> squashfs-tools, but to point out that when David started work on
> squashfs-tools-ng, it's not unreasonable that he might have gotten the
> impression that development had ceased --- especially if he followed
> the documentation in the kernel sources, and found an extremely
> cobwebby website, and a git tree on git.kernel.org that hadn't been
> updated since 2017, with substantive heavy development basically
> ending in 2014 (which is also when the last release of squashfs-tools
> was cut).
In 2012-2014 I was put into a difficult position. The previous year I
had started work @ Redhat as a kernel maintainer, which was leaving me
very little free time.
But the tools (especially Mksquashfs) which I had written in my spare
time between 2002-2009, when Squashfs was purely a hobbyist
filesystem, were increasingly breaking. There were issues with stack
size, overflows, and basically dozens of things which fuzzers and
others like to exploit to produce crashes etc.
I decided I had to do a major rewrite of Mksquashfs. It took over two
years, working all evenings available (from work) and most weekends
and at the end of it I felt completely burnt out. My intention was
this should give me space to concentrate on other things for a while,
having cleared up all the issues.
I went on vacation a couple of months after release, deliberately
where there was no internet (off grid). When I came back I found a
number of highly abusive emails from people, that had gotten more
abusive as I'd not answered them for a week. I then had what I now
recognise to be a nervous breakdown. I destroyed all my development
files and notes, and I couldn't bear to look at a line of Squashfs
code for a couple of months.
I obviously came back after a couple of months. But, I took the
decision to not let Squashfs become the nightmare it had been for a
couple of years. I would step back and cease to do pro-active
development and concentrate on security issues, bug fixes and
correspondence.
I genuinely felt things were going well and I was getting a good
balance between my life, my job, and Squashfs, until now.
Perhaps that is why I have reacted "so badly" now, it is called dismay.
I can't write anymore as it is already very late.
Best Regards
Phillip
> You don't need to ascribe malice to what might just simply
> be an impression by looking in the official locations in the official
> kernel documentation!
>
> As a fellow kernel file system developer, let me make a few
> suggestions.
>
> * Don't worry about with "competing" software projects. For a while,
> a multi-billion dollar company attempted to maintain a BSD-licensed
> "competition" to some of the programs in e2fsprogs. This was
> because Andy Rubin was highly allergic to the GPL way back when. I
> pointed the independent implementation was creating invalid file
> systems, and was buggy, and in general was making that billion
> dollar company's life harder, not easier. They eventually gave up
> on it, and Android uses e2fsprogs these days.
>
> The whole point of open source is "may the best code win". If
> you're convinced that you, as the upstream kernel developer, can do
> a better job maintaining the userspace tools, then instead of
> complaining and threatening to sue, just keep your head down, and
> keep improving your code, and in the end, the best code will win.
>
> * I'd suggest that you make sure there is a single canonical git tree.
> It appears it's the github version of your git tree. So... starting
> with your github tree, do a "git merge" of the master branch from
> git.kernel.org, and then push updates to github, git.kernel.org, and
> git.sf.code.net. It's fine to have multiple mirrors of your git
> tree. I maintain multiple copies of git e2fsprogs repo on
> git.kernel.org, github, and sourceforge.
>
> * Please consider tagging your releases. There are git tags for
> squashfs 3.1 and 3.2, but none of your 4.x releases. It's going to
> make it much easier for other people to know which git commits
> correspond to your official releases. For bonus points, you could
> use signed git tags. If you need help getting that set up, please
> contact me off-line. I'd be delighted to help you with that.
>
> * Please consider updating the squashfs web site. I acutally keep a
> copy of the e2fsprogs.sourceforge.net web site in the e2fsprogs git
> tree, under the "web" branch. You can see it here:
> https://github.com/tytso/e2fsprogs/tree/web
>
> * It can be handy to audit and apply/merge patches being carried by
> distributions. Before I took over the Debian maintainership of
> e2fsprogs, I would periodically scan the debian patches (and I still
> keep an eye to see what Ubuntu and Fedora have in their local
> patches). If any of those patches fix real bugs, I'll pull them
> into the e2fsprogs git repo, and then send a note to the
> distribution maintainer that I've done so, and let them know that in
> the next release, I've included their patch, and so they can drop it
> from their tree. That way, I can find and fix bugs introduced by
> distribution patches.
>
> In general, I've found that keeping on good terms with the
> distribution packagers is really good thing from the perspective of
> the upstream author.
>
> Best regards,
>
> - Ted
>
^ permalink raw reply
* Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores
From: Phillip Lougher @ 2019-08-02 4:17 UTC (permalink / raw)
To: László Böszörményi (GCS)
Cc: 921146, Chris Lamb, hartmans, debian-ctte, Alexander Couzens,
linux-fsdevel, linux-embedded, LKML
In-Reply-To: <CAKjSHr0vTK6En1_n6GwArV0N6=kM=Czbx2SYat9vK71HuyzMAA@mail.gmail.com>
On Thu, Aug 1, 2019 at 11:41 PM László Böszörményi (GCS) <gcs@debian.org> wrote:
> Let me add Chris Lamb then the previous Debian Project Leader (also
> British just like you [as I know] and you may sit down and talk about
> this in person) who asked for the reproducibility patch / build in the
> first place.
>
If Chris Lamb or anyone else wants a face-to-face meeting I'm more
than happy to do so.
I coincidentally have a week's holiday (vacation) next week, and I'm
happy to spend a day of it travelling and meeting to discuss the
situation.
I do want to de-escalate this situation if possible.
Phillip
> > What else do I have to do to make you stop bad-mouthing Squashfs? Sue?
> If you feel yourself better with that, be my guest. I don't know who
> is the lawyer of Debian, but I'm sure s/he can show you that it's only
> you who dance this storm.
>
> Regards,
> Laszlo/GCS
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#5
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#83
> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#88
> [4] https://sourceforge.net/p/squashfs/code/ci/e38956b92f738518c29734399629e7cdb33072d3/log/?path=
> [5] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921146#75
> [6] https://github.com/AgentD/squashfs-tools-ng
> [7] https://lwn.net/Articles/651775/
> [8] https://github.com/plougher/squashfs-tools/commit/f95864afe8833fe3ad782d714b41378e860977b1
> [9] https://github.com/plougher/squashfs-tools/commit/ba215d73e153a6f237088b4ecb88c702bb4d4183
^ permalink raw reply
* Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores
From: Theodore Y. Ts'o @ 2019-08-02 0:34 UTC (permalink / raw)
To: Phillip Lougher
Cc: 921146, hartmans, debian-ctte,
László Böszörményi (GCS),
Alexander Couzens, linux-fsdevel, linux-embedded, LKML
In-Reply-To: <CAB3wodfF=Gc3FbKedU4JBWi8hZLxcBPUtVPipsCVnaHdFXGk8Q@mail.gmail.com>
Phillip,
Peace.
You may not like the fact that David Oberhollenzer (GitHub username
AgentD) started an effort to implement a new set of tools to generate
squashfs images on April 30th, 2019, and called it squashfs-tools-ng.
However, it's really not fair to complain that there is a "violation
of copyright" given that all of squashfs-tools was released is under
the GPL. Using some text from squashfs-tools in the package
description or documentation of squashfs-tools-ng is totally allowed
under the GPL. You could complain that they didn't include an
acknowledgement that text was taken from your program. But then give
them time to fix up the acknowledgements. Assuming good faith is
always a good default.
The other thing that you've complained about is that some folks have
(inaccurately, in your view) described squashfs-tools as not being
maintained. I'd encourage you to take a step back, and consider how
this might be quite understandable how they might have gotten that
impression.
First of all, let's look on the documentation in kernel source tree,
located at Documentation/filesystems/squashfs.txt. It states that
squashfs-tools's web site is www.squashfs.org, and the git tree is at
git://git.kernel.org/pub/scm/fs/squashfs/squashfs-tools.git.
The web site www.squashfs.org is not currently responding, but
according to the Internet Archive, it was redirecting to
http://squashfs.sourceforge.net/. This web site describes the latest
version of squashfs-tools is 4.2, released in February 2011, It
apparently wasn't updated when squashfs-tools 4.3 was released in May
2014.
The git.kernel.org tree is identical to the sourceforge.net's git
tree. That tree's most recent commit is from August 2017, e38956b9
("squashfs-tools: Add zstd support"). Now, the fascinating thing is
that the github tree has a completely different commit-id for the same
commit is 61133613 ("squashfs-tools: Add zstd support"). The git
commit that the two trees have in common is 9c1db6d1 from September
2014.
Reconstructing the git history, you didn't make any commits between
September 2014 and March 2017. At that time, you merged a number of
github pull requests between 2014 and 2017, but then exported them as
patches and applied them on the kernel.org/sourceforge git trees.
Why, I'm not sure.
In August 2017, you stopped updating the kernel.org and sourceforge
git trees, and abandoned them. After that for the rest of 2017, you
merged one more pull request, and applied one commit to add the -nold
option. In 2018, there were only two commits, in February and June.
And then nothing until April 2019 (about the time that squash-tools-ng
was started/announced), there has been a flurry of activity, including
merging github pull requests from 2017 and 2018, antd you've done a
lot of work since then.
I say this not to criticize the amount of attention you've paid to
squashfs-tools, but to point out that when David started work on
squashfs-tools-ng, it's not unreasonable that he might have gotten the
impression that development had ceased --- especially if he followed
the documentation in the kernel sources, and found an extremely
cobwebby website, and a git tree on git.kernel.org that hadn't been
updated since 2017, with substantive heavy development basically
ending in 2014 (which is also when the last release of squashfs-tools
was cut). You don't need to ascribe malice to what might just simply
be an impression by looking in the official locations in the official
kernel documentation!
As a fellow kernel file system developer, let me make a few
suggestions.
* Don't worry about with "competing" software projects. For a while,
a multi-billion dollar company attempted to maintain a BSD-licensed
"competition" to some of the programs in e2fsprogs. This was
because Andy Rubin was highly allergic to the GPL way back when. I
pointed the independent implementation was creating invalid file
systems, and was buggy, and in general was making that billion
dollar company's life harder, not easier. They eventually gave up
on it, and Android uses e2fsprogs these days.
The whole point of open source is "may the best code win". If
you're convinced that you, as the upstream kernel developer, can do
a better job maintaining the userspace tools, then instead of
complaining and threatening to sue, just keep your head down, and
keep improving your code, and in the end, the best code will win.
* I'd suggest that you make sure there is a single canonical git tree.
It appears it's the github version of your git tree. So... starting
with your github tree, do a "git merge" of the master branch from
git.kernel.org, and then push updates to github, git.kernel.org, and
git.sf.code.net. It's fine to have multiple mirrors of your git
tree. I maintain multiple copies of git e2fsprogs repo on
git.kernel.org, github, and sourceforge.
* Please consider tagging your releases. There are git tags for
squashfs 3.1 and 3.2, but none of your 4.x releases. It's going to
make it much easier for other people to know which git commits
correspond to your official releases. For bonus points, you could
use signed git tags. If you need help getting that set up, please
contact me off-line. I'd be delighted to help you with that.
* Please consider updating the squashfs web site. I acutally keep a
copy of the e2fsprogs.sourceforge.net web site in the e2fsprogs git
tree, under the "web" branch. You can see it here:
https://github.com/tytso/e2fsprogs/tree/web
* It can be handy to audit and apply/merge patches being carried by
distributions. Before I took over the Debian maintainership of
e2fsprogs, I would periodically scan the debian patches (and I still
keep an eye to see what Ubuntu and Fedora have in their local
patches). If any of those patches fix real bugs, I'll pull them
into the e2fsprogs git repo, and then send a note to the
distribution maintainer that I've done so, and let them know that in
the next release, I've included their patch, and so they can drop it
from their tree. That way, I can find and fix bugs introduced by
distribution patches.
In general, I've found that keeping on good terms with the
distribution packagers is really good thing from the perspective of
the upstream author.
Best regards,
- Ted
^ permalink raw reply
* Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores
From: Phillip Lougher @ 2019-08-01 22:41 UTC (permalink / raw)
To: Don Armstrong; +Cc: owner, listmaster, linux-embedded, LKML, linux-fsdevel
In-Reply-To: <20190801222613.iugdsswxbn2pawgd@qor.donarmstrong.com>
On Thu, Aug 1, 2019 at 11:26 PM Don Armstrong <don@debian.org> wrote:
>
> On Thu, 01 Aug 2019, Phillip Lougher wrote:
> > That patch is a laughable piece of rubbish. I believe both you
> > people (the Debian maintainer and author) are in total denial
> > about your incompetence in this matter. This is obviously just my
> > opinion I've formed over the last couple of months, in case you want to
> > claim that it is libellous.
>
> This isn't an appropriate tone for Debian mailing lists or the our bug
> tracking system.
>
> It's fine to disagree on technical matters, but it's not appropriate to
> claim that people are incompetent or that they are making rubbish.
>
I am only defending myself from the slurs and false information being
spread by your maintainer.
I would not be doing this otherwise.
Cheers
Phillip
> Please stop.
>
> --
> Don Armstrong https://www.donarmstrong.com
>
> To punish me for my contempt of authority, Fate has made me an
> authority myself
> -- Albert Einstein
^ permalink raw reply
* Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores
From: László Böszörményi (GCS) @ 2019-08-01 22:41 UTC (permalink / raw)
To: Phillip Lougher, 921146, Chris Lamb
Cc: hartmans, debian-ctte, Alexander Couzens, linux-fsdevel,
linux-embedded, LKML
In-Reply-To: <CAB3wodfF=Gc3FbKedU4JBWi8hZLxcBPUtVPipsCVnaHdFXGk8Q@mail.gmail.com>
On Thu, Aug 1, 2019 at 5:27 PM Phillip Lougher
<phillip.lougher@gmail.com> wrote:
> On Wed, 31 Jul 2019 17:43:44 +0200 Alexander Couzens <lynxis@fe80.eu> wrote:
> > > On Wed, Jul 31, 2019 at 3:06 AM Steven Shiau <steven@nchc.org.tw>
> > > wrote: [...]
> > > As mentioned in the past, it's the
> > > 0016-remove-frag_deflator_thread.patch that needs either reverted or
> > > fixed by it's original author, Alexander Couzens (added to this mail).
> >
> > Since I've full weeks ahead, I can not make any estimation when
> > I've time to look onto the performance issue.
>
> That patch is a laughable piece of rubbish. I believe both you
> people (the Debian maintainer and author) are in total denial
> about your incompetence in this matter. This is obviously just my
> opinion I've formed over the last couple of months, in case you want to
> claim that it is libellous.
Good to know that you could form your opinion in the last couple of
months as on the other side I didn't know you until last week.
> It is, obvious, from the patch where the problem lies. You *remove*
> the parallel fragment compressor threads, and move the work onto the
> main thread.
>
> Now what do you think that does? It completely removes a significant
> amount of the parallelism of Mksquashfs and makes the main thread
> a bottleneck.
>
> This is your fault and your problem, and it lies in your lack of
> understanding.
Who said we don't know where is the bottleneck? Alexander only stated
he can't work on fixing the bottleneck for some weeks.
While I don't know Alexander in person, according to my information he
contributes to 227 projects. Those programmed in C, C++, Python and
Rust; I wouldn't say he lacks any understanding in code.
> Yet you continually blame your inability to make Mksquashfs work
> correctly on *my* code being old and unmaintained and badly written.
Never said that 1) mksquashfs doesn't work correctly, 2) it has a
problem which you don't fix and not possible to fix by others because
your code quality is bad. Please prove it where I have said such
things.
> This can be seen from the following excerpt from a post in this
> Debian bug thread made by the Debian maintainer.
>
> "> First of all, as squashfs-tools wasn't written in the last years, when
> > reproducible builds became more famous. So it's not written
> > with reproducible building in mind.
> > For me is reproducible builds more important than using all cpu cores.
> > But I don't use it with gigabytes images.
> Yeah, it's quite an old software without real development in the recent years."
Just for the record, this is Debian bugreport #919207 [1] started by
a third person (not me or Alexander), Chris Lamb. Neither written by
me, but Alexander[2]. More importantly it only states that development
goals was not that the result image should always be the same (be
reproducible). It doesn't say anything about code quality.
> " This sounds more complex work than it can be achieved in this week.
> Maybe a complete rewrite would be better then on the long run."
Written by me, talking _about the patch_ used for the
reproducibility[3] and _not_ about your code, see which part of the
previous email was referred above.
> Constantly pointing the blame at my tools and my code.
Again, where is the blame on your code? You didn't code it with
reproducibility in mind, but it doesn't mean in any way that your code
would be bad.
> This is typical of the poor worker who chooses to blame everyone
> else but himself.
Let me ask, who is the one who puff it up, cross post to unrelated
mailing lists (like LKML, which doesn't involved with Debian matters
or squashfs-tools application development)?
> None of that is the case. 50% of the code-base of Squashfs-tools
> was (re-)written in the last 9 years as part of on-going improvements.
> It has also been maintained across that period.
Until recently the know VCS location was at SourceForge[4]. The last
commit there from 2017-08-15, adding zstd support by Sean Purcell. In
2017 there were eight commits, the one mentioned previously and thinks
like "Make all compressor functions static", "Fix pseudo format error
message", "add pseudo definition format to the help text" and "improve
the error message when filenames with spaces are used" etc. Doesn't
look like a rewrite.
> As for your specific claim that Mksquashfs can't be made to produce
> reproducible builds because it is old and written before reproducible
> builds became popular. That is abject nonsense.
Where do you see the statement it can't be made to produce
reproducible images? Again, it only said when you originally coded
squashfs-tools this was not in your goals.
> I added reproducible builds to Mksquashfs. It took one weekend.
>
> https://github.com/plougher/squashfs-tools/commit/24da0c63c80be64e1adc3f24c27459ebe18a19af
On June the 30th, 2019 just for the record. For some reason you
didn't notice anyone about this, that you have the correct solution
when talking about this matter[5].
> Squashfs-tools-ng, this is a rogue project masquerading as an
> official new upstream . It is neither official nor a new upstream. As
> Squashfs author and maintainer I completely disassociate that project
> with the Squashfs project.
It has a different name, immediately stating it's "a new set of tools
for working with SquashFS images"[6]. Where do you see it saying it's
the main project for squashfs images and/or your code is no more or
just shouldn't be used? Then your code is under the GPL-2 or later
license meaning it can be changed, extended and/or reimplemented with
a different name. Nothing happened against this license or against
your personal life / promotion.
What else David should add to prevent all of your misunderstandings?
> I also have publicly stated that this project is spreading falsehoods and
> that it is defamatory to me as the Squashfs author and maintainer.
I still don't see these nor a personal offense against you.
> I have lived for a couple of months with you two people bad-mouthing
> Squashfs tools, it's code-base and my maintenance.
As for me, I've stated only public facts. Like that two security
vulnerabilities, CVE-2015-4645 and CVE-2015-4646 were reported on July
20, 2015 [7] (maybe even before) and you only fixed these on July 15,
2019 [8] with last bits on July 20, 2019 [9] (exactly) four years
later. Didn't judged you on this or questioned why it took so long if
you are such an active maintainer of squashfs-tools.
> I have absolutely had enough.
>
> I have CC'd this to the Debian project leader and the Debian technical
> commitee, and also to linux-kernel, linux-fsdevel and linux-embedded
> for wider attention.
Let me add Chris Lamb then the previous Debian Project Leader (also
British just like you [as I know] and you may sit down and talk about
this in person) who asked for the reproducibility patch / build in the
first place.
> What else do I have to do to make you stop bad-mouthing Squashfs? Sue?
If you feel yourself better with that, be my guest. I don't know who
is the lawyer of Debian, but I'm sure s/he can show you that it's only
you who dance this storm.
Regards,
Laszlo/GCS
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#5
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#83
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#88
[4] https://sourceforge.net/p/squashfs/code/ci/e38956b92f738518c29734399629e7cdb33072d3/log/?path=
[5] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921146#75
[6] https://github.com/AgentD/squashfs-tools-ng
[7] https://lwn.net/Articles/651775/
[8] https://github.com/plougher/squashfs-tools/commit/f95864afe8833fe3ad782d714b41378e860977b1
[9] https://github.com/plougher/squashfs-tools/commit/ba215d73e153a6f237088b4ecb88c702bb4d4183
^ permalink raw reply
* Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores
From: Phillip Lougher @ 2019-08-01 15:23 UTC (permalink / raw)
To: 921146
Cc: hartmans, debian-ctte,
László Böszörményi (GCS),
Alexander Couzens, linux-fsdevel, linux-embedded, LKML
On Wed, 31 Jul 2019 17:43:44 +0200 Alexander Couzens <lynxis@fe80.eu> wrote:
> Hi L치szl칩,
>
> > On Wed, Jul 31, 2019 at 3:06 AM Steven Shiau <steven@nchc.org.tw>
> > wrote: [...]
> > As mentioned in the past, it's the
> > 0016-remove-frag_deflator_thread.patch that needs either reverted or
> > fixed by it's original author, Alexander Couzens (added to this mail).
>
> Since I've full weeks ahead, I can not make any estimation when
> I've time to look onto the performance issue.
>
That patch is a laughable piece of rubbish. I believe both you
people (the Debian maintainer and author) are in total denial
about your incompetence in this matter. This is obviously just my
opinion I've formed over the last couple of months, in case you want to
claim that it is libellous.
It is, obvious, from the patch where the problem lies. You *remove*
the parallel fragment compressor threads, and move the work onto the
main thread.
Now what do you think that does? It completely removes a significant
amount of the parallelism of Mksquashfs and makes the main thread
a bottleneck.
This is your fault and your problem, and it lies in your lack of
understanding.
Yet you continually blame your inability to make Mksquashfs work
correctly on *my* code being old and unmaintained and badly written.
This can be seen from the following excerpt from a post in this
Debian bug thread made by the Debian maintainer.
"> First of all, as squashfs-tools wasn't written in the last years, when
> reproducible builds became more famous. So it's not written
> with reproducible building in mind.
> For me is reproducible builds more important than using all cpu cores.
> But I don't use it with gigabytes images.
Yeah, it's quite an old software without real development in the recent years."
and
" This sounds more complex work than it can be achieved in this week.
Maybe a complete rewrite would be better then on the long run."
Constantly pointing the blame at my tools and my code.
This is typical of the poor worker who chooses to blame everyone
else but himself.
None of that is the case. 50% of the code-base of Squashfs-tools
was (re-)written in the last 9 years as part of on-going improvements.
It has also been maintained across that period.
As for your specific claim that Mksquashfs can't be made to produce
reproducible builds because it is old and written before reproducible
builds became popular. That is abject nonsense.
I added reproducible builds to Mksquashfs. It took one weekend.
https://github.com/plougher/squashfs-tools/commit/24da0c63c80be64e1adc3f24c27459ebe18a19af
> > > Or shall we gradually switch to squashfs-tools-ng is the upstream
> > > is more active:
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965
> > It's under investigation. I'm traveling and will only arrive back
> > home on Sunday evening. Expect more details on this next week.
>
> I also seen the upcoming squashfs-tools-ng rising and quite interested
> to test it and reading the code.
> Depending on tests & the code, I could imagine I'll put my effort and
> time towards squashfs-tools-ng.
>
> @L치szl칩 It should be your call to revert or not. For sure there are
> also downstream users who need reproducible builds (e.g. tails) who
> may also change to squashfs-tools-ng if -ng is the only reproducible
> squashfs generator in debian.
>
Upstream Squashfs-tools now produces reproducible builds.
Squashfs-tools-ng, this is a rogue project masquerading as an
official new upstream . It is neither official nor a new upstream. As
Squashfs author and maintainer I completely disassociate that project
with the Squashfs project.
I also have publicly stated that this project is spreading falsehoods and
that it is defamatory to me as the Squashfs author and maintainer.
See:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965#29
and
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965#34
I have lived for a couple of months with you two people bad-mouthing
Squashfs tools, it's code-base and my maintenance.
I have absolutely had enough.
I have CC'd this to the Debian project leader and the Debian technical
commitee, and also to linux-kernel, linux-fsdevel and linux-embedded
for wider attention.
What else do I have to do to make you stop bad-mouthing Squashfs? Sue?
Yours
Dr. Phillip Lougher
> Hopefully I'll find time to look into it.
>
> Best Regards,
> lynxis
> --
> Alexander Couzens
>
> mail: lynxis@fe80.eu
> jabber: lynxis@fe80.eu
> gpg: 390D CF78 8BF9 AA50 4F8F F1E2 C29E 9DA6 A0DF 8604
^ permalink raw reply
* Re: [PATCH] tools: fix cross-compile var export
From: Dan Carpenter @ 2018-01-08 11:45 UTC (permalink / raw)
To: Martin Kelly
Cc: Paul Gortmaker, linux-kbuild, kernel-janitors, linux-embedded,
David Woodhouse, Matt Mackall, Masahiro Yamada, Michal Marek,
linux-kernel
In-Reply-To: <c021c0f9-4504-6323-98ed-455a196463fc@martingkelly.com>
On Sun, Jan 07, 2018 at 10:51:21AM -0800, Martin Kelly wrote:
> Urg, I accidentally sent to kernel-kbuild instead of linux-kbuild, changed
> now. It appears that past changes to tools/scripts/Makefile.include have
> been handled by linux-kbuild and often Masahiro Yamada.
>
> Perhaps the best sequence here is to send a patch to kbuild adding the
> call-override function and calls to it to the main common Makefile. Then I
> can send individual subsystem patches dropping the individual CC = lines and
> similar. It will be 13 patches instead of 1 but will eventually result in
> the same thing.
>
> Paul, any thoughts?
>
That sounds good.
regards,
dan carpenter
^ permalink raw reply
* Re: [PATCH] tools: fix cross-compile var export
From: Martin Kelly @ 2018-01-07 21:43 UTC (permalink / raw)
To: Paul Gortmaker
Cc: linux-kbuild, kernel-janitors, linux-embedded, David Woodhouse,
Matt Mackall, Masahiro Yamada, Michal Marek, linux-kernel
In-Reply-To: <20180107190453.GN6273@windriver.com>
On 01/07/2018 11:04 AM, Paul Gortmaker wrote:
> [Re: [PATCH] tools: fix cross-compile var export] On 07/01/2018 (Sun 10:31) Martin Kelly wrote:
>
> [...]
>
>> With the change, we add do CC = $(CROSS_COMPILE)gcc if and only if CC is not
>> already set. I'm happy to add all these details to the commit description.
>
> That is probably a step in the right direction. I contribute to yocto
> on a regular basis and hence am sympathetic to these frustrating SDK
> type issues. But after a quick scan of this patch it wasn't obvious to
> me that existing behaviour was preserved, or that it would be a pain to
> separate it into chunks. (key word here being "quick")
>
> So I'd recommend updating the commit log, and adding a Cc: line for the
> maintainers of each subsystem below, and then if nobody complains you
> might get akpm to pick it up as he does for other patches w/o a clear
> maintainer or subsystem.
>
> P
Thanks, I have done so. I also contribute to Yocto and indeed, SDK
issues are all-to-common.
^ permalink raw reply
* Re: [PATCH] tools: fix cross-compile var export
From: Paul Gortmaker @ 2018-01-07 19:04 UTC (permalink / raw)
To: Martin Kelly
Cc: kernel-kbuild, kernel-janitors, linux-embedded, David Woodhouse,
Matt Mackall, Masahiro Yamada, Michal Marek, linux-kernel
In-Reply-To: <549cf839-9ae7-78ce-58df-3d84fc7b3d05@martingkelly.com>
[Re: [PATCH] tools: fix cross-compile var export] On 07/01/2018 (Sun 10:31) Martin Kelly wrote:
[...]
> With the change, we add do CC = $(CROSS_COMPILE)gcc if and only if CC is not
> already set. I'm happy to add all these details to the commit description.
That is probably a step in the right direction. I contribute to yocto
on a regular basis and hence am sympathetic to these frustrating SDK
type issues. But after a quick scan of this patch it wasn't obvious to
me that existing behaviour was preserved, or that it would be a pain to
separate it into chunks. (key word here being "quick")
So I'd recommend updating the commit log, and adding a Cc: line for the
maintainers of each subsystem below, and then if nobody complains you
might get akpm to pick it up as he does for other patches w/o a clear
maintainer or subsystem.
P.
--
>
> >>
> >> tools/cgroup/Makefile | 1 -
> >> tools/gpio/Makefile | 2 --
> >> tools/hv/Makefile | 1 -
> >> tools/iio/Makefile | 2 --
> >> tools/laptop/freefall/Makefile | 1 -
> >> tools/leds/Makefile | 1 -
> >> tools/perf/Makefile.perf | 6 ------
> >> tools/power/acpi/Makefile.config | 3 ---
> >> tools/scripts/Makefile.include | 18 ++++++++++++++++++
> >> tools/spi/Makefile | 2 --
> >> tools/usb/Makefile | 1 -
> >> tools/vm/Makefile | 1 -
> >> tools/wmi/Makefile | 1 -
> >> 13 files changed, 18 insertions(+), 22 deletions(-)
> >>
^ permalink raw reply
* Re: [PATCH] tools: fix cross-compile var export
From: Martin Kelly @ 2018-01-07 18:51 UTC (permalink / raw)
To: Paul Gortmaker
Cc: linux-kbuild, kernel-janitors, linux-embedded, David Woodhouse,
Matt Mackall, Masahiro Yamada, Michal Marek, linux-kernel
In-Reply-To: <549cf839-9ae7-78ce-58df-3d84fc7b3d05@martingkelly.com>
On 01/07/2018 10:31 AM, Martin Kelly wrote:
> On 01/07/2018 08:11 AM, Paul Gortmaker wrote:
>> [[PATCH] tools: fix cross-compile var export] On 06/01/2018 (Sat
>> 12:16) Martin Kelly wrote:
>>
>>> From: Martin Kelly <martin@martingkelly.com>
>>>
>>> Currently in a number of Makefiles, we clobber the CC, LD, and/or
>>> STRIP env vars
>>> when cross-compiling, which breaks any additional flags that might be
>>> set (such
>>> as sysroot). This easily shows up by using, for instance, a Yocto SDK.
>>>
>>> Fix this by more carefully overriding the flags in the way that the perf
>>> Makefile does.
>>>
>>> This patch does not fix cross-compile for all the tools (some have
>>> other bugs),
>>> but it does appear to fix it for these:
>>>
>>> - cgroup
>>> - freefall
>>> - gpio
>>> - hv
>>> - iio
>>> - leds
>>> - spi
>>> - vm
>>> - wmi
>>>
>>> Signed-off-by: Martin Kelly <martin@martingkelly.com>
>>> ---
>>> This patch touches multiple subsystems, and there doesn't appear to
>>> be a global
>>> tools maintainer, so I sent this to linux-embedded kernel-janitors, and
>>> linux-kbuild. Please let me know if there is a better place for it.
>>
>> I think you will have better luck if you go subsystem by subsystem, e.g.
>> send gpio change to gpio maintainer, usb change to usb maintainer and so
>> on. In addition, you'll want to try and understand/explain why those
>> lines were added in the 1st place and how it won't break other existing
>> use cases by removing them. Maybe "git blame" will help for that...
>>
>> P.
>> --
>>
>
> I can send patches subsystem-by-subsystem, but this patch fixes a single
> bug that applies to many Makefiles. It fixes it by putting the correct
> logic into tools/scripts/Makefile.include, which each Makefiles
> includes. If I do this subsystem-by-subsystem, we will just duplicate
> these lines into every Makefile:
>
> +$(call allow-override,CC,$(CROSS_COMPILE)gcc)
> +$(call allow-override,AR,$(CROSS_COMPILE)ar)
> +$(call allow-override,LD,$(CROSS_COMPILE)ld)
> +$(call allow-override,CXX,$(CROSS_COMPILE)g++)
> +$(call allow-override,CXX,$(CROSS_COMPILE)strip)
>
> We could do this, but it's a lot of code duplication and means that
> anyone making a new Makefile must also copy the lines or their Makefile
> will have this bug. Since the bug fix generically applies everywhere, I
> think it's most correct for it to go into the common Makefile.
>
> If there is really no maintainer for the tools/scripts/Makefile.include
> file, then maybe I have to send subsystem-by-subsystem, but that seems
> less than ideal.
>
> I do understand why the CC = $(CROSS_COMPILE)gcc lines exist; it is to
> enable cross-compile when you set the $CROSS_COMPILE env var and to use
> native gcc when that is not set. The bug is that if you already have $CC
> set to your cross-compiler (which is what the Yocto SDK and other
> toolchains do), then most of your command-line gets clobbered. Here's an
> example from an ARM Yocto SDK I have:
>
> $ echo $CC
> arm-poky-linux-gnueabi-gcc -march=armv7-a -mfpu=neon -mfloat-abi=hard
> -mcpu=cortex-a8
> --sysroot=/home/martin/src/poky/build/sdk/sysroots/cortexa8hf-neon-poky-linux-gnueabi
>
>
> $ echo $CROSS_COMPILE
> arm-poky-linux-gnueabi-
>
> $ echo ${CROSS_COMPILE}gcc
> arm-poky-linux-gnueabi-gcc
>
> Although arm-poky-linux-gnueabi-gcc is a cross-compiler, we've lost many
> build flags, notably --sysroot, which enables us to find the right
> libraries to link against. Without this change, I get this kind of error
> in all the affected Makefiles:
>
> martin@cascade:~/src/linux/tools$ make iio
> [snip]
> iio_event_monitor.c:18:10: fatal error: unistd.h: No such file or directory
> #include <unistd.h>
> ^~~~~~~~~~
>
> This occurs because unistd.h is in the sysroot, but we've dropped the
> --sysroot flag:
> $ find /home/martin/src/poky/build/sdk/sysroots -path
> '*/usr/include/unistd.h'
> /home/martin/src/poky/build/sdk/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/include/unistd.h
>
>
> With the change, we add do CC = $(CROSS_COMPILE)gcc if and only if CC is
> not already set. I'm happy to add all these details to the commit
> description.
>
Urg, I accidentally sent to kernel-kbuild instead of linux-kbuild,
changed now. It appears that past changes to
tools/scripts/Makefile.include have been handled by linux-kbuild and
often Masahiro Yamada.
Perhaps the best sequence here is to send a patch to kbuild adding the
call-override function and calls to it to the main common Makefile. Then
I can send individual subsystem patches dropping the individual CC =
lines and similar. It will be 13 patches instead of 1 but will
eventually result in the same thing.
Paul, any thoughts?
>>>
>>> tools/cgroup/Makefile | 1 -
>>> tools/gpio/Makefile | 2 --
>>> tools/hv/Makefile | 1 -
>>> tools/iio/Makefile | 2 --
>>> tools/laptop/freefall/Makefile | 1 -
>>> tools/leds/Makefile | 1 -
>>> tools/perf/Makefile.perf | 6 ------
>>> tools/power/acpi/Makefile.config | 3 ---
>>> tools/scripts/Makefile.include | 18 ++++++++++++++++++
>>> tools/spi/Makefile | 2 --
>>> tools/usb/Makefile | 1 -
>>> tools/vm/Makefile | 1 -
>>> tools/wmi/Makefile | 1 -
>>> 13 files changed, 18 insertions(+), 22 deletions(-)
>>>
>>> diff --git a/tools/cgroup/Makefile b/tools/cgroup/Makefile
>>> index 860fa151640a..ffca068e4a76 100644
>>> --- a/tools/cgroup/Makefile
>>> +++ b/tools/cgroup/Makefile
>>> @@ -1,7 +1,6 @@
>>> # SPDX-License-Identifier: GPL-2.0
>>> # Makefile for cgroup tools
>>> -CC = $(CROSS_COMPILE)gcc
>>> CFLAGS = -Wall -Wextra
>>> all: cgroup_event_listener
>>> diff --git a/tools/gpio/Makefile b/tools/gpio/Makefile
>>> index 805a2c0cf4cd..240eda014b37 100644
>>> --- a/tools/gpio/Makefile
>>> +++ b/tools/gpio/Makefile
>>> @@ -12,8 +12,6 @@ endif
>>> # (this improves performance and avoids hard-to-debug behaviour);
>>> MAKEFLAGS += -r
>>> -CC = $(CROSS_COMPILE)gcc
>>> -LD = $(CROSS_COMPILE)ld
>>> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>>> ALL_TARGETS := lsgpio gpio-hammer gpio-event-mon
>>> diff --git a/tools/hv/Makefile b/tools/hv/Makefile
>>> index 31503819454d..68c2d7b059b3 100644
>>> --- a/tools/hv/Makefile
>>> +++ b/tools/hv/Makefile
>>> @@ -1,7 +1,6 @@
>>> # SPDX-License-Identifier: GPL-2.0
>>> # Makefile for Hyper-V tools
>>> -CC = $(CROSS_COMPILE)gcc
>>> WARNINGS = -Wall -Wextra
>>> CFLAGS = $(WARNINGS) -g $(shell getconf LFS_CFLAGS)
>>> diff --git a/tools/iio/Makefile b/tools/iio/Makefile
>>> index a08e7a47d6a3..332ed2f6c2c2 100644
>>> --- a/tools/iio/Makefile
>>> +++ b/tools/iio/Makefile
>>> @@ -12,8 +12,6 @@ endif
>>> # (this improves performance and avoids hard-to-debug behaviour);
>>> MAKEFLAGS += -r
>>> -CC = $(CROSS_COMPILE)gcc
>>> -LD = $(CROSS_COMPILE)ld
>>> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>>> ALL_TARGETS := iio_event_monitor lsiio iio_generic_buffer
>>> diff --git a/tools/laptop/freefall/Makefile
>>> b/tools/laptop/freefall/Makefile
>>> index 5f758c489a20..b572d94255f6 100644
>>> --- a/tools/laptop/freefall/Makefile
>>> +++ b/tools/laptop/freefall/Makefile
>>> @@ -2,7 +2,6 @@
>>> PREFIX ?= /usr
>>> SBINDIR ?= sbin
>>> INSTALL ?= install
>>> -CC = $(CROSS_COMPILE)gcc
>>> TARGET = freefall
>>> diff --git a/tools/leds/Makefile b/tools/leds/Makefile
>>> index c379af003807..7b6bed13daaa 100644
>>> --- a/tools/leds/Makefile
>>> +++ b/tools/leds/Makefile
>>> @@ -1,7 +1,6 @@
>>> # SPDX-License-Identifier: GPL-2.0
>>> # Makefile for LEDs tools
>>> -CC = $(CROSS_COMPILE)gcc
>>> CFLAGS = -Wall -Wextra -g -I../../include/uapi
>>> all: uledmon led_hw_brightness_mon
>>> diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
>>> index 68cf1360a3f3..3035ce5f6b36 100644
>>> --- a/tools/perf/Makefile.perf
>>> +++ b/tools/perf/Makefile.perf
>>> @@ -144,12 +144,6 @@ define allow-override
>>> $(eval $(1) = $(2)))
>>> endef
>>> -# Allow setting CC and AR and LD, or setting CROSS_COMPILE as a prefix.
>>> -$(call allow-override,CC,$(CROSS_COMPILE)gcc)
>>> -$(call allow-override,AR,$(CROSS_COMPILE)ar)
>>> -$(call allow-override,LD,$(CROSS_COMPILE)ld)
>>> -$(call allow-override,CXX,$(CROSS_COMPILE)g++)
>>> -
>>> LD += $(EXTRA_LDFLAGS)
>>> HOSTCC ?= gcc
>>> diff --git a/tools/power/acpi/Makefile.config
>>> b/tools/power/acpi/Makefile.config
>>> index a1883bbb0144..2cccbba64418 100644
>>> --- a/tools/power/acpi/Makefile.config
>>> +++ b/tools/power/acpi/Makefile.config
>>> @@ -56,9 +56,6 @@ INSTALL_SCRIPT = ${INSTALL_PROGRAM}
>>> # to compile vs uClibc, that can be done here as well.
>>> CROSS = #/usr/i386-linux-uclibc/usr/bin/i386-uclibc-
>>> CROSS_COMPILE ?= $(CROSS)
>>> -CC = $(CROSS_COMPILE)gcc
>>> -LD = $(CROSS_COMPILE)gcc
>>> -STRIP = $(CROSS_COMPILE)strip
>>> HOSTCC = gcc
>>> # check if compiler option is supported
>>> diff --git a/tools/scripts/Makefile.include
>>> b/tools/scripts/Makefile.include
>>> index 3fab179b1aba..09b2d0f07f66 100644
>>> --- a/tools/scripts/Makefile.include
>>> +++ b/tools/scripts/Makefile.include
>>> @@ -42,6 +42,24 @@ EXTRA_WARNINGS += -Wformat
>>> CC_NO_CLANG := $(shell $(CC) -dM -E -x c /dev/null | grep -Fq
>>> "__clang__"; echo $$?)
>>> +# Makefiles suck: This macro sets a default value of $(2) for the
>>> +# variable named by $(1), unless the variable has been set by
>>> +# environment or command line. This is necessary for CC and AR
>>> +# because make sets default values, so the simpler ?= approach
>>> +# won't work as expected.
>>> +define allow-override
>>> + $(if $(or $(findstring environment,$(origin $(1))),\
>>> + $(findstring command line,$(origin $(1)))),,\
>>> + $(eval $(1) = $(2)))
>>> +endef
>>> +
>>> +# Allow setting various cross-compile vars or setting CROSS_COMPILE
>>> as a prefix.
>>> +$(call allow-override,CC,$(CROSS_COMPILE)gcc)
>>> +$(call allow-override,AR,$(CROSS_COMPILE)ar)
>>> +$(call allow-override,LD,$(CROSS_COMPILE)ld)
>>> +$(call allow-override,CXX,$(CROSS_COMPILE)g++)
>>> +$(call allow-override,CXX,$(CROSS_COMPILE)strip)
>>> +
>>> ifeq ($(CC_NO_CLANG), 1)
>>> EXTRA_WARNINGS += -Wstrict-aliasing=3
>>> endif
>>> diff --git a/tools/spi/Makefile b/tools/spi/Makefile
>>> index 90615e10c79a..815d15589177 100644
>>> --- a/tools/spi/Makefile
>>> +++ b/tools/spi/Makefile
>>> @@ -11,8 +11,6 @@ endif
>>> # (this improves performance and avoids hard-to-debug behaviour);
>>> MAKEFLAGS += -r
>>> -CC = $(CROSS_COMPILE)gcc
>>> -LD = $(CROSS_COMPILE)ld
>>> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>>> ALL_TARGETS := spidev_test spidev_fdx
>>> diff --git a/tools/usb/Makefile b/tools/usb/Makefile
>>> index 4e6506078494..01d758d73b6d 100644
>>> --- a/tools/usb/Makefile
>>> +++ b/tools/usb/Makefile
>>> @@ -1,7 +1,6 @@
>>> # SPDX-License-Identifier: GPL-2.0
>>> # Makefile for USB tools
>>> -CC = $(CROSS_COMPILE)gcc
>>> PTHREAD_LIBS = -lpthread
>>> WARNINGS = -Wall -Wextra
>>> CFLAGS = $(WARNINGS) -g -I../include
>>> diff --git a/tools/vm/Makefile b/tools/vm/Makefile
>>> index be320b905ea7..20f6cf04377f 100644
>>> --- a/tools/vm/Makefile
>>> +++ b/tools/vm/Makefile
>>> @@ -6,7 +6,6 @@ TARGETS=page-types slabinfo page_owner_sort
>>> LIB_DIR = ../lib/api
>>> LIBS = $(LIB_DIR)/libapi.a
>>> -CC = $(CROSS_COMPILE)gcc
>>> CFLAGS = -Wall -Wextra -I../lib/
>>> LDFLAGS = $(LIBS)
>>> diff --git a/tools/wmi/Makefile b/tools/wmi/Makefile
>>> index e664f1167388..e0e87239126b 100644
>>> --- a/tools/wmi/Makefile
>>> +++ b/tools/wmi/Makefile
>>> @@ -2,7 +2,6 @@ PREFIX ?= /usr
>>> SBINDIR ?= sbin
>>> INSTALL ?= install
>>> CFLAGS += -D__EXPORTED_HEADERS__ -I../../include/uapi -I../../include
>>> -CC = $(CROSS_COMPILE)gcc
>>> TARGET = dell-smbios-example
>>> --
>>> 2.11.0
>>>
^ permalink raw reply
* Re: [PATCH] tools: fix cross-compile var export
From: Martin Kelly @ 2018-01-07 18:31 UTC (permalink / raw)
To: Paul Gortmaker
Cc: kernel-kbuild, kernel-janitors, linux-embedded, David Woodhouse,
Matt Mackall, Masahiro Yamada, Michal Marek, linux-kernel
In-Reply-To: <20180107161140.GM6273@windriver.com>
On 01/07/2018 08:11 AM, Paul Gortmaker wrote:
> [[PATCH] tools: fix cross-compile var export] On 06/01/2018 (Sat 12:16) Martin Kelly wrote:
>
>> From: Martin Kelly <martin@martingkelly.com>
>>
>> Currently in a number of Makefiles, we clobber the CC, LD, and/or STRIP env vars
>> when cross-compiling, which breaks any additional flags that might be set (such
>> as sysroot). This easily shows up by using, for instance, a Yocto SDK.
>>
>> Fix this by more carefully overriding the flags in the way that the perf
>> Makefile does.
>>
>> This patch does not fix cross-compile for all the tools (some have other bugs),
>> but it does appear to fix it for these:
>>
>> - cgroup
>> - freefall
>> - gpio
>> - hv
>> - iio
>> - leds
>> - spi
>> - vm
>> - wmi
>>
>> Signed-off-by: Martin Kelly <martin@martingkelly.com>
>> ---
>> This patch touches multiple subsystems, and there doesn't appear to be a global
>> tools maintainer, so I sent this to linux-embedded kernel-janitors, and
>> linux-kbuild. Please let me know if there is a better place for it.
>
> I think you will have better luck if you go subsystem by subsystem, e.g.
> send gpio change to gpio maintainer, usb change to usb maintainer and so
> on. In addition, you'll want to try and understand/explain why those
> lines were added in the 1st place and how it won't break other existing
> use cases by removing them. Maybe "git blame" will help for that...
>
> P.
> --
>
I can send patches subsystem-by-subsystem, but this patch fixes a single
bug that applies to many Makefiles. It fixes it by putting the correct
logic into tools/scripts/Makefile.include, which each Makefiles
includes. If I do this subsystem-by-subsystem, we will just duplicate
these lines into every Makefile:
+$(call allow-override,CC,$(CROSS_COMPILE)gcc)
+$(call allow-override,AR,$(CROSS_COMPILE)ar)
+$(call allow-override,LD,$(CROSS_COMPILE)ld)
+$(call allow-override,CXX,$(CROSS_COMPILE)g++)
+$(call allow-override,CXX,$(CROSS_COMPILE)strip)
We could do this, but it's a lot of code duplication and means that
anyone making a new Makefile must also copy the lines or their Makefile
will have this bug. Since the bug fix generically applies everywhere, I
think it's most correct for it to go into the common Makefile.
If there is really no maintainer for the tools/scripts/Makefile.include
file, then maybe I have to send subsystem-by-subsystem, but that seems
less than ideal.
I do understand why the CC = $(CROSS_COMPILE)gcc lines exist; it is to
enable cross-compile when you set the $CROSS_COMPILE env var and to use
native gcc when that is not set. The bug is that if you already have $CC
set to your cross-compiler (which is what the Yocto SDK and other
toolchains do), then most of your command-line gets clobbered. Here's an
example from an ARM Yocto SDK I have:
$ echo $CC
arm-poky-linux-gnueabi-gcc -march=armv7-a -mfpu=neon -mfloat-abi=hard
-mcpu=cortex-a8
--sysroot=/home/martin/src/poky/build/sdk/sysroots/cortexa8hf-neon-poky-linux-gnueabi
$ echo $CROSS_COMPILE
arm-poky-linux-gnueabi-
$ echo ${CROSS_COMPILE}gcc
arm-poky-linux-gnueabi-gcc
Although arm-poky-linux-gnueabi-gcc is a cross-compiler, we've lost many
build flags, notably --sysroot, which enables us to find the right
libraries to link against. Without this change, I get this kind of error
in all the affected Makefiles:
martin@cascade:~/src/linux/tools$ make iio
[snip]
iio_event_monitor.c:18:10: fatal error: unistd.h: No such file or directory
#include <unistd.h>
^~~~~~~~~~
This occurs because unistd.h is in the sysroot, but we've dropped the
--sysroot flag:
$ find /home/martin/src/poky/build/sdk/sysroots -path
'*/usr/include/unistd.h'
/home/martin/src/poky/build/sdk/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/include/unistd.h
With the change, we add do CC = $(CROSS_COMPILE)gcc if and only if CC is
not already set. I'm happy to add all these details to the commit
description.
>>
>> tools/cgroup/Makefile | 1 -
>> tools/gpio/Makefile | 2 --
>> tools/hv/Makefile | 1 -
>> tools/iio/Makefile | 2 --
>> tools/laptop/freefall/Makefile | 1 -
>> tools/leds/Makefile | 1 -
>> tools/perf/Makefile.perf | 6 ------
>> tools/power/acpi/Makefile.config | 3 ---
>> tools/scripts/Makefile.include | 18 ++++++++++++++++++
>> tools/spi/Makefile | 2 --
>> tools/usb/Makefile | 1 -
>> tools/vm/Makefile | 1 -
>> tools/wmi/Makefile | 1 -
>> 13 files changed, 18 insertions(+), 22 deletions(-)
>>
>> diff --git a/tools/cgroup/Makefile b/tools/cgroup/Makefile
>> index 860fa151640a..ffca068e4a76 100644
>> --- a/tools/cgroup/Makefile
>> +++ b/tools/cgroup/Makefile
>> @@ -1,7 +1,6 @@
>> # SPDX-License-Identifier: GPL-2.0
>> # Makefile for cgroup tools
>>
>> -CC = $(CROSS_COMPILE)gcc
>> CFLAGS = -Wall -Wextra
>>
>> all: cgroup_event_listener
>> diff --git a/tools/gpio/Makefile b/tools/gpio/Makefile
>> index 805a2c0cf4cd..240eda014b37 100644
>> --- a/tools/gpio/Makefile
>> +++ b/tools/gpio/Makefile
>> @@ -12,8 +12,6 @@ endif
>> # (this improves performance and avoids hard-to-debug behaviour);
>> MAKEFLAGS += -r
>>
>> -CC = $(CROSS_COMPILE)gcc
>> -LD = $(CROSS_COMPILE)ld
>> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>>
>> ALL_TARGETS := lsgpio gpio-hammer gpio-event-mon
>> diff --git a/tools/hv/Makefile b/tools/hv/Makefile
>> index 31503819454d..68c2d7b059b3 100644
>> --- a/tools/hv/Makefile
>> +++ b/tools/hv/Makefile
>> @@ -1,7 +1,6 @@
>> # SPDX-License-Identifier: GPL-2.0
>> # Makefile for Hyper-V tools
>>
>> -CC = $(CROSS_COMPILE)gcc
>> WARNINGS = -Wall -Wextra
>> CFLAGS = $(WARNINGS) -g $(shell getconf LFS_CFLAGS)
>>
>> diff --git a/tools/iio/Makefile b/tools/iio/Makefile
>> index a08e7a47d6a3..332ed2f6c2c2 100644
>> --- a/tools/iio/Makefile
>> +++ b/tools/iio/Makefile
>> @@ -12,8 +12,6 @@ endif
>> # (this improves performance and avoids hard-to-debug behaviour);
>> MAKEFLAGS += -r
>>
>> -CC = $(CROSS_COMPILE)gcc
>> -LD = $(CROSS_COMPILE)ld
>> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>>
>> ALL_TARGETS := iio_event_monitor lsiio iio_generic_buffer
>> diff --git a/tools/laptop/freefall/Makefile b/tools/laptop/freefall/Makefile
>> index 5f758c489a20..b572d94255f6 100644
>> --- a/tools/laptop/freefall/Makefile
>> +++ b/tools/laptop/freefall/Makefile
>> @@ -2,7 +2,6 @@
>> PREFIX ?= /usr
>> SBINDIR ?= sbin
>> INSTALL ?= install
>> -CC = $(CROSS_COMPILE)gcc
>>
>> TARGET = freefall
>>
>> diff --git a/tools/leds/Makefile b/tools/leds/Makefile
>> index c379af003807..7b6bed13daaa 100644
>> --- a/tools/leds/Makefile
>> +++ b/tools/leds/Makefile
>> @@ -1,7 +1,6 @@
>> # SPDX-License-Identifier: GPL-2.0
>> # Makefile for LEDs tools
>>
>> -CC = $(CROSS_COMPILE)gcc
>> CFLAGS = -Wall -Wextra -g -I../../include/uapi
>>
>> all: uledmon led_hw_brightness_mon
>> diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
>> index 68cf1360a3f3..3035ce5f6b36 100644
>> --- a/tools/perf/Makefile.perf
>> +++ b/tools/perf/Makefile.perf
>> @@ -144,12 +144,6 @@ define allow-override
>> $(eval $(1) = $(2)))
>> endef
>>
>> -# Allow setting CC and AR and LD, or setting CROSS_COMPILE as a prefix.
>> -$(call allow-override,CC,$(CROSS_COMPILE)gcc)
>> -$(call allow-override,AR,$(CROSS_COMPILE)ar)
>> -$(call allow-override,LD,$(CROSS_COMPILE)ld)
>> -$(call allow-override,CXX,$(CROSS_COMPILE)g++)
>> -
>> LD += $(EXTRA_LDFLAGS)
>>
>> HOSTCC ?= gcc
>> diff --git a/tools/power/acpi/Makefile.config b/tools/power/acpi/Makefile.config
>> index a1883bbb0144..2cccbba64418 100644
>> --- a/tools/power/acpi/Makefile.config
>> +++ b/tools/power/acpi/Makefile.config
>> @@ -56,9 +56,6 @@ INSTALL_SCRIPT = ${INSTALL_PROGRAM}
>> # to compile vs uClibc, that can be done here as well.
>> CROSS = #/usr/i386-linux-uclibc/usr/bin/i386-uclibc-
>> CROSS_COMPILE ?= $(CROSS)
>> -CC = $(CROSS_COMPILE)gcc
>> -LD = $(CROSS_COMPILE)gcc
>> -STRIP = $(CROSS_COMPILE)strip
>> HOSTCC = gcc
>>
>> # check if compiler option is supported
>> diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
>> index 3fab179b1aba..09b2d0f07f66 100644
>> --- a/tools/scripts/Makefile.include
>> +++ b/tools/scripts/Makefile.include
>> @@ -42,6 +42,24 @@ EXTRA_WARNINGS += -Wformat
>>
>> CC_NO_CLANG := $(shell $(CC) -dM -E -x c /dev/null | grep -Fq "__clang__"; echo $$?)
>>
>> +# Makefiles suck: This macro sets a default value of $(2) for the
>> +# variable named by $(1), unless the variable has been set by
>> +# environment or command line. This is necessary for CC and AR
>> +# because make sets default values, so the simpler ?= approach
>> +# won't work as expected.
>> +define allow-override
>> + $(if $(or $(findstring environment,$(origin $(1))),\
>> + $(findstring command line,$(origin $(1)))),,\
>> + $(eval $(1) = $(2)))
>> +endef
>> +
>> +# Allow setting various cross-compile vars or setting CROSS_COMPILE as a prefix.
>> +$(call allow-override,CC,$(CROSS_COMPILE)gcc)
>> +$(call allow-override,AR,$(CROSS_COMPILE)ar)
>> +$(call allow-override,LD,$(CROSS_COMPILE)ld)
>> +$(call allow-override,CXX,$(CROSS_COMPILE)g++)
>> +$(call allow-override,CXX,$(CROSS_COMPILE)strip)
>> +
>> ifeq ($(CC_NO_CLANG), 1)
>> EXTRA_WARNINGS += -Wstrict-aliasing=3
>> endif
>> diff --git a/tools/spi/Makefile b/tools/spi/Makefile
>> index 90615e10c79a..815d15589177 100644
>> --- a/tools/spi/Makefile
>> +++ b/tools/spi/Makefile
>> @@ -11,8 +11,6 @@ endif
>> # (this improves performance and avoids hard-to-debug behaviour);
>> MAKEFLAGS += -r
>>
>> -CC = $(CROSS_COMPILE)gcc
>> -LD = $(CROSS_COMPILE)ld
>> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>>
>> ALL_TARGETS := spidev_test spidev_fdx
>> diff --git a/tools/usb/Makefile b/tools/usb/Makefile
>> index 4e6506078494..01d758d73b6d 100644
>> --- a/tools/usb/Makefile
>> +++ b/tools/usb/Makefile
>> @@ -1,7 +1,6 @@
>> # SPDX-License-Identifier: GPL-2.0
>> # Makefile for USB tools
>>
>> -CC = $(CROSS_COMPILE)gcc
>> PTHREAD_LIBS = -lpthread
>> WARNINGS = -Wall -Wextra
>> CFLAGS = $(WARNINGS) -g -I../include
>> diff --git a/tools/vm/Makefile b/tools/vm/Makefile
>> index be320b905ea7..20f6cf04377f 100644
>> --- a/tools/vm/Makefile
>> +++ b/tools/vm/Makefile
>> @@ -6,7 +6,6 @@ TARGETS=page-types slabinfo page_owner_sort
>> LIB_DIR = ../lib/api
>> LIBS = $(LIB_DIR)/libapi.a
>>
>> -CC = $(CROSS_COMPILE)gcc
>> CFLAGS = -Wall -Wextra -I../lib/
>> LDFLAGS = $(LIBS)
>>
>> diff --git a/tools/wmi/Makefile b/tools/wmi/Makefile
>> index e664f1167388..e0e87239126b 100644
>> --- a/tools/wmi/Makefile
>> +++ b/tools/wmi/Makefile
>> @@ -2,7 +2,6 @@ PREFIX ?= /usr
>> SBINDIR ?= sbin
>> INSTALL ?= install
>> CFLAGS += -D__EXPORTED_HEADERS__ -I../../include/uapi -I../../include
>> -CC = $(CROSS_COMPILE)gcc
>>
>> TARGET = dell-smbios-example
>>
>> --
>> 2.11.0
>>
^ permalink raw reply
* Re: [PATCH] tools: fix cross-compile var export
From: Paul Gortmaker @ 2018-01-07 16:11 UTC (permalink / raw)
To: Martin Kelly
Cc: kernel-kbuild, kernel-janitors, linux-embedded, David Woodhouse,
Matt Mackall, Masahiro Yamada, Michal Marek, linux-kernel
In-Reply-To: <20180106201601.11074-1-martin@martingkelly.com>
[[PATCH] tools: fix cross-compile var export] On 06/01/2018 (Sat 12:16) Martin Kelly wrote:
> From: Martin Kelly <martin@martingkelly.com>
>
> Currently in a number of Makefiles, we clobber the CC, LD, and/or STRIP env vars
> when cross-compiling, which breaks any additional flags that might be set (such
> as sysroot). This easily shows up by using, for instance, a Yocto SDK.
>
> Fix this by more carefully overriding the flags in the way that the perf
> Makefile does.
>
> This patch does not fix cross-compile for all the tools (some have other bugs),
> but it does appear to fix it for these:
>
> - cgroup
> - freefall
> - gpio
> - hv
> - iio
> - leds
> - spi
> - vm
> - wmi
>
> Signed-off-by: Martin Kelly <martin@martingkelly.com>
> ---
> This patch touches multiple subsystems, and there doesn't appear to be a global
> tools maintainer, so I sent this to linux-embedded kernel-janitors, and
> linux-kbuild. Please let me know if there is a better place for it.
I think you will have better luck if you go subsystem by subsystem, e.g.
send gpio change to gpio maintainer, usb change to usb maintainer and so
on. In addition, you'll want to try and understand/explain why those
lines were added in the 1st place and how it won't break other existing
use cases by removing them. Maybe "git blame" will help for that...
P.
--
>
> tools/cgroup/Makefile | 1 -
> tools/gpio/Makefile | 2 --
> tools/hv/Makefile | 1 -
> tools/iio/Makefile | 2 --
> tools/laptop/freefall/Makefile | 1 -
> tools/leds/Makefile | 1 -
> tools/perf/Makefile.perf | 6 ------
> tools/power/acpi/Makefile.config | 3 ---
> tools/scripts/Makefile.include | 18 ++++++++++++++++++
> tools/spi/Makefile | 2 --
> tools/usb/Makefile | 1 -
> tools/vm/Makefile | 1 -
> tools/wmi/Makefile | 1 -
> 13 files changed, 18 insertions(+), 22 deletions(-)
>
> diff --git a/tools/cgroup/Makefile b/tools/cgroup/Makefile
> index 860fa151640a..ffca068e4a76 100644
> --- a/tools/cgroup/Makefile
> +++ b/tools/cgroup/Makefile
> @@ -1,7 +1,6 @@
> # SPDX-License-Identifier: GPL-2.0
> # Makefile for cgroup tools
>
> -CC = $(CROSS_COMPILE)gcc
> CFLAGS = -Wall -Wextra
>
> all: cgroup_event_listener
> diff --git a/tools/gpio/Makefile b/tools/gpio/Makefile
> index 805a2c0cf4cd..240eda014b37 100644
> --- a/tools/gpio/Makefile
> +++ b/tools/gpio/Makefile
> @@ -12,8 +12,6 @@ endif
> # (this improves performance and avoids hard-to-debug behaviour);
> MAKEFLAGS += -r
>
> -CC = $(CROSS_COMPILE)gcc
> -LD = $(CROSS_COMPILE)ld
> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>
> ALL_TARGETS := lsgpio gpio-hammer gpio-event-mon
> diff --git a/tools/hv/Makefile b/tools/hv/Makefile
> index 31503819454d..68c2d7b059b3 100644
> --- a/tools/hv/Makefile
> +++ b/tools/hv/Makefile
> @@ -1,7 +1,6 @@
> # SPDX-License-Identifier: GPL-2.0
> # Makefile for Hyper-V tools
>
> -CC = $(CROSS_COMPILE)gcc
> WARNINGS = -Wall -Wextra
> CFLAGS = $(WARNINGS) -g $(shell getconf LFS_CFLAGS)
>
> diff --git a/tools/iio/Makefile b/tools/iio/Makefile
> index a08e7a47d6a3..332ed2f6c2c2 100644
> --- a/tools/iio/Makefile
> +++ b/tools/iio/Makefile
> @@ -12,8 +12,6 @@ endif
> # (this improves performance and avoids hard-to-debug behaviour);
> MAKEFLAGS += -r
>
> -CC = $(CROSS_COMPILE)gcc
> -LD = $(CROSS_COMPILE)ld
> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>
> ALL_TARGETS := iio_event_monitor lsiio iio_generic_buffer
> diff --git a/tools/laptop/freefall/Makefile b/tools/laptop/freefall/Makefile
> index 5f758c489a20..b572d94255f6 100644
> --- a/tools/laptop/freefall/Makefile
> +++ b/tools/laptop/freefall/Makefile
> @@ -2,7 +2,6 @@
> PREFIX ?= /usr
> SBINDIR ?= sbin
> INSTALL ?= install
> -CC = $(CROSS_COMPILE)gcc
>
> TARGET = freefall
>
> diff --git a/tools/leds/Makefile b/tools/leds/Makefile
> index c379af003807..7b6bed13daaa 100644
> --- a/tools/leds/Makefile
> +++ b/tools/leds/Makefile
> @@ -1,7 +1,6 @@
> # SPDX-License-Identifier: GPL-2.0
> # Makefile for LEDs tools
>
> -CC = $(CROSS_COMPILE)gcc
> CFLAGS = -Wall -Wextra -g -I../../include/uapi
>
> all: uledmon led_hw_brightness_mon
> diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
> index 68cf1360a3f3..3035ce5f6b36 100644
> --- a/tools/perf/Makefile.perf
> +++ b/tools/perf/Makefile.perf
> @@ -144,12 +144,6 @@ define allow-override
> $(eval $(1) = $(2)))
> endef
>
> -# Allow setting CC and AR and LD, or setting CROSS_COMPILE as a prefix.
> -$(call allow-override,CC,$(CROSS_COMPILE)gcc)
> -$(call allow-override,AR,$(CROSS_COMPILE)ar)
> -$(call allow-override,LD,$(CROSS_COMPILE)ld)
> -$(call allow-override,CXX,$(CROSS_COMPILE)g++)
> -
> LD += $(EXTRA_LDFLAGS)
>
> HOSTCC ?= gcc
> diff --git a/tools/power/acpi/Makefile.config b/tools/power/acpi/Makefile.config
> index a1883bbb0144..2cccbba64418 100644
> --- a/tools/power/acpi/Makefile.config
> +++ b/tools/power/acpi/Makefile.config
> @@ -56,9 +56,6 @@ INSTALL_SCRIPT = ${INSTALL_PROGRAM}
> # to compile vs uClibc, that can be done here as well.
> CROSS = #/usr/i386-linux-uclibc/usr/bin/i386-uclibc-
> CROSS_COMPILE ?= $(CROSS)
> -CC = $(CROSS_COMPILE)gcc
> -LD = $(CROSS_COMPILE)gcc
> -STRIP = $(CROSS_COMPILE)strip
> HOSTCC = gcc
>
> # check if compiler option is supported
> diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
> index 3fab179b1aba..09b2d0f07f66 100644
> --- a/tools/scripts/Makefile.include
> +++ b/tools/scripts/Makefile.include
> @@ -42,6 +42,24 @@ EXTRA_WARNINGS += -Wformat
>
> CC_NO_CLANG := $(shell $(CC) -dM -E -x c /dev/null | grep -Fq "__clang__"; echo $$?)
>
> +# Makefiles suck: This macro sets a default value of $(2) for the
> +# variable named by $(1), unless the variable has been set by
> +# environment or command line. This is necessary for CC and AR
> +# because make sets default values, so the simpler ?= approach
> +# won't work as expected.
> +define allow-override
> + $(if $(or $(findstring environment,$(origin $(1))),\
> + $(findstring command line,$(origin $(1)))),,\
> + $(eval $(1) = $(2)))
> +endef
> +
> +# Allow setting various cross-compile vars or setting CROSS_COMPILE as a prefix.
> +$(call allow-override,CC,$(CROSS_COMPILE)gcc)
> +$(call allow-override,AR,$(CROSS_COMPILE)ar)
> +$(call allow-override,LD,$(CROSS_COMPILE)ld)
> +$(call allow-override,CXX,$(CROSS_COMPILE)g++)
> +$(call allow-override,CXX,$(CROSS_COMPILE)strip)
> +
> ifeq ($(CC_NO_CLANG), 1)
> EXTRA_WARNINGS += -Wstrict-aliasing=3
> endif
> diff --git a/tools/spi/Makefile b/tools/spi/Makefile
> index 90615e10c79a..815d15589177 100644
> --- a/tools/spi/Makefile
> +++ b/tools/spi/Makefile
> @@ -11,8 +11,6 @@ endif
> # (this improves performance and avoids hard-to-debug behaviour);
> MAKEFLAGS += -r
>
> -CC = $(CROSS_COMPILE)gcc
> -LD = $(CROSS_COMPILE)ld
> CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
>
> ALL_TARGETS := spidev_test spidev_fdx
> diff --git a/tools/usb/Makefile b/tools/usb/Makefile
> index 4e6506078494..01d758d73b6d 100644
> --- a/tools/usb/Makefile
> +++ b/tools/usb/Makefile
> @@ -1,7 +1,6 @@
> # SPDX-License-Identifier: GPL-2.0
> # Makefile for USB tools
>
> -CC = $(CROSS_COMPILE)gcc
> PTHREAD_LIBS = -lpthread
> WARNINGS = -Wall -Wextra
> CFLAGS = $(WARNINGS) -g -I../include
> diff --git a/tools/vm/Makefile b/tools/vm/Makefile
> index be320b905ea7..20f6cf04377f 100644
> --- a/tools/vm/Makefile
> +++ b/tools/vm/Makefile
> @@ -6,7 +6,6 @@ TARGETS=page-types slabinfo page_owner_sort
> LIB_DIR = ../lib/api
> LIBS = $(LIB_DIR)/libapi.a
>
> -CC = $(CROSS_COMPILE)gcc
> CFLAGS = -Wall -Wextra -I../lib/
> LDFLAGS = $(LIBS)
>
> diff --git a/tools/wmi/Makefile b/tools/wmi/Makefile
> index e664f1167388..e0e87239126b 100644
> --- a/tools/wmi/Makefile
> +++ b/tools/wmi/Makefile
> @@ -2,7 +2,6 @@ PREFIX ?= /usr
> SBINDIR ?= sbin
> INSTALL ?= install
> CFLAGS += -D__EXPORTED_HEADERS__ -I../../include/uapi -I../../include
> -CC = $(CROSS_COMPILE)gcc
>
> TARGET = dell-smbios-example
>
> --
> 2.11.0
>
^ permalink raw reply
* [PATCH] tools: fix cross-compile var export
From: Martin Kelly @ 2018-01-06 20:16 UTC (permalink / raw)
To: kernel-kbuild, kernel-janitors, linux-embedded
Cc: Paul Gortmaker, David Woodhouse, Matt Mackall, Masahiro Yamada,
Michal Marek, linux-kernel, Martin Kelly
From: Martin Kelly <martin@martingkelly.com>
Currently in a number of Makefiles, we clobber the CC, LD, and/or STRIP env vars
when cross-compiling, which breaks any additional flags that might be set (such
as sysroot). This easily shows up by using, for instance, a Yocto SDK.
Fix this by more carefully overriding the flags in the way that the perf
Makefile does.
This patch does not fix cross-compile for all the tools (some have other bugs),
but it does appear to fix it for these:
- cgroup
- freefall
- gpio
- hv
- iio
- leds
- spi
- vm
- wmi
Signed-off-by: Martin Kelly <martin@martingkelly.com>
---
This patch touches multiple subsystems, and there doesn't appear to be a global
tools maintainer, so I sent this to linux-embedded kernel-janitors, and
linux-kbuild. Please let me know if there is a better place for it.
tools/cgroup/Makefile | 1 -
tools/gpio/Makefile | 2 --
tools/hv/Makefile | 1 -
tools/iio/Makefile | 2 --
tools/laptop/freefall/Makefile | 1 -
tools/leds/Makefile | 1 -
tools/perf/Makefile.perf | 6 ------
tools/power/acpi/Makefile.config | 3 ---
tools/scripts/Makefile.include | 18 ++++++++++++++++++
tools/spi/Makefile | 2 --
tools/usb/Makefile | 1 -
tools/vm/Makefile | 1 -
tools/wmi/Makefile | 1 -
13 files changed, 18 insertions(+), 22 deletions(-)
diff --git a/tools/cgroup/Makefile b/tools/cgroup/Makefile
index 860fa151640a..ffca068e4a76 100644
--- a/tools/cgroup/Makefile
+++ b/tools/cgroup/Makefile
@@ -1,7 +1,6 @@
# SPDX-License-Identifier: GPL-2.0
# Makefile for cgroup tools
-CC = $(CROSS_COMPILE)gcc
CFLAGS = -Wall -Wextra
all: cgroup_event_listener
diff --git a/tools/gpio/Makefile b/tools/gpio/Makefile
index 805a2c0cf4cd..240eda014b37 100644
--- a/tools/gpio/Makefile
+++ b/tools/gpio/Makefile
@@ -12,8 +12,6 @@ endif
# (this improves performance and avoids hard-to-debug behaviour);
MAKEFLAGS += -r
-CC = $(CROSS_COMPILE)gcc
-LD = $(CROSS_COMPILE)ld
CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
ALL_TARGETS := lsgpio gpio-hammer gpio-event-mon
diff --git a/tools/hv/Makefile b/tools/hv/Makefile
index 31503819454d..68c2d7b059b3 100644
--- a/tools/hv/Makefile
+++ b/tools/hv/Makefile
@@ -1,7 +1,6 @@
# SPDX-License-Identifier: GPL-2.0
# Makefile for Hyper-V tools
-CC = $(CROSS_COMPILE)gcc
WARNINGS = -Wall -Wextra
CFLAGS = $(WARNINGS) -g $(shell getconf LFS_CFLAGS)
diff --git a/tools/iio/Makefile b/tools/iio/Makefile
index a08e7a47d6a3..332ed2f6c2c2 100644
--- a/tools/iio/Makefile
+++ b/tools/iio/Makefile
@@ -12,8 +12,6 @@ endif
# (this improves performance and avoids hard-to-debug behaviour);
MAKEFLAGS += -r
-CC = $(CROSS_COMPILE)gcc
-LD = $(CROSS_COMPILE)ld
CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
ALL_TARGETS := iio_event_monitor lsiio iio_generic_buffer
diff --git a/tools/laptop/freefall/Makefile b/tools/laptop/freefall/Makefile
index 5f758c489a20..b572d94255f6 100644
--- a/tools/laptop/freefall/Makefile
+++ b/tools/laptop/freefall/Makefile
@@ -2,7 +2,6 @@
PREFIX ?= /usr
SBINDIR ?= sbin
INSTALL ?= install
-CC = $(CROSS_COMPILE)gcc
TARGET = freefall
diff --git a/tools/leds/Makefile b/tools/leds/Makefile
index c379af003807..7b6bed13daaa 100644
--- a/tools/leds/Makefile
+++ b/tools/leds/Makefile
@@ -1,7 +1,6 @@
# SPDX-License-Identifier: GPL-2.0
# Makefile for LEDs tools
-CC = $(CROSS_COMPILE)gcc
CFLAGS = -Wall -Wextra -g -I../../include/uapi
all: uledmon led_hw_brightness_mon
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 68cf1360a3f3..3035ce5f6b36 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -144,12 +144,6 @@ define allow-override
$(eval $(1) = $(2)))
endef
-# Allow setting CC and AR and LD, or setting CROSS_COMPILE as a prefix.
-$(call allow-override,CC,$(CROSS_COMPILE)gcc)
-$(call allow-override,AR,$(CROSS_COMPILE)ar)
-$(call allow-override,LD,$(CROSS_COMPILE)ld)
-$(call allow-override,CXX,$(CROSS_COMPILE)g++)
-
LD += $(EXTRA_LDFLAGS)
HOSTCC ?= gcc
diff --git a/tools/power/acpi/Makefile.config b/tools/power/acpi/Makefile.config
index a1883bbb0144..2cccbba64418 100644
--- a/tools/power/acpi/Makefile.config
+++ b/tools/power/acpi/Makefile.config
@@ -56,9 +56,6 @@ INSTALL_SCRIPT = ${INSTALL_PROGRAM}
# to compile vs uClibc, that can be done here as well.
CROSS = #/usr/i386-linux-uclibc/usr/bin/i386-uclibc-
CROSS_COMPILE ?= $(CROSS)
-CC = $(CROSS_COMPILE)gcc
-LD = $(CROSS_COMPILE)gcc
-STRIP = $(CROSS_COMPILE)strip
HOSTCC = gcc
# check if compiler option is supported
diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index 3fab179b1aba..09b2d0f07f66 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -42,6 +42,24 @@ EXTRA_WARNINGS += -Wformat
CC_NO_CLANG := $(shell $(CC) -dM -E -x c /dev/null | grep -Fq "__clang__"; echo $$?)
+# Makefiles suck: This macro sets a default value of $(2) for the
+# variable named by $(1), unless the variable has been set by
+# environment or command line. This is necessary for CC and AR
+# because make sets default values, so the simpler ?= approach
+# won't work as expected.
+define allow-override
+ $(if $(or $(findstring environment,$(origin $(1))),\
+ $(findstring command line,$(origin $(1)))),,\
+ $(eval $(1) = $(2)))
+endef
+
+# Allow setting various cross-compile vars or setting CROSS_COMPILE as a prefix.
+$(call allow-override,CC,$(CROSS_COMPILE)gcc)
+$(call allow-override,AR,$(CROSS_COMPILE)ar)
+$(call allow-override,LD,$(CROSS_COMPILE)ld)
+$(call allow-override,CXX,$(CROSS_COMPILE)g++)
+$(call allow-override,CXX,$(CROSS_COMPILE)strip)
+
ifeq ($(CC_NO_CLANG), 1)
EXTRA_WARNINGS += -Wstrict-aliasing=3
endif
diff --git a/tools/spi/Makefile b/tools/spi/Makefile
index 90615e10c79a..815d15589177 100644
--- a/tools/spi/Makefile
+++ b/tools/spi/Makefile
@@ -11,8 +11,6 @@ endif
# (this improves performance and avoids hard-to-debug behaviour);
MAKEFLAGS += -r
-CC = $(CROSS_COMPILE)gcc
-LD = $(CROSS_COMPILE)ld
CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
ALL_TARGETS := spidev_test spidev_fdx
diff --git a/tools/usb/Makefile b/tools/usb/Makefile
index 4e6506078494..01d758d73b6d 100644
--- a/tools/usb/Makefile
+++ b/tools/usb/Makefile
@@ -1,7 +1,6 @@
# SPDX-License-Identifier: GPL-2.0
# Makefile for USB tools
-CC = $(CROSS_COMPILE)gcc
PTHREAD_LIBS = -lpthread
WARNINGS = -Wall -Wextra
CFLAGS = $(WARNINGS) -g -I../include
diff --git a/tools/vm/Makefile b/tools/vm/Makefile
index be320b905ea7..20f6cf04377f 100644
--- a/tools/vm/Makefile
+++ b/tools/vm/Makefile
@@ -6,7 +6,6 @@ TARGETS=page-types slabinfo page_owner_sort
LIB_DIR = ../lib/api
LIBS = $(LIB_DIR)/libapi.a
-CC = $(CROSS_COMPILE)gcc
CFLAGS = -Wall -Wextra -I../lib/
LDFLAGS = $(LIBS)
diff --git a/tools/wmi/Makefile b/tools/wmi/Makefile
index e664f1167388..e0e87239126b 100644
--- a/tools/wmi/Makefile
+++ b/tools/wmi/Makefile
@@ -2,7 +2,6 @@ PREFIX ?= /usr
SBINDIR ?= sbin
INSTALL ?= install
CFLAGS += -D__EXPORTED_HEADERS__ -I../../include/uapi -I../../include
-CC = $(CROSS_COMPILE)gcc
TARGET = dell-smbios-example
--
2.11.0
^ permalink raw reply related
* Re: POSIX Message Queue Priority Scheduling
From: Jonathan Haws @ 2017-11-14 23:29 UTC (permalink / raw)
To: kevin.dankwardt@gmail.com; +Cc: linux-embedded@vger.kernel.org
In-Reply-To: <1510340694.15160.7.camel@sdl.usu.edu>
> Yeah - I was looking at that code and was having a hard time
> following
> it. It appears to me that if the queue is full, it goes into this
> wq_sleep(), but then never comes back to post - however it has to
> come
> back at some point when it wakes. The wq_sleep() function calls
> wq_add
> which appears to add in priority order, however it uses static_prio
> of
> the task_struct.
>
> Looking at task_struct, there are four priorities defined --
>
> int prio, static_prio, normal_prio;
> unsigned int rt_priority;
>
> What I'd like to know is the details of what each of those mean. It
> seems that if the static_prio of each thread is the same, even if the
> rt_priority is different, they would wake in FIFO order (since all
> static_prios are equal). Some searching doesn't quite answer the
> question fully. Does the message queue implementation ignore the
> fact
> that some threads are RT threads? It sure seems to...
Did a lot more digging and found the following, which I'll post here
for reference.
prio - holds the actual priority in use (after priority boosting/RT
considerations/etc.) of the process or thread
static_prio - holds the static priority (based on niceness) of the
process or thread
normal_prio - holds the dynamic priority of the process or thread. In
the testing I did this always equaled prio, but this isn't necessarily
always the case
rt_priority - holds the user-specified RT priority of the thread and
plays into the prio value if using a RT scheduler (such as SCHED_FIFO
or SCHED_RR).
I made a simple change to the ipc/mqueue.c:wq_add() routine. I changed
static_prio to prio in the if statement comparing priorities. This was
all it took.
I'm working on getting that change into mainline, but hopefully this
thread on this list will be of help to someone else.
Thanks all!
^ permalink raw reply
* Re: POSIX Message Queue Priority Scheduling
From: Jonathan Haws @ 2017-11-10 19:04 UTC (permalink / raw)
To: kevin.dankwardt@gmail.com; +Cc: linux-embedded@vger.kernel.org
In-Reply-To: <CAJUC52HYx1pWt9iHFN6Lc0-SB6wA2zBxX9nQYxzPh+R7tqdVgw@mail.gmail.com>
Yeah - I was looking at that code and was having a hard time following
it. It appears to me that if the queue is full, it goes into this
wq_sleep(), but then never comes back to post - however it has to come
back at some point when it wakes. The wq_sleep() function calls wq_add
which appears to add in priority order, however it uses static_prio of
the task_struct.
Looking at task_struct, there are four priorities defined --
int prio, static_prio, normal_prio;
unsigned int rt_priority;
What I'd like to know is the details of what each of those mean. It
seems that if the static_prio of each thread is the same, even if the
rt_priority is different, they would wake in FIFO order (since all
static_prios are equal). Some searching doesn't quite answer the
question fully. Does the message queue implementation ignore the fact
that some threads are RT threads? It sure seems to...
I'm going to post this on the linux-kernel list as well. It seems that
there may be a bug here, or maybe the thread's static_prio isn't
getting set right.
On Thu, 2017-11-09 at 21:15 -0800, kevin dankwardt wrote:
> I believe its in the kernel code ipc/mqueue.c.
>
> It looks like a sender of a message gets the receiver.
>
> receiver = wq_get_first_waiter(info, RECV);
>
> -kevin
>
> On Thu, Nov 9, 2017 at 7:13 PM, Jonathan Haws <Jonathan.Haws@sdl.usu.
> edu> wrote:
> > POSIX - I would expect them to wake in thread priority order, but
> > that
> > isn't what I'm seeing. Is it possible that while the Linux
> > scheduler
> > is priority-based, the queues do not support priority?
> >
> > A typical Linux man-page for mq_send doesn't say anything about how
> > multiple blockers wake on mq_send, but this link talks about it:
> >
> > http://pubs.opengroup.org/onlinepubs/009695399/functions/mq_send.ht
> > ml
> >
> > It seems to me that Linux does not support "Priority Scheduling"
> > when
> > it comes to POSIX message queues, but rather wakes them in FIFO
> > order.
> >
> > I'm not even sure what code-base to look at to see. It seems that
> > this
> > is a kernel thing - or are POSIX messages queue completely user-
> > space?
> > Would I want to look at the source for glibc?
> >
> >
> >
> > On Fri, 2017-11-10 at 11:56 +0900, Kevin Dankwardt wrote:
> > > Jonathan,
> > >
> > > The wakeup order need not be based on scheduler priority. The
> > > queueing mechanism determines the wake order. As I recall SYS V
> > > semaphores wake in FIFO order but POSIX semaphors wake in
> > > process/thread priority order.
> > >
> > > Are you really using POSIX msg queues or sys V.
> > >
> > > -kevin
> > >
> > > >
> > > > On Nov 10, 2017, at 11:43 AM, Jonathan Haws <Jonathan.Haws@sdl.
> > usu.
> > > > edu> wrote:
> > > >
> > > > Kevin,
> > > >
> > > > I understand that - they are different. What I'm doing is
> > having
> > > > multiple threads block on a full queue and those threads should
> > > > wake up
> > > > in *thread priority order* when the queue becomes available. I
> > > > have
> > > > those threads post a message with a *message priority* equal to
> > the
> > > > *thread priority*.
> > > >
> > > > When messages come out of the queue, I should see that the
> > threads
> > > > woke
> > > > up in thread priority order. However, I'm seeing them wake up
> > in
> > > > the
> > > > order that they blocked on the queue which seems incorrect to
> > me
> > > > since
> > > > Linux runs a priority-based scheduler.
> > > >
> > > >
> > > > >
> > > > > On Fri, 2017-11-10 at 11:39 +0900, Kevin Dankwardt wrote:
> > > > > Jonathan,
> > > > >
> > > > > Check the man page to see that priority is not process
> > scheduling
> > > > > priority.
> > > > >
> > > > > -kevin
> > > > >
> > > > > >
> > > > > >
> > > > > > On Nov 10, 2017, at 11:34 AM, Jonathan Haws <Jonathan.Haws@
> > sdl.
> > > > > > usu.
> > > > > > edu> wrote:
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Can someone explain to me how message queues handle
> > waking
> > > > > > > > multiple
> > > > > > > > threads blocked on a single message queue?
> > > > > > > >
> > > > > > > > My situation is I have multiple writers blocking on a
> > full
> > > > > > > > message
> > > > > > > > queue, each posting messages with priority equal to the
> > > > > > > > thread
> > > > > > > > priority. I want to make sure they wake and post in
> > > > > > > > priority
> > > > > > > > order,
> > > > > > > > however my application is behaving as if they are
> > waking in
> > > > > > > > FIFO
> > > > > > > > order
> > > > > > > > (i.e. the order in which they blocked). Each blocking
> > > > > > > > thread
> > > > > > > > is
> > > > > > > > scheduled with the SCHED_FIFO policy with a different
> > > > > > > > priority
> > > > > > > > with
> > > > > > > > system level scope.
> > > > > > > >
> > > > > > > > I've searched the Internet high and low for something
> > > > > > > > describing
> > > > > > > > how
> > > > > > > > this should work and all I can find is POSIX man pages
> > > > > > > > describing
> > > > > > > > that
> > > > > > > > multiple blockers wake in priority order **if Priority
> > > > > > > > Scheduling
> > > > > > > > is
> > > > > > > > supported**. Since the kernel scheduler is a priority
> > > > > > > > scheduler I
> > > > > > > > would think that the threads would wake in priority
> > order
> > > > > > > > and
> > > > > > > > post
> > > > > > > > to
> > > > > > > > the queue, however that doesn't appear to be the
> > case. I'm
> > > > > > > > sure
> > > > > > > > I'm
> > > > > > > > just missing some subtle detail and was hoping the
> > experts
> > > > > > > > here
> > > > > > > > on
> > > > > > > > this list can help shine some light on what I'm seeing,
> > > > > > > > since
> > > > > > > > its
> > > > > > > > at
> > > > > > > > the kernel level that these threads are made ready to
> > run.
> > > > > > > >
> > > > > > > > I have a small test application that I can post here if
> > > > > > > > necessary. If
> > > > > > > > this isn't the right place to ask this, by all means
> > let me
> > > > > > > > know
> > > > > > > > and
> > > > > > > > direct me to the right forum.
> > > > > > Bump...anyone have any thoughts on this? I haven't been
> > able
> > > > > > to
> > > > > > find
> > > > > > anything on this.
> >
>
>
^ permalink raw reply
* Re: POSIX Message Queue Priority Scheduling
From: Jonathan Haws @ 2017-11-10 2:34 UTC (permalink / raw)
To: kevin.dankwardt@gmail.com; +Cc: linux-embedded@vger.kernel.org
In-Reply-To: <CAJUC52E3gsjSzdkvJD+gL8WSp93ZtabhShX_BHBZJRLDYWZbAg@mail.gmail.com>
> > Can someone explain to me how message queues handle waking multiple
> > threads blocked on a single message queue?
> >
> > My situation is I have multiple writers blocking on a full message
> > queue, each posting messages with priority equal to the thread
> > priority. I want to make sure they wake and post in priority
> > order,
> > however my application is behaving as if they are waking in FIFO
> > order
> > (i.e. the order in which they blocked). Each blocking thread is
> > scheduled with the SCHED_FIFO policy with a different priority with
> > system level scope.
> >
> > I've searched the Internet high and low for something describing
> > how
> > this should work and all I can find is POSIX man pages describing
> > that
> > multiple blockers wake in priority order **if Priority Scheduling
> > is
> > supported**. Since the kernel scheduler is a priority scheduler I
> > would think that the threads would wake in priority order and post
> > to
> > the queue, however that doesn't appear to be the case. I'm sure
> > I'm
> > just missing some subtle detail and was hoping the experts here on
> > this list can help shine some light on what I'm seeing, since its
> > at
> > the kernel level that these threads are made ready to run.
> >
> > I have a small test application that I can post here if
> > necessary. If
> > this isn't the right place to ask this, by all means let me know
> > and
> > direct me to the right forum.
Bump...anyone have any thoughts on this? I haven't been able to find
anything on this.
^ permalink raw reply
* Re: POSIX Message Queue Priority Scheduling
From: Jonathan Haws @ 2017-10-31 16:49 UTC (permalink / raw)
To: kevin.dankwardt@gmail.com; +Cc: linux-embedded@vger.kernel.org
In-Reply-To: <CAJUC52E3gsjSzdkvJD+gL8WSp93ZtabhShX_BHBZJRLDYWZbAg@mail.gmail.com>
You're right about that - but my thread priorities are set such that I
have one thread at priority 25, one at 30, one at 35, etc. Each of
those ends up blocking on the mq_send at the same time. When space
becomes available in the queue, I would expect the thread with the
highest priority to wake up and post its message. The priority of the
message posted simply matched the priority of the thread posting so
that I could tell who posted.
I'm seeing that the thread who blocks first wakes up first, even if
they have a lower priority. This is not as expected - I expect the
threads to wake in priority order, but that doesn't seem to be the
case.
Would it help if I sent over my test application?
^ permalink raw reply
* POSIX Message Queue Priority Scheduling
From: Jonathan Haws @ 2017-10-31 15:32 UTC (permalink / raw)
To: linux-embedded@vger.kernel.org
Can someone explain to me how message queues handle waking multiple
threads blocked on a single message queue?
My situation is I have multiple writers blocking on a full message
queue, each posting messages with priority equal to the thread
priority. I want to make sure they wake and post in priority order,
however my application is behaving as if they are waking in FIFO order
(i.e. the order in which they blocked). Each blocking thread is
scheduled with the SCHED_FIFO policy with a different priority with
system level scope.
I've searched the Internet high and low for something describing how
this should work and all I can find is POSIX man pages describing that
multiple blockers wake in priority order **if Priority Scheduling is
supported**. Since the kernel scheduler is a priority scheduler I
would think that the threads would wake in priority order and post to
the queue, however that doesn't appear to be the case. I'm sure I'm
just missing some subtle detail and was hoping the experts here on
this list can help shine some light on what I'm seeing, since its at
the kernel level that these threads are made ready to run.
I have a small test application that I can post here if necessary. If
this isn't the right place to ask this, by all means let me know and
direct me to the right forum.
Thanks!
Jon
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox