[PATCH 0/6] Address issues with SPDX requirements and PEP-263

[PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Mauro Carvalho Chehab]

The  description at Documentation/process/license-rules.rst is very strict
with regards to the position where the SPDX tags should be.

In the past several developers and maintainers interpreted it on a
more permissive way, placing the SPDX header between lines 1 to 15, 
with are the ones which the  scripts/spdxcheck.py script verifies.

However, recently, devs are becoming more strict about such
requirement and want it to strictly follow the rule, with states that
the SPDX rule should be at the first line ever on most files, and
at the second line for scripts.

Well, for Python script, such requirement causes violation to PEP-263, 
making regressions on scripts that contain encoding lines, as PEP-263
also states about the same.

This series addresses it.

Patches 1 to 3 fix some Python scripts that violates PEP-263;

Patch 4 mentions PEP-263 for Python scripts, allowing to go up to
line 3, when both "#!" and the encoding line is found;

Patch 5 changes the scripts/spdxcheck.py for it to identify on what
line each SPDX header is found, optinally allowing to print an histogram
about that;

Patch 6 adds a pedantic^Wstrict mode to scripts/spdxcheck.py,
making it to also check for violations at the line with contains the
SPDX header.

PS.: I sent already a RFC version for those patches along with this
thread:

    https://lore.kernel.org/lkml/b32c2e46b91e7bcda2a9bd140673f06d71b2487a.camel@perches.com/

Mauro Carvalho Chehab (6):
  docs: sphinx: add SPDX header for some sphinx extensions
  tools: perf: fix SPDX header in the light of PEP-263
  tools: intel_pstate_tracer.py: fix SPDX header in the light of PEP-263
  docs: license-rules.txt: cover SPDX headers on Python scripts
  scripts/spdxcheck.py: keep track on what line SPDX header was found
  scripts/spdxcheck.py: check if the line number follows the strict rule

 Documentation/process/license-rules.rst       |  7 ++-
 Documentation/sphinx/kernel_include.py        |  1 +
 Documentation/sphinx/rstFlatTable.py          |  1 +
 scripts/spdxcheck.py                          | 55 +++++++++++++++----
 tools/perf/python/tracepoint.py               |  3 +-
 tools/perf/python/twatch.py                   |  3 +-
 .../intel_pstate_tracer.py                    |  2 +-
 7 files changed, 55 insertions(+), 17 deletions(-)

-- 
2.21.0

[PATCH 1/6] docs: sphinx: add SPDX header for some sphinx extensions

[Mauro Carvalho Chehab]

Those extensions are released under GPLv2, as stated at the
:license: markup tag.

Add the corresponding SPDX tags for such license.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/sphinx/kernel_include.py | 1 +
 Documentation/sphinx/rstFlatTable.py   | 1 +
 2 files changed, 2 insertions(+)

diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/kernel_include.py
index f523aa68a36b..7aaea4e31f78 100755
--- a/Documentation/sphinx/kernel_include.py
+++ b/Documentation/sphinx/kernel_include.py
@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8; mode: python -*-
+# SPDX-License-Identifier: GPL-2.0
 # pylint: disable=R0903, C0330, R0914, R0912, E0401
 
 u"""
diff --git a/Documentation/sphinx/rstFlatTable.py b/Documentation/sphinx/rstFlatTable.py
index 2019a55f6b18..15769d01831b 100755
--- a/Documentation/sphinx/rstFlatTable.py
+++ b/Documentation/sphinx/rstFlatTable.py
@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8; mode: python -*-
+# SPDX-License-Identifier: GPL-2.0
 # pylint: disable=C0330, R0903, R0912
 
 u"""
-- 
2.21.0

[PATCH 4/6] docs: license-rules.txt: cover SPDX headers on Python scripts

[Mauro Carvalho Chehab]

The author of the license-rules.rst file wanted to be very restrict
with regards to the location of the SPDX header. It says that
the SPDX header "shall be added at the first  possible line in
a file which can contain a comment". Not happy with this already
restrictive requiement, it goes further:

"For the majority  of files this is the first line, except for
scripts", opening an exception to have the SPDX header at the
second line, if the first line starts with "#!".

Well, it turns that this is too restrictive for Python scripts,
and may cause regressions if this would be enforced.

As mentioned on:
	https://stackoverflow.com/questions/728891/correct-way-to-define-python-source-code-encoding

Python's PEP-263 [1] dictates that an script that needs to default to
UTF-8 encoding has to follow this rule:

	'Python will default to ASCII as standard encoding if no other
	 encoding hints are given.

	 To define a source code encoding, a magic comment must be placed
	 into the source files either as first or second line in the file'

And:
	'More precisely, the first or second line must match the following
	 regular expression:

	 ^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)'

[1] https://www.python.org/dev/peps/pep-0263/

If a script has both "#!" and the charset encoding line, we can't place
a SPDX tag without either violating license-rules.rst or breaking the
script by making it crash with non-ASCII characters.

So, add a sort notice saying that, for Python scripts, the SPDX
header may be up to the third line, in order to cover the case
where both "#!" and "# .*coding.*UTF-8" lines are found.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/process/license-rules.rst | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/Documentation/process/license-rules.rst b/Documentation/process/license-rules.rst
index 2ef44ada3f11..5d23e3498b1c 100644
--- a/Documentation/process/license-rules.rst
+++ b/Documentation/process/license-rules.rst
@@ -64,9 +64,12 @@ License identifier syntax
    possible line in a file which can contain a comment.  For the majority
    of files this is the first line, except for scripts which require the
    '#!PATH_TO_INTERPRETER' in the first line.  For those scripts the SPDX
-   identifier goes into the second line.
+   identifier goes into the second line\ [1]_.
 
-|
+.. [1] Please notice that Python scripts may also need an encoding rule
+   as defined on PEP-263, which should be defined either at the first
+   or the second line. So, for such scripts, the SPDX identifier may
+   go up to the third line.
 
 2. Style:
 
-- 
2.21.0

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Joe Perches]

On Thu, 2019-09-05 at 16:57 -0300, Mauro Carvalho Chehab wrote:
> The  description at Documentation/process/license-rules.rst is very strict
> with regards to the position where the SPDX tags should be.
[]
> PS.: I sent already a RFC version for those patches along with this
> thread:
> 
>     https://lore.kernel.org/lkml/b32c2e46b91e7bcda2a9bd140673f06d71b2487a.camel@perches.com/

Nice, thanks.

Now I guess I have to update checkpatch too.
(unless you want to... ;)

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Mauro Carvalho Chehab]

Em Thu, 05 Sep 2019 13:05:08 -0700
Joe Perches <joe@perches.com> escreveu:

> On Thu, 2019-09-05 at 16:57 -0300, Mauro Carvalho Chehab wrote:
> > The  description at Documentation/process/license-rules.rst is very strict
> > with regards to the position where the SPDX tags should be.  
> []
> > PS.: I sent already a RFC version for those patches along with this
> > thread:
> > 
> >     https://lore.kernel.org/lkml/b32c2e46b91e7bcda2a9bd140673f06d71b2487a.camel@perches.com/  
> 
> Nice, thanks.
> 
> Now I guess I have to update checkpatch too.
> (unless you want to... ;)

Yeah, it makes sense to update checkpatch too. Perhaps it could be
changed to use the "-s" flag on the check, instead of implementing
its own logic to handle the position.

Thanks,
Mauro

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Joe Perches]

On Fri, 2019-09-06 at 09:02 -0300, Mauro Carvalho Chehab wrote:
> Em Thu, 05 Sep 2019 13:05:08 -0700
> Joe Perches <joe@perches.com> escreveu:
> 
> > On Thu, 2019-09-05 at 16:57 -0300, Mauro Carvalho Chehab wrote:
> > > The  description at Documentation/process/license-rules.rst is very strict
> > > with regards to the position where the SPDX tags should be.  
> > []
> > > PS.: I sent already a RFC version for those patches along with this
> > > thread:
> > > 
> > >     https://lore.kernel.org/lkml/b32c2e46b91e7bcda2a9bd140673f06d71b2487a.camel@perches.com/  
> > 
> > Nice, thanks.
> > 
> > Now I guess I have to update checkpatch too.
> > (unless you want to... ;)
> 
> Yeah, it makes sense to update checkpatch too. Perhaps it could be
> changed to use the "-s" flag on the check, instead of implementing
> its own logic to handle the position.

I think checkpatch needs its own logic as its
source input is a patch file.

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Jonathan Corbet]

On Thu,  5 Sep 2019 16:57:47 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:

> The  description at Documentation/process/license-rules.rst is very strict
> with regards to the position where the SPDX tags should be.
> 
> In the past several developers and maintainers interpreted it on a
> more permissive way, placing the SPDX header between lines 1 to 15, 
> with are the ones which the  scripts/spdxcheck.py script verifies.
> 
> However, recently, devs are becoming more strict about such
> requirement and want it to strictly follow the rule, with states that
> the SPDX rule should be at the first line ever on most files, and
> at the second line for scripts.
> 
> Well, for Python script, such requirement causes violation to PEP-263, 
> making regressions on scripts that contain encoding lines, as PEP-263
> also states about the same.
> 
> This series addresses it.

So I really don't want to be overly difficult here, but I would like to
approach this from yet another angle...

> Patches 1 to 3 fix some Python scripts that violates PEP-263;

I just checked all of those scripts, and they are all just plain ASCII.
So it really doesn't matter whether the environment defaults to UTF-8 or
ASCII here.  So, in other words, we really shouldn't need to define the
encoding at all.

This suggests to me that we're adding a bunch of complications that we
don't necessarily need.  What am I missing here?

Educate me properly and I'll not try to stand in the way of all this...

Thanks,

jon

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Markus Heiser]

Am 07.09.19 um 15:34 schrieb Jonathan Corbet:
> On Thu,  5 Sep 2019 16:57:47 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> 
>> The  description at Documentation/process/license-rules.rst is very strict
>> with regards to the position where the SPDX tags should be.
>>
>> In the past several developers and maintainers interpreted it on a
>> more permissive way, placing the SPDX header between lines 1 to 15,
>> with are the ones which the  scripts/spdxcheck.py script verifies.
>>
>> However, recently, devs are becoming more strict about such
>> requirement and want it to strictly follow the rule, with states that
>> the SPDX rule should be at the first line ever on most files, and
>> at the second line for scripts.
>>
>> Well, for Python script, such requirement causes violation to PEP-263,
>> making regressions on scripts that contain encoding lines, as PEP-263
>> also states about the same.
>>
>> This series addresses it.
> 
> So I really don't want to be overly difficult here, but I would like to
> approach this from yet another angle...
> 
>> Patches 1 to 3 fix some Python scripts that violates PEP-263;
> 
> I just checked all of those scripts, and they are all just plain ASCII.
> So it really doesn't matter whether the environment defaults to UTF-8 or
> ASCII here.  So, in other words, we really shouldn't need to define the
> encoding at all.
> 

Thats what I mean [1] .. lets patch the description in the license-rules.rst::

- first line for the OS (shebang)
- second line for environment (python-encoding, editor-mode, ...)
- third and more lines for application (SPDX use) ..

[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg33240.html

-- Markus --

> This suggests to me that we're adding a bunch of complications that we
> don't necessarily need.  What am I missing here?
> 
> Educate me properly and I'll not try to stand in the way of all this...
> 
> Thanks,
> 
> jon
> 

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Mauro Carvalho Chehab]

Em Sat, 7 Sep 2019 16:36:36 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 07.09.19 um 15:34 schrieb Jonathan Corbet:
> > On Thu,  5 Sep 2019 16:57:47 -0300
> > Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> >   
> >> The  description at Documentation/process/license-rules.rst is very strict
> >> with regards to the position where the SPDX tags should be.
> >>
> >> In the past several developers and maintainers interpreted it on a
> >> more permissive way, placing the SPDX header between lines 1 to 15,
> >> with are the ones which the  scripts/spdxcheck.py script verifies.
> >>
> >> However, recently, devs are becoming more strict about such
> >> requirement and want it to strictly follow the rule, with states that
> >> the SPDX rule should be at the first line ever on most files, and
> >> at the second line for scripts.
> >>
> >> Well, for Python script, such requirement causes violation to PEP-263,
> >> making regressions on scripts that contain encoding lines, as PEP-263
> >> also states about the same.
> >>
> >> This series addresses it.  
> > 
> > So I really don't want to be overly difficult here, but I would like to
> > approach this from yet another angle...
> >   
> >> Patches 1 to 3 fix some Python scripts that violates PEP-263;  
> > 
> > I just checked all of those scripts, and they are all just plain ASCII.
> > So it really doesn't matter whether the environment defaults to UTF-8 or
> > ASCII here.  So, in other words, we really shouldn't need to define the
> > encoding at all.

I'm not a python expert, but, from what I researched, and from what I 
understood from Markus, if a script tries to print an UTF-8 but the
system's encoding is ASCII (or some other encoding), the python script
will crash.

At least on media, we define that some Kernel strings can be UTF-8. 
See, for example the model field at the media_entity struct:

	https://linuxtv.org/downloads/v4l-dvb-apis/kapi/mc-core.html

As stated there:

	"media_entity.model must be filled with the device model name as
	 a NUL-terminated UTF-8 string. The device/model revision must
	 not be stored in this field."

I've no idea if the two perf scripts that contain the encoding data are
meant to print some strings that may be UTF-8 encoding (like those that
we have at the media subsystem), or if it is just that whomever added
were using e-macs and wanted to make his life simpler. As it is better
to be safe then sorry, on patches 2 and 3, I'm assuming the first case.

In any case, we do need the encoding line at Sphinx extensions, 
although there, the shebang line is optional.

In other words, we have those alternatives:

1) Neither shebang nor coding -> SPDX will be at first line;
2) shebang + SPDX -> SPDX will be at the second line;
3) shebang + coding + SPDX -> SPDX will be at the third line;
4) coding + SPDX

   This is something that only makes sense for Sphinx extensions.

   IMHO, I would place SPDX at the second line too, but I *guess* Python
   may accept it at the first line and would still properly evaluate
   coding (as this technically satisfies the text at PEP-263).

> >   
> 
> Thats what I mean [1] .. lets patch the description in the license-rules.rst::
> 
> - first line for the OS (shebang)
> - second line for environment (python-encoding, editor-mode, ...)
> - third and more lines for application (SPDX use) ..
> 
> [1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg33240.html
> 
> -- Markus --
> 
> > This suggests to me that we're adding a bunch of complications that we
> > don't necessarily need.  What am I missing here?
> > 
> > Educate me properly and I'll not try to stand in the way of all this...
> > 
> > Thanks,
> > 
> > jon
> >   



Thanks,
Mauro

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Markus Heiser]

Am 07.09.19 um 18:22 schrieb Mauro Carvalho Chehab:
> Em Sat, 7 Sep 2019 16:36:36 +0200
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
> 
>> Am 07.09.19 um 15:34 schrieb Jonathan Corbet:
>>> On Thu,  5 Sep 2019 16:57:47 -0300
>>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
>>>    
>>>> The  description at Documentation/process/license-rules.rst is very strict
>>>> with regards to the position where the SPDX tags should be.
>>>>
>>>> In the past several developers and maintainers interpreted it on a
>>>> more permissive way, placing the SPDX header between lines 1 to 15,
>>>> with are the ones which the  scripts/spdxcheck.py script verifies.
>>>>
>>>> However, recently, devs are becoming more strict about such
>>>> requirement and want it to strictly follow the rule, with states that
>>>> the SPDX rule should be at the first line ever on most files, and
>>>> at the second line for scripts.
>>>>
>>>> Well, for Python script, such requirement causes violation to PEP-263,
>>>> making regressions on scripts that contain encoding lines, as PEP-263
>>>> also states about the same.
>>>>
>>>> This series addresses it.
>>>
>>> So I really don't want to be overly difficult here, but I would like to
>>> approach this from yet another angle...
>>>    
>>>> Patches 1 to 3 fix some Python scripts that violates PEP-263;
>>>
>>> I just checked all of those scripts, and they are all just plain ASCII.
>>> So it really doesn't matter whether the environment defaults to UTF-8 or
>>> ASCII here.  So, in other words, we really shouldn't need to define the
>>> encoding at all.
> 
> I'm not a python expert, but, from what I researched, and from what I
> understood from Markus, if a script tries to print an UTF-8 but the
> system's encoding is ASCII (or some other encoding), the python script
> will crash.

An (uncatched) exception is thrown, when writing UTF-8 to a stream which
do not support UTF-8 .. this is not a crash, it mostly indicates that the
developper makes some wrong assumption about the use-case.  There exists
also the possibility to encode the UTF-8 to ASCII and replace unknown
code points in the out-stream, or to catch the exception.

But this was only academical, where do we have such problems in practice?

> At least on media, we define that some Kernel strings can be UTF-8.
> See, for example the model field at the media_entity struct:
> 
> 	https://linuxtv.org/downloads/v4l-dvb-apis/kapi/mc-core.html
> 
> As stated there:
> 
> 	"media_entity.model must be filled with the device model name as
> 	 a NUL-terminated UTF-8 string. The device/model revision must
> 	 not be stored in this field."
> 
> I've no idea if the two perf scripts that contain the encoding data are
> meant to print some strings that may be UTF-8 encoding (like those that
> we have at the media subsystem), or if it is just that whomever added
> were using e-macs and wanted to make his life simpler. As it is better
> to be safe then sorry, on patches 2 and 3, I'm assuming the first case.

Hm, I'am unsure if I understand you correct: Using UTF-8 in the .rst
files are fine .. where do we have scripts generating UTF-8 outputs?
(except the HTML output).

> 
> In any case, we do need the encoding line at Sphinx extensions,
> although there, the shebang line is optional.
> 
> In other words, we have those alternatives:
> 
> 1) Neither shebang nor coding -> SPDX will be at first line;
> 2) shebang + SPDX -> SPDX will be at the second line;
> 3) shebang + coding + SPDX -> SPDX will be at the third line;
> 4) coding + SPDX
> 
>     This is something that only makes sense for Sphinx extensions.
> 
>     IMHO, I would place SPDX at the second line too, but I *guess* Python
>     may accept it at the first line and would still properly evaluate
>     coding (as this technically satisfies the text at PEP-263).

Why you are so restrictive .. what we normal do:

- write a shebang line if this file is called directly from the
   command line .. but we do not need shebangs on py modules which
   are imported from other modules or scripts

- write a encoding line if it is need or helpful / mostly it is helpful
   to know the encoding of a text/code file.

- add a SPDX tag

At the end we will have files with one, two or all three of this lines.
And the oder of this lines is, what I wrote:

>>
>> Thats what I mean [1] .. lets patch the description in the license-rules.rst::
>>
>> - first line for the OS (shebang)
>> - second line for environment (python-encoding, editor-mode, ...)
>> - third and more lines for application (SPDX use) ..
>>
>> [1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg33240.html
>>
>> -- Markus --
>>
>>> This suggests to me that we're adding a bunch of complications that we
>>> don't necessarily need.  What am I missing here?
>>>
>>> Educate me properly and I'll not try to stand in the way of all this...
>>>


It seems like it is not only me who is mising something .. what are
the use-cases we have py-Exceptions, what are the use-cases to be so
restrictive as you described above.

.. or did alice get lost in the cave?

Thanks for your patience with me

-- Markus --

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Mauro Carvalho Chehab]

Em Sat, 7 Sep 2019 19:33:06 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 07.09.19 um 18:22 schrieb Mauro Carvalho Chehab:
> > Em Sat, 7 Sep 2019 16:36:36 +0200
> > Markus Heiser <markus.heiser@darmarit.de> escreveu:
> >   
> >> Am 07.09.19 um 15:34 schrieb Jonathan Corbet:  
> >>> On Thu,  5 Sep 2019 16:57:47 -0300
> >>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> >>>      
> >>>> The  description at Documentation/process/license-rules.rst is very strict
> >>>> with regards to the position where the SPDX tags should be.
> >>>>
> >>>> In the past several developers and maintainers interpreted it on a
> >>>> more permissive way, placing the SPDX header between lines 1 to 15,
> >>>> with are the ones which the  scripts/spdxcheck.py script verifies.
> >>>>
> >>>> However, recently, devs are becoming more strict about such
> >>>> requirement and want it to strictly follow the rule, with states that
> >>>> the SPDX rule should be at the first line ever on most files, and
> >>>> at the second line for scripts.
> >>>>
> >>>> Well, for Python script, such requirement causes violation to PEP-263,
> >>>> making regressions on scripts that contain encoding lines, as PEP-263
> >>>> also states about the same.
> >>>>
> >>>> This series addresses it.  
> >>>
> >>> So I really don't want to be overly difficult here, but I would like to
> >>> approach this from yet another angle...
> >>>      
> >>>> Patches 1 to 3 fix some Python scripts that violates PEP-263;  
> >>>
> >>> I just checked all of those scripts, and they are all just plain ASCII.
> >>> So it really doesn't matter whether the environment defaults to UTF-8 or
> >>> ASCII here.  So, in other words, we really shouldn't need to define the
> >>> encoding at all.  
> > 
> > I'm not a python expert, but, from what I researched, and from what I
> > understood from Markus, if a script tries to print an UTF-8 but the
> > system's encoding is ASCII (or some other encoding), the python script
> > will crash.  
> 
> An (uncatched) exception is thrown, when writing UTF-8 to a stream which
> do not support UTF-8 .. this is not a crash, it mostly indicates that the
> developper makes some wrong assumption about the use-case.

A not-handled exception is a crash in Python. I've seen python scripts
crash countless times with non-English names.

> There exists
> also the possibility to encode the UTF-8 to ASCII and replace unknown
> code points in the out-stream, or to catch the exception.

Yeah, but getting this right is very painful. I use patchwork since 2013.
It took *years* for it to not crash with non-ASCII chars[1]. That's, btw,
the primary reason why I don't usually use python: with other languages,
an alien char doesn't cause a crash.

[1] I might be wrong, but the last patch I saw addressing an issue
    there was applied this year.

> 
> But this was only academical, where do we have such problems in practice?
> 
> > At least on media, we define that some Kernel strings can be UTF-8.
> > See, for example the model field at the media_entity struct:
> > 
> > 	https://linuxtv.org/downloads/v4l-dvb-apis/kapi/mc-core.html
> > 
> > As stated there:
> > 
> > 	"media_entity.model must be filled with the device model name as
> > 	 a NUL-terminated UTF-8 string. The device/model revision must
> > 	 not be stored in this field."
> > 
> > I've no idea if the two perf scripts that contain the encoding data are
> > meant to print some strings that may be UTF-8 encoding (like those that
> > we have at the media subsystem), or if it is just that whomever added
> > were using e-macs and wanted to make his life simpler. As it is better
> > to be safe then sorry, on patches 2 and 3, I'm assuming the first case.  
> 
> Hm, I'am unsure if I understand you correct: Using UTF-8 in the .rst
> files are fine .. where do we have scripts generating UTF-8 outputs?
> (except the HTML output).

In thesis, perf scripts may be reading strings from the Kernel, with 
might be using UTF-8 encoding.

> 
> > 
> > In any case, we do need the encoding line at Sphinx extensions,
> > although there, the shebang line is optional.
> > 
> > In other words, we have those alternatives:
> > 
> > 1) Neither shebang nor coding -> SPDX will be at first line;
> > 2) shebang + SPDX -> SPDX will be at the second line;
> > 3) shebang + coding + SPDX -> SPDX will be at the third line;
> > 4) coding + SPDX
> > 
> >     This is something that only makes sense for Sphinx extensions.
> > 
> >     IMHO, I would place SPDX at the second line too, but I *guess* Python
> >     may accept it at the first line and would still properly evaluate
> >     coding (as this technically satisfies the text at PEP-263).  
> 
> Why you are so restrictive .. 

No idea. I would actually prefer to just remove the restriction, and let
the SPDX header to be anywhere inside the first comment block inside a 
file [2].

That's basically how this thread started: other developers think 
that it is a good idea to be pedantic. So, be it, but let's then fix
the documentation, as the way it is, it is implicitly forbidding the
addition of encoding lines for Python scripts.

[2] I *suspect* that the restriction was added in order to make
    ./scripts/spdxcheck.py to run faster and to avoid false positives.
    Right now, if the maximum limit is removed (or set to a very high
    value), there will be one false positive:

	Documentation/dev-tools/kselftest.rst

    This doc has a SPDX-like tag at line 230, asking people to add SPDX
    headers on files, but the file itself doesn't have its own SPDX tag. 
	
> what we normal do:
> 
> - write a shebang line if this file is called directly from the
>    command line .. but we do not need shebangs on py modules which
>    are imported from other modules or scripts
> 
> - write a encoding line if it is need or helpful / mostly it is helpful
>    to know the encoding of a text/code file.
> 
> - add a SPDX tag

Yes, but this violates the current documentation, as it doesn't allow the
SPDX tag after line #2.

> 
> At the end we will have files with one, two or all three of this lines.
> And the oder of this lines is, what I wrote:
> 
> >>
> >> Thats what I mean [1] .. lets patch the description in the license-rules.rst::
> >>
> >> - first line for the OS (shebang)
> >> - second line for environment (python-encoding, editor-mode, ...)
> >> - third and more lines for application (SPDX use) ..
> >>
> >> [1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg33240.html
> >>
> >> -- Markus --
> >>  
> >>> This suggests to me that we're adding a bunch of complications that we
> >>> don't necessarily need.  What am I missing here?
> >>>
> >>> Educate me properly and I'll not try to stand in the way of all this...
> >>>  
> 
> 
> It seems like it is not only me who is mising something .. what are
> the use-cases we have py-Exceptions, what are the use-cases to be so
> restrictive as you described above.
> 
> .. or did alice get lost in the cave?
> 
> Thanks for your patience with me
> 
> -- Markus --



Thanks,
Mauro

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Markus Heiser]

Am 07.09.19 um 20:04 schrieb Mauro Carvalho Chehab:
> Em Sat, 7 Sep 2019 19:33:06 +0200
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
>> An (uncatched) exception is thrown, when writing UTF-8 to a stream which
>> do not support UTF-8 .. this is not a crash, it mostly indicates that the
>> developper makes some wrong assumption about the use-case.
> 
> A not-handled exception is a crash in Python. I've seen python scripts
> crash countless times with non-English names.

This has nothing to do with the language, ask the developer of those scripts.

>> There exists
>> also the possibility to encode the UTF-8 to ASCII and replace unknown
>> code points in the out-stream, or to catch the exception.
> 
> Yeah, but getting this right is very painful. I use patchwork since 2013.
> It took *years* for it to not crash with non-ASCII chars[1]. That's, btw,
> the primary reason why I don't usually use python: with other languages,
> an alien char doesn't cause a crash.

Python cares encoded (text) string-types while other languages and
application are just piping bytes to streams .. if you care about the
enconding you need exceptions when one whants write UTF-8 to ASCII out.

Anyway this is a bit of nitpicking / not helping here ..

> 
> [1] I might be wrong, but the last patch I saw addressing an issue
>      there was applied this year.

I alrady postet an example [1]

<snip>
This means your application has to know the encoding of a stream/file.
E.g. we handle the output from of the external Perl script
scripts/kernel-docs by encoding the byte stream from proc-call's
stdout into utf-8:

    out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8')

see patch 
https://github.com/torvalds/linux/commit/86c0f046a8b0c23fca65f77333c233a06c25ef9a

Again, this is talking about application development and has
nothing to do with the encoding of the source files.
<snap>

[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg33240.html

>>
>> But this was only academical, where do we have such problems in practice?
>>
>>> At least on media, we define that some Kernel strings can be UTF-8.
>>> See, for example the model field at the media_entity struct:
>>>
>>> 	https://linuxtv.org/downloads/v4l-dvb-apis/kapi/mc-core.html
>>>
>>> As stated there:
>>>
>>> 	"media_entity.model must be filled with the device model name as
>>> 	 a NUL-terminated UTF-8 string. The device/model revision must
>>> 	 not be stored in this field."
>>>
>>> I've no idea if the two perf scripts that contain the encoding data are
>>> meant to print some strings that may be UTF-8 encoding (like those that
>>> we have at the media subsystem), or if it is just that whomever added
>>> were using e-macs and wanted to make his life simpler. As it is better
>>> to be safe then sorry, on patches 2 and 3, I'm assuming the first case.
>>
>> Hm, I'am unsure if I understand you correct: Using UTF-8 in the .rst
>> files are fine .. where do we have scripts generating UTF-8 outputs?
>> (except the HTML output).
> 
> In thesis, perf scripts may be reading strings from the Kernel, with
> might be using UTF-8 encoding.
> 
>>
>>>
>>> In any case, we do need the encoding line at Sphinx extensions,
>>> although there, the shebang line is optional.
>>>
>>> In other words, we have those alternatives:
>>>
>>> 1) Neither shebang nor coding -> SPDX will be at first line;
>>> 2) shebang + SPDX -> SPDX will be at the second line;
>>> 3) shebang + coding + SPDX -> SPDX will be at the third line;
>>> 4) coding + SPDX
>>>
>>>      This is something that only makes sense for Sphinx extensions.
>>>
>>>      IMHO, I would place SPDX at the second line too, but I *guess* Python
>>>      may accept it at the first line and would still properly evaluate
>>>      coding (as this technically satisfies the text at PEP-263).
>>
>> Why you are so restrictive ..
> 
> No idea. I would actually prefer to just remove the restriction, and let
> the SPDX header to be anywhere inside the first comment block inside a
> file [2].
> 
> That's basically how this thread started: other developers think
> that it is a good idea to be pedantic. So, be it, but let's then fix
> the documentation, as the way it is, it is implicitly forbidding the
> addition of encoding lines for Python scripts.
> 
> [2] I *suspect* that the restriction was added in order to make
>      ./scripts/spdxcheck.py to run faster and to avoid false positives.
>      Right now, if the maximum limit is removed (or set to a very high
>      value), there will be one false positive:
> 
> 	Documentation/dev-tools/kselftest.rst
> 
>      This doc has a SPDX-like tag at line 230, asking people to add SPDX
>      headers on files, but the file itself doesn't have its own SPDX tag.
> 	
>> what we normal do:
>>
>> - write a shebang line if this file is called directly from the
>>     command line .. but we do not need shebangs on py modules which
>>     are imported from other modules or scripts
>>
>> - write a encoding line if it is need or helpful / mostly it is helpful
>>     to know the encoding of a text/code file.
>>
>> - add a SPDX tag
> 
> Yes, but this violates the current documentation, as it doesn't allow the
> SPDX tag after line #2.

Thats what I mean: The documentation was written with only a small use-cases
in mind .. there is no real need for SPDX to be in line one or two ... lets
fix the documentation as I described before.

Side note: if I can help you with perf or your build systems, don't hesitate
to contact me directly.

-- Markus --

>> At the end we will have files with one, two or all three of this lines.
>> And the oder of this lines is, what I wrote:
>>
>>>>
>>>> Thats what I mean [1] .. lets patch the description in the license-rules.rst::
>>>>
>>>> - first line for the OS (shebang)
>>>> - second line for environment (python-encoding, editor-mode, ...)
>>>> - third and more lines for application (SPDX use) ..
>>>>
>>>> [1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg33240.html
>>>>
>>>> -- Markus --
>>>>   
>>>>> This suggests to me that we're adding a bunch of complications that we
>>>>> don't necessarily need.  What am I missing here?
>>>>>
>>>>> Educate me properly and I'll not try to stand in the way of all this...
>>>>>   
>>
>>
>> It seems like it is not only me who is mising something .. what are
>> the use-cases we have py-Exceptions, what are the use-cases to be so
>> restrictive as you described above.
>>
>> .. or did alice get lost in the cave?
>>
>> Thanks for your patience with me
>>
>> -- Markus --
> 
> 
> 
> Thanks,
> Mauro
> 

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Thomas Gleixner]

On Sat, 7 Sep 2019, Markus Heiser wrote:
> Am 07.09.19 um 20:04 schrieb Mauro Carvalho Chehab:
> > No idea. I would actually prefer to just remove the restriction, and let
> > the SPDX header to be anywhere inside the first comment block inside a
> > file [2].
> 
> > That's basically how this thread started: other developers think
> > that it is a good idea to be pedantic. So, be it, but let's then fix
> > the documentation, as the way it is, it is implicitly forbidding the
> > addition of encoding lines for Python scripts.
> > 
> > [2] I *suspect* that the restriction was added in order to make
> >      ./scripts/spdxcheck.py to run faster and to avoid false positives.
> >      Right now, if the maximum limit is removed (or set to a very high
> >      value), there will be one false positive:

Nope. The intention was to have a well define place and format instead of
everyone and his dog deciding to put it somewhere. SPDX is not intended to
replace the existing licensing mess with some other randomly placed and
formatted licensing mess.

> > > - write a shebang line if this file is called directly from the
> > >     command line .. but we do not need shebangs on py modules which
> > >     are imported from other modules or scripts
> > > 
> > > - write a encoding line if it is need or helpful / mostly it is helpful
> > >     to know the encoding of a text/code file.
> > > 
> > > - add a SPDX tag
> > 
> > Yes, but this violates the current documentation, as it doesn't allow the
> > SPDX tag after line #2.
> 
> Thats what I mean: The documentation was written with only a small use-cases
> in mind .. there is no real need for SPDX to be in line one or two ... lets
> fix the documentation as I described before.

If there is a requirement from the language to have 2 lines right at the
top for conveying information then there is of course no reason to insist
on the SPDX identifier being on line 2.

So the documentation should say:

   The SPDX identifier must be at the first possible line at the top of the
   file which is not occupied by information which is required to be
   immediately at the top of the file by system constraints, e.g. shebang,
   or by the language, e.g. the encoding information for python.

or something to that effect.

Thanks,

	tglx

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Matthew Wilcox]

On Sat, Sep 07, 2019 at 11:17:22PM +0200, Thomas Gleixner wrote:
> On Sat, 7 Sep 2019, Markus Heiser wrote:
> > Am 07.09.19 um 20:04 schrieb Mauro Carvalho Chehab:
> > > No idea. I would actually prefer to just remove the restriction, and let
> > > the SPDX header to be anywhere inside the first comment block inside a
> > > file [2].
> > > [2] I *suspect* that the restriction was added in order to make
> > >      ./scripts/spdxcheck.py to run faster and to avoid false positives.
> > >      Right now, if the maximum limit is removed (or set to a very high
> > >      value), there will be one false positive:
> 
> Nope. The intention was to have a well define place and format instead of
> everyone and his dog deciding to put it somewhere. SPDX is not intended to
> replace the existing licensing mess with some other randomly placed and
> formatted licensing mess.

I find the current style quite unaesthetic:

// SPDX-License-Identifier: GPL-2.0-only
/*
 *  linux/mm/memory.c
 *
 *  Copyright (C) 1991, 1992, 1993, 1994  Linus Torvalds
 */

I'd much rather see

/*
 * SPDX-License-Identifier: GPL-2.0-only
 * Copyright (C) 1991, 1992, 1993, 1994  Linus Torvalds
 */

but I appreciate the desire to force it to be on the first line if at all
possible.

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Thomas Gleixner]

On Sun, 8 Sep 2019, Matthew Wilcox wrote:
> On Sat, Sep 07, 2019 at 11:17:22PM +0200, Thomas Gleixner wrote:
> > On Sat, 7 Sep 2019, Markus Heiser wrote:
> > > Am 07.09.19 um 20:04 schrieb Mauro Carvalho Chehab:
> > > > No idea. I would actually prefer to just remove the restriction, and let
> > > > the SPDX header to be anywhere inside the first comment block inside a
> > > > file [2].
> > > > [2] I *suspect* that the restriction was added in order to make
> > > >      ./scripts/spdxcheck.py to run faster and to avoid false positives.
> > > >      Right now, if the maximum limit is removed (or set to a very high
> > > >      value), there will be one false positive:
> > 
> > Nope. The intention was to have a well define place and format instead of
> > everyone and his dog deciding to put it somewhere. SPDX is not intended to
> > replace the existing licensing mess with some other randomly placed and
> > formatted licensing mess.
> 
> I find the current style quite unaesthetic:
> 
> // SPDX-License-Identifier: GPL-2.0-only
> /*
>  *  linux/mm/memory.c
>  *
>  *  Copyright (C) 1991, 1992, 1993, 1994  Linus Torvalds
>  */
> 
> I'd much rather see
> 
> /*
>  * SPDX-License-Identifier: GPL-2.0-only
>  * Copyright (C) 1991, 1992, 1993, 1994  Linus Torvalds
>  */
> 
> but I appreciate the desire to force it to be on the first line if at all
> possible.

That style is inflicted upon you by Penguin Emperor Decree. :)

Re: [PATCH 0/6] Address issues with SPDX requirements and PEP-263

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> On Sun, 8 Sep 2019, Matthew Wilcox wrote:
> > On Sat, Sep 07, 2019 at 11:17:22PM +0200, Thomas Gleixner wrote:
> > > On Sat, 7 Sep 2019, Markus Heiser wrote:
> > > > Am 07.09.19 um 20:04 schrieb Mauro Carvalho Chehab:
> > > > > No idea. I would actually prefer to just remove the restriction, and let
> > > > > the SPDX header to be anywhere inside the first comment block inside a
> > > > > file [2].
> > > > > [2] I *suspect* that the restriction was added in order to make
> > > > >      ./scripts/spdxcheck.py to run faster and to avoid false positives.
> > > > >      Right now, if the maximum limit is removed (or set to a very high
> > > > >      value), there will be one false positive:
> > > 
> > > Nope. The intention was to have a well define place and format instead of
> > > everyone and his dog deciding to put it somewhere. SPDX is not intended to
> > > replace the existing licensing mess with some other randomly placed and
> > > formatted licensing mess.
> > 
> > I find the current style quite unaesthetic:
> > 
> > // SPDX-License-Identifier: GPL-2.0-only
> > /*
> >  *  linux/mm/memory.c
> >  *
> >  *  Copyright (C) 1991, 1992, 1993, 1994  Linus Torvalds
> >  */
> > 
> > I'd much rather see
> > 
> > /*
> >  * SPDX-License-Identifier: GPL-2.0-only
> >  * Copyright (C) 1991, 1992, 1993, 1994  Linus Torvalds
> >  */
> > 
> > but I appreciate the desire to force it to be on the first line if at all
> > possible.
> 
> That style is inflicted upon you by Penguin Emperor Decree. :)

I'd also say that it's a rather tooling-friendly format which mandates a 
single-line representation, which will be less likely to be morphed into 
a zillion variants like the original boilerplates.

So the Penguin Emperor Decree also makes sense, which helps. ;-)

Thanks,

	Ingo