[RFC/PATCH] CodingGuidelines: add Python code guidelines

[RFC/PATCH] CodingGuidelines: add Python code guidelines

[John Keeping]

These are kept short by simply deferring to PEP-8.  Most of the Python
code in Git is already very close to this style (some things in contrib/
are not).

Rationale for version suggestions:

 - Amongst the noise in [2], there isn't any disagreement about using
   2.6 as a base (see also [3]).

 - The Git INSTALL document currently says:

      Python version 2.6 or later is needed to use the git-p4
      interface to Perforce.

 - Restricting ourselves to 2.6+ makes aiming for Python 3 compatibility
   significantly easier [4].

 - Following Pete's comment [5] I tested Python 2.6.0 and it does
   support bytes literals, as suggested by [4] but contradicted by [6].

 - Advocating Python 3 support in all scripts is currently unrealistic
   because:

     - 'p4 -G' provides output in a format that is very hard to use with
       Python 3 (and its documentation claims Python 3 is unsupported).

     - Mercurial does not support Python 3.

     - Bazaar does not support Python 3.

 - But we should try to make new scripts compatible with Python 3
   because all new Python development is happening on version 3 and the
   Python community will eventually stop supporting Python 2 [7].

I chose to recommend `from __future__ import unicode_literals` since it
provides the widest range of compatibility (2.6+ and 3.0+) while
allowing us to be explicit about bytes vs. Unicode.  The alternative
would be to advocate using the 'u' prefix on Unicode strings but this
isn't available in versions 3.0 - 3.2 (it is reintroduced in version 3.3
as a no-op in order to make it easier to write scripts targeting a wide
range of Python versions without needing to use 2to3 [1]).  In reality I
doubt we will ever need to worry about this since ASCII strings will
just work in both Python 2 and Python 3.

[1] http://www.python.org/dev/peps/pep-0414/
[2] http://thread.gmane.org/gmane.comp.version-control.git/210329
[3] http://article.gmane.org/gmane.comp.version-control.git/210429
[4] http://docs.python.org/3.3/howto/pyporting.html#try-to-support-python-2-6-and-newer-only
[5] http://article.gmane.org/gmane.comp.version-control.git/213830
[6] http://docs.python.org/2.6/reference/lexical_analysis.html#literals
[7] http://www.python.org/dev/peps/pep-0404/
---
 Documentation/CodingGuidelines | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines
index 69f7e9b..baf3b41 100644
--- a/Documentation/CodingGuidelines
+++ b/Documentation/CodingGuidelines
@@ -179,6 +179,22 @@ For C programs:
  - Use Git's gettext wrappers to make the user interface
    translatable. See "Marking strings for translation" in po/README.
 
+For Python scripts:
+
+ - We follow PEP-8 (http://www.python.org/dev/peps/pep-0008/).
+
+ - As a minimum, we aim to be compatible with Python 2.6 and 2.7.
+
+ - Where required libraries do not restrict us to Python 2, we try to
+   also be compatible with Python 3.  In this case we use
+   `from __future__ import unicode_literals` if we need to differentiate
+   Unicode string literals, rather than prefixing Unicode strings with
+   'u' since the latter is not supported in Python versions 3.0 - 3.2.
+
+ - We use the 'b' prefix for bytes literals.  Note that even though
+   the Python documentation for version 2.6 does not mention this
+   prefix it is supported since version 2.6.0.
+
 Writing Documentation:
 
  Every user-visible change should be reflected in the documentation.
-- 
1.8.1

Re: [RFC/PATCH] CodingGuidelines: add Python code guidelines

[Junio C Hamano]

John Keeping <john@keeping.me.uk> writes:

> diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines
> index 69f7e9b..baf3b41 100644
> --- a/Documentation/CodingGuidelines
> +++ b/Documentation/CodingGuidelines
> @@ -179,6 +179,22 @@ For C programs:
>   - Use Git's gettext wrappers to make the user interface
>     translatable. See "Marking strings for translation" in po/README.
>  
> +For Python scripts:
> +
> + - We follow PEP-8 (http://www.python.org/dev/peps/pep-0008/).
> +
> + - As a minimum, we aim to be compatible with Python 2.6 and 2.7.
> +
> + - Where required libraries do not restrict us to Python 2, we try to
> +   also be compatible with Python 3.  In this case we use
> +   `from __future__ import unicode_literals` if we need to differentiate
> +   Unicode string literals, rather than prefixing Unicode strings with
> +   'u' since the latter is not supported in Python versions 3.0 - 3.2.

"In this case"?  In what case?  This document will stay effective
long after you settle one particular backward incompatibility Python
3 introduced, namely, the unicode literal issues.  It is just one
"example".

That example somehow tells me that early versions of Python 3.x
series may be too buggy and not worth worrying about, and we may
want to set a floor for Python 3.x series, too, with something
like:

    - The code should be compatible with 2.6 and newer versions of
      Python 2.x series; 2.5 and older are not supported anymore.

    - The code should also be comptabile with 3.2 and newer versions
      of Python 3.x series; 3.1 and older are not mature enough and
      have too many problems to write scripts that work on it and
      solid 2.x at the same time.

I am not actively advocating to disqualify early 3.x; I am just
suggesting that doing so may be a viable escape hatch for us that
does not harm real users.  If you and others who know Python better
think there isn't any problem that makes it too cumbersome to
support both late 2.x and 3.0/3.1, there is no reason to set the
floor at 3.2.

I just have this feeling that we might be better off treating them
as 0.x releases of a new software called Python3, that happens to be
similar to the Python we know.

Re: [RFC/PATCH] CodingGuidelines: add Python code guidelines

[John Keeping]

On Fri, Jan 18, 2013 at 11:04:15AM -0800, Junio C Hamano wrote:
> John Keeping <john@keeping.me.uk> writes:
> 
>> diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines
>> index 69f7e9b..baf3b41 100644
>> --- a/Documentation/CodingGuidelines
>> +++ b/Documentation/CodingGuidelines
>> @@ -179,6 +179,22 @@ For C programs:
>>   - Use Git's gettext wrappers to make the user interface
>>     translatable. See "Marking strings for translation" in po/README.
>>  
>> +For Python scripts:
>> +
>> + - We follow PEP-8 (http://www.python.org/dev/peps/pep-0008/).
>> +
>> + - As a minimum, we aim to be compatible with Python 2.6 and 2.7.
>> +
>> + - Where required libraries do not restrict us to Python 2, we try to
>> +   also be compatible with Python 3.  In this case we use
>> +   `from __future__ import unicode_literals` if we need to differentiate
>> +   Unicode string literals, rather than prefixing Unicode strings with
>> +   'u' since the latter is not supported in Python versions 3.0 - 3.2.
> 
> "In this case"?  In what case?  This document will stay effective
> long after you settle one particular backward incompatibility Python
> 3 introduced, namely, the unicode literal issues.  It is just one
> "example".

I meant "in the case where you are supporting Python 3" but I suspect it
would be better to move the unicode_literals sentence to a new bullet.
Or maybe we should just remove it - I haven't seen a case in the current
Git source where we need Unicode literals.

> That example somehow tells me that early versions of Python 3.x
> series may be too buggy and not worth worrying about, and we may
> want to set a floor for Python 3.x series, too, with something
> like:
[snip]
> I am not actively advocating to disqualify early 3.x; I am just
> suggesting that doing so may be a viable escape hatch for us that
> does not harm real users.  If you and others who know Python better
> think there isn't any problem that makes it too cumbersome to
> support both late 2.x and 3.0/3.1, there is no reason to set the
> floor at 3.2.
> 
> I just have this feeling that we might be better off treating them
> as 0.x releases of a new software called Python3, that happens to be
> similar to the Python we know.

I originally thought about putting a floor of 3.3 (which is where
Unicode literals were reintroduced) but that was only released in
September and as far as I'm aware Unicode literals are the only reason
to have a restriction on Python 3 versions, given that we support Python
2.6 - standard library features should be equivalent.

I don't think Python 3.0 is any less stable than any other 3.x release,
it's just that it was the first release which attempted a clean break
from backwards compatibility.  From the point of view of features
supported, Python 2.6 and 3.0 should be roughly equivalent - they were
released together with the intent that 2.6 should make it possible to
write code that ports to 3.0 easily, using 2to3.

As more people have started trying to support Python 3 in the wild, it
has become clear that it is often easier to have a single codebase that
works with both Python 2 and Python 3, and not use 2to3.

It is for this reason that the Unicode literal prefix was reintroduced.
From the specification reintroducing it [1]:

   Complaint: Python 3 shouldn't be made worse just to support porting
   from Python 2

   This is indeed one of the key design principles of Python 3. However,
   one of the key design principles of Python as a whole is that
   "practicality beats purity".


[1] http://www.python.org/dev/peps/pep-0414/#complaint-python-3-shouldn-t-be-made-worse-just-to-support-porting-from-python-2


John

Re: [RFC/PATCH] CodingGuidelines: add Python code guidelines

[Junio C Hamano]

John Keeping <john@keeping.me.uk> writes:

>>> + - Where required libraries do not restrict us to Python 2, we try to
>>> +   also be compatible with Python 3.  In this case we use
>>> +   `from __future__ import unicode_literals` if we need to differentiate
>>> +   Unicode string literals, rather than prefixing Unicode strings with
>>> +   'u' since the latter is not supported in Python versions 3.0 - 3.2.
>> 
>> "In this case"?  In what case?  This document will stay effective
>> long after you settle one particular backward incompatibility Python
>> 3 introduced, namely, the unicode literal issues.  It is just one
>> "example".
>
> I meant "in the case where you are supporting Python 3" but I suspect it
> would be better to move the unicode_literals sentence to a new bullet.
> Or maybe we should just remove it - I haven't seen a case in the current
> Git source where we need Unicode literals.

Yeah, "we support 2.x" and "we suport 3.x" may want to be combined,
but listing individual specifics as separate points to watch out for
would make it much more readable.

> As more people have started trying to support Python 3 in the wild, it
> has become clear that it is often easier to have a single codebase that
> works with both Python 2 and Python 3, and not use 2to3.
>
> It is for this reason that the Unicode literal prefix was reintroduced.

Yes, and from that perspective, placing floor on earlier 3.x makes
tons of sense, no?

These early versions may not be unstable in the "this does not
behave as specified in the language specification for 3.x" sense,
but for the purpose of running scripts meant to be executable by
both 2.x and 3.x series, the early 3.x versions are not as good as
later versions where Python folks started making deliberate effort
to support them.

Re: [RFC/PATCH] CodingGuidelines: add Python code guidelines

[John Keeping]

On Fri, Jan 18, 2013 at 12:25:34PM -0800, Junio C Hamano wrote:
> John Keeping <john@keeping.me.uk> writes:
>> As more people have started trying to support Python 3 in the wild, it
>> has become clear that it is often easier to have a single codebase that
>> works with both Python 2 and Python 3, and not use 2to3.
>>
>> It is for this reason that the Unicode literal prefix was reintroduced.
> 
> Yes, and from that perspective, placing floor on earlier 3.x makes
> tons of sense, no?
> 
> These early versions may not be unstable in the "this does not
> behave as specified in the language specification for 3.x" sense,
> but for the purpose of running scripts meant to be executable by
> both 2.x and 3.x series, the early 3.x versions are not as good as
> later versions where Python folks started making deliberate effort
> to support them.

As far as I'm aware (and having reviewed the release notes for 3.1, 3.2
and 3.3 as well as the planned features for 3.4), Unicode literals are
the only feature to have been added that was intended to make it easier
to support Python 2 and 3 in the same codebase.

Given that no code currently on pu uses Unicode literals, I don't see a
reason to specify a minimum version of Python 3 since we're already
restricting ourselves to features in 2.6.


John

Re: [RFC/PATCH] CodingGuidelines: add Python code guidelines

[Junio C Hamano]

John Keeping <john@keeping.me.uk> writes:

> On Fri, Jan 18, 2013 at 12:25:34PM -0800, Junio C Hamano wrote:
>> John Keeping <john@keeping.me.uk> writes:
>>> As more people have started trying to support Python 3 in the wild, it
>>> has become clear that it is often easier to have a single codebase that
>>> works with both Python 2 and Python 3, and not use 2to3.
>>>
>>> It is for this reason that the Unicode literal prefix was reintroduced.
>> 
>> Yes, and from that perspective, placing floor on earlier 3.x makes
>> tons of sense, no?
>> 
>> These early versions may not be unstable in the "this does not
>> behave as specified in the language specification for 3.x" sense,
>> but for the purpose of running scripts meant to be executable by
>> both 2.x and 3.x series, the early 3.x versions are not as good as
>> later versions where Python folks started making deliberate effort
>> to support them.
>
> As far as I'm aware (and having reviewed the release notes for 3.1, 3.2
> and 3.3 as well as the planned features for 3.4), Unicode literals are
> the only feature to have been added that was intended to make it easier
> to support Python 2 and 3 in the same codebase.

So there may be some other incompatibility lurking that we may run
into later?

> Given that no code currently on pu uses Unicode literals, I don't see a
> reason to specify a minimum version of Python 3 since we're already
> restricting ourselves to features in 2.6.

OK, at least that reasoning need to be kept somewhere, either in the
documentation of in the log message.

Thanks.

Re: [RFC/PATCH] CodingGuidelines: add Python code guidelines

[John Keeping]

On Fri, Jan 18, 2013 at 02:26:06PM -0800, Junio C Hamano wrote:
> John Keeping <john@keeping.me.uk> writes:
>> On Fri, Jan 18, 2013 at 12:25:34PM -0800, Junio C Hamano wrote:
>>> These early versions may not be unstable in the "this does not
>>> behave as specified in the language specification for 3.x" sense,
>>> but for the purpose of running scripts meant to be executable by
>>> both 2.x and 3.x series, the early 3.x versions are not as good as
>>> later versions where Python folks started making deliberate effort
>>> to support them.
>>
>> As far as I'm aware (and having reviewed the release notes for 3.1, 3.2
>> and 3.3 as well as the planned features for 3.4), Unicode literals are
>> the only feature to have been added that was intended to make it easier
>> to support Python 2 and 3 in the same codebase.
> 
> So there may be some other incompatibility lurking that we may run
> into later?

I doubt it - enough projects are running on Python 2 and 3 now that I
doubt there's anything unexpected left to hit.

>> Given that no code currently on pu uses Unicode literals, I don't see a
>> reason to specify a minimum version of Python 3 since we're already
>> restricting ourselves to features in 2.6.
> 
> OK, at least that reasoning need to be kept somewhere, either in the
> documentation of in the log message.

I'll put it in the log message when I send this as a proper patch.


John