Python 3 bindings

Python 3 bindings

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 1962 bytes --]

Hi,

I'm adjusting python bindings to work on python3 too. This will require
few #if in the code (to compile for both python2 and python3), but it
isn't that bad. But there are some major changes in python3, which
require some decision about the bindings API:

1. Python3 has no longer separate 'int' and 'long' type - old 'long'
type was renamed to 'int' (but on C-API level, it uses PyLong_*). I see
two options:
  - switch to PyLong_* everywhere, including python2 bindings - this
    makes the code much cleaner, but it is an API change in python2
  - switch to PyLong_* only for python3 - this will introduce some
    #ifdefs, but python2 API will be unchanged

2. Python3 has no longer separate 'str' and 'unicode' type, new 'str' is
the same as 'unicode' (PyUnicode_* at C-API level). For things not
really unicode-aware, 'bytes' type should be used. On the other hand, in
python2 'bytes' type was the same as 'str'.
This affects various places, where in most cases 'bytes' type is
appropriate (for example cpuid). But I'm not sure about xenstore paths -
those should also be 'bytes', or maybe 'unicode' (which is implicitly
using 'utf-8' encoding)? I think the only reason to use 'unicode' is
convenience for API users - in python3 if you write 'some string' it
will be unicode type, to create bytes data you need to write b'some
string'.
As for python2, it should definitely be still 'str'/'bytes' type.

There is one more little detail - build process. Here I'm going to
follow popular standard - use $(PYTHON) variable - if that points to
python3, build for python3. Actually this means no change in the current
makefile. If someone want to build for both python2 and python3, will
need to call the build twice - at packaging level.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Python 3 bindings

[Wei Liu]

On Fri, Feb 17, 2017 at 01:36:01PM +0100, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> I'm adjusting python bindings to work on python3 too. This will require
> few #if in the code (to compile for both python2 and python3), but it
> isn't that bad. But there are some major changes in python3, which
> require some decision about the bindings API:
> 
> 1. Python3 has no longer separate 'int' and 'long' type - old 'long'
> type was renamed to 'int' (but on C-API level, it uses PyLong_*). I see
> two options:
>   - switch to PyLong_* everywhere, including python2 bindings - this
>     makes the code much cleaner, but it is an API change in python2
>   - switch to PyLong_* only for python3 - this will introduce some
>     #ifdefs, but python2 API will be unchanged

Could you be more specific? Like, provide a code snippet?

> 
> 2. Python3 has no longer separate 'str' and 'unicode' type, new 'str' is
> the same as 'unicode' (PyUnicode_* at C-API level). For things not
> really unicode-aware, 'bytes' type should be used. On the other hand, in
> python2 'bytes' type was the same as 'str'.
> This affects various places, where in most cases 'bytes' type is
> appropriate (for example cpuid). But I'm not sure about xenstore paths -
> those should also be 'bytes', or maybe 'unicode' (which is implicitly
> using 'utf-8' encoding)? I think the only reason to use 'unicode' is

According to docs/txt/misc/xenstore.txt, paths should be ASCII
alphanumerics plus four punctuation characters. Not sure if this is
relevant to what you describe.

> convenience for API users - in python3 if you write 'some string' it
> will be unicode type, to create bytes data you need to write b'some
> string'.
> As for python2, it should definitely be still 'str'/'bytes' type.
> 
> There is one more little detail - build process. Here I'm going to
> follow popular standard - use $(PYTHON) variable - if that points to
> python3, build for python3. Actually this means no change in the current
> makefile. If someone want to build for both python2 and python3, will
> need to call the build twice - at packaging level.
> 
> -- 
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?



> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Python 3 bindings

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 4122 bytes --]

On Mon, Feb 20, 2017 at 05:18:44PM +0000, Wei Liu wrote:
> On Fri, Feb 17, 2017 at 01:36:01PM +0100, Marek Marczykowski-Górecki wrote:
> > Hi,
> > 
> > I'm adjusting python bindings to work on python3 too. This will require
> > few #if in the code (to compile for both python2 and python3), but it
> > isn't that bad. But there are some major changes in python3, which
> > require some decision about the bindings API:
> > 
> > 1. Python3 has no longer separate 'int' and 'long' type - old 'long'
> > type was renamed to 'int' (but on C-API level, it uses PyLong_*). I see
> > two options:
> >   - switch to PyLong_* everywhere, including python2 bindings - this
> >     makes the code much cleaner, but it is an API change in python2
> >   - switch to PyLong_* only for python3 - this will introduce some
> >     #ifdefs, but python2 API will be unchanged
> 
> Could you be more specific? Like, provide a code snippet?

Here is compile tested only version:
https://github.com/marmarek/xen/tree/python3

It uses PyLong_* only for python3, here is how it looks in code (I've
skipped s/PyInt_/PyLongOrInt_/ for readability):

-----8<-----
--- a/tools/python/xen/lowlevel/xc/xc.c
+++ b/tools/python/xen/lowlevel/xc/xc.c
@@ -34,6 +34,17 @@
 
 #define FLASK_CTX_LEN 1024
 
+/* Python 2 compatibility */
+#if PY_VERSION_HEX >= 0x03000000
+#define PyLongOrInt_FromLong PyLong_FromLong
+#define PyLongOrInt_Check PyLong_Check
+#define PyLongOrInt_AsLong PyLong_AsLong
+#else
+#define PyLongOrInt_FromLong PyInt_FromLong
+#define PyLongOrInt_Check PyInt_Check
+#define PyLongOrInt_AsLong PyInt_AsLong
+#endif
+
 static PyObject *xc_error_obj, *zero;
 
 typedef struct {
--- a/tools/python/xen/lowlevel/xs/xs.c
+++ b/tools/python/xen/lowlevel/xs/xs.c
@@ -43,6 +43,14 @@
 #define PKG "xen.lowlevel.xs"
 #define CLS "xs"
 
+#if PY_VERSION_HEX < 0x03000000
+/* Python 2 compatibility */
+#define PyLong_FromLong PyInt_FromLong
+#undef PyLong_Check
+#define PyLong_Check PyInt_Check
+#define PyLong_AsLong PyInt_AsLong
+#endif
+
 static PyObject *xs_error;
 
 /** Python wrapper round an xs handle.
-----8<-----

> 
> > 
> > 2. Python3 has no longer separate 'str' and 'unicode' type, new 'str' is
> > the same as 'unicode' (PyUnicode_* at C-API level). For things not
> > really unicode-aware, 'bytes' type should be used. On the other hand, in
> > python2 'bytes' type was the same as 'str'.
> > This affects various places, where in most cases 'bytes' type is
> > appropriate (for example cpuid). But I'm not sure about xenstore paths -
> > those should also be 'bytes', or maybe 'unicode' (which is implicitly
> > using 'utf-8' encoding)? I think the only reason to use 'unicode' is
> 
> According to docs/txt/misc/xenstore.txt, paths should be ASCII
> alphanumerics plus four punctuation characters. Not sure if this is
> relevant to what you describe.

It's easy to make function accept both 'bytes' and 'unicode'. The
question is what should be return type (read_watch, ls etc) - given
limited character set used there, I'm in favor of 'unicode' - easier to
handle, but we shouldn't hit any unicode decoding problems.
Maybe the same should apply to path arguments (use 'unicode')? Most
file-handling methods in python3 use 'unicode' for paths, if that
matters.

> > convenience for API users - in python3 if you write 'some string' it
> > will be unicode type, to create bytes data you need to write b'some
> > string'.
> > As for python2, it should definitely be still 'str'/'bytes' type.
> > 
> > There is one more little detail - build process. Here I'm going to
> > follow popular standard - use $(PYTHON) variable - if that points to
> > python3, build for python3. Actually this means no change in the current
> > makefile. If someone want to build for both python2 and python3, will
> > need to call the build twice - at packaging level.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Python 3 bindings

[Wei Liu]

On Tue, Feb 21, 2017 at 02:03:00PM +0100, Marek Marczykowski-Górecki wrote:
> On Mon, Feb 20, 2017 at 05:18:44PM +0000, Wei Liu wrote:
> > On Fri, Feb 17, 2017 at 01:36:01PM +0100, Marek Marczykowski-Górecki wrote:
> > > Hi,
> > > 
> > > I'm adjusting python bindings to work on python3 too. This will require
> > > few #if in the code (to compile for both python2 and python3), but it
> > > isn't that bad. But there are some major changes in python3, which
> > > require some decision about the bindings API:
> > > 
> > > 1. Python3 has no longer separate 'int' and 'long' type - old 'long'
> > > type was renamed to 'int' (but on C-API level, it uses PyLong_*). I see
> > > two options:
> > >   - switch to PyLong_* everywhere, including python2 bindings - this
> > >     makes the code much cleaner, but it is an API change in python2
> > >   - switch to PyLong_* only for python3 - this will introduce some
> > >     #ifdefs, but python2 API will be unchanged
> > 
> > Could you be more specific? Like, provide a code snippet?
> 
> Here is compile tested only version:
> https://github.com/marmarek/xen/tree/python3
> 
> It uses PyLong_* only for python3, here is how it looks in code (I've
> skipped s/PyInt_/PyLongOrInt_/ for readability):
> 
> -----8<-----
> --- a/tools/python/xen/lowlevel/xc/xc.c
> +++ b/tools/python/xen/lowlevel/xc/xc.c
> @@ -34,6 +34,17 @@
>  
>  #define FLASK_CTX_LEN 1024
>  
> +/* Python 2 compatibility */
> +#if PY_VERSION_HEX >= 0x03000000
> +#define PyLongOrInt_FromLong PyLong_FromLong
> +#define PyLongOrInt_Check PyLong_Check
> +#define PyLongOrInt_AsLong PyLong_AsLong
> +#else
> +#define PyLongOrInt_FromLong PyInt_FromLong
> +#define PyLongOrInt_Check PyInt_Check
> +#define PyLongOrInt_AsLong PyInt_AsLong
> +#endif
> +
>  static PyObject *xc_error_obj, *zero;
>  
>  typedef struct {
> --- a/tools/python/xen/lowlevel/xs/xs.c
> +++ b/tools/python/xen/lowlevel/xs/xs.c
> @@ -43,6 +43,14 @@
>  #define PKG "xen.lowlevel.xs"
>  #define CLS "xs"
>  
> +#if PY_VERSION_HEX < 0x03000000
> +/* Python 2 compatibility */
> +#define PyLong_FromLong PyInt_FromLong
> +#undef PyLong_Check
> +#define PyLong_Check PyInt_Check
> +#define PyLong_AsLong PyInt_AsLong
> +#endif
> +
>  static PyObject *xs_error;
>  

If this is the recommended practice, then that's fine. I can't seem to
find such practice in https://docs.python.org/3/howto/cporting.html but
I'm no python binding expert.

BTW, I went through your python3 branch. It seems that some patches can
be submitted independently.


>  /** Python wrapper round an xs handle.
> -----8<-----
> 
> > 
> > > 
> > > 2. Python3 has no longer separate 'str' and 'unicode' type, new 'str' is
> > > the same as 'unicode' (PyUnicode_* at C-API level). For things not
> > > really unicode-aware, 'bytes' type should be used. On the other hand, in
> > > python2 'bytes' type was the same as 'str'.
> > > This affects various places, where in most cases 'bytes' type is
> > > appropriate (for example cpuid). But I'm not sure about xenstore paths -
> > > those should also be 'bytes', or maybe 'unicode' (which is implicitly
> > > using 'utf-8' encoding)? I think the only reason to use 'unicode' is
> > 
> > According to docs/txt/misc/xenstore.txt, paths should be ASCII
> > alphanumerics plus four punctuation characters. Not sure if this is
> > relevant to what you describe.
> 
> It's easy to make function accept both 'bytes' and 'unicode'. The
> question is what should be return type (read_watch, ls etc) - given
> limited character set used there, I'm in favor of 'unicode' - easier to
> handle, but we shouldn't hit any unicode decoding problems.
> Maybe the same should apply to path arguments (use 'unicode')? Most
> file-handling methods in python3 use 'unicode' for paths, if that
> matters.
> 

OK. Using unicode makes sense to me. Again, I'm no python expert and I
trust what you said. :-)

> > > convenience for API users - in python3 if you write 'some string' it
> > > will be unicode type, to create bytes data you need to write b'some
> > > string'.
> > > As for python2, it should definitely be still 'str'/'bytes' type.
> > > 
> > > There is one more little detail - build process. Here I'm going to
> > > follow popular standard - use $(PYTHON) variable - if that points to
> > > python3, build for python3. Actually this means no change in the current
> > > makefile. If someone want to build for both python2 and python3, will
> > > need to call the build twice - at packaging level.
> 
> -- 
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Python 3 bindings

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 4675 bytes --]

On Wed, Feb 22, 2017 at 11:34:16AM +0000, Wei Liu wrote:
> On Tue, Feb 21, 2017 at 02:03:00PM +0100, Marek Marczykowski-Górecki wrote:
> > On Mon, Feb 20, 2017 at 05:18:44PM +0000, Wei Liu wrote:
> > > On Fri, Feb 17, 2017 at 01:36:01PM +0100, Marek Marczykowski-Górecki wrote:
> > > > Hi,
> > > > 
> > > > I'm adjusting python bindings to work on python3 too. This will require
> > > > few #if in the code (to compile for both python2 and python3), but it
> > > > isn't that bad. But there are some major changes in python3, which
> > > > require some decision about the bindings API:
> > > > 
> > > > 1. Python3 has no longer separate 'int' and 'long' type - old 'long'
> > > > type was renamed to 'int' (but on C-API level, it uses PyLong_*). I see
> > > > two options:
> > > >   - switch to PyLong_* everywhere, including python2 bindings - this
> > > >     makes the code much cleaner, but it is an API change in python2
> > > >   - switch to PyLong_* only for python3 - this will introduce some
> > > >     #ifdefs, but python2 API will be unchanged
> > > 
> > > Could you be more specific? Like, provide a code snippet?
> > 
> > Here is compile tested only version:
> > https://github.com/marmarek/xen/tree/python3
> > 
> > It uses PyLong_* only for python3, here is how it looks in code (I've
> > skipped s/PyInt_/PyLongOrInt_/ for readability):
> > 
> > -----8<-----
> > --- a/tools/python/xen/lowlevel/xc/xc.c
> > +++ b/tools/python/xen/lowlevel/xc/xc.c
> > @@ -34,6 +34,17 @@
> >  
> >  #define FLASK_CTX_LEN 1024
> >  
> > +/* Python 2 compatibility */
> > +#if PY_VERSION_HEX >= 0x03000000
> > +#define PyLongOrInt_FromLong PyLong_FromLong
> > +#define PyLongOrInt_Check PyLong_Check
> > +#define PyLongOrInt_AsLong PyLong_AsLong
> > +#else
> > +#define PyLongOrInt_FromLong PyInt_FromLong
> > +#define PyLongOrInt_Check PyInt_Check
> > +#define PyLongOrInt_AsLong PyInt_AsLong
> > +#endif
> > +
> >  static PyObject *xc_error_obj, *zero;
> >  
> >  typedef struct {
> > --- a/tools/python/xen/lowlevel/xs/xs.c
> > +++ b/tools/python/xen/lowlevel/xs/xs.c
> > @@ -43,6 +43,14 @@
> >  #define PKG "xen.lowlevel.xs"
> >  #define CLS "xs"
> >  
> > +#if PY_VERSION_HEX < 0x03000000
> > +/* Python 2 compatibility */
> > +#define PyLong_FromLong PyInt_FromLong
> > +#undef PyLong_Check
> > +#define PyLong_Check PyInt_Check
> > +#define PyLong_AsLong PyInt_AsLong
> > +#endif
> > +
> >  static PyObject *xs_error;
> >  
> 
> If this is the recommended practice, then that's fine. I can't seem to
> find such practice in https://docs.python.org/3/howto/cporting.html but
> I'm no python binding expert.

"In the C-API, PyInt_* functions are replaced by their PyLong_*
equivalents."

> BTW, I went through your python3 branch. It seems that some patches can
> be submitted independently.

Yes, I've tried to separate cleanup commits from actual python3 support.

> >  /** Python wrapper round an xs handle.
> > -----8<-----
> > 
> > > 
> > > > 
> > > > 2. Python3 has no longer separate 'str' and 'unicode' type, new 'str' is
> > > > the same as 'unicode' (PyUnicode_* at C-API level). For things not
> > > > really unicode-aware, 'bytes' type should be used. On the other hand, in
> > > > python2 'bytes' type was the same as 'str'.
> > > > This affects various places, where in most cases 'bytes' type is
> > > > appropriate (for example cpuid). But I'm not sure about xenstore paths -
> > > > those should also be 'bytes', or maybe 'unicode' (which is implicitly
> > > > using 'utf-8' encoding)? I think the only reason to use 'unicode' is
> > > 
> > > According to docs/txt/misc/xenstore.txt, paths should be ASCII
> > > alphanumerics plus four punctuation characters. Not sure if this is
> > > relevant to what you describe.
> > 
> > It's easy to make function accept both 'bytes' and 'unicode'. The
> > question is what should be return type (read_watch, ls etc) - given
> > limited character set used there, I'm in favor of 'unicode' - easier to
> > handle, but we shouldn't hit any unicode decoding problems.
> > Maybe the same should apply to path arguments (use 'unicode')? Most
> > file-handling methods in python3 use 'unicode' for paths, if that
> > matters.
> > 
> 
> OK. Using unicode makes sense to me. Again, I'm no python expert and I
> trust what you said. :-)

Ok, I'll adjust patches to return unicode paths.

Then test them and actually send here :)

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel