[PATCH] netmap-modules: make deterministic builds for drivers

[PATCH] netmap-modules: make deterministic builds for drivers

[jackie.huang]

From: Jackie Huang <jackie.huang@windriver.com>

The driver builds are optional, but for deterministic builds,
we should should be able to explicitly enable/disable the
builds for them in a proper place (maybe in BSP).
But we can't use PACKAGECONFIG since there is no option for
each driver, and the options are:
 --no-drivers    do not compile any driver
 --no-drivers=   do not compile the given drivers (comma sep.)
 --drivers=      only compile the given drivers (comma sep.)

So use NETMAP_DRIVERS to list the needed drivers and add proper
configs to EXTRA_OECONF, the default is no drivers, and all
supported drivers are listed in NETMAP_ALL_DRIVERS.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
---
 .../recipes-kernel/netmap/netmap-modules_git.bb    | 31 +++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb b/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb
index 6365fee..cfe5ce9 100644
--- a/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb
+++ b/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb
@@ -15,7 +15,36 @@ EXTRA_OECONF = "--kernel-dir=${STAGING_KERNEL_BUILDDIR} \
                 --driver-suffix="-netmap" \
                 "
 
-EXTRA_OECONF += "--no-drivers=ixgbe --no-drivers=virtio_net.c"
+# The driver builds are optional, but for deterministic builds,
+# we should be able to explicitly enable/disable the builds
+# for them in a proper place (maybe in BSP).
+# But we can't use PACKAGECONFIG since there is no option for
+# each driver, and the options are:
+#  --no-drivers    do not compile any driver
+#  --no-drivers=   do not compile the given drivers (comma sep.)
+#  --drivers=      only compile the given drivers (comma sep.)
+#
+# So use NETMAP_DRIVERS and the following python code to add proper
+# configs to EXTRA_OECONF and skip arch test for kernel modules.
+#
+# The default is no-drivers, and all supported drivers are listed
+# in NETMAP_ALL_DRIVERS.
+NETMAP_DRIVERS ??= ""
+NETMAP_ALL_DRIVERS = "ixgbe igb e1000e e1000 veth.c forcedeth.c virtio_net.c r8169.c"
+
+python __anonymous () {
+    drivers_list = d.getVar("NETMAP_DRIVERS", True).split()
+    all_drivers_list = d.getVar("NETMAP_ALL_DRIVERS", True).split()
+    config_drivers = "--drivers=" + ",".join(drivers_list)
+
+    extra_oeconf_drivers = bb.utils.contains_any('NETMAP_DRIVERS', all_drivers_list, config_drivers, '--no-drivers', d)
+    d.appendVar("EXTRA_OECONF", extra_oeconf_drivers)
+
+    # skip the arch test for kernel modules
+    if drivers_list:
+        for driver in drivers_list:
+            d.setVar("INSANE_SKIP_kernel-module-%s-netmap" % driver, "arch")
+}
 
 LDFLAGS := "${@'${LDFLAGS}'.replace('-Wl,-O1', '')}"
 LDFLAGS := "${@'${LDFLAGS}'.replace('-Wl,--as-needed', '')}"
-- 
1.9.1

Re: [PATCH] netmap-modules: make deterministic builds for drivers

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3710 bytes --]

On Wed, Nov 18, 2015 at 06:02:21PM +0800, jackie.huang@windriver.com wrote:
> From: Jackie Huang <jackie.huang@windriver.com>
> 
> The driver builds are optional, but for deterministic builds,
> we should should be able to explicitly enable/disable the
> builds for them in a proper place (maybe in BSP).
> But we can't use PACKAGECONFIG since there is no option for
> each driver, and the options are:
>  --no-drivers    do not compile any driver
>  --no-drivers=   do not compile the given drivers (comma sep.)
>  --drivers=      only compile the given drivers (comma sep.)
> 
> So use NETMAP_DRIVERS to list the needed drivers and add proper
> configs to EXTRA_OECONF, the default is no drivers, and all
> supported drivers are listed in NETMAP_ALL_DRIVERS.
> 
> Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
> ---
>  .../recipes-kernel/netmap/netmap-modules_git.bb    | 31 +++++++++++++++++++++-
>  1 file changed, 30 insertions(+), 1 deletion(-)
> 
> diff --git a/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb b/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb
> index 6365fee..cfe5ce9 100644
> --- a/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb
> +++ b/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb
> @@ -15,7 +15,36 @@ EXTRA_OECONF = "--kernel-dir=${STAGING_KERNEL_BUILDDIR} \
>                  --driver-suffix="-netmap" \
>                  "
>  
> -EXTRA_OECONF += "--no-drivers=ixgbe --no-drivers=virtio_net.c"
> +# The driver builds are optional, but for deterministic builds,
> +# we should be able to explicitly enable/disable the builds
> +# for them in a proper place (maybe in BSP).
> +# But we can't use PACKAGECONFIG since there is no option for
> +# each driver, and the options are:
> +#  --no-drivers    do not compile any driver
> +#  --no-drivers=   do not compile the given drivers (comma sep.)
> +#  --drivers=      only compile the given drivers (comma sep.)
> +#
> +# So use NETMAP_DRIVERS and the following python code to add proper
> +# configs to EXTRA_OECONF and skip arch test for kernel modules.
> +#
> +# The default is no-drivers, and all supported drivers are listed
> +# in NETMAP_ALL_DRIVERS.
> +NETMAP_DRIVERS ??= ""
> +NETMAP_ALL_DRIVERS = "ixgbe igb e1000e e1000 veth.c forcedeth.c virtio_net.c r8169.c"
> +
> +python __anonymous () {
> +    drivers_list = d.getVar("NETMAP_DRIVERS", True).split()
> +    all_drivers_list = d.getVar("NETMAP_ALL_DRIVERS", True).split()
> +    config_drivers = "--drivers=" + ",".join(drivers_list)
> +
> +    extra_oeconf_drivers = bb.utils.contains_any('NETMAP_DRIVERS', all_drivers_list, config_drivers, '--no-drivers', d)
> +    d.appendVar("EXTRA_OECONF", extra_oeconf_drivers)

This part looks good to me, thanks for fixing virtio_net.c change (I've
merged that one by accident).

> +
> +    # skip the arch test for kernel modules
> +    if drivers_list:
> +        for driver in drivers_list:
> +            d.setVar("INSANE_SKIP_kernel-module-%s-netmap" % driver, "arch")

Why is this needed? Shouldn't the kernel modules build with correct
architecture? And e.g. r8169.c creates kernel-module-r8169.c-netmap package?
I hope the .c is stripped somewhere in the build.


> +}
>  
>  LDFLAGS := "${@'${LDFLAGS}'.replace('-Wl,-O1', '')}"
>  LDFLAGS := "${@'${LDFLAGS}'.replace('-Wl,--as-needed', '')}"
> -- 
> 1.9.1
> 
> -- 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [PATCH] netmap-modules: make deterministic builds for drivers

[Huang, Jie (Jackie)]

> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org [mailto:openembedded-devel-
> bounces@lists.openembedded.org] On Behalf Of Martin Jansa
> Sent: Wednesday, November 18, 2015 6:26 PM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] [PATCH] netmap-modules: make deterministic builds for drivers
> 
> On Wed, Nov 18, 2015 at 06:02:21PM +0800, jackie.huang@windriver.com wrote:
> > From: Jackie Huang <jackie.huang@windriver.com>
> >
> > The driver builds are optional, but for deterministic builds,
> > we should should be able to explicitly enable/disable the
> > builds for them in a proper place (maybe in BSP).
> > But we can't use PACKAGECONFIG since there is no option for
> > each driver, and the options are:
> >  --no-drivers    do not compile any driver
> >  --no-drivers=   do not compile the given drivers (comma sep.)
> >  --drivers=      only compile the given drivers (comma sep.)
> >
> > So use NETMAP_DRIVERS to list the needed drivers and add proper
> > configs to EXTRA_OECONF, the default is no drivers, and all
> > supported drivers are listed in NETMAP_ALL_DRIVERS.
> >
> > Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
> > ---
> >  .../recipes-kernel/netmap/netmap-modules_git.bb    | 31 +++++++++++++++++++++-
> >  1 file changed, 30 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb b/meta-
> networking/recipes-kernel/netmap/netmap-modules_git.bb
> > index 6365fee..cfe5ce9 100644
> > --- a/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb
> > +++ b/meta-networking/recipes-kernel/netmap/netmap-modules_git.bb
> > @@ -15,7 +15,36 @@ EXTRA_OECONF = "--kernel-dir=${STAGING_KERNEL_BUILDDIR} \
> >                  --driver-suffix="-netmap" \
> >                  "
> >
> > -EXTRA_OECONF += "--no-drivers=ixgbe --no-drivers=virtio_net.c"
> > +# The driver builds are optional, but for deterministic builds,
> > +# we should be able to explicitly enable/disable the builds
> > +# for them in a proper place (maybe in BSP).
> > +# But we can't use PACKAGECONFIG since there is no option for
> > +# each driver, and the options are:
> > +#  --no-drivers    do not compile any driver
> > +#  --no-drivers=   do not compile the given drivers (comma sep.)
> > +#  --drivers=      only compile the given drivers (comma sep.)
> > +#
> > +# So use NETMAP_DRIVERS and the following python code to add proper
> > +# configs to EXTRA_OECONF and skip arch test for kernel modules.
> > +#
> > +# The default is no-drivers, and all supported drivers are listed
> > +# in NETMAP_ALL_DRIVERS.
> > +NETMAP_DRIVERS ??= ""
> > +NETMAP_ALL_DRIVERS = "ixgbe igb e1000e e1000 veth.c forcedeth.c virtio_net.c r8169.c"
> > +
> > +python __anonymous () {
> > +    drivers_list = d.getVar("NETMAP_DRIVERS", True).split()
> > +    all_drivers_list = d.getVar("NETMAP_ALL_DRIVERS", True).split()
> > +    config_drivers = "--drivers=" + ",".join(drivers_list)
> > +
> > +    extra_oeconf_drivers = bb.utils.contains_any('NETMAP_DRIVERS', all_drivers_list, config_drivers,
> '--no-drivers', d)
> > +    d.appendVar("EXTRA_OECONF", extra_oeconf_drivers)
> 
> This part looks good to me, thanks for fixing virtio_net.c change (I've
> merged that one by accident).
> 
> > +
> > +    # skip the arch test for kernel modules
> > +    if drivers_list:
> > +        for driver in drivers_list:
> > +            d.setVar("INSANE_SKIP_kernel-module-%s-netmap" % driver, "arch")
> 
> Why is this needed? Shouldn't the kernel modules build with correct

My previous patch explains this:
"[meta-networking][PATCH] netmap-modules: skip the arch check for kernel module"

# When building on a multilib capable BSP, it's possible for the
# kernel bit size to be different then the userspace. Avoid the QA test
# when building modules
INSANE_SKIP_kernel-module-netmap = "arch"

And we have BSP allowing a 64 bit kernel + modules to be packaged against a
32 bit userspace.

> architecture? And e.g. r8169.c creates kernel-module-r8169.c-netmap package?
> I hope the .c is stripped somewhere in the build.

Sorry, I didn't test all the drivers, I will check if this is stripped and changed accordingly.

Thanks,
Jackie

> 
> 
> > +}
> >
> >  LDFLAGS := "${@'${LDFLAGS}'.replace('-Wl,-O1', '')}"
> >  LDFLAGS := "${@'${LDFLAGS}'.replace('-Wl,--as-needed', '')}"
> > --
> > 1.9.1
> >
> > --
> > _______________________________________________
> > Openembedded-devel mailing list
> > Openembedded-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> 
> --
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com